Topics
Artificial Intelligence and Business Strategy
In collaboration with
BCG
In collaboration with
When Zan Gilani came to the U.S. from Pakistan to complete his undergraduate studies, he chose to study Chinese because it was rumored to be a difficult language. At the time, the tech industry was booming, and he quickly became interested in applying his passion for foreign languages and learning more generally in a technology-rich environment.
Those interests led Zan to Duolingo, where he has been working in product management for eight years and now oversees the app company’s experiential AI team. What excites him about working at the language-learning app company is his ability to help build solutions that enable personalized education at scale: The app boasts over 16 million daily active users, and AI-driven functionality motivates them through frequent notifications, personalizes learning experiences by adjusting the difficulty of questions in practice sessions, and observes and critiques learners’ performance.
Zan joins the Me, Myself, and AI podcast to outline the specific ways Duolingo uses AI and machine learning to drive user engagement, and discuss how the technology can be used to support learning more generally.
Subscribe to Me, Myself, and AI on Apple Podcasts, Spotify, or Google Podcasts.
Transcript
Sam Ransbotham: Learning a new language is a complex process. One app company looks to AI to help with personalization, context, and motivation. Learn more on today’s episode.
Zan Gilani: I’m Zan Gilani from Duolingo, and you’re listening to Me, Myself, and AI.
Sam Ransbotham: Welcome to Me, Myself, and AI, a podcast on artificial intelligence in business. Each episode, we introduce you to someone innovating with AI. I’m Sam Ransbotham, professor of analytics at Boston College. I’m also the AI and business strategy guest editor at MIT Sloan Management Review.
Shervin Khodabandeh: And I’m Shervin Khodabandeh, senior partner with BCG and one of the leaders of our AI business. Together, MIT SMR and BCG have been researching and publishing on AI since 2017, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.
Sam Ransbotham: Welcome. Today, Shervin and I talk with Zan Gilani, principal product manager at Duolingo. I’m personally pretty excited about today’s episode because Duolingo is in a really interesting position. In a lot of our show, we end up talking about how organizations are using AI to augment or to improve human decision-making, and that’s great. But Duolingo goes further and focuses on how humans are learning with AI — personalized education at scale, which is something that appeals to me. Anyway, that’s enough from me. Let’s hear it from Zan. Zan, thanks for joining us.
Zan Gilani: Thank you. It’s a pleasure to be here.
Sam Ransbotham: Zan, first, can you give us an overview of Duolingo and the company, and what your role is?
Zan Gilani: Absolutely. Duolingo is basically an app that teaches languages. We teach 40-plus languages right now. Most of those people, by the way, are learning English, and most of our learners are also learning on mobile phones, so Android and iOS devices. And we are the most-downloaded and the most-used educational app there is, languages and otherwise. We have around 16 million DAU, or daily active users, and 40-plus million monthly active users as well. And I’ve been at the company just coming around to eight years, where I am a principal product manager for the Experimental AI team.
My primary role is leading this team that is working to teach more effectively using generative AI, and also, more broadly, I help set the company up for success when it comes to generative AI.
One of the main use cases we have now specifically for generative AI is a new subscription tier. Duolingo is a freemium product. The mission is to provide the highest-quality education possible [and] make it accessible to as many people [as possible]; all you need is an internet connection. So we have this free version of the app, which is what most people are on. That’s really good; most people at the company spend time on that. And then we have subscription tiers.
And we recently just launched a new AI-powered subscription tier called Duolingo Max, and I’m happy to go into the features [of that] as well.
Shervin Khodabandeh: So Experimental AI … it’s sort of like your R&D lab, if you will?
Zan Gilani: Well, I’d say actually more than half of the company is working on R&D broadly — kind of all of the people who work on product, and [among] the folks who work on learning or teaching effectively, you could think of it as a lab, yes.
Shervin Khodabandeh: Great. Maybe, for our audience, you could list the variety of use cases or places that AI is being used. I mean, of course, [aside] from the obvious one — personalization and gamification — that a lot of the apps have. But just tell us how AI is being used.
Zan Gilani: Duolingo has been using AI for as long as the app has been around, which is coming up to a decade. In fact, it was founded by a computer science professor, Luis von Ahn at Carnegie Mellon University, and his graduate student at the time, Severin Hacker. Right now, it’s used in a number of different ways. On the growth side, for example, a lot of work goes into optimizing notifications.
The thing about Duolingo that makes it different from other apps is that it really tries to help you stay motivated. For us, pedagogically, we think that the hardest thing about learning a language is actually staying motivated, because remember: This is something tough to do. It’s especially tough to do by yourself, on a phone — that’s really a tough thing. So notifications and the timing of when you get them, the messaging that it has, really does affect someone’s ability to stick with it in the long term. So we put a lot of effort into machine learning for notifications.
We also use it on the content-generation side, so we are trying to create useful, appropriate content; we use it there as well. And then, in a separate arm of the company, which is the Duolingo English Test, which is basically an English proficiency test that students have to take when going to university — kind of a competitor to the TOEFL or TOEIC or IELTS; most Americans haven’t had to experience this — we use it for detecting cheating. We use it for generating content in a bunch of different places as well.
Shervin Khodabandeh: That’s fascinating. And on the actual core of the app, which is teaching you language, how dynamic is that? How personalized is it when it adjusts to a person’s way of learning?
Zan Gilani: Personalization plays a role, but a lot of the experience is curated by learning scientists, by educational content developers. For most people, the order in which you want to learn things is going to be roughly similar. For example, everyone learns the present tense before you then go on into other complicated tenses. But where we do use AI for personalization specifically is for something called Birdbrain, and this is basically a system we have that is optimizing for the difficulty of questions that you are going to see in practice sessions.
And one thing to note over here about Duolingo is that it’s completely interactive. It’s meant to feel like a game. So every time you’re learning, you’re kind of answering a mini exercise, and then you get it right or wrong. You always get feedback. So the point of Birdbrain is to try to come up with a Goldilocks level of difficulty, because basically, if something is too hard, it’s frustrating and you lose motivation. If something’s too easy, it’s boring and you lose motivation. Birdbrain is one of those things that we use for personalizing practice in particular.
Sam Ransbotham: Shervin, one thing I think that’s funny about that is that, in describing his product, we just took for granted the speech synthesis, speech recognition. These were avant-garde things 10 years ago, and now, Zan, you didn’t even bother to mention those as applications of AI. I think that’s sort of stepping back [and] seeing what now is normal.
Shervin Khodabandeh: That’s right. And I think one thing you have that’s quite interesting and maybe unique is just the number of feedback loops you must have. I mean, usually, in a lot of other applications, when you think about corporations and retail and banks and telecom, etc., it takes some time to get feedback. And you’re almost getting instantaneous feedback on so many experiments or mini experiments from so many people. Maybe tell us a little bit more about just the speed of learning and the speed of adaptation of your algorithms.
Zan Gilani: There’s a nice bit of mirroring over here, which is that there are principles for learning effectively that we try to impart into the product itself. And then, as a company and as an organization, we are trying to operate on those principles as well. Getting fast feedback really matters when you’re learning a language. But it also matters when you are working on the product itself, and, from a very early time, we basically built out our own A/B testing infrastructure, and it’s pretty core to the product process; that’s the central component of how we do product at Duolingo.
At any given moment in time, you have hundreds of experiments being run on things that are very granular — the positioning of buttons [indicating] the pricing on the page for the price of our subscriptions, your classic bread-and-butter experiments — to also huge new features, new tabs. Basically, if it’s not a bug, it’s probably the case that we’re running this in the form of an experiment. We’ve gotten very, very good and very, very sophisticated at doing this.
Sam Ransbotham: Shervin’s going to roll his eyes because I’m going to go academic for a minute, but you’re a classic case of experiential learning. And actually, I’m going to give you a little bit of segue to talk about your Max product here, because I think there are two pretty exciting parts of that. There’s Roleplay, which is getting to the academic, concrete experience part of experiential learning, and then this Explain My Answer, which is a reflective observation and critique. So tell us a little bit about both of those and how those are tied to learning outcomes.
Zan Gilani: Definitely. I think it’s useful to talk about where we were before adding those two features. Pedagogically, Duolingo has always been about implicit learning, so learning without telling you explicitly, “This is what you’re going to learn, and this is how it works.” And that is the DNA of the experience, because we think it’s just more automatic when you do it that way. People also get really demotivated and put off by complicated grammar tables, etc. And then, the second is that you’re learning on an app, and you’re doing this interactive thing — these almost gamelike exercises.
The two things that have been missing or limited is that, one, some people do actually want explanations, and that can be really efficient and effective when done in the right amount and in the right way. So Explain My Answer is a solution to that problem, and the nice thing about Explain My Answer that really only works thanks to modern large language models is that the large language models are really good at providing jargon-free, concise explanations of the mistakes you make. It’s really tough, even for teachers, to try to do that without using jargon and also without being right on the money. And so we are able to do that with Explain My Answer pretty well. It can give you follow-up illustrations, it can give you examples — all of these things that make large language models, by the way, just good for anyone trying to learn anything by themselves. For me, it’s life-changing.
Shervin Khodabandeh: Or anyone trying to give a speech.
Zan Gilani: Or anyone trying to give a speech, yes. And then the second one, Roleplay, is that, ultimately, for most people, language learning is not an academic exercise. They’re trying to use it as a skill in the real world, and that’s a skill in and of itself. So how do you give people the ability to practice that? How do you get them to simulate this? Chatbots are exactly one way to do so, because now, what you have is something that can speak to you at the level of difficulty that you’re at, has infinite patience, and can then actually give you feedback afterward as well.
Even if people had access to native speakers, which they don’t generally, you’d still need a native speaker who has infinite patience and who really, really can understand what you’re looking for. So Roleplay is basically trying to provide that experience in the app, rather than just having a text interface like ChatGPT and just having you speak back and forth. Basically, what we’ve done is we’ve broken down conversations or scenarios into these bite-size experiences, where you’re talking to a chatbot who’s going to be a barista or a friend that you’re borrowing something from, and then you try to go through a very specific, short experience.
Shervin Khodabandeh: Sam, I have a question for you because, Sam, as you know, is a distinguished professor, and he’s been teaching people — including people that are not in his class, like me — a lot of things. So my question, Sam, is … in all seriousness, though, what makes a good teacher, in your opinion?
Sam Ransbotham: Actually, Zan hit on a couple things right there, and that’s part of why I was feeling somewhat convicted in his conversation. This infinite patience that he talks about — that’s really important, and it’s also really hard.
Shervin Khodabandeh: Impractical.
Sam Ransbotham: Yeah. And meeting people where they are is a fundamental problem that we have in education. I mean, I sit in front of a room with 35, 40 people, and they are not homogeneous. And meeting people where they are is exactly what Zan was talking about. So, yeah, you have to know the material well, you have to keep people engaged, but you also have to keep track of where people are and what they know — present stuff that’s not too easy and not too hard — and I’m very attracted to this model.
Shervin Khodabandeh: So, what’s next?
Sam Ransbotham: Well, yeah, that’s exactly where I was going. What’s next? I mean, if we can do this with language, what else can we do this with? Where does it work?
Shervin Khodabandeh: I meant just for language. So meeting people where they are, having very good knowledge of the student and their progress, and then being able to do role-playing and lots of experiential learning, and being able to explain and guide in a more jargon-free way and in a way that you could repeat it and I don’t have to rush to take notes. I mean, all of these things, right? What is missing? I’m hearing this and this is better than a regular teacher, it sounds like. But in your view, what is missing for this to be a perfect teacher — in Zan’s view?
Zan Gilani: I think we are still a very, very, very long way away from being the perfect teacher, for a number of reasons. I’d also say, we don’t believe, and I certainly don’t believe, that at the end of the day, all you need is a chat interface and most people will be able to get really far with whatever they’re trying to learn. So, to answer the first part about why we’re far away from a teacher, well, some of this, I think, is unknown, but there is a real impact to developing a relationship with another human being when you’re learning.
There is an emotional part of learning something new, and there is a human connection that can be as simple as the fact that you show up for your tutoring appointments because you don’t want to let your teacher down, to also the fact that that tutor can be really engaging with their facial expressions, with the cadence of their voice, with the illustrations that they can tell, the way they can relate to maybe their own lives.
So that, we’re still very far away from. And then, when it comes to just “Hey, why can’t a chatbot teach you everything?” for most people, you still have to be organized, and being organized is quite exhausting. For example, just deciding what to eat next is often what you spend all your time doing, or deciding what movie to watch. And, similarly, deciding what to learn next is tough. So someone on the product side has to figure out the right sequencing of things for you to learn.
Similarly, being creative is exhausting as well. For a real-life tutor, it would be really difficult for them to put on a lot of different personas and do a lot of role-playing. Then you’d need a tutor who is also an amazing actor. And AI gives you the ability to do that with a virtual teacher, but even then, we don’t want a learner to have to come up with these scenarios all the time themselves.
The last two things I’ll mention is that learning is also multisensory. It’s also contextual. What you’d want, ideally, is some kind of learning and context happening as well instead of just with a chatbot.
And the last thing to mention is, people really do need a sense of progression in order to stay bought into learning a language, because actual progress is so slow and so incremental, it’s really hard to see. And so what you have to do is, you have to show people progress bars that are filling up, and you have to check in. I think there are probably ways that you could simulate that in a tutor experience, but those are some of the reasons why I think we care about building the experience outside of that as well and building a path that you go along, and all of these other things that go into making Duolingo what it is.
Sam Ransbotham: Zan, I want to come back to one thing you were talking about that I’m pretty attracted to, [which] is this idea of figuring out what’s the next thing that you need to learn. I mean, historically, every class that I had in languages, we had a book and we moved from Chapter 1 to Chapter 2. And somebody, somewhere, really thought through the sequence of that.
But at the same time, I don’t know that that’s necessarily optimal for all students. And we think about recommendation systems and Netflix, or in music; you know, we’ve talked to Spotify, Shervin. This seems like a great opportunity here. What’s the one thing that I could learn right now that would make the most difference? Or what’s the one thing that I don’t know that’s keeping me from opening up this next section? That seems like a huge opportunity.
Zan Gilani: Yeah. And I would break that into two pieces as well. One of them is just, generally, what’s an effective order to learn in, and what’s an effective order to teach in? And it’s especially useful for people who have some prior proficiency, which is most people who are trying to learn a language in Duolingo — particularly English learners, because they’ve done it in school in some kind of way, and their knowledge is patchy, as we like to say.
And in those cases, the order is going to be quite different person to person because of different experiences. And then the second aspect of it is the things that you specifically are having trouble with in the moment that you need to focus on more. Maybe you’re just making way more mistakes for a particular concept. Maybe it’s verb conjugations that are the problem, or maybe you just don’t know how to introduce yourself well. And so there’s an opportunity to, say, actually spend more time in that part of the experience.
It’s not that every curriculum is custom made for a unique student, but it is that where you put your attention and your focus, and the amount you put in, and the order in which you do it in is directed by some kind of tutor. I will say that’s a tough thing to do. It really is a tough thing to do. We’re not there, but that’s part of the dream, certainly.
Shervin Khodabandeh: So the show is Me, Myself, and AI, and we didn’t learn a lot about you and yourself. So maybe tell us a bit about your background — how you ended up in this role, and what the journey was like.
Zan Gilani: Yes. How far back should I go? Do I start with grandparents? Parents? Myself?
Sam Ransbotham: Up to you.
Zan Gilani: OK. I’ll skip the extended biography. I grew up in Karachi, Pakistan. That’s where I was raised, and I came to the U.S. for undergraduate studies. And I actually studied political science and East Asian studies in college because, one, I was interested in languages, so I took Chinese, and when I was learning languages in college, one of the most exciting things was always trying out tools to see what was effective or not.
So it’s the interest in languages, it’s the interest in technology that got me to Duolingo. And actually, I started in marketing because I’d done some marketing and advertising internships before that. At that time, we were around 50 people at Duolingo, and we went from having just one team that worked on the app to having teams that worked on monetization, learning, and growth. And so I started working on growth as a marketing person, and then, about a year and a half in, I switched to being a product manager. And for most of my time at Duolingo, I have worked on growth, which is to say user retention, whether that’s new users or existing users, growing our presence in Asia; that was super interesting. Things like that. And motivation and what gets people motivated and what keeps them motivated has always just been an interest of mine.
And then, the switch to working on AI as the focus and leading this Experimental AI team came at the end of last year because, basically, we ended up partnering with OpenAI for their GPT-4 launch. And the reason why we ended up doing this partnership was actually because I knew the product manager for the GPT-4 launch at OpenAI, and when we got together, we quickly realized Duolingo is the perfect use case for the launch, because for the launch, what they really wanted was to show that companies are actually using this to transform their businesses and these are also companies that are trying to do good for the world, etc. And so we, as a company, based on that partnership and getting access to GPT-4 early, shifted a bunch of priorities [and] started really leaning into AI, and one of those things was to start this team that’s trying to work on experimental features.
Sam Ransbotham: Very exciting. And actually, from a motivation standpoint, you’ll be thrilled to know that my son is very proud of his Spanish language streak, so …
Zan Gilani: Amazing.
Sam Ransbotham: We’ve got a segment where we ask you a series of questions, and these are just rapid-fire; say the first thing that comes to your mind. What’s your proudest moment in AI?
Zan Gilani: We, as a company, launching Duolingo Max has been super exciting and just paving the way for a new era of teaching. It’s been created, actually, by the Max team at Duolingo, which is led by Edwin Bodge and Bill Peterson, and it’s super exciting.
I’d say personally, actually, the first product that my team has been working on is actually — the experiment is going live. You could kind of think of it like TikTok for reading, but it’s something where all the content is generated by GPT-4. It’s giving you a huge breadth of content because people need reading practice in lots of different varieties, and it’s also listening as well. So I’m very excited to see how that goes.
Sam Ransbotham: Very cool. What worries you about AI?
Zan Gilani: I’m generally very optimistic, but I think the thing that worries me is just that the pace of change is so quick that it will be destabilizing. And the goal for us as a society is to make sure we can stay stable in the face of very, very rapid change, because rapid change is what we want, actually. We want progress to come quickly but without all of the destabilizing effects. So that’s the fear. I think it’s a solvable problem, but that’s the fear.
Sam Ransbotham: What’s your favorite activity that does not involve technology?
Zan Gilani: I play a lot of soccer. Funnily, there is technology involved with that, as it turns out, but, yes, I play a lot of soccer. That’s one of my favorite activities to do.
Sam Ransbotham: So, what first career did you want? Poli-sci, I think you mentioned. What did you want to be when you grew up when you were a kid?
Zan Gilani: Lots of different things. There was never one answer. I wanted to be a writer, a journalist, a cricket player — a cricketer — and work in international development; I even wanted to be an architect.
Sam Ransbotham: You ran the table there.
Zan Gilani: Yeah, a lot of different things.
Sam Ransbotham: What’s your greatest wish for AI? What are you hoping we can get from this?
Zan Gilani: I think AI is the way that we can create the fastest progress, where [by] “progress” here I just mean as many human beings [as possible] are living the lives that they choose to live — that they’re flourishing. I think that is the highest-level goal, and that is the meaning of life for most people. I’m very excited to be part of that effort.
Sam Ransbotham: Well, it sounds like you are. Zan, thanks for taking the time to talk to us. I genuinely feel like this is a hugely different and exciting opportunity for how we as humans can get better. And it sounds like Duolingo is doing a lot of that. Thanks for joining us.
Zan Gilani: Thank you so much.
Shervin Khodabandeh: Thanks for joining us. Next time, Sam and I speak with Jeremy King, senior vice president and head of engineering at Pinterest. See you then.
Allison Ryder: Thanks for listening to Me, Myself, and AI. We believe, like you, that the conversation about AI implementation doesn’t start and stop with this podcast. That’s why we’ve created a group on LinkedIn specifically for listeners like you. It’s called AI for Leaders, and if you join us, you can chat with show creators and hosts, ask your own questions, share your insights, and gain access to valuable resources about AI implementation from MIT SMR and BCG. You can access it by visiting mitsmr.com/AIforLeaders. We’ll put that link in the show notes, and we hope to see you there.
