This will make you a better decision maker | Annie Duke (Thinking In Bets, former pro poker player)

[00:00] It's so incredibly necessary in improving decision quality to take what's implicit and make it explicit. It's not that intuition is crap. Your intuition is sometimes right. If you don't make it explicit, then you don't get to find out when it's wrong. When you look at companies that have read your book, what do you find are the frameworks or tactics that really stick? People generally think the purpose of a meeting is for three things. Discover, discuss, decide. The only thing that's ever supposed to happen in a meeting is the discussion part. Something that comes up in product a lot is this idea of premortems. [00:30] Only if you set up kill criteria. Commit to actions that you're going to take if you see those signals. You have a very interesting framework for how to think about decision quality when the outcome is very long term. There is no such thing as a long feedback loop. And the way you choose to shorten the feedback loop is to say, what are the things that are correlated with the outcome that I eventually desire? [00:53] Today my guest is Annie Duke. Annie is the author of the bestselling book, Thinking and Bets, and also her more recent book, Quit: The Power of Knowing When to Walk Away. [01:02] She's also a special partner at First Round Capital, which we spent some time on and is incredibly fascinating. [01:08] Prior to this part of her career, she was a professional poker player. She's won over $4 million in tournaments, including winning a World Series of Poker bracelet. And she's the only woman who's won the World Series of Poker Tournament of Champions and the National Poker Heads Up Championship. Currently, she spends her time helping companies make better decisions. In our conversation, we cover the many lessons that she's learned from her friend Daniel Kahneman, who recently passed away.

[01:38] how to make better quick decisions when the feedback loop is very long, and also why she doesn't actually believe in long feedback loops, how she changed the way that the partners at First Round Capital make decisions, which is incredibly interesting, plus why when you're thinking about quitting, that probably means you've already waited too long and you should have quit a while ago. I learned a ton from this conversation, and this is definitely going to change the way I think about a lot of things. With that, I bring you Annie Duke after a short word from our sponsors. [02:08] podcast don't forget to subscribe and follow it in your favorite podcasting app or youtube it's the best way to avoid missing future episodes and it helps the podcast tremendously [02:17] This episode is brought to you by Vanta. When it comes to ensuring your company has top-notch security practices, things get complicated fast. Now you can assess risk, secure the trust of your customers, and automate compliance for SOC 2, ISO 27001, HIPAA, and more with a single platform, Vanta. Vanta's market-leading trust management platform helps you continuously monitor compliance alongside reporting and tracking risks. [02:47] Plus, you can save hours by completing security questionnaires with Vanta AI. Join thousands of global companies that use Vanta to automate evidence collection, unify risk management, and streamline security reviews. Get $1,000 off Vanta when you go to vanta.com slash Lenny. That's V-A-N-T-A dot com slash Lenny.

[03:10] This episode is brought to you by User Testing. [03:17] development process so that you can build the right thing the first time. Companies are being asked to do more with less. They need to move more quickly while building experiences that meet changing customer expectations, all while minimizing risk and costly rework. With user testing, you have a trusted partner who provides user research teams with the skills, tools, and data to be able to articulate the value of user research to the business so that you can make an impact and build [03:47] Get started today at usertesting.com/lenny. [03:54] Annie, thank you so much for being here and welcome to the podcast. Thank you so much for having me. [03:59] It's my pleasure. I was telling a bunch of people that you were coming on this podcast, and every single one of them was so excited that we were doing this. [04:06] So I'm quite excited that this is happening. [04:09] I want to start with a question about Daniel Kahneman. [04:11] I know you two are very close. You did a podcast together. I know you grew to be good friends over the past few years. [04:18] Sadly, he passed away about a week ago at this point. Yeah. [04:21] I'm curious what are some lasting lessons that you took away from your time with him, maybe one or two. [04:28] things that stick with you and you think will stick with you. [04:31] It's hard for me to describe properly [04:34] how humble this man was. [04:37] And it seems so impossible because [04:40] his influence was so huge, you know, and his intellect was so huge and his,

[04:46] insight was so [04:49] change making. [04:52] in the areas that he worked. [04:54] But he was so humble. [04:56] he really wanted to hear what you thought he [04:59] He said, I don't know a lot. [05:01] He would change his mind? [05:03] You know, it's funny, I got asked in a class that I was teaching, [05:07] about [05:09] Thinking fast and slow. [05:11] And someone was talking about how [05:14] There's... [05:15] a lot of studies in there that didn't end up replicating. [05:19] and [05:20] What I've seen from a lot of other people in the field, where their work is defined by some line of work that didn't replicate, [05:28] where they're really arguing against the fact that it doesn't replicate, just sort of [05:33] trying to justify or say, but no, or this and that, like this real lack of open-mindedness to the idea that, [05:39] You know what? Maybe [05:41] Maybe it turns out not to be true and that should be okay. [05:44] But the thing about Danny was when you talked about it, he was the first one to tell you, [05:49] I wish priming wasn't in the book. [05:51] It doesn't replicate. [05:53] You know, like he was just so open-minded to the idea that he kind of, [05:58] He knows what he knows now, but he knows that he doesn't know [06:04] everything. Another thing that people don't know about him outside of academics, I think, are that he was one of the pioneers in something called an adversarial collaboration. One of his favorite things, and he's sort of been [06:16] Someone invented it.

[06:18] was find someone who really disagrees with you, [06:20] and then write a paper with them. [06:23] and try to figure out with them what's the study that would resolve the issue. So an example of that was Danny did work on happiness as it relates to [06:34] wealth? [06:35] Kahneman, [06:37] and a collaborator, Angus Deaton, [06:42] They published a result that got a lot of press. [06:45] about the relationship between money and happiness. And the idea was money doesn't buy happiness, [06:50] beyond $75,000. So that got published in 2010. I'm sure if you ran that exact same study, it would be [06:56] more than $75,000 now because you would adjust for inflation. But it was kind of like, [07:01] The idea being, [07:02] Once your basic needs are met and you're not that worried about paying rent and so on and so forth, it's not going to have that big of an effect. But then a decade later, Matthew Killingsworth, [07:13] said, that's not true. [07:16] that [07:17] Actually, money and happiness are kind of correlated [07:22] through all the income levels. The more money you have, the more happy you tend to be. Okay, so this is like a classic example where like people will tear each other down sometimes, and that guy's wrong and whatever. And instead it was, [07:36] Conovan's like, all right, let's do it together. And they joined forces along with Barb Mellers, [07:42] um [07:44] to basically try to, you know, to try to sort of resolve [07:47] that issue.

[07:48] He was just always seeking out. Like, it was just like everything was like, why am I wrong? [07:53] And he was eager and he was excited and he wanted to, you know, I remember having a lunch with him where he, [07:59] I realized after the lunch that the whole lunch was him asking me questions about my work. [08:05] Like, why is he asking me questions about my work, right? Like, he's Danny Gahneman. But he was just so curious about other people. [08:14] And he just made time for everybody. [08:17] He just loved the people around him so much. [08:21] And I think that in the end, that's [08:24] It's hard to believe, but I just think that that dwarfs his body of work, which is spectacular. But [08:30] she is a human being which is so spectacular. [08:33] And I do think you hear all the time like, [08:37] about people who are [08:39] so incredibly brilliant. It's sort of the brilliant but asshole [08:43] kind of thing. [08:45] And it was just so the opposite with him. [08:48] You're so kind. [08:49] Hmm. [08:51] It's amazing. And it feels like there's probably a strong correlation between people that are curious. [08:56] always looking for how they're wrong, [08:58] and finding time to spend with people to kind of debate and discuss and ask. [09:03] Feels like there's going to be a strong correlation with that and finding really interesting insights uncovering [09:08] New things. [09:09] Yeah, yeah, yeah. [09:11] Well, thank you. Thank you for sharing all that. I wish I had known him. [09:16] Let's talk about decision making.

[09:19] So there's kind of like two areas I want to spend time on, as one would not be surprised decision making and quitting. [09:25] And I know that you've been on a lot of podcasts. I imagine many listeners have read your book or have heard a lot about the stuff that you teach. I'm going to try to come at this from a bunch of different directions and do it a little differently. [09:36] First of all, I know you're a parent. As last I heard, you have four kids. [09:40] I just had a kid. He's almost 10 months now. Oh, congratulations. Thank you. [09:45] I'm curious what your frameworks... [09:47] you find most helpful in parenting and in raising your kids that have stuck with you there. [09:51] Thank you. [09:52] Well, let me just first off say, I get a lot of people who are pregnant who come and ask me for my advice about parenting. [10:00] And [10:01] I'll tell you the two pieces of advice I give people. [10:05] The first is this. [10:08] There's all sorts of parenting books. There's all sorts of styles, you know, like... [10:15] you know, a book needs to sell, so they need to sell you something new or different or whatever. Okay. [10:21] But the only thing that matters really [10:24] is that your kids know you love them. [10:27] like really know like deep in their mind. [10:30] bones. [10:31] Know you love them. [10:33] And like we can argue breastfeeding versus not breastfeeding or like co-sleeping versus not co-sleeping or attachment parenting or sleep training or. [10:43] We didn't talk about all that stuff. [10:45] Do you homeschool? Do you send them to private school? Do you send them to public school? We can have lots of discussion. But it all dwarfs in comparison to...

[10:53] your children, [10:55] know that [10:56] You would lay down in the street in the front of a bus. [11:00] if it meant that they would [11:01] be alive and happy. [11:04] Like, [11:04] That's the number one thing I say. [11:07] Thank you. [11:08] Number two is [11:10] At some point, you will drop your baby on its head. [11:14] So I bet this has happened to you already. Well, yeah, sort of. Sort of. [11:20] So here's the thing about babies is babies can't move. [11:25] And so you get very used to, you know, like, [11:29] you know, as a mom or a dad or like you're sitting on the couch and you're folding laundry and your baby is right here next to you. [11:36] And your baby can't move. [11:38] And then one day without warning, [11:41] your baby can roll over. [11:43] And you think it's fine because you put a pillow there, but you turn your back for a second because you're like, whatever, and then your baby falls on its head. It happened with every single one of my children. [11:56] And, you know, and they may they don't fall that far and whatever, but they're totally fine. And you're not a bad parent. [12:03] And it's just this like, [12:05] there are things you don't know and stuff's gonna happen, and it's not gonna be perfect. And every little thing isn't gonna have some lasting impression on your child, 'cause at some point you will drop your baby on its head. You will not mean to. [12:19] You didn't intend to, but in my case, it was always like, oh my gosh, my baby can roll over. I did not think that was going to happen today.

[12:27] And gosh, by the time I got to the fourth one, it's like I'm chasing after one baby to try to get the diaper on and this and that. And then the other baby hits its neck. [12:36] So, you know, I think there's just a lesson in that, which is like, it's a small thing, like don't sweat it. [12:41] and don't, [12:43] Don't... [12:44] get mad at yourself because you made a mistake. It's like the mistake would be [12:49] "Oh, I realize my baby can roll over now, so now you should make sure that you're taking extra precautions about your baby rolling over off the couch." But it's just a fun... I mean, it's really just a funny way for me to say, like, [13:02] your baby's going to hit its head or walk into a coffee table or get cut. [13:07] or, [13:08] you know, whatever. It's like, [13:10] They're very resilient. [13:11] And you didn't do anything wrong as a parent. [13:15] So don't beat yourself up about that. [13:17] And then I do actually have one other thing I say. - Yeah, go for it. - I go having four. [13:23] You have no influence whatsoever basically on what your child's personality is. [13:29] um [13:30] What you can do as a parent is make sure they say please and thank you. [13:33] Mm-hmm. [13:34] Thank you. [13:35] So, you know, what I find with like parents of one, [13:39] is [13:40] particularly if that one is very well behaved. [13:46] They think they're a great parent. [13:47] And they may be. [13:49] That would be resulting. They may be. But once you have four, what you realize is like, "Oh, my parenting really didn't have anything to do." [13:56] Yeah.

[13:57] But my children do say please and thank you. [14:00] This is really valuable, good advice. I need to hear this. My wife is going to need to hear this, too. [14:05] She's always worried about the soft spot on her kid's head and getting hit. It'll be fine. It's actually kind of why it's there. Really your baby will be fine. I mean, that's the thing. We used to cart babies around in the wilderness. We were living in Cape. We're fine. We're built for it. We're built for a much rougher world than... [14:28] the world that babies actually grow up in now. I will say my oldest child, [14:33] when she was in high school, just for whatever reason, she had no interest whatsoever in like alcohol or drugs or anything like that. [14:41] And I remember just feeling so smug about... [14:45] how well I had raised this child and how great she was. [14:49] And oh, these other parents who clearly weren't doing a good job. I mean, I didn't really, but like kind of. [14:55] Because it's hard not to. And then my next child went through high school and I was like, "Oh, now that was..." [15:04] That's just a temperament thing. [15:06] It's a good example of resulting in action. It is a good example of resulting. But all my kids have turned out fine and they're all very different. [15:13] Mm-hmm. [15:14] But yeah, but as far as decision making is concerned, which is a totally different thing, [15:20] I'll actually quote Danny Kahneman on this one. Nothing is as important as it seems when you're thinking about it. [15:26] That's a really important one.

[15:27] um [15:28] One of the things that I used to try to do with my kids all the time was mental time travel, which is actually a very good decision tool. [15:34] So they would be really upset about something. And it could be... [15:38] something that happened at school, but, or it could be like, [15:42] "they were grounded for something or whatever, [15:45] And, [15:47] You know, and the thing is, when you're in the moment, it just feels like so big. [15:51] And the thing that I used to say and they say all the time now is, [15:54] This is going to be so great for you when you're 40 at Thanksgiving. You're going to be able to tell these stories to your children, and it's going to be the best. [16:03] Like, you should be thanking me so that you can talk about your like crazy mother to the Thanksgiving table, you know, and it allowed them to get some time and space from it to sort of realize this is going to be funny at some point. [16:18] At some point, it's going to be a hilarious story that you tell people, no matter how horrible it seems now. [16:24] It's going to be a hilarious story at some point. [16:26] when you're older. And it sounds like kind of a fun thing I was doing with them, but it's actually a really good decision-making tool for us in general, right? Like, [16:34] Look, there are absolutely things that [16:36] 20 years from now are going to matter. [16:39] For sure. [16:40] I, [16:41] But most things, no. Like if you think about most things that ever happened to you, like, you know, you're 21 and your girlfriend breaks up with you or whatever, and you just think it's the end of the world and you're never going to recover, you know, and all these things, but [16:54] you know, [16:55] if you could get yourself like, well, how do I really think I'm going to think about this in 10 years?

[17:00] Am I still going to be heartbroken in 10 years? Probably not. And I think that we need to get that perspective of time. [17:07] so that we can get out of the moment, you know, cause in the moment it just feels so important and the feelings are so big and it's like, [17:14] The focusing effect on this second and the feelings you're feeling right now are so huge. [17:20] that we forget the scope of time. [17:23] And I think that that's absolutely one of the best things you can do with your kids. And you can do it in small ways too. [17:28] Right. Like, [17:29] you can choose to [17:31] play this video game, or you can choose to study for your task. [17:34] a week from now when you get that test back. [17:37] How do you think you're going to feel about those two choices? [17:41] Thank you. [17:42] Right? So, I mean, that was a tool that I used a ton. [17:46] Thank you. [17:47] And it's just generally an amazing decision-making tool, just generally for people. It's one that I happen to use with my kids all the time. [17:54] And then I would say that the other one was, um, [17:57] using the word "nevertheless." [18:00] which is, this is a great leadership skill, the word nevertheless. [18:05] So, [18:07] Let's say my child got caught doing something wrong. I don't know. I found a bag of red solo cups in my backyard that I forgot to throw out after a weekend that I was away. [18:18] That's hypothetical. So. [18:21] You know, so I'm grounding them and... [18:26] It's a lot of argument back, right? So they think this is a debate and they're giving me all of their input and their opinions on

[18:34] why it's unfair and [18:37] all the other kids do it and the other parents don't get mad and [18:40] Whatever, whatever the argument is. [18:43] You have to have the balance between [18:46] them feeling heard, which I think is incredibly important that your children feel heard. [18:50] And... [18:52] following through on what you know or believe is right. So it's [18:57] I hear you. [18:58] Nevertheless, [18:59] You're grounded for two weeks. [19:02] Like, I hear what you're saying. [19:04] And I understand. [19:05] Nevertheless, [19:07] This is what's going to happen. [19:09] And obviously, like the words that you can use for that might be different. You have so much more authority, obviously, over your children. [19:16] than you do in other places. But in the workplace, this is very good because employees gripe all the time at decisions that you make in leadership. [19:23] Because they think they're right and they want their way. [19:26] And to have the ability to say, I heard you. [19:30] and your input, trust me, was incorporated into the decision. [19:33] Nevertheless, this is the path we're going to take. [19:36] Right or wrong? [19:37] That's what we're going to do. [19:39] So, you know, I think there's a lot of things that basically you can say, like, these are good things to do with kids, but they're actually just generally good, really good decision strategy. [19:48] I'm taking notes on all these things. These are going to be extremely... Nevertheless is a really good one with children. [19:53] It connects to something Dr. Becky teaches, if you know her, of telling someone, I believe you. When your kid says something and they're upset about something, start with, I believe you. [20:02] Nevertheless. Nevertheless, exactly.

[20:05] I think we're going to start having to pivot this podcast to parenting podcast because there's so much... [20:10] stuff here, but I'm going to try to resist. [20:12] something that [20:14] So we've been talking about decision making and frameworks and things like that. [20:17] Something I wanted to ask you is, [20:19] how much better can someone get at making better decisions? [20:23] So for example, say someone listens to all of your... [20:26] Podcast reads your book, studies it intensely. [20:30] What's kind of the delta you find in somebody being able to make a better decision? And where this sort of comes from is... [20:36] Daniel Kahneman, of all people, people ask him, like, you just live such an optimal life now that you know all of these biases we have and all these mental errors we make. And he's like, no, I can't. [20:45] I think he said famously, I can't use these in practice. This is just stuff that I have learned, but it doesn't actually impact my day to day. [20:52] So I'm curious just... [20:54] How much better can someone get? What kind of delta have you found in terms of making decisions? [20:59] Okay, so it depends on whether you actually do the thing. [21:03] I mean, I think that that's what the issue is. [21:05] So, you know, I mean, I think, you know, Kahneman did some work way back when, like, that sort of started him on this journey. [21:13] That was basically work on hiring. [21:16] and [21:19] you know, is taking it from completely unstructured [21:23] Lenny saying, I just know a great [21:25] product manager when I see one. [21:27] whatever that means. [21:29] to [21:32] me going to Lenny and saying, "Okay, I understand that you think that you see a great product manager.

[21:37] you know one when you see one, but can you explain to me what that means? Like in the abstract, [21:41] What are the things that you're looking for? [21:44] in someone that you want to fill that role. And we can then sort of excavate that, right, and make what [21:50] is implicit because you're applying some sort of implicit like model [21:54] to how you're thinking about the person that you're interviewing and how they map onto the role. [22:02] that they're going to be in, but let's make that explicit. So we can make that explicit. We can turn it into a decision rubric. We can create a structured interview process out of that. [22:10] And then, you know, what he found was after you've gone through that process, you then sort of use your intuition after having done that, not before. [22:19] that then you actually can really... [22:21] drastically improve [22:23] sort of your hit rate on hiring. And in that case, it was from about a 50% hit rate to 65%. [22:30] That's pretty huge. [22:31] So that's a really big difference. [22:33] Problem is nobody does it. So I can tell you that most of the conversations that I had with [22:39] Dani, that were about my work in particular, [22:42] We're just him saying, how do you get anybody to do it? [22:47] Now, [22:47] That's not to say that I would be better at that than him, it's that he didn't do my job, right? So he was an academic doing research and so on and so forth. I'm like living in companies. [22:57] like embedding for years. [22:59] um, [23:01] You know, I've been with one client for five years, one client for four years. [23:06] one client for

[23:10] four years also, almost four years, three and a half years. And then I just took on a new client who I've been with for six months. And that's my whole client roster. [23:18] Because I stay with them so long. [23:21] So if you do it, the answer is quite a bit, like at least in terms of, [23:27] Danny's work in relation to something pretty noisy like hiring, you went from 50% hit rate to 65% hit rate, which is huge. [23:35] It's enormous. [23:37] The problem is that, [23:38] the way that you can make decisions in order to be better at them [23:43] is not supernatural to the way that humans make decisions. [23:47] I think we think a lot more highly of our intuition than we really ought to. [23:51] We think... [23:53] very highly of her ability to notice things in the moment and act rationally toward them. [23:58] in a way that really we ought not to. [24:01] we tend to think that we have insight that other people don't have when actually the other people probably have more insight into our situation than we do. So, you know, I think that the, I think the answer is a lot. [24:14] with a big if, with an if you can get people to do it. [24:18] And I think that's where the issue comes in. Now, the good news is, [24:23] that [24:24] you can then [24:26] reverse that and say, [24:28] If I'm willing to do it, [24:31] think about what an edge I'm gonna have over people who aren't willing to do it. [24:35] Right. So we it's like we think there's people who don't know about it.

[24:41] There's people who do know about it, but don't do it. [24:45] And then there's people who do know about it, but do do it. And it's that last group that's so tiny. [24:50] So the answer is I think a lot, but nobody does it. [24:54] So maybe following that thread, when you look at companies that you've worked with or companies that have read your book and really dove in, [25:02] and started to implement some of your piece of advice to make better decisions. [25:06] If you could look back at the ones that [25:08] had an impact what do you find are the the mental models or frameworks or tactics that really stick [25:14] that most often have an impact and the biggest impact. [25:17] I think the one that's easiest to implement is this. [25:20] And it's so easy and I just wish more people would do it. [25:27] the best way to get somebody's opinion [25:31] is independently. [25:33] of other people's opinions independently, asynchronously. [25:37] So, [25:39] The way that I put it is I want people to stop talking to each other so much. [25:43] when we think about like what did people generally think the purpose of a meeting is they think it's for three things [25:49] discovery, [25:50] which is I want to discover... [25:52] what your opinions are, what your judgments are. [25:56] of something. [25:57] So I want to find out what you think about something. So like, for example, if we're, if we're, um, [26:06] If we're in product development and we're trying to figure out a timeline,

[26:12] You know, like, so we've got like, we're trying to develop a product roadmap and we're trying to figure out how long it's going to take to release certain features or, [26:19] something like that, we all come into a room and we start yelling it out. [26:23] right together. [26:25] you know, [26:27] It's just like it's so bad for decision making, like I can't even tell you. [26:33] There's cross influence. The loudest person in the room tends to then have an outsized influence on the decision. The most confident person in the room tends to have an outside influence, outsized influence on the decision. And that's great if the most confident person is also right. [26:47] But the problem is, [26:49] That's not always so. [26:52] And so that's the discovery piece, right? And we tend to do that in a group. [26:55] Then there's discussion, which is I've now discovered the way that Lenny is modeling this problem or what his [27:01] judgments are about certain things or as forecasts are as estimates are. [27:05] And now we're going to discuss [27:07] those ideas and we're going to discuss [27:10] Your idea is in comparison to my ideas, and that's going to happen in the group setting. [27:14] And then there's also now we're going to decide. [27:18] in the group setting. [27:19] So we're going to make some... [27:21] set of decisions about [27:23] say what the roadmap's gonna look like. [27:26] Here's the thing that I think is the easiest to change. [27:29] is to realize the only thing that's ever supposed to happen in a meeting [27:33] is the discussion part. [27:36] So we should absolutely [27:39] Becoming together as human beings to discuss

[27:42] everybody's judgments and opinions and the way they're modeling the problem and their forecasts [27:48] In particular, it's really good to come in and discuss the places where people are different. [27:54] So you've been in many meetings, I'm sure about 80% of the time is double clicking. [27:59] Oh, I agree. And I want to now refl- [28:02] literally just use different words to say the exact same thing because I'm in agreement. [28:07] If we can focus the discussion on places that people disagree, we're much better off. [28:11] But- [28:13] So now we can take this discover, discuss, decide. So I'm saying discuss is only supposed to happen in the meeting. That's the only thing that's supposed to happen in the meeting. So what's happening with the other stuff, the discover and the decide? [28:24] Well, and this is the thing that I think I actually have been able to get people to do is discover what people think before you get in a room independently of each other. So how would you do that? [28:36] Well, let's imagine that you're going to have a meeting about the product roadmap. [28:43] So you would say to yourself, in this meeting, what are the opinions that I need to get? [28:48] from the people in the room. [28:50] It could be a brainstorm. What are all the different features that we could develop? [28:55] It could be a brainstorm. Fine. [28:58] write to them independently and say hey [29:00] free for off, [29:02] Just, [29:03] Come up with all the different features that you think would be reasonable for us to consider developing. [29:08] and then give me a fourth rank, [29:11] from best to worst of your own ideas with some three to five sentence rationale as to why you have these things in this order.

[29:17] And so I could do that, but maybe we already have a list. [29:20] I can send that out for a force rank to everybody. Okay, so here's the final list of things we're really considering. [29:25] prioritize those for me, just force rank them. [29:30] And then again, give me a rationale. Give me a little bit of free writing as to why you think this should be that you have these things in this order. [29:36] Maybe we've decided now on what our top five priorities are. [29:40] Great. [29:41] I could before the meeting, [29:43] send the top five priorities out. Hey, we've decided on this stuff. For each thing, I'd like to understand what you think a reasonable timeline is, how many sprints it's gonna take, [29:53] so on and so forth, right? So we can ask for that type of information. [29:57] so that we can start making estimates because that's going to affect like budget and what we're going back to the board with and so on and so forth. But regardless, like just figure out what what what is the thing that we're going to be discussing in this meeting? And I'm going to send it out to everybody independently and I'm going to say, don't reply all. [30:12] right? Just send it back to me independently. If it's a repeated decision, you can actually create a rubric that lives on like, you know, [30:18] Airtable or Coda or [30:21] Google Sheets or whatever, I don't really care. And people can input their decisions there. [30:25] where they can't see anybody else's decision. So like with Google, you can use Google Forms, [30:31] in order to do it, and then it dumps into a spreadsheet that only you can see. That's a great way to do it. [30:35] So anyway, so you do that. [30:37] And now you [30:39] can now see everybody's opinions. [30:42] that you then now send out to the group and say, everybody look this over before we come in and discuss.

[30:48] So notice you're still working as a group, but you're working as a group where you're not in the same room together and talking at the same time. [30:54] And there's a word for that, which is nominal group. So it's a group that at that moment is working independently and asynchronously of each other. [31:03] So if you can get people to do that, [31:05] And I do have companies where I don't consult with them. [31:10] but I've just come in and talked to them briefly or whatever that do absolutely implement that piece of it. [31:16] That is a huge piece of it. [31:18] That's ginormous. [31:20] And then you do the same thing for deciding. [31:23] So the decision should not be made in a room. [31:26] It's made either [31:29] I love the one decision maker model, but not everybody's down with that. But you can have like a vote forum where people go vote in private about the way they're leaning. [31:37] you know, [31:38] You can do a variety of things with that, but just don't do it in the meeting. [31:44] And then just the last thing that I'll add, which is a muscle that you really have to exercise, [31:50] is [31:51] I think it's really important to understand [31:54] that [31:55] the word alignment [31:57] in terms of we're all aligned as a group, right? [32:00] The word alignment is stupid and it shouldn't be used. [32:04] And I know I'm saying that very harshly, [32:06] But it's true. [32:08] It's dumb. [32:09] because it doesn't exist. [32:11] like you have 10 people in a room, [32:14] and they're all really different people with different opinions

[32:18] and [32:20] They're never gonna come out of the room agreeing with each other. [32:23] And it's really bad if the expectation is that they're supposed to. [32:28] And it's really bad for a few reasons. One is it isn't reality. [32:32] And I don't like coming out of things without reality, actually. [32:37] kicking. So it's not reality. People don't actually agree. They're not actually aligned. [32:43] on the decision. That's just the thing that makes you feel better. [32:45] right so i you know i think that that's problem number one but problem number two is that if the goal is alignment if the goal is agreement [32:53] then the meeting becomes coercive. [32:56] And you never want that. [32:58] So the way that I'm supposed to... [33:02] talk about my ideas is to convey that. [33:05] why I believe what I do. [33:07] not to convince anybody that I'm right. [33:10] Because if I'm working to convince people that I'm right, it becomes coercive. [33:14] And that's horrible. [33:16] So you have to get comfortable with walking out of the room. This is the nevertheless. [33:21] Walking out of the room, understanding that, [33:24] Once you have that discussion, it's not that, Lenny, your opinion might change. It could. [33:29] I could say my thing. [33:31] And it could be just so damn brilliant. [33:33] that [33:34] you change your mind somewhat. So you come, maybe you're like, "Oh, I'm thinking about this differently." And actually, [33:41] But maybe you don't. [33:44] And you still believe a thing that's very different than me. [33:47] And leadership has to say, that's fine. I've heard both of you.

[33:52] And I know that this isn't going Annie's way, [33:57] Nevertheless, I'm going to be happy. [33:58] trying to think about what all of our goals are. This is what the decision is going to be. And it's totally fine that you ended up not agreeing with each other. [34:05] Because it's reality and what it allows me to do is get a better sense of what the space of decisions is. So those things I have been able to get people to do. [34:13] And they're actually quite impactful. [34:17] Imagine a place where you can find all your potential customers and get your message in front of them in a cost-efficient way. If you're a B2B business, that place exists, and it's called LinkedIn. LinkedIn Ads allows you to build the right relationships, drive results, and reach your customers in a respectful environment. Two of my portfolio companies, Webflow and Census, are LinkedIn success stories. [34:43] team. For Webflow, after ramping up on LinkedIn in Q4, they had the highest marketing source revenue quarter to date. With LinkedIn ads, you'll have direct access to and can build relationships with decision makers, including 950 million members, 180 million senior execs, and over 10 million C-level executives. [35:03] You'll be able to drive results with targeting and measurement tools built specifically for B2B. In tech, LinkedIn generated a 2-5x higher return on ad spend than any other social media platforms. [35:14] Audiences on LinkedIn have two times the buying power of the average web audience, and you'll work with a partner who respects the B2B world you operate in. Make B2B marketing everything it can be and get $100 credit on your next campaign. Just go to linkedin.com slash pod Lenny to claim your credit. That's linkedin.com slash pod Lenny. Terms and conditions apply.

[35:37] There's kind of an implication here, you touched on a bit, that there's a DRI, essentially, there's one decision maker. [35:43] Sometimes people start to feel like, oh, my voice isn't heard. I don't have a lot of say. I can't be part of this decision. I know you talked a bunch about just how to make people feel included and get feedback along the way. But any advice there, if you try to move to this model of making... [35:55] people feel like, "Okay, I actually have impact on where this goes." Yeah. Here's the really wonderful thing. [36:01] about moving from a [36:04] a coercive model to a, to a, [36:08] I [36:09] really, I guess you could say a model of curiosity, right? Like you want to be curious, not coercive. [36:15] which means that the way that people in the meeting are talking about what their opinions are is in the mode of conveying information, not trying to convince anybody. [36:25] So, [36:27] Once we move away from that coercive model, [36:30] And when I say coercive, I'm not saying anybody... [36:34] is purposely trying to set up a culture of coercion. [36:38] Sometimes that's true, but for the most part, everybody's trying to do a good job and nobody's trying to set up a culture of coercion. But as soon as you say, are we all in agreement or are we all in alignment? [36:49] And as soon as you're allowing people, and I'm sure you've been in these meetings, right? If you allow people to interrupt, [36:55] If you allow people to say this, I think you're wrong. [37:00] I disagree. [37:02] here's why. [37:05] Those are all very coercive things.

[37:07] things to have happen. [37:09] You know, interrupting someone is silencing them. [37:15] Saying you're wrong? [37:17] Well, now you become tribal and people aren't going to be open-minded and they're going to stake their ground and it's all really bad. So let's move away from... [37:24] that number one, that's already gonna help. [37:27] But, [37:29] When we think about the way that meetings normally happen, again, there's all this crosstalk and [37:34] Some people aren't speaking and some people are and so on and so forth. And then, of course, not everybody feels equally heard. [37:40] But when you work as a nominal group, [37:43] Let's talk about something as simple as... [37:47] We're going to make a forecast of how long it's going to take to [37:52] to launch this product feature. Okay, so [37:55] I'm going to send out to everybody. [37:57] What's your point estimate? What's your lower bound? What's your upper bound in terms of timeline? [38:02] And you could do it in some way, like how many sprints do you think is going to take? Whatever language you want to use. [38:07] Now, everybody independently [38:09] now gives their forecast. [38:12] with [38:13] a rationale for why they believe that. [38:16] and then you come into the room, [38:19] And you run a discussion where everybody's getting to say what their estimate was and why they believe that. And people are getting to ask questions. [38:27] I have a question. Always is a clarification. It's just I don't understand. [38:32] And as the leadership, the way that you would do it is, [38:37] Let me give you an example from a real one because that'll be easy. I did one of these discussions for a question about remote work, like what did the company want in terms of remote versus hybrid and that kind of thing.

[38:49] And so like there was a lot of disagreement. [38:52] about whether whatever the policy was, [38:56] it should be consistent across functions. [38:59] Lots of disagreement. [39:01] So, [39:03] you know, let's take somebody who was on the, no, it shouldn't be consistent across functions. So they now say why they believe it shouldn't be consistent across functions. And they say things like, [39:13] Well, different functions have different requirements, right? So there's some functions that have to be in the office, like if you're IT, [39:20] as an example, but there's other functions which are more collaborative and creative and whatnot. [39:28] at [39:29] where [39:30] it makes sense for people to be in the office. [39:33] Whereas like engineers, [39:35] Doesn't really matter. [39:36] So they're like, there's different needs of different functions in terms of how much they need to be in the same space. So I'm not agreeing or disagreeing with this. I'm just saying what somebody said. [39:45] So now as a facilitator, [39:47] I never would I never say, oh, I agree. [39:51] What I say is, [39:53] I just want to make sure I understood what you said. [39:56] And I reflect it back. So I say, so what I heard you say is that not all functions are created equal in terms of how well they work remotely or how well they work in person or what the flexibility might be. So what you're saying is there's some there are some functions that have to be in person. [40:11] Period. [40:13] And then other functions where, [40:17] that sort of collaborative element [40:19] being in the same office would be more important

[40:22] versus [40:23] some functions where being remote is totally fine. Is that what you meant? And then they have the ability to say, "Yes, that is what I meant," or [40:31] actually, [40:32] I meant something slightly different. Let me say what that is. And then I reflect that back. [40:36] Okay, so literally you're just going around, you're calling on people to do that, and then you're reflecting back what they say without offering your own opinion. [40:44] I don't know, Lenny, tell me how someone doesn't feel hurt in that situation. Absolutely. [40:49] I would love it. I would feel so good if somebody just clarified and made it clear they know exactly what I'm saying. [40:54] Even if the decision doesn't end up the way it wants. Right. So that's the thing. And then... [40:59] And then what actually ends up happening is that, [41:01] The people in the room feel more, the psychological term would be endowed, more endowed to the decision. In other words, they feel like they have ownership over the decision. [41:10] And whatever the decision is, [41:12] they generally will see their selves reflected in it. [41:16] because they were heard. [41:18] you know and and [41:22] They also will generally understand that nobody, no one really ends up with exactly [41:29] every single thing that they wanted to see in the decision because they also get to see the true spread of opinions [41:35] on the team. [41:36] and and what you [41:37] see in that situation is that there's lots and lots and lots and lots of disagreement. [41:42] which you don't see if you talk in a group, it narrows the space. For example, particularly if you're senior,

[41:52] If you say, I think this is going to take [41:54] three months to launch. And I was thinking, [41:57] four weeks you're never going to hear four weeks from me [42:00] Right. [42:01] Ever. [42:02] But if we get those opinions independently, you will actually hear that I think it's going to take four weeks. [42:08] You may tend to be more right there being more senior, but I may have something interesting to say. Like you should hear what I have to say. [42:15] And this also allows me to learn from you too. But because I haven't heard your opinion first, I'm not. [42:21] I'm not going to [42:22] conform my opinion to yours. So it actually spreads the surface area of disagreement that you see on the team, which then makes people feel much better about the decision not being exactly what they want because [42:36] They recognize like, oh, this is actually a hard problem. Like people really disagree on this stuff. [42:41] Amazing. I think this is going to be really helpful to a lot of people. Just to kind of close the thread on this little summary maybe is... [42:47] Core advice here is brainstorm separately. So I guess first there's Discover. [42:52] Discuss. Decide. [42:53] to discover ideas. [42:54] completely independently basically brainstorms you're a big advocate of brainstorming independently sitting [43:00] and around thinking through ideas. [43:01] and then bringing people together to discuss all the things they've come up with, and especially where they disagree. And then ideally having one person that makes a decision once they're, [43:10] She or he has taken all the input. I mean, I work with people where it's like a partnership and there's six people who are going to vote. [43:17] But they still do it independently. They go to a forum. [43:20] So I don't know what your vote is.

[43:22] And in the end, all six people don't have to agree. I just think it's really important that once you get above an N of one, you shouldn't necessarily expect the people to agree, which I think is just really important. This is good for more than just brainstorming. It's for forecasting, [43:38] any kind of project planning, [43:40] uh budgeting, [43:42] Um [43:43] Yeah, I mean, we do this, like, for example, at... [43:46] First round, we do this for, we have a structured, [43:49] forum for evaluating a company. [43:52] in terms of whether you should invest in it. [43:56] you know, [43:57] there are facts and then there are the way that you model those facts. And so it's a lot, you know, what are the investors opinion of the founder and the product and, [44:05] that kind of thing, and we just have structure around [44:09] how we're eliciting those opinions. [44:11] which is really helpful. The other thing that I'll just kind of add to what you said is that you can actually be in the same room together and still discover information independently. So this will happen on the fly all the time. [44:26] where people are talking and they start talking, you know, [44:28] something comes up, like someone suggests some new feature or something like that. And then people start saying, yeah, but that's going to take... [44:36] And you go, stop. [44:37] Okay, everybody take out a piece of paper. [44:41] So you can still get that same independence on the fly. [44:44] as well. [44:45] is doing it in advance. But yes, that was a very nice summary. [44:49] If this is just the one thing people take away from this conversation already, I think that could have a big impact.

[44:54] Speaking of first round, one, we actually just had Todd Jackson on the podcast talking about- Love Todd. Love Todd talking about product market fit. That episode will have come out before this episode. [45:03] Also, so I asked Brett Burson, your colleague at First Round, what to ask you. [45:08] Oh, okay. [45:09] Yeah, yeah. By the way, your title of first round is amazing. Special partner. I've never seen that before. [45:15] I am. It's a title just for me. I adore Brett Burson. [45:19] Me too. So I'm interested in what he said you said. Nothing too spicy. [45:24] It's kind of along the lines of what you talked about. So you said that you have a very interesting framework for how to think about decision quality in the short term when the outcome is very long term. For example, [45:34] investing. Also, many decisions we make in business, things you need to decide now that you only find out years from now. [45:40] Can you talk about your advice there and your insights here? Oh my gosh, I'm so happy that that's the question that he asked. [45:46] Okay, so can I give a tiny bit of background to this? - Absolutely. - So prior to talking to First Round, I have another client too, they're amazing, Renegade Partners at Roseanne. [45:57] Winstack and Renata Quintini, they're [46:00] Incredible. [46:01] before finally hooking up with them. And they sort of work at different stages. First round of seed, obviously, [46:08] Renegade is more like A, B, C. [46:11] tiny bit of dabble and see but uh [46:15] Before running into them, I talked to... [46:17] quite a few... [46:20] venture firms. [46:22] you know, who are interested in talking to me,

[46:25] sort of post thinking and bets having come out. So this would be like 2018. [46:30] And there was a theme. [46:33] There was a theme across them all. [46:36] The first one was... [46:38] Well, the kind of decision making you're talking about, we don't need to do because [46:42] We just know a good founder when we see one. So that is a sentence that came out of many people's mouths. [46:47] And as I just said to you, like, okay, I have no doubt, but don't you want to make that explicit? There's all sorts of great things that come from making it explicit, not just in terms of [46:57] the increase in decision quality in the moment, but it actually allows you to close feedback loops much better. [47:02] So that was one thing that I found quite surprising. [47:05] But the one that I really found very interesting was being told, [47:10] Well, [47:11] What you're talking about doesn't apply. [47:13] because our feedback loops are a decade [47:17] And in poker, you got an answer right away. [47:20] You won or lost the hand. [47:22] right away. [47:23] So the way that you're thinking about decision making doesn't doesn't really apply. [47:29] Until... [47:31] I, [47:31] met first round. [47:33] and then renegade. [47:35] where they actually heard what I had to say because I gave the same answer to everybody. [47:40] So we'll just put aside that wouldn't you want to make that explicit? [47:43] The first thing that I would say is, "Oh, poker is much noisier than you think, because when I win a hand, I have no idea why." [47:50] So I do actually have to wait a long time. [47:52] Because I have to play many, many, many hands.

[47:55] before I actually know, like, [47:57] do I actually have an edge? Because I actually don't know very much. [48:01] On one hand, for one thing, I almost never see my opponent's cards. [48:05] So I'm kind of left in a dust of uncertainty. But separately from that, [48:09] The main thing that I said was, "How could you possibly think that the feedback loop is 10 years?" [48:17] And this is what I think really caught first round eye. [48:20] Because when I was talking to Josh Koppelman about it, he said, [48:24] "Well, what do you mean? We don't get an exit for 10 years." [48:26] And I said, [48:27] "Oh, I'm sorry, do you invest and then you go to sleep like Rip Van Winkle? And then you wake up 10 years later and you go, 'Hey, how'd that go?'" [48:35] Or are there all sorts of things that happen in between? [48:39] The simplest thing, [48:40] thing. The simplest thing is, does it fund at Series A? [48:45] you know, and [48:47] The little pushback that I would get there is, [48:51] But we're not investing for Series A. [48:53] And I said, well, I know that. [48:55] But have you ever had a company that exited for more than a billion dollars that [49:00] did not fund at Series A. [49:02] And the answer is no. [49:03] And I'm like, okay. [49:04] So it sounds like that's necessary. Might not be sufficient, but it's necessary. And it's certainly a signal. [49:11] that is actually more highly correlated. [49:13] with that today. [49:15] out well. [49:16] than the investment at sea. [49:18] And then, oh, right, you have series B. [49:22] And that's separate from all the other things that you can look at, like,

[49:28] you know, what you talked to Todd Jackson about. [49:30] What's going... Is it achieving product market fit? [49:33] We know that eventually for it to be [49:36] to be successful. [49:38] It's going to have to achieve product market fit. [49:42] So you can look at what's happening with that. Just general things about traction, [49:47] I, [49:48] what's happening with net new ARR, [49:51] ability to retain top talent, churn. I mean, there's so many different things that you can look at. [49:58] All of which, [50:00] are things that [50:01] Thank you. [50:02] you know must happen in order for the big thing to happen. [50:06] Okay, so what that means is that [50:08] This is the big, I'm going to make a bold statement. [50:12] here. [50:14] There is no such thing as a long feedback loop. [50:17] you can make a decision about how long the feedback loop is. [50:23] That is your choice to live in a long feedback loop. [50:27] and you can choose to shorten the feedback loop. And the way you choose to shorten the feedback loop is to say, what are the things that are necessary but not sufficient? [50:35] That's one thing for getting a good exit. Or what are the things that are correlated with? [50:39] the outcome that i eventually desire and [50:43] What that means is that when you're at the decision point, [50:47] In First Round's case, I'm going to invest in a company. [50:51] What you have to understand is that you are making a prediction about [50:54] how the world is going to unfold, how the future is going to unfold.

[50:58] And those things that you're predicting, [51:00] You can track. [51:02] and you can track them back to the decision, and you can do it pretty darn fast, mind you. I mean, think about like, [51:08] Think about being in 2021. [51:11] There were companies that were raising an A six months after seed. [51:15] Today, it's a little more 16-month [51:18] ish, but even so, [51:21] Let's just say that that was the only thing that you decided to do. [51:24] I'm going to forecast the probability that this company is going to fund at Series A. And then obviously those companies start to fund or not fund at Series A. [51:32] And you're finding that out in 16 months. [51:34] Here's my question for you, Lenny. Is 16 months shorter than 10 years? [51:39] Mm-hmm. [51:41] So, I mean, that's, you know, I mean, it's probably why he, [51:44] said to ask me that question because I just really do. I mean, I have a very strong opinion about this. [51:50] The feedback loop is as long as you choose it to be. [51:53] and [51:55] If I take that back to some of the things that I heard [51:59] early on when I was talking to people, [52:02] What I would say is that [52:06] I think that there is a certain amount of psychological safety. [52:12] And [52:12] allowing the feedback loop to stay long. [52:17] because [52:21] really have two main factors. [52:23] One is that [52:26] Look, if I was early into Uber,

[52:29] And now I'm like, [52:31] you know, a [52:32] celebrity investor or something. [52:34] I don't really want to know if I'm good or not. [52:37] Do I? [52:39] I don't really want the world to know that. If I'm good, that's great, but it feels like they already believe that I'm good because I happen to be early into Uber. [52:47] So so since people already think I'm good, like I'm kind of just losing to that decision. [52:52] psychologically speaking. [52:54] not [52:56] investment quality speaking, but psychologically speaking. [52:59] Okay, so here's the problem though. [53:01] Why was I early into Uber? [53:04] Did I have an insight? [53:07] into a real pain point. [53:10] in a developing market, [53:13] Blah, blah, blah. [53:14] or [53:16] Did my buddy start Uber? And I was like, sure, I'll give you some money. [53:20] Right. I mean, obviously, I'm talking about the extremes here. [53:22] But we don't actually know what the decision quality was. All we know is that you had a good result. And given that you had a good result, and people think very highly of you, [53:32] What are you gonna gain? [53:35] Right. So so it's like it's so nice to just sort of let that feedback loop sit there. [53:39] and allow people to have the opinion of you. [53:42] that is really nice, feels good, right? [53:46] and not actually find out the answer. [53:48] Because why would I want to? [53:51] Unless you're really super focused on decision quality, then you would want to do that. So that's, you know, that's part of the psychological safety. [53:59] And what that goes to is the real core of it.

[54:03] is [54:05] that it's very, very difficult [54:08] for human beings. [54:10] to deal with feeling wrong in the moment, even if it helps them in the long run. [54:15] It's just hard. [54:17] And the tighter the feedback loop, [54:19] the more that you risk finding out you were wrong in the short run. [54:23] Now, that helps you to learn and improve your decision making if you're focused on it and you're good at it, right? [54:29] that's going to help you. And then in the long run, you're actually going to do better. [54:32] But human beings are notoriously good at trading off the long run just to feel good in the short run. That's why we're all eating chocolate and cupcakes and stuff that we know is bad for us. [54:42] Because it feels good. [54:45] And so much of our decision making is [54:48] tried to advance this positive self-narrative. [54:51] Mm-hmm. [54:52] And the idea that, yeah, we're going to have a more positive narrative ourselves in five years if we do some stuff. [54:59] You know, most of us are like, no, I don't really want to make that trade. I'd rather just kind of feel good now. [55:05] And [55:06] I can kind of use the fact that [55:09] we're living in power law. [55:12] you know, under the influence of power law, I can kind of use that to sort of [55:17] just sort of confirm a lot of things that I wish to [55:20] to believe that are true of me. [55:23] And if you kind of take that away from me and you take the uncertainty away from me, [55:27] It's going to be really hard. [55:29] And I will tell you, that's what I love about both Renegade and First Round is that they're just like,

[55:34] I want to know. It would be such a horror for me [55:37] to think that I was [55:39] making good decisions when I actually wasn't. [55:42] And that's what really matters to me. And I think that it's just so special. [55:47] Hearing all this makes me want to be an LP in first round. Not that they would let me in there, but just knowing that you're in there. [55:53] futzing with everything and the way they're thinking is... [55:56] is really inspiring. [55:57] I'm curious if you can share an example of anything that they tweaked. [56:01] as a result of this analysis that you did and how they evaluate [56:04] Thank you. [56:05] Well, first of all, let me just say, [56:08] They didn't really record a lot about their decisions. [56:11] when I first came in. [56:12] They voted. [56:14] And they had a record of the vote. So they knew who said yes and who said no. [56:17] But they didn't really have any, they didn't have a lot of other information. So the first thing that happened was just that. [56:24] okay what do we really think uh is [56:28] the way that you would model whether you should invest or not invest. And broadly, very broadly, you would say you're rating the market, the team, the founder, the product. [56:39] broadly. We're going to make sure that those opinions aren't just like, it's good or it's bad, but it's on a scale of one to seven so that we can actually get some precision and [56:48] some spread, [56:50] among the partners in terms of say strength of market. We're going to make sure that we have shared definitions. [56:56] of those things, which you'd be surprised, people don't. So when I'm thinking about market and market quality, I might be thinking about something very different than you. So we want to make sure that

[57:06] We have a shared definition of that, and that's reflected in something that we call mediating judgments, which are, [57:11] judgments that you make [57:13] related to market prior to actually judging what you think of the market in general. So you could think of something like competitive landscape. [57:21] So you would judge that. So that turns into what these mediating judgments are, which is basically an implied definition. So you create that and then you also think about what are the forecasts that are important. One thing that you already know is [57:35] You're going to forecast the probability the company funds at Series X. [57:39] So that was like a huge change. [57:41] which is a very different way of making decisions. [57:45] What we've been able to do with that now, because I've been there for five years, [57:49] is, [57:50] we now can actually look, say, at the partnership as a whole, [57:54] and look at these ratings that they're making of the component pieces of parts of how they're modeling, like what makes a good investment, these forecasts that they're making. We actually know how these companies have now unfolded. [58:06] So in the simplest sense, we know whether hundreds of companies [58:11] have funded at Series A or not. [58:14] And now we can actually look and say, [58:16] How good are the partners? [58:19] at actually forecasting this thing. [58:21] right are they random are they better than random and we we actually know that and we can feed that back to them so that they can understand their own accuracy around these pieces of the decision because the fact is that [58:36] Whatever a seed investor says to you,

[58:40] Whether that company is going to fund it series A is part of their decision. It's included in the decision. So they're making that forecast, whether they make the forecast explicitly or not. So what we're saying over at first round is let's make it explicit. [58:53] because you're doing it implicitly anyway. And then we can actually start to look at your accuracy. We can now feed that back to you and let you know. [59:00] how accurate you are. [59:02] which will then help you to become... [59:05] more accurate. We can also look because we know in any given vintage, [59:11] sort of what are the best companies or what are the worst companies. Remember, we're having people do these ratings like on a scale of one to seven of say the quality of the market. And we can look across the partnership and say, look, [59:22] Lenny, when you think the market is great, how does that map onto? [59:27] how well the company is doing. [59:30] in the future. [59:31] When you say the market is terrible, [59:33] How well does that map onto? [59:36] how that company is doing, how that company ends up doing in the future. And I can come to you and I can say, "Oh, Lenny, by the way, [59:44] Your judgments about market are amazing. [59:47] You know a good market when you see one and it's really mapping on in a great way. [59:53] onto how that company unfolds. [59:55] but you're not so great with Founder. [59:58] or maybe you're great across the board, or maybe you're not. So we can now start to give people [1:00:04] we can give the partners insight into their own decision making, not only to allow them to improve the decision making, but also to allow them

[1:00:11] to understand [1:00:13] that they not to over index on certain things that maybe aren't as predictive as an example for them. It allows us to we also can change the rubric based on the evidence now. So the first version of the rubric is always. [1:00:25] taking the intuitions of the partners and making those explicit. But then we can start to loop those back together. [1:00:33] and understand [1:00:34] uh, [1:00:36] well maybe this thing that the partners thought was important actually isn't predictive across any partner for example we can start to [1:00:42] develop the rubric based on the data. These were all things that weren't possible. [1:00:47] Because, [1:00:48] Prior to that, [1:00:49] If I had come in and said, well, let's look at decision quality, how would I do that? [1:00:55] I have no idea why people were – I just know whether they said yes or no. And so it's very difficult to then to start to do some really serious refining of the decisions. [1:01:05] if I don't have that information. [1:01:08] I desperately want to know which partner makes the best decisions. I know you're not going to share that. No, I'm not going to. [1:01:14] The partnership as a whole is excellent, as we know. [1:01:18] And this is what I will tell you is that, [1:01:20] All partners have strengths and all partners have weaknesses. [1:01:24] and they are not perfectly overlapping which is wonderful right i mean that's one of the things it's [1:01:29] What's really wonderful, and I think it shows the power of why would you have more than one person having input into a decision, is some people are very strong on rating a particular aspect of market, or some people are very strong on rating a particular aspect of business.

[1:01:45] the founder, there's overlap and there's, [1:01:48] you know, and then there's, [1:01:50] things where Todd is uniquely great at something. [1:01:53] or Josh is uniquely great at something, or Bill is uniquely great at something. So that's the wonderful thing about it is that everybody has strengths and everybody has weaknesses, and they're not perfectly overlapping. [1:02:03] So this is where you can see like getting that spread of opinions and really understanding. [1:02:08] breaking that decision down into its component parts [1:02:12] really shows you the value of diverse opinions [1:02:17] you know, as input into a decision. [1:02:21] I could talk about this thread forever. Maybe let me just ask one more question just because I'm super curious. Is there anything surprising that stands out that came out of this analysis so far? Just like, oh, wow, maybe... [1:02:31] market isn't as important as we thought or this person is amazing at it. Yes. Just generally speaking, [1:02:38] there, you know, when you're creating the initial decision rubric, there are people that there are things that people are really pounding the table about that they think is like especially important. [1:02:47] in making a decision. [1:02:49] and [1:02:50] One of the things that we found is that sometimes, [1:02:55] their intuition was absolutely right. [1:02:57] the thing that they were pounding the table about, [1:03:00] is incredibly predictive. [1:03:02] not just for them, but other other partners. [1:03:05] But sometimes it's not at all predictive. [1:03:09] And, you know, and these are equal table pounding situations. So let's say you're pounding the table about something. Sometimes the thing you're pounding a table about it is predictive for you.

[1:03:22] And for all the other partners that it's actually quite predictive about how the company does. But sometimes when you're really pounding the table about something, it's not just it's not predictive for the other partners. It's not predictive for you. [1:03:32] And I think that what's really important to understand about this, and this is why it's so incredibly important, [1:03:37] necessary. [1:03:38] in improving decision quality, [1:03:40] to take what's implicit and make it explicit, is that our intuition is sometimes right and sometimes wrong. [1:03:48] It's not that intuition is crap and your intuition is just completely wrong. I mean, obviously that can't be true. We would die. Right. So your intuition is sometimes right. [1:03:58] but it's also sometimes wrong. [1:04:00] And if you don't, [1:04:02] Make sure. [1:04:02] inexplicit. [1:04:04] then you don't get to find out when it's wrong. [1:04:07] You don't get to find out when it's off base. And that's a disaster. [1:04:11] So that's the thing that I think is really interesting is that like you have equal, you [1:04:17] Be a man. [1:04:18] and confidence that this particular factor is really important. [1:04:22] And sometimes it is and sometimes it isn't. [1:04:25] And it's so surprising because we're talking about people who are true experts. [1:04:31] who are great. [1:04:33] and [1:04:34] I think that we all just... [1:04:37] have intuition about intuition, right? Like you just intuit that, [1:04:41] If they're so amazing, clearly their intuition about what's important would have to be good. [1:04:49] But not necessarily. That's the thing. Not necessarily.

[1:04:52] Yeah, I think that was probably the most exciting thing. [1:04:56] keeping it mysterious, but I still appreciate you sharing. Well, I have to keep it mysterious. I can't give away the trade secrets. [1:05:06] I wanted to touch on a different framework that I've heard a lot of companies actually using, and something that comes up in product a lot is this idea of, [1:05:13] Premortems. [1:05:14] which is essentially think ahead of time what might go wrong. Can you talk about this? Because I think this is something that's easy to implement, really powerful. [1:05:21] And a lot of people are actually doing this. [1:05:24] Thank you. [1:05:25] Yes. Okay. So a pre-mortem is great, but only if you attach a pre-commitment with it. [1:05:32] So I just want to be super clear about that. What you find with premortems is actually work. I actually did this work with Marie Schweitzer and Linnea Gombe, who are both at Penn. [1:05:42] Then when you have people do pre-mortems, [1:05:44] uh, [1:05:45] it generally doesn't actually change their plan very much. [1:05:49] or change their behavior. So I think that we have the feeling that if you do a pre-mortem and you think like what are the ways that things might go wrong, that that's going to change your plan. But probably not unless [1:06:01] you're specifically using it for that purpose. [1:06:04] And you say, okay, we're going to do this, but... [1:06:07] Let's think about how we might change our plan in light of this information. But I think what's actually more important than that, [1:06:13] is what a premortem allows you to do is to set up kill criteria. [1:06:17] So, [1:06:19] Kill criteria are just [1:06:21] a set of

[1:06:22] signals. [1:06:24] that you might see. [1:06:25] that would tell you that it's time to pivot or stop. [1:06:29] Because once we actually launch something, [1:06:33] We're very, very slow. [1:06:35] to decide to quit. [1:06:37] I'm sure that everybody has felt that way before. Things go on way too long, even like they're over budget and you've blown the timeline and, [1:06:44] When you finally shut it down, you realize you should have done it a lot earlier. [1:06:48] And this is kind of true across the board because a variety of biases, the most well-known and probably the biggest influence is something called sunk cost. [1:06:56] which is that feeling, but then I'll have wasted all the time and effort [1:07:00] that I've put into this already. So it's taking into account what you've put in in the past and deciding whether to continue [1:07:06] on in the future. [1:07:08] So what we want to do is actually just get better at that thing. So sort of understanding that when you've gotten to the premortem process, [1:07:17] you probably are going to launch it. [1:07:20] like it's probably going to be the case. [1:07:22] Use the premortem. [1:07:24] to set up kill criteria. [1:07:26] So I'll give you an example from a sales team that I worked with. [1:07:30] Basically, I sent them out a prompt, all the ICs a prompt. [1:07:35] that was imagine that you got a lead through an RFP or RFI, [1:07:41] and you worked on it for six months, [1:07:43] And now it's six months later and the deal is dead. [1:07:46] Looking back, you realize there were early signals that that was going to. [1:07:49] be the case, what were they? So this is a premortem, like what are the things that you saw

[1:07:54] that would tell you that this was going to go south. [1:07:57] And they came up with all sorts of ideas. [1:07:59] Notice this doesn't mean they're not gonna, [1:08:02] start off pursuing the lead, right? But they came up with all sorts of signals. [1:08:08] So I'll just give you three of them. [1:08:11] I... [1:08:12] The RFP RFI was clearly written with a competitor in mind. [1:08:15] So they felt that was a very bad signal. [1:08:17] that was like probably gonna go badly. [1:08:20] Another one was the customer didn't want a demo, they only want to talk about price. [1:08:25] Obviously, that's quite bad. [1:08:27] And another one was after the first few meetings, they couldn't get a decision maker in the room. [1:08:31] It was a much longer list than this, but those are three. For each of those, that now becomes a kill criteria. If I see this thing, [1:08:40] And now you attach an action with it. [1:08:43] So in the case of price, [1:08:45] They actually just said we should kill it. [1:08:48] If they literally don't want a demo and they're only asking about price, they're just trying to beat up somebody else on price. Like that's, [1:08:54] there were a box checking exercise. So there they just said, we're going to kill it. We're not going to pursue the deal anymore. [1:08:59] So this is great because salespeople, [1:09:01] will not [1:09:02] will pursue deals forever. And leadership is like, "Well, why did you stop pursuing that deal?" and they get in trouble for it. So this is going to help with that problem, right? [1:09:11] In the case of [1:09:13] The RFP RFI was written with a competitor in mind. They have an action associated with that as well, which is, [1:09:18] Ask them directly. [1:09:19] if they're working with a competitor, [1:09:21] and how far down the road they are. [1:09:23] depending on the answer you would kill or pursue. In the case of we couldn't get a decision maker in the room, offer up executive alignment at the next meeting.

[1:09:31] And if they say, sure, great. And if they say no, [1:09:34] Kill. [1:09:35] So that's actually what I feel is the best use of a premortem is to say, I'm going to try to figure out. [1:09:42] what those signals are along the way, [1:09:44] that things are going badly. [1:09:46] And now, instead of just kind of hoping, [1:09:49] that when I see those signals, I actually act rationally, which is [1:09:52] I hope that [1:09:53] It will not come true. That's why there's many people who climb Everest in the middle of a blizzard, even though [1:09:59] they shouldn't be doing that. [1:10:01] Use the premortem to now create structure around those signals that you've spotted. [1:10:07] and [1:10:08] Commit to actions that you're going to take if you see those signals. [1:10:12] And I think that's the best use of a premortem. [1:10:15] That's really helpful. And it's interesting how many of your examples come back to [1:10:19] just a framework that you often talk about, which is make explicit what is implicit. [1:10:23] This is another example that the first round example is a great example that where it's just [1:10:28] Here's all our assumptions that just make them actually explicit and just shows how much power there is. Then you can examine them. [1:10:33] You can examine them. You can... [1:10:35] People can discuss them. You can figure out if they're wrong or right or [1:10:39] whatever it's like [1:10:41] I want to be very clear. I'm not anti what... [1:10:45] you know, anti your gut. [1:10:47] Or your intuition. [1:10:49] I think it's probably sometimes pretty good. [1:10:52] I just want you to make it explicit, that's all. [1:10:55] Okay, so we didn't have time to get into quitting, which is your more recent book. [1:10:59] Maybe we'll do a follow-up episode specifically thinking about quitting. It's my fault because my answers are long. I apologize to everybody. No apologies necessary. We'll have plenty of time in the future, hopefully.

[1:11:10] Let me just ask one question. I found this one quote from you. [1:11:13] where you said, "You should assume that if you're thinking about quitting, it's already probably past the time that you should have quit." [1:11:20] Do you still believe that? Is that generally a good rule of thumb in just any... [1:11:23] Take away. [1:11:25] lesson on quitting as our one question on quitting. [1:11:29] So, [1:11:30] The data is pretty strong. [1:11:33] that [1:11:34] By the time you quit, it's probably long after you should have. [1:11:38] I [1:11:38] And it's really just because [1:11:41] Look, [1:11:42] when we start things, we're starting things under difficult circumstances. [1:11:46] which come from the uncertainty of the decision to start something. [1:11:51] So when we start something, luck is going to have an influence on the outcome, which obviously we have no control over. [1:11:56] because luck isn't in our control. And then there's also hidden information. So what happens is that after the fact, [1:12:01] We know that we're going to learn new information and it can make it very hard to start things. [1:12:06] Because we sort of want to be more sure than we actually need to be. [1:12:09] It's why, you know, Bezos has the 70% rule to try to roll people back and be willing to accept that uncertainty in the starting decision. Now, the good news is, [1:12:18] that [1:12:18] when you learn that new stuff. [1:12:21] and the new stuff that you learn is, "Ooh, if I had known this, I wouldn't have started it," you have the option to quit generally. [1:12:28] So that's the good news. The bad news is that the same difficulties [1:12:31] that apply to the decision to start apply to the decision to stop. In other words, we're making that decision under uncertainty as well. [1:12:38] So we're not gonna know for sure whether it would have turned out well.

[1:12:42] or poorly. [1:12:43] unless we continue [1:12:45] to do the thing that we already started. And we don't like to walk away from things unless we know for sure. [1:12:51] So as Richard Thaler put it, [1:12:53] most people won't quit until it actually isn't a decision. In other words, the whole thing is blown up, or the startup has no money, or you're up on Everest and the blizzard is literally upon you. [1:13:07] and you're stuck in it. [1:13:08] or I think as he said, they won't until you've fallen in the crevasse already, then you'll make the decision to quit because then you know how it was really bad. So people generally like for example, [1:13:21] don't quit their jobs until they sort of feel they have no other choice or, [1:13:24] relationships or [1:13:27] projects or [1:13:28] products that they're developing, it all applies. [1:13:31] Because we want to know, we're just sort of, we want to know for sure. And then on top of that is the fact that, [1:13:37] There is this issue of sunk cost, which is, [1:13:40] When we walk away, we feel like we'll have wasted everything that we've already put into what we're doing. But of course, waste is a prospective problem, not a retrospective one, even if we treat it like a retrospective one, because it's the prospective one. Well, if you wouldn't start this today. [1:13:54] then that means that everything that you're putting into this going forward is the actual waste. [1:13:58] But we do that all the time. We go forward with things. [1:14:02] that we ought not to be going forward with because we're trying to protect the resources that we've already sunk into it in the past. Then there's other issues that have to do with, for example, endowment, the ownership over the things that we've built. This is particularly bad in product because we're building things.

[1:14:17] And once we build things, we own them. [1:14:20] And once we own them, we don't want to give them up, and we actually value them more highly than identical things that we don't own. [1:14:26] And then there's issues of internal and external validity, which is really just a fancy way to say your identity. [1:14:31] How do other people view you? How do you view yourself? Do you feel like you failed? [1:14:36] And what that means is that by the time [1:14:38] There's so many biases against stopping. [1:14:41] That by the time you're actually even thinking about quitting, [1:14:44] Like that, it's probably already past the time that you ought to have quit. [1:14:48] But we'll still continue on until we know for sure we didn't have any choice. Because here's the thing. When you walk away from something and someone's like, hey, why'd you stop that? [1:14:58] and you're like, "Oh, I had no choice," and you tell them everything that went wrong and [1:15:02] nobody's going to question you. They're going to be like, ah, well, it sounds like you put in your [1:15:07] your best effort. [1:15:09] But if you walk away early, people are like, what? [1:15:12] So just quickly, I'll tell you, I think, just one of the best stories of this. [1:15:16] that I've got in my pocket here. Let me pull it out before the end. So, [1:15:22] Stuart Butterfield. [1:15:25] creating a product called Glitch. [1:15:27] And Glitch is a... [1:15:29] a massive multiplayer online world-building cooperative game. [1:15:34] releases it, this is in the [1:15:36] thoughts. It's like a huge hit with the critics. It's Monty Python meets Dr. Seuss. It's an incredible whatever. They're getting tons and tons of great word of mouth and PR,

[1:15:48] not doing any paid marketing. They have incredible investors in Andreessen Horitz and Excel. They have $6 million in the bank and they have 5,000 diehard users, meaning users who use the game [1:16:01] who play over 20 hours a week. [1:16:03] The issue is, [1:16:05] that [1:16:06] Customer acquisition was a beast. [1:16:09] that for every one person, [1:16:12] who was playing over 20 hours a week, [1:16:14] There were like between 95 and 99 people. [1:16:17] who came for five minutes. [1:16:20] left. So obviously this is a customer acquisition problem which which everybody kind of knows. So they make an agreement in 2012 and [1:16:29] that they're going to do paid marketing, which they do for six weeks. And during that six weeks, their acquisition of new users, it's growing like six to 7% week over week, which is amazing. And at the end of that six weeks, [1:16:42] This is November of 2012. At the end of that six weeks, [1:16:47] that Monday morning. [1:16:48] Stuart Butterfield writes a note to his investors and co-founders and says, "I woke up this morning, [1:16:53] with the dead certainty that Glitch was over. [1:16:58] Now notice nobody would do this in this case, right? [1:17:02] But this is what happened is that the issue is that you really kind of have to see, like, is this worthwhile or not? Would I start this today? [1:17:09] That's a forecast of the future, right? And what he did was some back of the envelope math. And he said, look, [1:17:14] If we continue to acquire customers at the weight that we've been acquiring them, at the cost that we've been acquiring them,

[1:17:20] It's going to be 31 weeks till we break even. But that's an absurd assumption because [1:17:26] Customer acquisition cost, it's going to go up. Cap has to rise because we're going to saturate the core gaming market. [1:17:32] So it's got to rise. [1:17:34] So what he realized at that point was that this was not a venture-scale business, and he was in this for a venture-scale business. [1:17:40] So even though nobody else saw that he was supposed to shut it down, [1:17:43] He saw he was supposed to shut it down. And not only that, he saw that he was supposed to shut it down for his employees who were working for equity. And he had now realized that the equity wasn't worth their time. [1:17:53] and that it wasn't fair to them to keep going with it. So he shuts it down. [1:17:58] Obviously, that feels, who does that, right? But that, and he will actually tell you that he knows he should have shut it down before the marketing push. [1:18:06] but he needed the marketing push to prove it to himself. [1:18:10] that he was seeing the future clearly. Now, the coda to this story, just quickly, is two days later, he's like, "Well, I'm a startup guy. "I wanna start something." And he's got this internal communication tool that his team is using in order to develop this product that everybody loves, and he says, [1:18:25] "Actually, they really like that. "Maybe that should be the next product." So he goes and talks to the investors. They roll their money over into that. [1:18:32] And that thing, which had no name at the time, [1:18:35] now gets a name, which is searchable log of all company knowledge, [1:18:39] which is Slack. [1:18:40] And so I think this is the important thing to realize is that we get so focused on [1:18:44] But what about everything that I've put into it? [1:18:47] when what we forget is that when you're doing something,

[1:18:51] There's not just the cost of doing something that's not worthwhile, that's direct. [1:18:56] But there's also the cost of not being able to devote your attention [1:19:00] to other opportunities [1:19:01] that might be available to you. And as smart as Stuart Butterfield is, he couldn't see Slack until he quit Glitch. [1:19:10] And that is a true cost that he would have borne of continuing with Glitch. If he had continued with Glitch, Slack would not be something that we're all using today. [1:19:21] That is an incredibly beautiful way to end our conversation. [1:19:25] I feel like I have at least a billion more questions to ask you. And on the other hand, I feel like we've also helped a lot of people make much better decisions. [1:19:33] through this chat. [1:19:34] So I'm really thankful that you made time for this. [1:19:36] Two last questions. Where can folks find you online if they want to learn about the stuff you're up to in case they want to work with you? And how can listeners be useful to you? [1:19:43] You can find me at AnnieDuke.com if you're interested in working with me. I have a sub stack, Thinking and Bets. Please go check that out. I teach a class on Maven.com twice a year on effective decision making. [1:19:56] So if you're interested in that, you can go to Maven and check it out. My next cohort at the moment is in September, although I might do one in May. I'm not sure. But in terms of how people can help me, I co-founded an organization called the Alliance for Decision Education. We're trying to bring the kinds of knowledge that we have about improving human decision making in adults. [1:20:18] to K-12 education to make the world a better place. So I would love it if people could go look at that. If you're interested in it, get the word out.

[1:20:26] Amazing. So we'll link to all those things in the show notes. [1:20:29] Annie, again, thank you so much for being here. Thank you. Thank you so much. This was so fun. Same. Bye, everyone.