Build for the Memo, Not the Demo — Shawn Chan, China Resources Holdings

>> Good afternoon. Thank you for being >> Good afternoon. Thank you for being here, day four of our conference, a room here, day four of our conference, a room here, day four of our conference, a room with no windows. with no windows. with no windows. Right just after lunch. Right just after lunch. Right just after lunch. You are the strongest people in this You are the strongest people in this You are the strongest people in this building. building. building. Let me start with a confession. For 15 Let me start with a confession. For 15 Let me start with a confession. For 15 years, my job has been one thing. I years, my job has been one thing. I years, my job has been one thing. I sitting in a room, a very smart, sitting in a room, a very smart, sitting in a room, a very smart, confident person hands me a piece of confident person hands me a piece of confident person hands me a piece of paper, paper, paper, and before my company spends a hundred and before my company spends a hundred and before my company spends a hundred million dollars, I have to decide, do I million dollars, I have to decide, do I million dollars, I have to decide, do I believe this paper? This year, my job believe this paper? This year, my job believe this paper? This year, my job still exactly that. Except now, some of time, the very Except now, some of time, the very confident person handing me confident person handing me confident person handing me the paper is a chatbot. the paper is a chatbot. the paper is a chatbot. And honestly, And honestly, And honestly, the chatbot is often better written than the chatbot is often better written than the chatbot is often better written than humans, better grammar, nicer humans, better grammar, nicer humans, better grammar, nicer formatting, formatting, formatting, never get defensive when you ask a never get defensive when you ask a never get defensive when you ask a follow-up question, very very sure of follow-up question, very very sure of follow-up question, very very sure of itself. itself. itself. The problem is being sure of yourself The problem is being sure of yourself The problem is being sure of yourself and being right are two different and being right are two different and being right are two different skills. Some of most the confident skills. Some of most the confident skills. Some of most the confident people I have ever met in finance were people I have ever met in finance were people I have ever met in finance were also the most wrong. AI just learned also the most wrong. AI just learned also the most wrong. AI just learned that trick faster than rest of us.

that trick faster than rest of us. that trick faster than rest of us. So, here my whole war whole talk in two So, here my whole war whole talk in two So, here my whole war whole talk in two sentence. First, almost every AI finance sentence. First, almost every AI finance sentence. First, almost every AI finance product is built to impress people for product is built to impress people for product is built to impress people for five minutes, and almost none are built five minutes, and almost none are built five minutes, and almost none are built to survive a room whose entire job is to survive a room whose entire job is to survive a room whose entire job is to not be impressed. Second, and this is to not be impressed. Second, and this is to not be impressed. Second, and this is the part I promised the organizers added the part I promised the organizers added the part I promised the organizers added the exact same skills that fix your the exact same skills that fix your the exact same skills that fix your product is the skills that gets product is the skills that gets product is the skills that gets investors like me to write you a check. investors like me to write you a check. investors like me to write you a check. Same muscle. I've proved both. And I'm Same muscle. I've proved both. And I'm Same muscle. I've proved both. And I'm telling this now, this week, because a telling this now, this week, because a telling this now, this week, because a lot of you lot of you lot of you are about to get pulled into exactly are about to get pulled into exactly are about to get pulled into exactly this. Your CEO saw a demo somewhere. this. Your CEO saw a demo somewhere. this. Your CEO saw a demo somewhere. Your biggest comes a customer suddenly Your biggest comes a customer suddenly Your biggest comes a customer suddenly has a compliance department, or you are has a compliance department, or you are has a compliance department, or you are 3 months from raising your next round.

3 months from raising your next round. 3 months from raising your next round. When any of those days arrive, I'd When any of those days arrive, I'd When any of those days arrive, I'd rather you to hear the hard part from rather you to hear the hard part from rather you to hear the hard part from someone who sit on the other side of the someone who sit on the other side of the someone who sit on the other side of the table, table, table, than discover them live in a room in than discover them live in a room in than discover them live in a room in front of the people front of the people front of the people who bill by hour. A quick A quick quick a bit about me. I promise this is quick a bit about me. I promise this is quick a bit about me. I promise this is the boring part, and I will keep it the boring part, and I will keep it the boring part, and I will keep it fast. 15 years of cross-border deals, fast. 15 years of cross-border deals, fast. 15 years of cross-border deals, Hong Kong, mainland China, the UK, the Hong Kong, mainland China, the UK, the Hong Kong, mainland China, the UK, the US. US. US. Mergers, IPOs, big strategic investment Mergers, IPOs, big strategic investment Mergers, IPOs, big strategic investment on names you'd actually recognize, the on names you'd actually recognize, the on names you'd actually recognize, the kind of companies kind of companies kind of companies that go public, and your LinkedIn feed that go public, and your LinkedIn feed that go public, and your LinkedIn feed won't shut up and about it for a week. won't shut up and about it for a week. won't shut up and about it for a week. Along the way, I've sit in about 200 Along the way, I've sit in about 200 Along the way, I've sit in about 200 investment committee meetings. investment committee meetings. investment committee meetings. I have I have I have the grey hairs to prove it. I checked.

the grey hairs to prove it. I checked. the grey hairs to prove it. I checked. It's not genetics. It's not genetics. It's not genetics. And here And here And here the part that matters for second half of the part that matters for second half of the part that matters for second half of this talk. I've also read hundreds and this talk. I've also read hundreds and this talk. I've also read hundreds and hundreds of pitch decks, founders' hundreds of pitch decks, founders' hundreds of pitch decks, founders' decks, decks, decks, bankers' decks. bankers' decks. bankers' decks. 47 slide seed decks. 47 slide seed decks. 47 slide seed decks. I have seen fonts that should be I have seen fonts that should be I have seen fonts that should be illegal. illegal. illegal. >> [sighs and gasps] >> [sighs and gasps] >> [sighs and gasps] >> 200 committee meetings taught me one >> 200 committee meetings taught me one >> 200 committee meetings taught me one thing. thing. thing. No textbook says outlawed. The number on No textbook says outlawed. The number on No textbook says outlawed. The number on the page is not what gets a deal the page is not what gets a deal the page is not what gets a deal approved. Trust is a number. What gets a approved. Trust is a number. What gets a approved. Trust is a number. What gets a deal approved? Money doesn't deal approved? Money doesn't deal approved? Money doesn't follow intelligence. follow intelligence. follow intelligence. Money Money Money follows trust, and the trust is fragile, follows trust, and the trust is fragile, follows trust, and the trust is fragile, especially when the thing especially when the thing especially when the thing that wrote the page has never once that wrote the page has never once that wrote the page has never once in its entire life said the words, "I'm in its entire life said the words, "I'm in its entire life said the words, "I'm not sure."

not sure." not sure." Let me give you Let me give you Let me give you one small taste of what those rooms feel one small taste of what those rooms feel one small taste of what those rooms feel like. Early in my career, a very like. Early in my career, a very like. Early in my career, a very polished, polished, polished, very expensive banker present a very expensive banker present a very expensive banker present a beautiful slides full of confident beautiful slides full of confident beautiful slides full of confident wrong numbers. wrong numbers. wrong numbers. One senior person in room asked a one One senior person in room asked a one One senior person in room asked a one quiet question, "Where does this number quiet question, "Where does this number quiet question, "Where does this number come from?" The banker posed uh come from?" The banker posed uh come from?" The banker posed uh what felt like an entire physical what felt like an entire physical what felt like an entire physical quarter. quarter. quarter. The pose taught me more about finance The pose taught me more about finance The pose taught me more about finance than 3 years of exams did. So, than 3 years of exams did. So, than 3 years of exams did. So, I'm not sure as a I'm not sure as a I'm not sure as a So, I'm not here as a builder. I don't So, I'm not here as a builder. I don't So, I'm not here as a builder. I don't build these systems. I sit across the build these systems. I sit across the build these systems. I sit across the table from them and from the founders table from them and from the founders table from them and from the founders selling them, deciding whether to trust selling them, deciding whether to trust selling them, deciding whether to trust them. them. them. Today, Today, Today, I'm going to give you both I'm going to give you both I'm going to give you both halves of that. What breaks trust in halves of that. What breaks trust in halves of that. What breaks trust in your product and uh what builds trust in your product and uh what builds trust in your product and uh what builds trust in your pitch.

Let's define two machines that moves Let's define two machines that moves everyone keep confusing. Machine one is everyone keep confusing. Machine one is everyone keep confusing. Machine one is a demo. One clean document in, one a demo. One clean document in, one a demo. One clean document in, one fluent answer out is the whole job is to fluent answer out is the whole job is to fluent answer out is the whole job is to make a room go oh for 5 minutes. make a room go oh for 5 minutes. make a room go oh for 5 minutes. Machine two is a is a memo. The real Machine two is a is a memo. The real Machine two is a is a memo. The real document is a real committee race before document is a real committee race before document is a real committee race before real money moves. Hundreds of pages, real money moves. Hundreds of pages, real money moves. Hundreds of pages, filings, transcripts, broken nose, filings, transcripts, broken nose, filings, transcripts, broken nose, scratches, someone's rushed nose from a scratches, someone's rushed nose from a scratches, someone's rushed nose from a call last Tuesday. Half call last Tuesday. Half call last Tuesday. Half the sources disagree with each other and the sources disagree with each other and the sources disagree with each other and the memo's job is not to make you the memo's job is not to make you the memo's job is not to make you go oh. Its job's to survive an argument. go oh. Its job's to survive an argument. go oh. Its job's to survive an argument. Basically, a family dinner except Basically, a family dinner except Basically, a family dinner except somebody's uncle brought a spreadsheet. somebody's uncle brought a spreadsheet. somebody's uncle brought a spreadsheet. Here's everyday worship of the gap. Ask Here's everyday worship of the gap. Ask Here's everyday worship of the gap. Ask your phone to summarize a long email. your phone to summarize a long email. your phone to summarize a long email. That's a demo. It's not just sound That's a demo. It's not just sound That's a demo. It's not just sound plausible for 10 seconds.

plausible for 10 seconds. plausible for 10 seconds. Now, imagine that some Now, imagine that some Now, imagine that some same phone has to stand in front of same phone has to stand in front of same phone has to stand in front of a bank and defend out loud why you a bank and defend out loud why you a bank and defend out loud why you deserve a mortgage. Suddenly, plausible deserve a mortgage. Suddenly, plausible deserve a mortgage. Suddenly, plausible isn't enough. Now, it has to be right isn't enough. Now, it has to be right isn't enough. Now, it has to be right and it has to and it has to and it has to prove it. prove it. prove it. That second situation is what a memo That second situation is what a memo That second situation is what a memo actually is. Now, you might think the demo world and Now, you might think the demo world and the memo world never touch. Let me tell the memo world never touch. Let me tell the memo world never touch. Let me tell you about the most you about the most you about the most expensive typo in history. February expensive typo in history. February expensive typo in history. February 2023, one of the biggest tech company on 2023, one of the biggest tech company on 2023, one of the biggest tech company on the earth launches its shiny new AI the earth launches its shiny new AI the earth launches its shiny new AI assistant with a promotional demo. In assistant with a promotional demo. In assistant with a promotional demo. In that demo, the assistant answers a that demo, the assistant answers a that demo, the assistant answers a simple question about a space telescope simple question about a space telescope simple question about a space telescope and it gets a wrong the wrong answer. and it gets a wrong the wrong answer. and it gets a wrong the wrong answer. One sentence One sentence One sentence one wrong fact about a telescope in um one wrong fact about a telescope in um one wrong fact about a telescope in um marketing demo, the market noticed the marketing demo, the market noticed the marketing demo, the market noticed the company's stock dropped around 8% in a company's stock dropped around 8% in a company's stock dropped around 8% in a day.

day. day. That's roughly $100 billion That's roughly $100 billion That's roughly $100 billion of value gone because of unchecked of value gone because of unchecked of value gone because of unchecked sentence. $100 billion sentence. $100 billion sentence. $100 billion for one sentence. Nobody for one sentence. Nobody for one sentence. Nobody in that company asked the one question. in that company asked the one question. in that company asked the one question. Every junior analyst on my team is Every junior analyst on my team is Every junior analyst on my team is trained to ask before anything leaves trained to ask before anything leaves trained to ask before anything leaves the building, wait. the building, wait. the building, wait. Where does this claim come from? Did Where does this claim come from? Did Where does this claim come from? Did everyone check it? everyone check it? everyone check it? So, here's the punchline. Even the demo So, here's the punchline. Even the demo So, here's the punchline. Even the demo failed the memo test. The moment failed the memo test. The moment failed the memo test. The moment real money is watching and the real real money is watching and the real real money is watching and the real money is always watching. money is always watching. money is always watching. Every sentence become a memo sentence. Every sentence become a memo sentence. Every sentence become a memo sentence. There is no safe demo anymore. There is no safe demo anymore. There is no safe demo anymore. Keep that story in your head because of Keep that story in your head because of Keep that story in your head because of the same trust breaking moment happens the same trust breaking moment happens the same trust breaking moment happens in six in six in six smaller, quieter, very predictable ways smaller, quieter, very predictable ways smaller, quieter, very predictable ways inside of your product every day. Let's inside of your product every day. Let's inside of your product every day. Let's go through them fast.

Six ways trust quietly breaks. Six ways trust quietly breaks. Which source do you believe? Which source do you believe? Which source do you believe? Do the numbers agree? Do the numbers agree? Do the numbers agree? Do you hide Do you hide Do you hide contradictions or show them? Is that the contradictions or show them? Is that the contradictions or show them? Is that the a fact or a guess? Can you prove it in a fact or a guess? Can you prove it in a fact or a guess? Can you prove it in 30 seconds? 30 seconds? 30 seconds? And whose name is actually on the And whose name is actually on the And whose name is actually on the decision? Six. decision? Six. decision? Six. Keep count with me. I will keep each one Keep count with me. I will keep each one Keep count with me. I will keep each one short and I will bring receipts. Model one. Model one. Not every source deserves the same Not every source deserves the same Not every source deserves the same trust, but most AI system treat them trust, but most AI system treat them trust, but most AI system treat them like they do. Think of like this, a like they do. Think of like this, a like they do. Think of like this, a number from an audit filing is your number from an audit filing is your number from an audit filing is your accountant accountant accountant speaking under oath. speaking under oath. speaking under oath. A number from a analyst note is a friend A number from a analyst note is a friend A number from a analyst note is a friend at a party at a party at a party confident probably from someone's confident probably from someone's confident probably from someone's internal email is a thing you overheard internal email is a thing you overheard internal email is a thing you overheard in in a elevator.

in in a elevator. in in a elevator. Most of Most of Most of retrieval systems can't tell them these retrieval systems can't tell them these retrieval systems can't tell them these apart. apart. apart. They grab whichever text is close to They grab whichever text is close to They grab whichever text is close to your question and hand it over like a your question and hand it over like a your question and hand it over like a gospel. gospel. gospel. True story. Anonymized. I know I once True story. Anonymized. I know I once True story. Anonymized. I know I once watched a watched a watched a um um um very expensive AI tool confidently very expensive AI tool confidently very expensive AI tool confidently called a number from a group chat called a number from a group chat called a number from a group chat someone's rough guess someone's rough guess someone's rough guess texted six months earlier. The model texted six months earlier. The model texted six months earlier. The model loved the confident freezing. loved the confident freezing. loved the confident freezing. The real audit number was the three rows The real audit number was the three rows The real audit number was the three rows away in the actual filings. away in the actual filings. away in the actual filings. The AI just liked the group chat version The AI just liked the group chat version The AI just liked the group chat version better. better. better. It sounded more enthusiastic. If your It sounded more enthusiastic. If your It sounded more enthusiastic. If your system can can't tell an accountant system can can't tell an accountant system can can't tell an accountant under oath from a rumor in a group chat, under oath from a rumor in a group chat, under oath from a rumor in a group chat, it it is not ready for real money.

Model two. The Model two. The numbers have to agree each other numbers have to agree each other numbers have to agree each other everywhere, every time. Here's a memo everywhere, every time. Here's a memo everywhere, every time. Here's a memo that already died. Page one says revenue that already died. Page one says revenue that already died. Page one says revenue growth 18% 18% page 11 18% 18% page 11 little table nobody reads for fun says little table nobody reads for fun says little table nobody reads for fun says 17.4 17.4 17.4 nobody in the room cares about the nobody in the room cares about the nobody in the room cares about the missing 0.6. missing 0.6. missing 0.6. They care about They care about They care about what it what it means if this person what it what it means if this person what it what it means if this person didn't check the easy mathematics what didn't check the easy mathematics what didn't check the easy mathematics what did they did they did they not check on the hard stuff. not check on the hard stuff. not check on the hard stuff. That memo didn't pass That memo didn't pass That memo didn't pass not because of the number because of not because of the number because of not because of the number because of what number you implied. what number you implied. what number you implied. And if you want the industrial strength And if you want the industrial strength And if you want the industrial strength version of this failure remember the version of this failure remember the version of this failure remember the giant American real estate company that giant American real estate company that giant American real estate company that let algorithm to let algorithm to let algorithm to buy houses at scale.

buy houses at scale. buy houses at scale. The The The algorithm was algorithm was algorithm was extremely confident about the house extremely confident about the house extremely confident about the house prices the house disagreed. prices the house disagreed. prices the house disagreed. The company ended up writing off around The company ended up writing off around The company ended up writing off around half a billion dollars shut the whole half a billion dollars shut the whole half a billion dollars shut the whole unit down and let a quarter of the staff unit down and let a quarter of the staff unit down and let a quarter of the staff go. The model wasn't stupid the model go. The model wasn't stupid the model go. The model wasn't stupid the model was unsupervised nobody built a boring was unsupervised nobody built a boring was unsupervised nobody built a boring mechanic mechanic mechanic machinery that forces the numbers to machinery that forces the numbers to machinery that forces the numbers to keep agreeing with reality after lunch keep agreeing with reality after lunch keep agreeing with reality after lunch day. Fluent and a confident remember is not Fluent and a confident remember is not the same as right. Model three Model three surprises people. surprises people. surprises people. A contradiction is not a bug a A contradiction is not a bug a A contradiction is not a bug a contradiction is a gift. If the CEO says contradiction is a gift. If the CEO says contradiction is a gift. If the CEO says one gross number on the earning call and one gross number on the earning call and one gross number on the earning call and the official filing says the different a the official filing says the different a the official filing says the different a different one, the gap is the single different one, the gap is the single different one, the gap is the single most interesting thing in the real story most interesting thing in the real story most interesting thing in the real story highs.

highs. highs. Real diligence leaves for that gap. AI Real diligence leaves for that gap. AI Real diligence leaves for that gap. AI does the opposite. does the opposite. does the opposite. It is a trained to sound smooth and It is a trained to sound smooth and It is a trained to sound smooth and helpful. So, when it hears a helpful. So, when it hears a helpful. So, when it hears a conflict, it quietly picks whichever conflict, it quietly picks whichever conflict, it quietly picks whichever version reads nicer and moves on. You version reads nicer and moves on. You version reads nicer and moves on. You never even learn there was a never even learn there was a never even learn there was a disagreement. I've sit disagreement. I've sit disagreement. I've sit through exactly these CEO's number through exactly these CEO's number through exactly these CEO's number and the filing numbers and the filing numbers and the filing numbers meaningfully different. Neither one uh meaningfully different. Neither one uh meaningfully different. Neither one uh Nobody Nobody Nobody Nobody flagged it. Everyone just used Nobody flagged it. Everyone just used Nobody flagged it. Everyone just used the nicer one. the nicer one. the nicer one. We caught it because one person happened We caught it because one person happened We caught it because one person happened to have both documents open at once. to have both documents open at once. to have both documents open at once. Pure luck. Luck Pure luck. Luck Pure luck. Luck is not a control. Your job as a builder is not a control. Your job as a builder is not a control. Your job as a builder isn't resolve the argument.

isn't resolve the argument. isn't resolve the argument. It's to make sure that It's to make sure that It's to make sure that the argument happens in front of a human the argument happens in front of a human the argument happens in front of a human instead of quietly along inside box. Model four. Model four. Facts Facts Facts and the guesses have to live in separate and the guesses have to live in separate and the guesses have to live in separate box. box. box. And the fluent AI loves melting them And the fluent AI loves melting them And the fluent AI loves melting them into one smooth sentence. Example, the into one smooth sentence. Example, the into one smooth sentence. Example, the company will likely receive approval company will likely receive approval company will likely receive approval next quarter. next quarter. next quarter. Reads like a fact. Reads like a fact. Reads like a fact. Sounds like a fact. it is a guess. Sounds like a fact. it is a guess. Sounds like a fact. it is a guess. Somebody buys the estimator wearing fact Somebody buys the estimator wearing fact Somebody buys the estimator wearing fact shaped the clothing. shaped the clothing. shaped the clothing. A committee's entire job is to agree A committee's entire job is to agree A committee's entire job is to agree with the guesses while trusting the with the guesses while trusting the with the guesses while trusting the facts. If your system melts them facts. If your system melts them facts. If your system melts them together, the committee together, the committee together, the committee can't find them can't find them can't find them seems, and then all seems, and then all seems, and then all all they can do is to is is approve or all they can do is to is is approve or all they can do is to is is approve or reject the genuine map of the document.

reject the genuine map of the document. reject the genuine map of the document. You should not spend a hundred million You should not spend a hundred million You should not spend a hundred million dollars on maps. I watched approval dollars on maps. I watched approval dollars on maps. I watched approval accept soon. Turn across three drafts of a demo into Turn across three drafts of a demo into approval received. Nobody lied. The approval received. Nobody lied. The approval received. Nobody lied. The guess just rose its fact costume a guess just rose its fact costume a guess just rose its fact costume a little longer little longer little longer each rewrite until nobody remembered it each rewrite until nobody remembered it each rewrite until nobody remembered it then then then it started as a guess. it started as a guess. it started as a guess. The approval did The approval did The approval did didn't arrive on schedule. That was an didn't arrive on schedule. That was an didn't arrive on schedule. That was an uncomfortable phone call. This fix uncomfortable phone call. This fix uncomfortable phone call. This fix almost embarrassingly cheap. Label your almost embarrassingly cheap. Label your almost embarrassingly cheap. Label your guesses, a tag, a color, anything that guesses, a tag, a color, anything that guesses, a tag, a color, anything that survives being copy pasted into someone survives being copy pasted into someone survives being copy pasted into someone else's slides three weeks later. Model five, if nobody can find where a Model five, if nobody can find where a claim come from claim come from claim come from it doesn't matter how right it is.

You've all heard about the New York's You've all heard about the New York's lawyer. lawyer. lawyer. Uh he filed a legal brief written with a Uh he filed a legal brief written with a Uh he filed a legal brief written with a chat bot's help. The brief cited six chat bot's help. The brief cited six chat bot's help. The brief cited six court cases, beautiful court cases, beautiful court cases, beautiful citations citations citations proper proper formatting, very proper proper formatting, very proper proper formatting, very convincing. One small issue, the cases convincing. One small issue, the cases convincing. One small issue, the cases didn't exist. The AI invented didn't exist. The AI invented didn't exist. The AI invented invented all six. invented all six. invented all six. And here, my favorite detail, the part And here, my favorite detail, the part And here, my favorite detail, the part that should be taught in school. Before that should be taught in school. Before that should be taught in school. Before filing, the lawyer got suspicious, so he filing, the lawyer got suspicious, so he filing, the lawyer got suspicious, so he asked the chatbot, "Are these cases asked the chatbot, "Are these cases asked the chatbot, "Are these cases real?" And the chatbot said, "Yes." real?" And the chatbot said, "Yes." real?" And the chatbot said, "Yes." That is like asking the guy who sold you That is like asking the guy who sold you That is like asking the guy who sold you the watch whether the watch is real. The the watch whether the watch is real. The the watch whether the watch is real. The judge fined him. judge fined him. judge fined him. The story went around the world, and my The story went around the world, and my The story went around the world, and my second favorite detail, the fake cases second favorite detail, the fake cases second favorite detail, the fake cases even had a real even had a real even had a real real estate surrounding names and the real estate surrounding names and the real estate surrounding names and the page numbers. The AI didn't just lie, it page numbers. The AI didn't just lie, it page numbers. The AI didn't just lie, it just just just the formatting. It cited itself the formatting. It cited itself the formatting. It cited itself beautifully, wrong but beautifully.

beautifully, wrong but beautifully. beautifully, wrong but beautifully. The license is a 30-second test. The license is a 30-second test. The license is a 30-second test. When someone points at a sentence and When someone points at a sentence and When someone points at a sentence and says, "Show me where this come from." says, "Show me where this come from." says, "Show me where this come from." You either click once or land on exact You either click once or land on exact You either click once or land on exact source paragraph, or you open seven source paragraph, or you open seven source paragraph, or you open seven browser tabs and start swiping. browser tabs and start swiping. browser tabs and start swiping. I have personally been the guy with I have personally been the guy with I have personally been the guy with seven tabs in a real meeting while the seven tabs in a real meeting while the seven tabs in a real meeting while the room full of people watched my scroll. room full of people watched my scroll. room full of people watched my scroll. 10 out of 10. 10 out of 10. 10 out of 10. Wouldn't Wouldn't Wouldn't would not recommend. If you remember would not recommend. If you remember would not recommend. If you remember only one sentence from this whole talk. only one sentence from this whole talk. only one sentence from this whole talk. The click-through is a product. The click-through is a product. The click-through is a product. Everything else is well-written Everything else is well-written Everything else is well-written packaging. Model six, my favorite, because Model six, my favorite, because it's the most human someone has to sign.

Here's the story you probably know. An Here's the story you probably know. An airline website chatbot told a grieving airline website chatbot told a grieving airline website chatbot told a grieving customer he could book a full-price customer he could book a full-price customer he could book a full-price ticket now and claim a bereavement ticket now and claim a bereavement ticket now and claim a bereavement discount afterwards, that the policy discount afterwards, that the policy discount afterwards, that the policy didn't exist. didn't exist. didn't exist. The chatbot made it up politely, fluently, confidently. The politely, fluently, confidently. The customer took the airlines customer took the airlines customer took the airlines to a tribunal. to a tribunal. to a tribunal. And the airlines' defense, And the airlines' defense, And the airlines' defense, this is real, was this is real, was this is real, was was that the chatbot is called was that the chatbot is called was that the chatbot is called a separate legal entity responsible for a separate legal entity responsible for a separate legal entity responsible for its own action. its own action. its own action. That is the corporate version of my dog That is the corporate version of my dog That is the corporate version of my dog ate my homework. ate my homework. ate my homework. The tribunal didn't buy it. The airline The tribunal didn't buy it. The airline The tribunal didn't buy it. The airline paid, and every one of us in our paid, and every one of us in our paid, and every one of us in our boardroom quietly took a note that day. boardroom quietly took a note that day. boardroom quietly took a note that day. You can't outsource accountability to You can't outsource accountability to You can't outsource accountability to your own software. At the bottom your own software. At the bottom your own software. At the bottom of every real decision, a human signs.

of every real decision, a human signs. of every real decision, a human signs. If your architecture doesn't have a If your architecture doesn't have a If your architecture doesn't have a fundable human at the end of it, fundable human at the end of it, fundable human at the end of it, you you you you have not built a product. You have you have not built a product. You have you have not built a product. You have have built an excuse generator. So, have built an excuse generator. So, have built an excuse generator. So, build your AI around that accountable build your AI around that accountable build your AI around that accountable person, not instead of them. Okay, the fix. Okay, the fix. Five things. Five things. Five things. Each one is a direct cure for a story Each one is a direct cure for a story Each one is a direct cure for a story you have just heard. you have just heard. you have just heard. This is almost a word for word. This is almost a word for word. This is almost a word for word. What What What I demand from our vendor before lighting I demand from our vendor before lighting I demand from our vendor before lighting their system near a live deal. One, their system near a live deal. One, their system near a live deal. One, every claim comes with a receipt. Each every claim comes with a receipt. Each every claim comes with a receipt. Each sentence linked us sentence linked us sentence linked us straight to its source paragraph with straight to its source paragraph with straight to its source paragraph with the source trust level attached. Not a the source trust level attached. Not a the source trust level attached. Not a citation tab at the end. Two, fact citation tab at the end. Two, fact citation tab at the end. Two, fact and the guesses stay visibility and the guesses stay visibility and the guesses stay visibility separate. At glance at the page, I separate. At glance at the page, I separate. At glance at the page, I insistently see what's proven and what's insistently see what's proven and what's insistently see what's proven and what's somebody's best estimate. Three, somebody's best estimate. Three, somebody's best estimate. Three, numbers agree with each other numbers agree with each other numbers agree with each other automatically. The system refuse automatically. The system refuse automatically. The system refuse to ship a memo where the figures don't to ship a memo where the figures don't to ship a memo where the figures don't match. No human checking at 2:00 in the

match. No human checking at 2:00 in the match. No human checking at 2:00 in the morning. Four, contradictions get morning. Four, contradictions get morning. Four, contradictions get surfaced, never smoothed over. When surfaced, never smoothed over. When surfaced, never smoothed over. When sources disagree, the system raises its sources disagree, the system raises its sources disagree, the system raises its hand instead of hand instead of hand instead of picking the picking the picking the front layer answer. Five, a real human front layer answer. Five, a real human front layer answer. Five, a real human approval gate approval gate approval gate and is logged. Who's Who reviewed what and is logged. Who's Who reviewed what and is logged. Who's Who reviewed what change when they signed. That log is change when they signed. That log is change when they signed. That log is audit trail. audit trail. audit trail. Notice what's not on the list. A smart Notice what's not on the list. A smart Notice what's not on the list. A smart promoter not not One of these is a bigger brain problems. One of these is a bigger brain problems. One of these is a bigger brain problems. All five plumbing and honesty problems. All five plumbing and honesty problems. All five plumbing and honesty problems. The winners The winners The winners in the in this category won't win on in the in this category won't win on in the in this category won't win on benchmark points. They will win because benchmark points. They will win because benchmark points. They will win because a a a tired, skeptical finance person tired, skeptical finance person tired, skeptical finance person finance person can trust that their finance person can trust that their finance person can trust that their output at 11:00 at night without opening output at 11:00 at night without opening output at 11:00 at night without opening seven tabs.

Now, the part I promised, the money. Now, the part I promised, the money. Many of you Many of you Many of you are not just the building AI products. are not just the building AI products. are not just the building AI products. You are raising for them You are raising for them You are raising for them or you will be. So, or you will be. So, or you will be. So, let me tell you what actually happens let me tell you what actually happens let me tell you what actually happens after you leave the pitch meeting. after you leave the pitch meeting. after you leave the pitch meeting. Your deck becomes a memo. Literally, Your deck becomes a memo. Literally, Your deck becomes a memo. Literally, literally, literally, someone like me literally, literally, someone like me literally, literally, someone like me sits down and writes an internal memo sits down and writes an internal memo sits down and writes an internal memo about you. about you. about you. Every number you said Every number you said Every number you said out loud gets checked out loud gets checked out loud gets checked get get checked against your data room, get get checked against your data room, get get checked against your data room, what which means everything I just told what which means everything I just told what which means everything I just told you about the documents apply to you you about the documents apply to you you about the documents apply to you personally. Four lessons. personally. Four lessons. personally. Four lessons. Okay, my time's up. Thank you. Okay, my time's up. Thank you. Okay, my time's up. Thank you. >> [applause]