Deploying AI Inside a Bank Is Not an Engineering Problem — Paden Gayle, Applied AI Engineers

Congratulations. Today Congratulations. Today we will talk about we will talk about we will talk about implementing AI in a implementing AI in a implementing AI in a bank and why bank and why bank and why it is not purely an engineering it is not purely an engineering it is not purely an engineering problem. Before we problem. Before we problem. Before we get to that, I get to that, I get to that, I want to introduce myself. want to introduce myself. want to introduce myself. My name is Peyton My name is Peyton My name is Peyton Gale. I have worked in Gale. I have worked in Gale. I have worked in many different many different many different industries. I started in industries. I started in industries. I started in consulting at Accenture and consulting at Accenture and consulting at Accenture and then moved to then moved to then moved to Bloomberg as a Bloomberg as a software engineer, where I software engineer, where I software engineer, where I worked on the worked on the worked on the trading trading trading infrastructure and infrastructure and infrastructure and data platform teams. data platform teams. data platform teams. After that, I After that, I After that, I worked at Google on worked at Google on worked at Google on security and security and security and deployment issues. I am currently a co- deployment issues. I am currently a co- founder of a startup founder of a startup and writing a book about and writing a book about and writing a book about advanced engineering advanced engineering advanced engineering implementation for implementation for implementation for O'Reilly. In O'Reilly. In O'Reilly. In this example, this example, this example, the system reviews the system reviews the system reviews customer documents and customer documents and customer documents and creates exceptions for creates exceptions for creates exceptions for the analyst. the analyst. the analyst. The demonstration shows the The demonstration shows the The demonstration shows the planned planned planned workflow and everyone workflow and everyone workflow and everyone understands why it is understands why it is understands why it is useful. Then the useful. Then the useful. Then the security team asked, " security team asked, " Okay, what kind of environment Okay, what kind of environment Okay, what kind of environment will host this?" will host this?" will host this?" Data owners Data owners Data owners ask: "Who ask: "Who ask: "Who approved this approved this approved this use of use of use of the documents?" And someone the documents?" And someone the documents?" And someone in the risk department in the risk department in the risk department asked, "Who asked, "Who asked, "Who will check the will check the will check the false exception false exception false exception before it before it before it gets to the gets to the gets to the next system?" next system?" next system?" These are valid questions, are These are valid questions, are These are valid questions, are n't they? Every n't they? Every n't they? Every question can question can question can change what we are change what we are change what we are allowed to create allowed to create . The team . The team . The team has already invested in the design, has already invested in the design, has already invested in the design, even though many of the people even though many of the people even though many of the people responsible for it have responsible for it have responsible for it have never never never seen it. So, in this seen it. So, in this seen it. So, in this case, the demonstration case, the demonstration case, the demonstration works. The model was works. The model was works. The model was okay. However, it okay. However, it okay. However, it still did not go into still did not go into still did not go into production. I want to production. I want to production. I want to show you how to show you how to show you how to identify these solutions identify these solutions identify these solutions in the first week in the first week in the first week so you can so you can so you can design a system design a system design a system with a reliable path to with a reliable path to with a reliable path to operation. These operation. These operation. These four categories four categories four categories reflect the decisions that reflect the decisions that your deployment depends on. For your deployment depends on. For each of them, I each of them, I each of them, I need someone who need someone who need someone who can explain can explain can explain the decision, the the decision, the the decision, the evidence needed, and the next evidence needed, and the next evidence needed, and the next step. So, let's start with the step. So, let's start with the step. So, let's start with the data. Data: who data. Data: who data. Data: who owns the source, who owns the source, who owns the source, who can grant access, can grant access, can grant access, who can approve who can approve who can approve its use in the its use in the its use in the workflow. These workflow. These workflow. These responsibilities may responsibilities may responsibilities may belong to different belong to different belong to different people, but the ability

people, but the ability people, but the ability to create a dataset to create a dataset to create a dataset does not solve all does not solve all does not solve all the questions about how the questions about how the questions about how an application can an application can an application can use them. use them. use them. Regarding model risks, Regarding model risks, Regarding model risks, find someone who find someone who find someone who can tell you what can tell you what can tell you what verification verification verification applies to your applies to your applies to your intended intended intended use. use. use. Ask what the Ask what the Ask what the previous previous previous task was like. Find out task was like. Find out task was like. Find out who will own who will own who will own the system after the system after the system after launch and what evidence launch and what evidence launch and what evidence that that that owner will need to owner will need to owner will need to support it. Regarding support it. Regarding support it. Regarding third- third- third- party risks, draw a party risks, draw a party risks, draw a real chain of real chain of real chain of dependencies. dependencies. dependencies. Include the Include the Include the model provider model provider , hosting conditions, and , hosting conditions, and , hosting conditions, and your role in your role in your role in operational operational operational maintenance. maintenance. maintenance. Find out which Find out which Find out which relationships have already been relationships have already been relationships have already been tested and which tested and which tested and which proposed changes proposed changes proposed changes require a separate require a separate require a separate decision. For an audit— decision. For an audit— just ask just ask just ask someone, okay? someone, okay? For an audit, ask For an audit, ask what would be needed to what would be needed to what would be needed to reproduce a critical reproduce a critical reproduce a critical action: what input data action: what input data action: what input data was used, was used, was used, what configuration what configuration what configuration created the output created the output created the output data, what data, what data, what the reviewer saw, and who the reviewer saw, and who the reviewer saw, and who approved the action. These are approved the action. These are approved the action. These are issues that issues that issues that affect the records, affect the records, affect the records, and you need to and you need to and you need to collect them from the collect them from the collect them from the beginning. Now beginning. Now beginning. Now imagine imagine imagine getting these getting these getting these answers after you answers after you answers after you 've built the 've built the 've built the entire system, entire system, entire system, deployment boundaries, deployment boundaries, deployment boundaries, data path, or action that data path, or action that data path, or action that might need to be might need to be might need to be changed. A useful changed. A useful changed. A useful step is to involve these step is to involve these step is to involve these people and include them in the people and include them in the people and include them in the design process design process design process at an early stage.

at an early stage. at an early stage. Their responsibilities Their responsibilities Their responsibilities help us help us help us define a system define a system define a system that we can truly that we can truly that we can truly manage. And here, manage. And here, manage. And here, discovery is discovery is discovery is engineering. So engineering. So engineering. So now you want now you want now you want to talk about what to talk about what to talk about what discovery is and who discovery is and who discovery is and who exactly do we need to exactly do we need to exactly do we need to find to understand find to understand what exactly do we need to what exactly do we need to what exactly do we need to build. So it's worth build. So it's worth build. So it's worth remembering that every remembering that every remembering that every data set has data set has data set has an owner. The work an owner. The work an owner. The work I did at Bloomberg on the I did at Bloomberg on the data platform team made this data platform team made this statement statement statement concrete for me. concrete for me. concrete for me. So, creating So, creating So, creating the infrastructure for LLM the infrastructure for LLM the infrastructure for LLM meant understanding meant understanding meant understanding which which which datasets could be datasets could be datasets could be used and used and used and who could approve who could approve who could approve such use. such use. such use. The idea we want to The idea we want to The idea we want to convey is convey is convey is that every that every that every data set has an owner. data set has an owner. data set has an owner. My work at Bloomberg on the My work at Bloomberg on the data platform team made this data platform team made this obvious to me. obvious to me. obvious to me. Creating Creating Creating the infrastructure for LLM the infrastructure for LLM the infrastructure for LLM meant understanding meant understanding meant understanding which which which datasets could be datasets could be datasets could be used and used and used and who could authorize that who could authorize that who could authorize that use. use. use. Apply this to Apply this to Apply this to every banking every banking every banking deployment, and deployment, and deployment, and indeed to any indeed to any indeed to any deployment. So deployment. So deployment. So before designing your before designing your before designing your data search, data search, data search, find the person find the person find the person responsible for responsible for responsible for each source. each source. each source. Confirm the Confirm the Confirm the permitted permitted permitted use and use and use and the environment where the environment where the environment where processing can processing can processing can actually begin actually begin actually begin to occur. If these to occur. If these to occur. If these answers change, answers change, answers change, your project your project your project will change with will change with will change with them. Imagine that the key them. Imagine that the key them. Imagine that the key on this slide on this slide on this slide represents a real represents a real represents a real permission with a real permission with a real permission with a real owner. We need to be owner. We need to be owner. We need to be able to name this able to name this able to name this owner to know owner to know owner to know where exactly where exactly where exactly this key should come from.

this key should come from. When you start When you start defining defining defining architecture, architecture, architecture, many questions can be many questions can be many questions can be answered answered answered by looking at the by looking at the by looking at the organizational organizational organizational structure. structure. structure. Read the top Read the top Read the top line first. Documents line first. Documents line first. Documents are fed into the model, and are fed into the model, and are fed into the model, and the analyst checks the the analyst checks the the analyst checks the output results. output results. output results. You can probably You can probably You can probably sketch out this system sketch out this system sketch out this system in a few minutes. in a few minutes. in a few minutes. Now read Now read Now read the line below it. the line below it. the line below it. Documents have a Documents have a Documents have a data owner. The data owner. The data owner. The model service model service model service requires a requires a requires a platform owner. The platform owner. The analyst workflow requires analyst workflow requires someone who can someone who can someone who can validate the validate the validate the relevant relevant relevant risk requirements. And there's still a question mark next to the risk requirements. And there's still a question mark next to the risk requirements. And there's still a question mark next to the last name last name . This . This question mark is question mark is question mark is addiction. We addiction. We addiction. We developed a developed a developed a workflow without finding workflow without finding workflow without finding someone who could someone who could someone who could tell us whether tell us whether tell us whether its its its intended intended intended use was acceptable and what use was acceptable and what use was acceptable and what evidence would be evidence would be evidence would be required. So I would required. So I would required. So I would go through this go through this go through this with the sponsor with the sponsor with the sponsor or someone you or someone you or someone you trust and trust and trust and find out, "Okay, find out, "Okay, find out, "Okay, who owns the who owns the who owns the documents? Have documents? Have documents? Have they heard of this they heard of this they heard of this use?" Who use?" Who use?" Who can help us can help us can help us find the right find the right find the right reviewer? Then I reviewer? Then I reviewer? Then I confirm confirm confirm the answers the answers the answers directly with these directly with these directly with these people. The real people. The real people. The real map will have more map will have more map will have more connections than this one. The connections than this one. The connections than this one. The technical component technical component technical component may depend on may depend on may depend on several approvals. several approvals. several approvals. Some solutions Some solutions Some solutions cover the entire cover the entire cover the entire system. The point is to system. The point is to system. The point is to make these make these make these dependencies visible dependencies visible dependencies visible while the design is still easy while the design is still easy while the design is still easy to change. The nameless to change. The nameless to change. The nameless owner owner owner is transformed into a is transformed into a is transformed into a specific task with specific task with specific task with a person a person a person responsible for responsible for responsible for solving it. This is solving it. This is solving it. This is how the research phase how the research phase how the research phase begins begins begins to transform to transform to transform uncertainty into an uncertainty into an uncertainty into an implementation plan. And implementation plan. And implementation plan. And with that in mind, you with that in mind, you with that in mind, you can start can start can start asking yourself, " asking yourself, " Okay, Peyton, what Okay, Peyton, what Okay, Peyton, what questions should I questions should I questions should I ask?" So here are ask?" So here are ask?" So here are some questions some questions some questions I would ask during an I would ask during an I would ask during an installation call.

installation call. installation call. Where is the data stored Where is the data stored Where is the data stored and who can grant and who can grant and who can grant access to it? Who access to it? Who access to it? Who stopped this and stopped stopped this and stopped stopped this and stopped my project before, and my project before, and my project before, and why? What existing why? What existing why? What existing controls controls controls can cover this can cover this can cover this workflow and who workflow and who workflow and who owns them? Where owns them? Where owns them? Where will this will this will this product version run and product version run and product version run and who are who are who are the administrators of the administrators of the administrators of this environment? Who this environment? Who this environment? Who signs the relevant signs the relevant signs the relevant risk review and what risk review and what risk review and what was signed was signed was signed previously? What previously? What previously? What obligation is obligation is obligation is personally concerning to personally concerning to personally concerning to the sponsor and who can the sponsor and who can the sponsor and who can explain it? Pay explain it? Pay explain it? Pay attention to what I attention to what I attention to what I want to stay with. It's want to stay with. It's want to stay with. It's either names, or either names, or either names, or decisions, or subsequent decisions, or subsequent decisions, or subsequent conversations. If someone conversations. If someone conversations. If someone says security will says security will says security will be fine, be fine, be fine, ask who ask who ask who will actually review will actually review will actually review the design and when you the design and when you the design and when you can meet with them can meet with them can meet with them . Then . Then . Then complete the cycle. complete the cycle. complete the cycle. Send each Send each Send each owner a brief owner a brief owner a brief description of the proposed description of the proposed description of the proposed use and use and use and confirm what they confirm what they confirm what they need from you. By need from you. By need from you. By Friday, people Friday, people Friday, people whose decisions whose decisions whose decisions the project depends on must know the project depends on must know that the project exists. This that the project exists. This that the project exists. This gives the gives the gives the scope of work definition a scope of work definition a scope of work definition a much stronger much stronger much stronger starting point. And starting point. And starting point. And once all of that is once all of that is once all of that is done and we know done and we know done and we know who is responsible for all the who is responsible for all the who is responsible for all the decisions, I can decisions, I can decisions, I can determine the scope of determine the scope of determine the scope of work. When a customer work. When a customer work. When a customer can't write or can't write or can't write or provide a specification, provide a specification, provide a specification, I rely on these I rely on these I rely on these three things: three things: three things: observe, observe, observe, draft, and draft, and draft, and reduce. reduce. reduce. For example, at Bloomberg I For example, at Bloomberg I For example, at Bloomberg I worked on a worked on a worked on a system that system that system that processed cash processed cash processed cash flows. This work flows. This work flows. This work had to be had to be had to be coordinated. This gave coordinated. This gave coordinated. This gave me a concrete me a concrete me a concrete starting point. So, it all starting point. So, it all starting point. So, it all started with started with started with imitation.

imitation. imitation. For example, at Bloomberg I For example, at Bloomberg I For example, at Bloomberg I worked on a worked on a worked on a system that system that system that processed cash processed cash processed cash flows. The work flows. The work flows. The work needed to be needed to be needed to be coordinated, and this gave coordinated, and this gave coordinated, and this gave me a concrete me a concrete me a concrete starting point. Follow starting point. Follow starting point. Follow the flow and the people the flow and the people the flow and the people responsible for responsible for responsible for it. I use it. I use it. I use this approach when a this approach when a this approach when a sponsor wants AI for sponsor wants AI for sponsor wants AI for onboarding. Sit down with onboarding. Sit down with onboarding. Sit down with the analyst, the analyst, the analyst, see what see what see what systems they systems they systems they open, where they open, where they open, where they draw conclusions, and draw conclusions, and draw conclusions, and when they ask for when they ask for when they ask for permission, ask what permission, ask what permission, ask what happens if happens if happens if the document is missing. the document is missing. the document is missing. Write down the actual Write down the actual Write down the actual sequence. This is the sequence. This is the sequence. This is the beginning of the specification beginning of the specification . Now imagine that we . Now imagine that we . Now imagine that we create a helper create a helper create a helper to work with to work with to work with exceptions in exceptions in exceptions in documents. This is a documents. This is a documents. This is a hypothetical example hypothetical example , but I would still , but I would still , but I would still present these three present these three present these three lines to a sponsor. First lines to a sponsor. First : : : document verification. Second: document verification. Second: document verification. Second: preparing exceptions. preparing exceptions. preparing exceptions. And third: the analyst And third: the analyst And third: the analyst submits an application. Then I submits an application. Then I submits an application. Then I detail each detail each detail each point. What point. What point. What documents does documents does documents does the analyst have access to? Is the analyst have access to? Is the analyst have access to? Is such such such access approved for this access approved for this access approved for this processing? What should the processing? What should the processing? What should the model produce? This model produce? This model produce? This will be a highlighted will be a highlighted will be a highlighted passage, a draft passage, a draft passage, a draft form, or both form, or both form, or both . Who . Who . Who checks the draft? And checks the draft? And checks the draft? And also, who is serving it? also, who is serving it? also, who is serving it? For the first version, For the first version, For the first version, let's assume that let's assume that let's assume that the result is a the result is a the result is a highlighted exception and a highlighted exception and a highlighted exception and a draft form. draft form. draft form. The analyst checks The analyst checks The analyst checks the source, edits the source, edits the source, edits the form, and submits it. These are the form, and submits it. These are the form, and submits it. These are boundaries that someone can boundaries that someone can boundaries that someone can adjust. I would adjust. I would adjust. I would also suggest also suggest also suggest criteria for success.

criteria for success. criteria for success. For example: less time For example: less time For example: less time to test without to test without to test without increasing the number of increasing the number of increasing the number of missed exceptions. missed exceptions. missed exceptions. We still need to We still need to We still need to agree on how agree on how agree on how to measure this to measure this to measure this in relation to the current in relation to the current in relation to the current workflow, because that's workflow, because that's workflow, because that's really the really the really the point. This is just point. This is just point. This is just a suggestion until we a suggestion until we a suggestion until we reach an agreement. reach an agreement. reach an agreement. Show the draft to Show the draft to Show the draft to the sponsor and the sponsor and the sponsor and controlling owners. controlling owners. controlling owners. They can They can They can cross out the cross out the cross out the document type, change document type, change document type, change the reviewer, or the reviewer, or the reviewer, or simply reject simply reject simply reject the result. Each the result. Each the result. Each fix gives us a fix gives us a fix gives us a solution on which to solution on which to solution on which to build build build further steps. So, further steps. So, further steps. So, once the workflow once the workflow once the workflow is complete, the scope is complete, the scope is complete, the scope becomes becomes becomes a series of issues to a series of issues to a series of issues to agree on. Now agree on. Now agree on. Now let's let's let's develop develop develop our hypothetical our hypothetical our hypothetical system step by step. First, system step by step. First, system step by step. First, she summarizes she summarizes she summarizes the document that the document that the document that the analyst already has. We will the analyst already has. We will the analyst already has. We will still confirm still confirm still confirm whether the available whether the available whether the available access covers this access covers this access covers this use. Next, use. Next, use. Next, it reads data from the it reads data from the it reads data from the second system. Now second system. Now second system. Now another another another data path is emerging, and someone data path is emerging, and someone data path is emerging, and someone has to approve it. has to approve it. has to approve it. After that, she After that, she After that, she prepares a response that prepares a response that prepares a response that the client will see. Who the client will see. Who the client will see. Who reviews the content and who reviews the content and who reviews the content and who will review the rules will review the rules will review the rules we can we can we can use? Finally, use? Finally, use? Finally, it sends it sends it sends the response the response the response automatically. Who is automatically. Who is automatically. Who is responsible for this action? responsible for this action? responsible for this action? What happens if What happens if What happens if she sends the wrong thing she sends the wrong thing she sends the wrong thing ? And how do we stop ? And how do we stop ? And how do we stop this and fix the this and fix the this and fix the consequences? These are four consequences? These are four consequences? These are four different engineering different engineering different engineering levels. Appropriate levels. Appropriate levels. Appropriate checks depend checks depend checks depend on the institution and on the institution and usage scenario. Therefore, I will usage scenario. Therefore, I will coordinate them with those who coordinate them with those who coordinate them with those who make these decisions.

make these decisions. make these decisions. Then I ask, which Then I ask, which Then I ask, which step creates useful step creates useful step creates useful value value value today? Can I today? Can I today? Can I create drafts create drafts create drafts in the current in the current in the current analyst workflow analyst workflow analyst workflow to solve to solve to solve enough of the enough of the enough of the problem? If so, problem? If so, problem? If so, start with this. start with this. start with this. Confirm that Confirm that Confirm that existing controls existing controls existing controls are appropriate and create a are appropriate and create a are appropriate and create a minimum viable minimum viable minimum viable version, or MVP, with a version, or MVP, with a version, or MVP, with a robust robust robust approval path. This approval path. This approval path. This brings me to brings me to brings me to autonomy. I will start autonomy. I will start autonomy. I will start with the least with the least with the least authority to authority to authority to solve a problem, and solve a problem, and solve a problem, and then ask what then ask what then ask what problem can be problem can be problem can be proven as a basis for proven as a basis for proven as a basis for giving the system giving the system giving the system more freedom. When more freedom. When more freedom. When I was working on I was working on I was working on cash flow at cash flow at cash flow at Bloomberg, I was constantly Bloomberg, I was constantly Bloomberg, I was constantly thinking about thinking about thinking about automation. Therefore, automation. Therefore, automation. Therefore, each each each automated step automated step automated step requires control, and requires control, and requires control, and behind each indicator behind each indicator behind each indicator there is a responsible there is a responsible there is a responsible person. I brought this person. I brought this person. I brought this discipline into the field of AI discipline into the field of AI . So, find a . So, find a . So, find a defining action, defining action, defining action, assign someone assign someone assign someone responsible, and responsible, and responsible, and decide what decide what decide what happens when happens when happens when the system goes wrong. the system goes wrong. the system goes wrong. For example, in our For example, in our For example, in our hypothetical hypothetical exception handling workflow , submitting a request is , submitting a request is , submitting a request is such an action. The analyst such an action. The analyst such an action. The analyst reviewing the reviewing the reviewing the draft needs draft needs draft needs evidence and visible evidence and visible evidence and visible uncertainty to uncertainty to uncertainty to make an informed make an informed make an informed decision. The " decision. The " check" button check" button check" button only helps when only helps when only helps when a person a person a person can actually evaluate the work. can actually evaluate the work. can actually evaluate the work. So, when thinking So, when thinking So, when thinking about autonomy, I about autonomy, I about autonomy, I ask you to read these ask you to read these ask you to read these steps from the bottom up. steps from the bottom up. steps from the bottom up. "Propose" "Propose" "Propose" means that the analyst means that the analyst means that the analyst receives a recommendation receives a recommendation receives a recommendation and chooses what and chooses what and chooses what to do with it. " to do with it. " to do with it. " Draft" means that Draft" means that Draft" means that the system prepares the system prepares the system prepares a document that a document that a document that the analyst reviews and the analyst reviews and the analyst reviews and submits. This is the submits. This is the submits. This is the selected stage. " selected stage. " Acting with approval" Acting with approval" Acting with approval" means that a person means that a person means that a person authorizes each action authorizes each action authorizes each action before carrying it out.

before carrying it out. before carrying it out. "Operating" means that "Operating" means that "Operating" means that the system is operating within the system is operating within the system is operating within the limits that have already been the limits that have already been the limits that have already been agreed upon. Each stage agreed upon. Each stage agreed upon. Each stage changes the consequences of changes the consequences of changes the consequences of the error. However, it the error. However, it the error. However, it also changes which also changes which also changes which controls controls controls need to work and need to work and need to work and which auditors which auditors which auditors need to be convinced. need to be convinced. need to be convinced. I would start at the I would start at the I would start at the lowest stage that lowest stage that lowest stage that creates value and creates value and creates value and then gather evidence. Are then gather evidence. Are then gather evidence. Are there checks that there checks that there checks that reject drafts? reject drafts? reject drafts? What values What values are constantly repeated? are constantly repeated? Can people Can people Can people resume work resume work resume work when something goes when something goes when something goes wrong? Use wrong? Use wrong? Use these observations to these observations to these observations to make your next make your next make your next decision about decision about decision about authority. The model authority. The model authority. The model does not become more does not become more does not become more capable on its own capable on its own capable on its own . Answer . Answer how much authority how much authority how much authority this particular this particular this particular workflow should workflow should workflow should grant her. Now let's make grant her. Now let's make grant her. Now let's make this concrete using the this concrete using the this concrete using the same same same hypothetical exception hypothetical exception hypothetical exception helper as an example. helper as an example. helper as an example. Let's say the Let's say the Let's say the initial request is an initial request is an initial request is an agent that reads agent that reads agent that reads customer documents, customer documents, customer documents, determines which ones determines which ones determines which ones fall outside fall outside fall outside the policy, and the policy, and the policy, and automatically issues automatically issues automatically issues exceptions. In such a exceptions. In such a exceptions. In such a project, the project, the project, the model's judgments can become model's judgments can become model's judgments can become operational action without operational action without operational action without each each each case being checked by an analyst. case being checked by an analyst. case being checked by an analyst. Now leave the model, Now leave the model, Now leave the model, but change the boundaries of the actions. but change the boundaries of the actions. but change the boundaries of the actions. She identifies a She identifies a She identifies a potential exception, potential exception, potential exception, cites the appropriate cites the appropriate cites the appropriate source, and prepares a source, and prepares a source, and prepares a draft form. draft form. draft form. The analyst reviews The analyst reviews The analyst reviews the evidence, corrects the evidence, corrects the evidence, corrects the form, and decides whether to the form, and decides whether to the form, and decides whether to submit it. submit it. submit it. Follow this Follow this Follow this sequence on the sequence on the sequence on the slide: model, slide: model, slide: model, person, presentation. I person, presentation. I person, presentation. I will present this will present this will present this project to the people we project to the people we project to the people we found during the found during the found during the research phase. Can research phase. Can research phase. Can this leverage this leverage an analyst's existing workflow? Does an analyst's existing workflow? Does approved approved approved data access cover such data access cover such data access cover such processing? What controls processing? What controls are used? And what are used? And what evidence do evidence do evidence do the inspectors need?

the inspectors need? the inspectors need? Human review does not Human review does not Human review does not guarantee approval and guarantee approval and guarantee approval and does not determine does not determine does not determine regulatory regulatory regulatory category on its own category on its own . We still need to . We still need to . We still need to confirm the intended confirm the intended confirm the intended use and the use and the use and the necessary checks. A necessary checks. A necessary checks. A useful question useful question useful question is whether is whether is whether this this this draft version solves draft version solves draft version solves enough problems enough problems enough problems for a first run. for a first run. for a first run. If so, you will be able to If so, you will be able to If so, you will be able to collect evidence without collect evidence without collect evidence without errors, deviations, and errors, deviations, and errors, deviations, and recoveries by working recoveries by working recoveries by working within these boundaries. The next within these boundaries. The next within these boundaries. The next decision about decision about decision about authority can be authority can be authority can be based on based on based on real-world real-world real-world usage experience. This usage experience. This usage experience. This makes the assessment an makes the assessment an makes the assessment an integral integral integral part of the part of the part of the exploitation plan. This exploitation plan. This exploitation plan. This brings us to the brings us to the brings us to the fourth promise: fourth promise: fourth promise: an assessment that the an assessment that the an assessment that the risk team can risk team can risk team can use as evidence of what the use as evidence of what the risks they are responsible for managing actually mean. At they are responsible for managing actually mean. At Google, the question that Google, the question that Google, the question that needed to be needed to be needed to be answered answered answered was: what happens was: what happens was: what happens when the system when the system when the system fails? Who fails? Who fails? Who notices this? What notices this? What notices this? What stops? Who takes stops? Who takes responsibility? responsibility? How long will it How long will it How long will it take to take to take to get back to work? For get back to work? For get back to work? For AI implementation, I AI implementation, I AI implementation, I want to get the same want to get the same want to get the same specific specific specific report, right? report, right? report, right? Intended Intended Intended use, use, use, threat model, failure modes, threat model, failure modes, threat model, failure modes, and response plan. and response plan. and response plan. Show me how Show me how Show me how the controls work the controls work the controls work when the model when the model when the model makes makes makes an error. Name an error. Name an error. Name the person who receives the person who receives the person who receives the notification and can the notification and can the notification and can stop the stop the stop the workflow. This provides a workflow. This provides a workflow. This provides a practical way practical way practical way to evaluate our example.

to evaluate our example. to evaluate our example. So, going back to So, going back to So, going back to our great our great our great example with the example with the example with the hypothetical hypothetical hypothetical exception assistant: exception assistant: exception assistant: if it throws if it throws if it throws an exception, the analyst an exception, the analyst an exception, the analyst checks the original checks the original checks the original passage and rejects passage and rejects passage and rejects the draft. If the draft. If the draft. If the extraction produces the extraction produces the extraction produces unusable text, unusable text, unusable text, stop and stop and stop and request a clean request a clean request a clean file. If it file. If it file. If it misses exceptions, misses exceptions, misses exceptions, targeted targeted targeted offline cases offline cases offline cases help identify help identify help identify this weakness. They don't this weakness. They don't this weakness. They don't detect every detect every detect every pass in pass in pass in real time. real time. real time. Leave the relevant Leave the relevant Leave the relevant work under manual work under manual work under manual control. For control. For control. For each failure, indicate the each failure, indicate the each failure, indicate the detection path, detection path, detection path, backup plan, and backup plan, and backup plan, and responsible person. responsible person. responsible person. Add these solutions to the Add these solutions to the Add these solutions to the evidence package. These evidence package. These evidence package. These five files make five files make five files make the check complete. the check complete. the check complete. Intended Intended Intended use, use, use, record users, record users, record users, limits, and prohibited limits, and prohibited limits, and prohibited actions. Data origin actions. Data origin , records of each , records of each , records of each example source, and example source, and usage consent. usage consent. Failure modes, Failure modes, Failure modes, detection records, and detection records, and detection records, and backup plans. backup plans. backup plans. Results by Results by Results by segment, segment, segment, coverage and error records. coverage and error records. coverage and error records. Controls, Controls, Controls, inspection records, inspection records, inspection records, thresholds, thresholds, thresholds, stopping rules and stopping rules and stopping rules and owners. So, with all of owners. So, with all of owners. So, with all of this, we will this, we will this, we will generate evidence generate evidence generate evidence during each during each during each evaluation run. evaluation run. evaluation run. Along with this, I Along with this, I Along with this, I store the input data, store the input data, store the input data, query, model version, query, model version, query, model version, configuration, configuration, configuration, output data, and output data, and output data, and reviewer actions. This reviewer actions. This reviewer actions. This allows us allows us allows us to recreate a historical to recreate a historical to recreate a historical decision.

decision. decision. Rerunning the model Rerunning the model Rerunning the model later may give a later may give a later may give a different answer, so different answer, so different answer, so we keep what we keep what we keep what actually happened. actually happened. actually happened. The reviewer can The reviewer can The reviewer can expect such a expect such a expect such a confirmation confirmation confirmation note. So now we note. So now we note. So now we can look at what can look at what can look at what exactly is exactly is exactly is used to used to used to organize this organize this organize this output, which is output, which is output, which is contained in the contained in the contained in the fourth file. For a fourth file. For a fourth file. For a hypothetical hypothetical hypothetical workflow, I would workflow, I would workflow, I would separate clean separate clean separate clean documents, documents, exception scans, and new exception scans, and new document types. document types. document types. Measure the errors and Measure the errors and Measure the errors and behavior of behavior of behavior of controls in each controls in each controls in each group, then group, then group, then target the target the target the weakest result weakest result . An average . An average . An average can hide a can hide a can hide a serious gap in serious gap in serious gap in one segment. If one segment. If one segment. If the group has little the group has little the group has little evidence, indicate this and evidence, indicate this and evidence, indicate this and limit the use of limit the use of until you have until you have until you have enough. This enough. This enough. This prompts prompts prompts users to users to users to confirm confirm confirm the results, and also the results, and also the results, and also brings us back to our brings us back to our brings us back to our decisions from week one. decisions from week one. decisions from week one. To sum it all up: To sum it all up: To sum it all up: I want to I want to I want to have five have five have five answers by Friday. Is the answers by Friday. Is the data owner involved and aware data owner involved and aware of this project? Are the of this project? Are the workspace and its workspace and its administrators named? administrators named? administrators named? Can the sponsor describe Can the sponsor describe Can the sponsor describe the workflow without the workflow without the workflow without mentioning AI? Does mentioning AI? Does mentioning AI? Does the smallest the smallest the smallest useful volume fit within the useful volume fit within the useful volume fit within the existing controls, as existing controls, as existing controls, as confirmed by their confirmed by their confirmed by their owners? Do owners? Do owners? Do we know who is signing we know who is signing we know who is signing the risks, and what happened to the risks, and what happened to the risks, and what happened to the person who the person who the person who signed last signed last signed last time? Use time? Use time? Use these answers as these answers as these answers as heuristics for heuristics for heuristics for planning. Four planning. Four planning. Four or five "yes"s or five "yes"s or five "yes"s indicate a indicate a indicate a real path to real path to real path to release. Three means release. Three means that unresolved that unresolved that unresolved decisions require decisions require decisions require responsible persons responsible persons responsible persons and deadlines. Zero and deadlines. Zero and deadlines. Zero to two means: to two means: to two means: address these issues address these issues address these issues before before before supply commitments, perhaps supply commitments, perhaps supply commitments, perhaps by reducing or by reducing or by reducing or even even even rethinking the scope rethinking the scope . This score does not guarantee . This score does not guarantee . This score does not guarantee approval, but it does approval, but it does approval, but it does give you an idea of ​​what give you an idea of ​​what give you an idea of ​​what to spend the next to spend the next to spend the next week on. Here's what I would week on. Here's what I would week on. Here's what I would take with me into these take with me into these take with me into these next conversations.

next conversations. So, to sum up: I So, to sum up: I want you to take away want you to take away want you to take away the fact that the fact that the fact that research is what research is what research is what engineering is. Make engineering is. Make engineering is. Make a specification, then a specification, then a specification, then shorten it. Give shorten it. Give shorten it. Give the system only the the system only the the system only the powers it powers it powers it needs. Create a needs. Create a needs. Create a proof package in the proof package in the proof package in the first sprint. first sprint. first sprint. The demonstration The demonstration The demonstration worked. The model worked. The model worked. The model was normal. However, was normal. However, was normal. However, the release never the release never the release never took place. So what should we took place. So what should we took place. So what should we do? We go do? We go do? We go looking for people. Thank you for looking for people. Thank you for looking for people. Thank you for going through this going through this going through this presentation with presentation with presentation with me. My name is me. My name is me. My name is Peyton Gale. I am one of the Peyton Gale. I am one of the Peyton Gale. I am one of the founders of the founders of the founders of the forward deployed engineer movement and am writing a forward deployed engineer movement and am writing a forward deployed engineer movement and am writing a book about FTE for O'Reilly. book about FTE for O'Reilly. book about FTE for O'Reilly. And yes, I'd be happy And yes, I'd be happy And yes, I'd be happy to talk about FTE and to talk about FTE and to talk about FTE and everything everything everything related to it. If you are related to it. If you are related to it. If you are interested, you can interested, you can interested, you can find me on LinkedIn, find me on LinkedIn, find me on LinkedIn, or on my website or on my website or on my website peytongayl.com, where all the peytongayl.com, where all the peytongayl.com, where all the information is. information is. information is. Thanks again for watching Thanks again for watching Thanks again for watching today, and yes, today, and yes, today, and yes, I hope to be in touch I hope to be in touch .