AI Cut Our Research Time From 7 Days to 15 Minutes — Gareth Yoder, Vanguard

My name is Gareth My name is Gareth Yoder. I'm the technical Yoder. I'm the technical Yoder. I'm the technical lead for the money lead for the money lead for the money program here at Vanguard. program here at Vanguard. program here at Vanguard. I help manage I help manage I help manage four to six four to six four to six teams in the cash teams in the cash teams in the cash program. This is the "no program. This is the "no program. This is the "no hryvnia left aside" hryvnia left aside" hryvnia left aside" path of path of path of Vanguard modernization. This journey Vanguard modernization. This journey Vanguard modernization. This journey truly began truly began truly began when we when we when we introduced our introduced our introduced our “Cash Plus” offering to “Cash Plus” offering to Vanguard brokerage account clients in Vanguard brokerage account clients in 2022. After the launch, 2022. After the launch, 2022. After the launch, we started hearing that we started hearing that we started hearing that the upgrade was the upgrade was the upgrade was coming, and this would be coming, and this would be coming, and this would be our next our next our next big project. big project. How many people here How many people here have outdated have outdated have outdated code bases? Truth? There are code bases? Truth? There are code bases? Truth? There are many of you. How many of many of you. How many of many of you. How many of you wish this you wish this you wish this outdated base would outdated base would outdated base would simply simply simply be modernized? be modernized? be modernized? Truth? Many of Truth? Many of Truth? Many of you would say the same. you would say the same. you would say the same. So, looking at this So, looking at this So, looking at this path, we'll go path, we'll go path, we'll go through this graph in through this graph in through this graph in today's today's today's presentation, but presentation, but presentation, but first I want first I want first I want to explain what a to explain what a to explain what a cash book is. cash book is. The cash book The cash book includes all includes all includes all transactions of transactions of transactions of funds flowing into our funds flowing into our Vanguard brokerage accounts from the outside Vanguard brokerage accounts from the outside world. It also world. It also world. It also includes transfers of includes transfers of includes transfers of funds between accounts.

funds between accounts. funds between accounts. So, payment So, payment So, payment systems: think ACH, systems: think ACH, systems: think ACH, bank transfers, bank transfers, bank transfers, etc. We support etc. We support etc. We support web channels, mobile web channels, mobile web channels, mobile channels, and packet channels, and packet channels, and packet channels. Our system channels. Our system channels. Our system operates in an active operates in an active operates in an active multi-regional multi-regional multi-regional fault-tolerant fault-tolerant fault-tolerant ecosystem. Our ecosystem. Our ecosystem. Our services receive up to services receive up to services receive up to 30 million calls 30 million calls 30 million calls per day. Our clients per day. Our clients per day. Our clients also generate half a also generate half a also generate half a million million million ledger entries every day. So, ledger entries every day. So, ledger entries every day. So, when we started this when we started this modernization approach, we modernization approach, we broke it down into three broke it down into three broke it down into three pillars: pillars: pillars: data migration, data migration, data migration, producer migration, and producer migration, and producer migration, and consumer migration. consumer migration. But since it was a But since it was a company priority company priority , we had to , we had to , we had to do all three in do all three in do all three in parallel. So, parallel. So, parallel. So, when looking at the when looking at the when looking at the research analysis, research analysis, research analysis, remember, we were at remember, we were at remember, we were at the beginning of our journey, the beginning of our journey, the beginning of our journey, AI tools didn't exist back then AI tools didn't exist back then AI tools didn't exist back then . So often . So often . So often we worked with we worked with we worked with mainframe engineers mainframe engineers mainframe engineers and architects and architects and architects trying trying trying to determine what the to determine what the to determine what the data schema model would be. A turning point in data schema model would be. A turning point in data schema model would be. A turning point in this modernization this modernization this modernization occurred occurred occurred when our when our when our legacy legacy legacy codebase became accessible codebase became accessible codebase became accessible to AI tools to AI tools to AI tools via the MCP server. And via the MCP server. And via the MCP server. And now we've seen now we've seen now we've seen faster faster faster feedback loops.

feedback loops. What used to take a mainframe engineer 7 days engineer 7 days engineer 7 days to dig into the code to dig into the code , analyze it, and , analyze it, and , analyze it, and come back with an come back with an come back with an answer, answer, answer, could now be done with an could now be done with an AI prompt. We saw the AI prompt. We saw the feedback cycle feedback cycle feedback cycle reduce from 7 reduce from 7 reduce from 7 days to 15 minutes. This days to 15 minutes. This days to 15 minutes. This was a huge was a huge was a huge win for our win for our win for our product owners, product owners, product owners, technical leads, technical leads, technical leads, and developers. and developers. and developers. They all believe that this They all believe that this They all believe that this tool has contributed to the tool has contributed to the tool has contributed to the success of our success of our success of our modernization. So, modernization. So, modernization. So, let's talk let's talk let's talk about about about data modernization. We data modernization. We data modernization. We use the use the use the standard CDC standard CDC standard CDC (tracking (tracking (tracking data changes) approach common in the data changes) approach common in the data changes) approach common in the industry, so we industry, so we industry, so we had to scan had to scan had to scan both databases. both databases. both databases. The team conducted The team conducted The team conducted daily audits. daily audits. We first We first ensured ensured ensured consistency by consistency by checking checking the amount of data at the the amount of data at the the amount of data at the row level, and then row level, and then row level, and then ensured ensured ensured integrity integrity integrity by checking individual by checking individual by checking individual columns: whether columns: whether columns: whether the amounts in the old and the amounts in the old and the amounts in the old and new systems matched, whether the new systems matched, whether the account information, state and account information, state and federal withholdings matched federal withholdings matched . All attributes in the . All attributes in the . All attributes in the legacy legacy legacy database had to database had to database had to match the cloud ones.

match the cloud ones. Important tip: Don't overcomplicate this overcomplicate this solution, as solution, as this code this code will be will be will be decommissioned once the upgrade is complete. Now decommissioned once the upgrade is complete. Now decommissioned once the upgrade is complete. Now let's talk about let's talk about let's talk about the migration of producers. the migration of producers. In the legacy system, In the legacy system, our manufacturers our manufacturers our manufacturers accessed accessed accessed infrastructure in infrastructure in infrastructure in three ways. three ways. three ways. The first is through a request The first is through a request The first is through a request to the service. The second is to the service. The second is through the API that wrapped through the API that wrapped through the API that wrapped this request. The last, and this request. The last, and probably probably probably the most difficult, is the most difficult, is the most difficult, is direct insertion into direct insertion into direct insertion into tables, which means we tables, which means we tables, which means we saw the data in the database, saw the data in the database, saw the data in the database, but didn't know how it but didn't know how it but didn't know how it got there or got there or got there or who created it. So who created it. So who created it. So we took a shared we took a shared we took a shared data schema and data schema and data schema and placed it on a placed it on a placed it on a gateway that wrapped gateway that wrapped gateway that wrapped our APIs that our APIs that our APIs that performed cloud performed cloud performed cloud data inserts. data inserts. Use your Use your modernization journey as an modernization journey as an modernization journey as an opportunity opportunity opportunity to rethink your to rethink your to rethink your data ecosystem. We data ecosystem. We data ecosystem. We borrowed concepts borrowed concepts borrowed concepts from fintech, such as from fintech, such as from fintech, such as idempotence. We idempotence. We idempotence. We also ensured also ensured also ensured reliable consistency reliable consistency reliable consistency across multiple across multiple across multiple recorders. To do recorders. To do recorders. To do this, we used this, we used this, we used a paradigm similar to a paradigm similar to a paradigm similar to the one provided by Snowflake the one provided by Snowflake the one provided by Snowflake ID. We ID. We ID. We rethought everything, not rethought everything, not rethought everything, not just ported it over just ported it over just ported it over without any changes.

without any changes. Now we have a modern Now we have a modern offer for offer for offer for manufacturers. How do we manufacturers. How do we manufacturers. How do we attract manufacturers? attract manufacturers? attract manufacturers? We had 35 We had 35 We had 35 manufacturers who manufacturers who manufacturers who needed to needed to needed to upgrade to upgrade to upgrade to our ecosystem. Initially, our ecosystem. Initially, we had to we had to create create create documentation manually documentation manually documentation manually to facilitate this to facilitate this to facilitate this transition. And when transition. And when AI tools appeared, we AI tools appeared, we provided this provided this provided this documentation on the documentation on the documentation on the company wiki, and then company wiki, and then company wiki, and then exposed it through the MCP exposed it through the MCP server. This was server. This was server. This was great because previously great because previously great because previously many teams many teams many teams were communicating with us in were communicating with us in were communicating with us in chats and it was difficult for us to chats and it was difficult for us to organize it all. organize it all. Communication and Communication and Communication and discussion now discussion now discussion now takes place through takes place through takes place through prompts on the prompts on the prompts on the manufacturer's side, and we've manufacturer's side, and we've manufacturer's side, and we've noticed that noticed that noticed that the number of the number of the number of chat requests and support chat requests and support chat requests and support provided by our provided by our provided by our team team team has decreased significantly since has decreased significantly since has decreased significantly since implementing MCP. implementing MCP. AI tools have also done a AI tools have also done a great job of great job of great job of moving state moving state moving state from the legacy from the legacy from the legacy ecosystem to the cloud.

ecosystem to the cloud. We made the cloud We made the cloud ecosystem and ecosystem and ecosystem and registry values registry values registry values understandable to understandable to understandable to people, unlike the people, unlike the people, unlike the simple simple simple status codes in the legacy status codes in the legacy status codes in the legacy system. So, our system. So, our system. So, our AI tools have done AI tools have done AI tools have done a great job of a great job of bridging the gap bridging the gap between the intentions of the between the intentions of the between the intentions of the legacy legacy legacy ecosystem and ecosystem and ecosystem and cloud cloud cloud technologies. Now technologies. Now technologies. Now let's look at the let's look at the consumer migration path. We consumer migration path. We had three ways to had three ways to had three ways to access data at the access data at the access data at the consumer level. We consumer level. We consumer level. We had had real- real- time usage scenarios, batch time usage scenarios, batch time usage scenarios, batch scenarios, and streaming scenarios, and streaming scenarios, and streaming scenarios. The common thread scenarios. The common thread scenarios. The common thread here is that we have provided a here is that we have provided a here is that we have provided a registry of schemas, registry of schemas, registry of schemas, regardless of the regardless of the regardless of the channel through which your channel through which your channel through which your consumer consumer consumer accesses this data. accesses this data. So, regarding available So, regarding available funds—this is our funds—this is our funds—this is our most most most loaded loaded loaded real-time API that real-time API that real-time API that determines the balance determines the balance determines the balance displayed in displayed in displayed in your account. They your account. They your account. They will call us will call us will call us via GraphQL and via GraphQL and via GraphQL and subscribe subscribe subscribe only to the attributes only to the attributes only to the attributes that meet that meet that meet their needs.

their needs. their needs. For batch For batch For batch scripts, think scripts, think scripts, think about data reconciliation or about data reconciliation or about data reconciliation or analytics. We have a analytics. We have a analytics. We have a data warehouse that data warehouse that data warehouse that provides separate provides separate provides separate views of data to views of data to views of data to each specific each specific each specific consumer. Each consumer. Each consumer. Each consumer tells consumer tells consumer tells us their us their us their usage scenario, and we usage scenario, and we usage scenario, and we provide a provide a provide a representation representation representation specific to their specific to their specific to their data needs. This was data needs. This was data needs. This was supported by our supported by our data governance team, data governance team, who were adamant who were adamant who were adamant about ensuring about ensuring about ensuring consumers consumers consumers only saw the data they only saw the data they only saw the data they needed. The last needed. The last needed. The last aspect concerned aspect concerned aspect concerned streaming. streaming. streaming. We have an event topic We have an event topic We have an event topic in Kafka where we in Kafka where we in Kafka where we broadcast all broadcast all broadcast all inserts and updates to inserts and updates to inserts and updates to this database. Our this database. Our fraud prevention teams were able to fraud prevention teams were able to leverage this leverage this leverage this technology to technology to technology to provide provide real-time data to real-time data to their tools. their tools. their tools. We ensured that We ensured that We ensured that messages were delivered messages were delivered messages were delivered at least once. at least once. at least once. So, our concepts of So, our concepts of So, our concepts of idempotence, which idempotence, which idempotence, which we implemented on the we implemented on the we implemented on the producer side, we have producer side, we have producer side, we have also transferred to also transferred to event streaming. Since they event streaming. Since they have signed up for have signed up for have signed up for delivery at least delivery at least delivery at least once, we once, we once, we have ensured that they have ensured that they have ensured that they have secure have secure have secure idempotent idempotent idempotent operations. So if a operations. So if a operations. So if a batch job batch job batch job is interrupted is interrupted is interrupted in the middle of a process, these in the middle of a process, these in the middle of a process, these commands can commands can commands can retry it, and we retry it, and we retry it, and we guarantee that guarantee that guarantee that there are no there are no there are no duplicate entries in the duplicate entries in the duplicate entries in the registers. At the same time, registers. At the same time, registers. At the same time, delivery at least delivery at least delivery at least once from the once from the once from the event system side event system side event system side guaranteed that guaranteed that guaranteed that they would receive all they would receive all they would receive all created records.

created records. When we were connecting When we were connecting consumers, we consumers, we consumers, we had 50 different had 50 different had 50 different departments in the departments in the departments in the organization that organization that organization that needed our needed our needed our data. In Vanguard, when data. In Vanguard, when data. In Vanguard, when registering for the program, registering for the program, registering for the program, you need to have a you need to have a you need to have a three-character three-character three-character access code. This access code. This access code. This program code describes what the program code describes what the program code describes what the program is and what program is and what program is and what functions it performs. functions it performs. functions it performs. But we went even further. But we went even further. We asked our We asked our customers to provide customers to provide customers to provide a description and business a description and business case for case for case for the need to the need to the need to obtain this data. obtain this data. So, think about it, if So, think about it, if I have a trade order I have a trade order I have a trade order that needs a that needs a that needs a cash register to cash register to cash register to fulfill fulfill fulfill the obligations of the deal the obligations of the deal , that could be a business , that could be a business case. Or for a case. Or for a case. Or for a retirement account retirement account retirement account that must make a that must make a that must make a mandatory mandatory mandatory distribution. A distribution. A 529 account can 529 account can contribute to contribute to college tuition. By grouping all of college tuition. By grouping all of these business descriptions, these business descriptions, these business descriptions, AI tools AI tools AI tools can now pull them can now pull them can now pull them together and provide an accurate together and provide an accurate together and provide an accurate description of which cases description of which cases description of which cases have been modernized in have been modernized in have been modernized in our ecosystem, and our ecosystem, and our ecosystem, and we now know we now know what core business what core business functionality functionality functionality we support. We we support. We we support. We can also go further by can also go further by can also go further by asking what the asking what the

asking what the impact would be if impact would be if impact would be if our SLAs were violated. And our SLAs were violated. And our SLAs were violated. And this is critical this is critical this is critical because if you have a because if you have a because if you have a major incident and major incident and major incident and we see the we see the we see the application code starting application code starting application code starting to fall below the to fall below the five-nines availability level, we five-nines availability level, we can go in and can go in and can go in and see: okay, for a see: okay, for a see: okay, for a batch application, batch application, batch application, we might have we might have we might have a few hours to a few hours to a few hours to respond. For a respond. For a respond. For a real-time API where real-time API where real-time API where the client on the other side the client on the other side the client on the other side gets errors, gets errors, gets errors, we have minutes we have minutes we have minutes and time is and time is and time is critical for critical for critical for self-healing. We self-healing. We self-healing. We continue continue continue to evaluate how to to evaluate how to to evaluate how to accelerate accelerate accelerate response times by combining response times by combining response times by combining business business case justification with our case justification with our surveillance systems and surveillance systems and trace logs. trace logs. So we realized: So we realized: although we started this although we started this although we started this journey before the advent of journey before the advent of journey before the advent of AI tools, we have AI tools, we have AI tools, we have adapted and adapted and adapted and continue to continue to continue to implement new implement new implement new technologies. This technologies. This technologies. This has helped us significantly in has helped us significantly in has helped us significantly in research, research, research, development, and development, and development, and testing. Today, testing. Today, testing. Today, in our in our in our cash program, 90% of cash program, 90% of our engineers our engineers our engineers use use use these AI tools every day.

these AI tools every day. Our teams Our teams use use use multi-agent multi-agent multi-agent workflows and we have workflows and we have workflows and we have shared knowledge bases. shared knowledge bases. shared knowledge bases. We continue to We continue to We continue to develop this develop this develop this AI toolkit AI toolkit AI toolkit to meet to meet to meet our our our modernization needs. We have modernization needs. We have modernization needs. We have a concept of a “ a concept of a “ team brain”: team brain”: team brain”: when engineers when engineers when engineers create create create product features, this knowledge product features, this knowledge product features, this knowledge goes into a goes into a goes into a shared repository shared repository shared repository accessible to all accessible to all accessible to all team members. And team members. And team members. And if a if a if a new new new engineer joins the team, they can engineer joins the team, they can engineer joins the team, they can tap into this tap into this tap into this shared knowledge base and shared knowledge base and shared knowledge base and get the same context in their query get the same context in their query as an engineer who has as an engineer who has been here for the last 5 been here for the last 5 been here for the last 5 years. Now that you years. Now that you years. Now that you are in the are in the are in the process of upgrading process of upgrading , maintaining two , maintaining two , maintaining two codebases is a codebases is a codebases is a daunting task. And what daunting task. And what daunting task. And what you need to you need to you need to do is do is do is keep keep keep moving forward and moving forward and moving forward and fixing bugs in the fixing bugs in the fixing bugs in the new code. Don't new code. Don't new code. Don't fix the past. fix the past. When encountering bugs When encountering bugs that could have existed that could have existed that could have existed for decades, for decades, for decades, continue to continue to continue to fix the fix the fix the modernized state modernized state modernized state and let it be an and let it be an and let it be an incentive for your incentive for your incentive for your developers and developers and developers and consumers consumers consumers to move to the cloud.

to move to the cloud. This can affect This can affect your audit your audit your audit logs, so you logs, so you logs, so you need to have need to have need to have the ability to the ability to the ability to whitelist, and this is very whitelist, and this is very whitelist, and this is very important. And finally, important. And finally, important. And finally, make sure you make sure you make sure you have the right have the right have the right metrics, alerts, metrics, alerts, metrics, alerts, and and and tracking capabilities, as you tracking capabilities, as you tracking capabilities, as you will be will be will be checking them frequently as your checking them frequently as your checking them frequently as your product grows. product grows. Now, since this is a Now, since this is a guiding direction, I guiding direction, I guiding direction, I want to highlight five want to highlight five want to highlight five main points. main points. main points. The first is to make your The first is to make your The first is to make your legacy legacy legacy codebase accessible to codebase accessible to codebase accessible to AI queries. Providing AI queries. Providing AI queries. Providing access to our access to our access to our legacy legacy legacy codebase through the MCP server codebase through the MCP server codebase through the MCP server reduced research cycle time reduced research cycle time reduced research cycle time and and and accelerated accelerated accelerated feedback. That's 7 days feedback. That's 7 days feedback. That's 7 days converted into 15 converted into 15 converted into 15 minutes. It was a minutes. It was a minutes. It was a huge victory. huge victory. huge victory. Treat Treat migration documentation as migration documentation as part of your part of your part of your product. Your product. Your migration documentation is as migration documentation is as important as your business important as your business functionality, because it is what functionality, because it is what all your users will use all your users will use to connect to your to connect to your to connect to your application. Eventually, application. Eventually, application. Eventually, yes, they will become obsolete, yes, they will become obsolete, yes, they will become obsolete, but during the but during the but during the transition process, transition process, transition process, this documentation is this documentation is this documentation is critically important.

critically important. Third point: Third point: rethink, not rethink, not rethink, not just carry just carry just carry everything “as is.” The consumers everything “as is.” The consumers everything “as is.” The consumers and manufacturers and manufacturers and manufacturers we had the hardest time with during the we had the hardest time with during the we had the hardest time with during the transition to a transition to a transition to a modernized modernized modernized state were the ones who state were the ones who state were the ones who tried to simply tried to simply tried to simply port everything from the port everything from the port everything from the old to the new, as old to the new, as old to the new, as opposed to those who opposed to those who opposed to those who stopped and asked, stopped and asked, stopped and asked, “How can we better “How can we better “How can we better develop our develop our develop our product product product by taking advantage of the by taking advantage of the by taking advantage of the modern modern modern capabilities of the capabilities of the capabilities of the cash ledger ecosystem?” cash ledger ecosystem?” The fourth component is the gradual gradual gradual introduction of AI. With introduction of AI. With introduction of AI. With the advent of the advent of the advent of AI tools, you continue AI tools, you continue AI tools, you continue to improve your to improve your to improve your product. In today's product. In today's product. In today's world, there will never be a world, there will never be a world, there will never be a perfect perfect perfect starting point. It is important to starting point. It is important to starting point. It is important to get started and get started and get started and continue continue continue to improve and to improve and to improve and adapt to changes adapt to changes adapt to changes in technology. in technology.

One last point: One last point: observability is observability is observability is mandatory. You mandatory. You mandatory. You need to know who need to know who need to know who is creating your data and is creating your data and is creating your data and who is consuming it. This who is consuming it. This who is consuming it. This becomes even more important becomes even more important becomes even more important as the product as the product as the product transitions to transitions to transitions to normal normal normal operation, and once the operation, and once the upgrade is complete, your upgrade is complete, your teams need to teams need to teams need to maintain this as maintain this as maintain this as part of their daily part of their daily part of their daily operations. Thank you very much operations. Thank you very much operations. Thank you very much everyone. If we have time, everyone. If we have time, everyone. If we have time, we will answer we will answer we will answer questions. Otherwise, questions. Otherwise, questions. Otherwise, I will be outside the I will be outside the I will be outside the hall. You can hall. You can hall. You can chat with chat with chat with me later. And me later. And me later. And thank you again for the thank you again for the thank you again for the opportunity.