Skills are new features: Building Skill-Centric Harness — Yogendra Miraje, FactSet

Hi everyone, I am Yogi. I work at Faxet Hi everyone, I am Yogi. I work at Faxet as principal AI engineer. We are a as principal AI engineer. We are a as principal AI engineer. We are a financial data and research company. I'm financial data and research company. I'm financial data and research company. I'm going to talk about how to build skill going to talk about how to build skill going to talk about how to build skill centric agentic products and I'm going centric agentic products and I'm going centric agentic products and I'm going to post slides so you don't have to keep to post slides so you don't have to keep to post slides so you don't have to keep uh taking photos. So that's my exandle uh taking photos. So that's my exandle uh taking photos. So that's my exandle yogi not the bear. Um yogi not the bear. Um yogi not the bear. Um so so so let's connect there and uh let's begin. So in the last year's talk in this very So in the last year's talk in this very conference I talked about blueprints and conference I talked about blueprints and conference I talked about blueprints and what blueprints were really a simple set what blueprints were really a simple set what blueprints were really a simple set of steps or recipe that you can hand of steps or recipe that you can hand of steps or recipe that you can hand over to agents over to agents over to agents so that agent doesn't have to discover so that agent doesn't have to discover so that agent doesn't have to discover its path every time and when I look back its path every time and when I look back its path every time and when I look back it was simply a skill in a very naive it was simply a skill in a very naive it was simply a skill in a very naive form form form and a lot has changed since Then and a lot has changed since Then and a lot has changed since Then Antropic has shipped skills in last year Antropic has shipped skills in last year Antropic has shipped skills in last year October and even warned us not to build October and even warned us not to build October and even warned us not to build agents.

agents. agents. But on serious note since Antropic had But on serious note since Antropic had But on serious note since Antropic had already open source the skills there was already open source the skills there was already open source the skills there was no point in trying to maintain our own no point in trying to maintain our own no point in trying to maintain our own standards. So we moved away from standards. So we moved away from standards. So we moved away from blueprints and just adopted skills blueprints and just adopted skills blueprints and just adopted skills fully. And this talk really inspired me fully. And this talk really inspired me fully. And this talk really inspired me to build skill centric agent and I'm to build skill centric agent and I'm to build skill centric agent and I'm going to share some of my learnings from going to share some of my learnings from going to share some of my learnings from that journey. So a quick raise of hands. How many of So a quick raise of hands. How many of you really build skills here? So almost all of you. Now lower your So almost all of you. Now lower your hands if you have built that only in hands if you have built that only in hands if you have built that only in context of cloud code and codeex. and context of cloud code and codeex. and context of cloud code and codeex. and raise your hand if you have like built raise your hand if you have like built raise your hand if you have like built your own harness and added you know your own harness and added you know your own harness and added you know skills to that. skills to that. skills to that. So okay so I see a few hands. So reason So okay so I see a few hands. So reason So okay so I see a few hands. So reason I'm asking is when I see most of the I'm asking is when I see most of the I'm asking is when I see most of the online discourse it's about you know online discourse it's about you know online discourse it's about you know coding harnesses and skills in context coding harnesses and skills in context coding harnesses and skills in context of the coding agents and how to write of the coding agents and how to write of the coding agents and how to write great skills like so we need that but great skills like so we need that but great skills like so we need that but today I'm not going to talk about that today I'm not going to talk about that today I'm not going to talk about that what I'm going to focus on is skills in what I'm going to focus on is skills in what I'm going to focus on is skills in the context of agentic products and how the context of agentic products and how the context of agentic products and how do you really add support for skills in do you really add support for skills in do you really add support for skills in in your own harness and how to scale it in your own harness and how to scale it in your own harness and how to scale it at enterprise scale.

at enterprise scale. at enterprise scale. So traditionally product used to look So traditionally product used to look So traditionally product used to look like this. A surface made of screens, like this. A surface made of screens, like this. A surface made of screens, buttons, forms and dashboards. The user buttons, forms and dashboards. The user buttons, forms and dashboards. The user navigated this UI. navigated this UI. navigated this UI. But nowadays we are seeing more and more But nowadays we are seeing more and more But nowadays we are seeing more and more these kind of interfaces where agent is these kind of interfaces where agent is these kind of interfaces where agent is at the forefront. at the forefront. at the forefront. The user either talks to the agent or The user either talks to the agent or The user either talks to the agent or agent behind the scene is the main agent behind the scene is the main agent behind the scene is the main decision maker helping users navigate decision maker helping users navigate decision maker helping users navigate your product. your product. your product. Now if the agent becomes the main Now if the agent becomes the main Now if the agent becomes the main interface for your product then where do interface for your product then where do interface for your product then where do features live? This framing of who, what, and how This framing of who, what, and how really helps to answer that question. really helps to answer that question. really helps to answer that question. Prompts define who the agent is. Tools Prompts define who the agent is. Tools Prompts define who the agent is. Tools define what it can connect to. And define what it can connect to. And define what it can connect to. And skills really tell you how a task gets skills really tell you how a task gets skills really tell you how a task gets done. And this is the great place to done. And this is the great place to done. And this is the great place to keep your business logic that shapes keep your business logic that shapes keep your business logic that shapes your agents behavior. So skills are the your agents behavior. So skills are the your agents behavior. So skills are the new features. And you can see this with new features. And you can see this with new features. And you can see this with this example. So equity research and this example. So equity research and this example. So equity research and wealth management are two very important wealth management are two very important wealth management are two very important workflows in uh finance and these all workflows in uh finance and these all workflows in uh finance and these all used to be the buttons, dropdowns and used to be the buttons, dropdowns and used to be the buttons, dropdowns and screens and now they are merely skills.

One of the most uh underrated thing One of the most uh underrated thing about skills especially when talking about skills especially when talking about skills especially when talking about agentric products is how it has about agentric products is how it has about agentric products is how it has enabled to build skills for anyone who enabled to build skills for anyone who enabled to build skills for anyone who has good understanding of the products. has good understanding of the products. has good understanding of the products. So if skills are new features and this So if skills are new features and this So if skills are new features and this new features can be shipped by anyone in new features can be shipped by anyone in new features can be shipped by anyone in the company. The question is what's the the company. The question is what's the the company. The question is what's the role of engineer? Then role of engineer? Then role of engineer? Then the role of engineer is shifting from the role of engineer is shifting from the role of engineer is shifting from shipping features to shipping harnesses. shipping features to shipping harnesses. shipping features to shipping harnesses. Harnesses that are smooth vehicles for Harnesses that are smooth vehicles for Harnesses that are smooth vehicles for your skills to run. But before diving your skills to run. But before diving your skills to run. But before diving into that, let's just get to the basics into that, let's just get to the basics into that, let's just get to the basics and try to understand what skill is. The and try to understand what skill is. The and try to understand what skill is. The dictionary meaning of skill is the dictionary meaning of skill is the dictionary meaning of skill is the ability to do something well. And what ability to do something well. And what ability to do something well. And what it means is that your model can live it means is that your model can live it means is that your model can live with it without it. But it obviously with it without it. But it obviously with it without it. But it obviously going to do better in presence of skill. going to do better in presence of skill. going to do better in presence of skill. A better definition for agent skill is a A better definition for agent skill is a A better definition for agent skill is a standardized way to teach AI agents how standardized way to teach AI agents how standardized way to teach AI agents how to do a specific task well.

to do a specific task well. to do a specific task well. And a simple skill could contain just And a simple skill could contain just And a simple skill could contain just like a markdown but a very complex can like a markdown but a very complex can like a markdown but a very complex can have multiple references to the files have multiple references to the files have multiple references to the files and executable scripts. The skill.md is the heart of your skill. The skill.md is the heart of your skill. Name and description in the front matter Name and description in the front matter Name and description in the front matter are the key uh things that will help you are the key uh things that will help you are the key uh things that will help you to discover the skill and the business to discover the skill and the business to discover the skill and the business logic and instruction goes into the body logic and instruction goes into the body logic and instruction goes into the body of the skill and which will also contain of the skill and which will also contain of the skill and which will also contain references to the files and scripts. references to the files and scripts. references to the files and scripts. So let's see how to add a skill support So let's see how to add a skill support So let's see how to add a skill support in your harness. in your harness. in your harness. So what do you need to do like a bare So what do you need to do like a bare So what do you need to do like a bare minimum skill in your harness? You only minimum skill in your harness? You only minimum skill in your harness? You only need these three things like skill need these three things like skill need these three things like skill registry, a system prompt, and a basic registry, a system prompt, and a basic registry, a system prompt, and a basic file read tool. If you're running file read tool. If you're running file read tool. If you're running scripts, then obviously you're going to scripts, then obviously you're going to scripts, then obviously you're going to need either bash or maybe a code uh need either bash or maybe a code uh need either bash or maybe a code uh running sandbox environment. But this is running sandbox environment. But this is running sandbox environment. But this is the bare minimum requirement for adding the bare minimum requirement for adding the bare minimum requirement for adding support of skills like in your harness.

support of skills like in your harness. support of skills like in your harness. A simple skill registry looks like this. A simple skill registry looks like this. A simple skill registry looks like this. So what is a skill registry? It just So what is a skill registry? It just So what is a skill registry? It just like collection of skills with their like collection of skills with their like collection of skills with their name, description and path. And we are name, description and path. And we are name, description and path. And we are going to see this with three example going to see this with three example going to see this with three example skills. skills. skills. Company research skill Company research skill Company research skill which is supposed to do a very basic web which is supposed to do a very basic web which is supposed to do a very basic web search for a company and produce a search for a company and produce a search for a company and produce a markdown. markdown. markdown. a report HTML skill that will turn the a report HTML skill that will turn the a report HTML skill that will turn the markdown into an HTML. A report PDF markdown into an HTML. A report PDF markdown into an HTML. A report PDF skill that will take that markdown and skill that will take that markdown and skill that will take that markdown and turn into PDF. Now how does really the agent discover Now how does really the agent discover the skill? So you have your registry the skill? So you have your registry the skill? So you have your registry then you form a skill concat then you form a skill concat then you form a skill concat concatenating the name description path concatenating the name description path concatenating the name description path and put it in your system prompt. So if and put it in your system prompt. So if and put it in your system prompt. So if you notice like we are only using the you notice like we are only using the you notice like we are only using the name and description path in in this name and description path in in this name and description path in in this system prompt and not the skill body and system prompt and not the skill body and system prompt and not the skill body and that's what what they call about is that's what what they call about is that's what what they call about is progressive disclosure.

progressive disclosure. progressive disclosure. Agent is going to read the skills and Agent is going to read the skills and Agent is going to read the skills and only pick the skills that it is going to only pick the skills that it is going to only pick the skills that it is going to read and follow the instructions from read and follow the instructions from read and follow the instructions from there. there. there. And then you need your agentic loop And then you need your agentic loop And then you need your agentic loop that will run this system prompt. Here that will run this system prompt. Here that will run this system prompt. Here we're keeping track of all the messages we're keeping track of all the messages we're keeping track of all the messages in uh in that messages array and we're in uh in that messages array and we're in uh in that messages array and we're going to call the model with messages going to call the model with messages going to call the model with messages and agent tools and for every turn it's and agent tools and for every turn it's and agent tools and for every turn it's either looking for like making a tool either looking for like making a tool either looking for like making a tool call and all the tool calls get appended call and all the tool calls get appended call and all the tool calls get appended to the messages and if there is like no to the messages and if there is like no to the messages and if there is like no tool call we are just going to output tool call we are just going to output tool call we are just going to output the end of the uh program and going to the end of the uh program and going to the end of the uh program and going to show the output. show the output. show the output. So in this case what does like agent see So in this case what does like agent see So in this case what does like agent see in our example. in our example. in our example. So it sees that the skills that are So it sees that the skills that are So it sees that the skills that are available available available and and and then the activation part is when it then the activation part is when it then the activation part is when it looks for the company research skill for looks for the company research skill for looks for the company research skill for take this example of if you're asking to take this example of if you're asking to take this example of if you're asking to publish a report of NVDI it is going to publish a report of NVDI it is going to publish a report of NVDI it is going to call company research do the web call company research do the web call company research do the web searches and then use the build report searches and then use the build report searches and then use the build report skill to produce the report HTML skill to produce the report HTML skill to produce the report HTML And the output looks like something like And the output looks like something like And the output looks like something like this. Now what are the learnings from this. Now what are the learnings from this. Now what are the learnings from some of this is the descriptions are some of this is the descriptions are some of this is the descriptions are really the routing signals. And what I

really the routing signals. And what I really the routing signals. And what I mean by that if you noticed I had like mean by that if you noticed I had like mean by that if you noticed I had like two different skills report HTML and two different skills report HTML and two different skills report HTML and report PDF. But when I showed the report PDF. But when I showed the report PDF. But when I showed the example, you saw only HTML. And the example, you saw only HTML. And the example, you saw only HTML. And the reason is I have this description saying reason is I have this description saying reason is I have this description saying that use the skill only when user ask that use the skill only when user ask that use the skill only when user ask for a PDF report. Focus on this word for a PDF report. Focus on this word for a PDF report. Focus on this word PDF. Right? So that is the trigger word PDF. Right? So that is the trigger word PDF. Right? So that is the trigger word that helps agent to know which skill to that helps agent to know which skill to that helps agent to know which skill to pick and that's why descriptions are pick and that's why descriptions are pick and that's why descriptions are called routing signals. And it's very called routing signals. And it's very called routing signals. And it's very important to keep your descriptions important to keep your descriptions important to keep your descriptions aligned to the user request and not aligned to the user request and not aligned to the user request and not about about about the skill itself. the skill itself. the skill itself. It's also important to keep your It's also important to keep your It's also important to keep your descriptions distinct enough so that descriptions distinct enough so that descriptions distinct enough so that agent doesn't get confused and make sure agent doesn't get confused and make sure agent doesn't get confused and make sure don't let your skills get stale because don't let your skills get stale because don't let your skills get stale because these are the reasons why your skills these are the reasons why your skills these are the reasons why your skills don't get triggered.

don't get triggered. don't get triggered. One more like very important difference One more like very important difference One more like very important difference between like skills when we talk in between like skills when we talk in between like skills when we talk in agentic product context is most of the agentic product context is most of the agentic product context is most of the skills are only model driven because for skills are only model driven because for skills are only model driven because for nontechnical users we're not adding that nontechnical users we're not adding that nontechnical users we're not adding that cognitive load to remember them cognitive load to remember them cognitive load to remember them uh to keep the track of all the skills. Another learning that had was cut by Another learning that had was cut by user intent and not by data model. So user intent and not by data model. So user intent and not by data model. So when I started building the skill when I started building the skill when I started building the skill library, I had very narrow use cases. So library, I had very narrow use cases. So library, I had very narrow use cases. So add skill for estimation analysis or add add skill for estimation analysis or add add skill for estimation analysis or add for fundamentals. But I got real use for fundamentals. But I got real use for fundamentals. But I got real use cases and those use cases were not cases and those use cases were not cases and those use cases were not reflecting the data model. Those were reflecting the data model. Those were reflecting the data model. Those were really about the real use cases and I really about the real use cases and I really about the real use cases and I had to refactor this multiple times and had to refactor this multiple times and had to refactor this multiple times and that is okay, right? you start simply that is okay, right? you start simply that is okay, right? you start simply with narrower use cases and as you with narrower use cases and as you with narrower use cases and as you discover more use cases you start discover more use cases you start discover more use cases you start refactoring your skill library.

refactoring your skill library. refactoring your skill library. So in practice it means that you know So in practice it means that you know So in practice it means that you know instead of having an estimate analysis instead of having an estimate analysis instead of having an estimate analysis skill you should have earning skill you should have earning skill you should have earning preparation skill instead of having a preparation skill instead of having a preparation skill instead of having a skill for news and analysis skill for news and analysis skill for news and analysis analyst rating skill you should have a analyst rating skill you should have a analyst rating skill you should have a pre-market briefing skill. So we updated our uh stack to a new So we updated our uh stack to a new model and our agent start uh failing model and our agent start uh failing model and our agent start uh failing because it was not obeying the skills. because it was not obeying the skills. because it was not obeying the skills. Nothing was changed. Not a single line Nothing was changed. Not a single line Nothing was changed. Not a single line in the skill was changed but still it in the skill was changed but still it in the skill was changed but still it failed. And when when we dig under the failed. And when when we dig under the failed. And when when we dig under the hood what was happening it was that this hood what was happening it was that this hood what was happening it was that this like new model was very focusing on like new model was very focusing on like new model was very focusing on beginning of the skill and we had very beginning of the skill and we had very beginning of the skill and we had very critical instruction at the end of the critical instruction at the end of the critical instruction at the end of the skill. So that's why it's very important skill. So that's why it's very important skill. So that's why it's very important to run evals and skills without evals to run evals and skills without evals to run evals and skills without evals are really just wishful thinking.

are really just wishful thinking. are really just wishful thinking. Skills are not the documentation and a Skills are not the documentation and a Skills are not the documentation and a lot of people treat them like that and lot of people treat them like that and lot of people treat them like that and skills are really the contracts skills are really the contracts skills are really the contracts versioned to a model. So whenever you're versioned to a model. So whenever you're versioned to a model. So whenever you're upgrading a model, make sure to rerun uh upgrading a model, make sure to rerun uh upgrading a model, make sure to rerun uh your eval. So So when you have like a few skills, shoving when you have like a few skills, shoving when you have like a few skills, shoving them in the system prompt really works. them in the system prompt really works. them in the system prompt really works. But as soon as you start growing your But as soon as you start growing your But as soon as you start growing your skill stack, this falls apart. When you skill stack, this falls apart. When you skill stack, this falls apart. When you have like more than 10 skill, maybe have like more than 10 skill, maybe have like more than 10 skill, maybe that's like a good point to think start that's like a good point to think start that's like a good point to think start thinking about, you know, how can you thinking about, you know, how can you thinking about, you know, how can you shortlist the skills that you're going shortlist the skills that you're going shortlist the skills that you're going to add to the system prompt. And this to add to the system prompt. And this to add to the system prompt. And this could mean just like having embeddings could mean just like having embeddings could mean just like having embeddings and have similarity search and short and have similarity search and short and have similarity search and short lists or a smaller model that can lists or a smaller model that can lists or a smaller model that can shortlist the skills and add to the shortlist the skills and add to the shortlist the skills and add to the system prompt. The real trouble really system prompt. The real trouble really system prompt. The real trouble really starts when you have hundreds of skills.

starts when you have hundreds of skills. starts when you have hundreds of skills. At that point, you really need hierarchy At that point, you really need hierarchy At that point, you really need hierarchy of skills and metadata filters and the of skills and metadata filters and the of skills and metadata filters and the governance in the place to keep your uh governance in the place to keep your uh governance in the place to keep your uh library searchable and coherent. So there are like five aspect of skill So there are like five aspect of skill library governance library governance library governance and these are admission, ownership, and these are admission, ownership, and these are admission, ownership, boundaries, life cycle and coherence. boundaries, life cycle and coherence. boundaries, life cycle and coherence. Sounds very enterprisey right but each Sounds very enterprisey right but each Sounds very enterprisey right but each of this really answer like a very core of this really answer like a very core of this really answer like a very core question question question and we will get to that in a second. But and we will get to that in a second. But and we will get to that in a second. But when you hear governance, when you hear governance, when you hear governance, it really doesn't need to be a red tape it really doesn't need to be a red tape it really doesn't need to be a red tape bottleneck. It really depends on how bottleneck. It really depends on how bottleneck. It really depends on how you're implementing it. How much you're implementing it. How much you're implementing it. How much automation is in place with proper human automation is in place with proper human automation is in place with proper human in the loop. in the loop. in the loop. And the good news is that we can borrow And the good news is that we can borrow And the good news is that we can borrow a lot of good practices from code and a lot of good practices from code and a lot of good practices from code and apply them to the skills. And these apply them to the skills. And these apply them to the skills. And these coding practices like has worked for coding practices like has worked for coding practices like has worked for decades.

decades. decades. So what does admission even mean? What So what does admission even mean? What So what does admission even mean? What it means that should this skill even it means that should this skill even it means that should this skill even exist or it should go to an already exist or it should go to an already exist or it should go to an already existing skill. And in practice we build existing skill. And in practice we build existing skill. And in practice we build automated get for the registries with automated get for the registries with automated get for the registries with human in the loop. And this is very human in the loop. And this is very human in the loop. And this is very analog log to how we do the PR review analog log to how we do the PR review analog log to how we do the PR review process. Who maintains your skill? Who maintains your skill? Just like how features are maintained by Just like how features are maintained by Just like how features are maintained by application teams, we need skills to be application teams, we need skills to be application teams, we need skills to be maintained by application teams. And maintained by application teams. And maintained by application teams. And like code owners, we need to have a like code owners, we need to have a like code owners, we need to have a dedicated skill owners like named dedicated skill owners like named dedicated skill owners like named maintainers for your skills. What happens to the skill over time? You What happens to the skill over time? You need to have semantic version of the need to have semantic version of the need to have semantic version of the skills and also whenever you're getting skills and also whenever you're getting skills and also whenever you're getting rid of skills you need to have rid of skills you need to have rid of skills you need to have deprecation warnings and make sure that deprecation warnings and make sure that deprecation warnings and make sure that your changes are reflected in the change your changes are reflected in the change your changes are reflected in the change logs.

So when you have large number of skills, So when you have large number of skills, the library should still make sense. And the library should still make sense. And the library should still make sense. And just like features are cohesive in a just like features are cohesive in a just like features are cohesive in a good product, we need to make sure that good product, we need to make sure that good product, we need to make sure that we are conducting periodic audits and we are conducting periodic audits and we are conducting periodic audits and skill validation checks so that your skill validation checks so that your skill validation checks so that your skills would really make sense. Whenever you have a lot of skills, it's Whenever you have a lot of skills, it's important to have the allow list tools important to have the allow list tools important to have the allow list tools in in the skills and your tools are in in the skills and your tools are in in the skills and your tools are supposed to be access control and this supposed to be access control and this supposed to be access control and this is very important to keep the correct is very important to keep the correct is very important to keep the correct boundaries around skills. So what are the main takeaways? The main So what are the main takeaways? The main takeaways are skills are the features in takeaways are skills are the features in takeaways are skills are the features in your agentic products. Our role is your agentic products. Our role is your agentic products. Our role is shifting from features that is shipping shifting from features that is shipping shifting from features that is shipping features to shipping harnesses features to shipping harnesses features to shipping harnesses and routing mechanism doesn't get tuned and routing mechanism doesn't get tuned and routing mechanism doesn't get tuned as you scale. It changes the mechanism as you scale. It changes the mechanism as you scale. It changes the mechanism itself and at enterprise scale the skill itself and at enterprise scale the skill itself and at enterprise scale the skill library governance is really library governance is really library governance is really non-negotiable.

That was my talk. Thank you very much That was my talk. Thank you very much for listening. for listening. for listening. [applause]