The End of the Static Screen: Architecting Intent-Driven UX — Gus Iwanaga, commercetools

>> How's it going? >> How's it going? The end of uh the end of the conference, The end of uh the end of the conference, The end of uh the end of the conference, how's everybody feeling? Tired, drinking how's everybody feeling? Tired, drinking how's everybody feeling? Tired, drinking from the fire hose as well? from the fire hose as well? from the fire hose as well? Are you guys a little bit tired of Are you guys a little bit tired of Are you guys a little bit tired of hearing loop engineering, harness hearing loop engineering, harness hearing loop engineering, harness engineering, software factory, evolves, engineering, software factory, evolves, engineering, software factory, evolves, pre-training, pre-training, pre-training, post-training data, and whatnot? Uh but post-training data, and whatnot? Uh but post-training data, and whatnot? Uh but anyways, those topics were more than anyways, those topics were more than anyways, those topics were more than valid, right? Uh hi. valid, right? Uh hi. valid, right? Uh hi. Some uh really uh common faces here. By Some uh really uh common faces here. By Some uh really uh common faces here. By the way, I'm super excited to be here the way, I'm super excited to be here the way, I'm super excited to be here and talk to you guys about this title. and talk to you guys about this title. and talk to you guys about this title. And And I'm sorry, when I was when I And And I'm sorry, when I was when I And And I'm sorry, when I was when I submitted the application, I was submitted the application, I was submitted the application, I was thinking of potentially a catchy title, thinking of potentially a catchy title, thinking of potentially a catchy title, but I don't like this at all. So, with but I don't like this at all. So, with but I don't like this at all. So, with all due respect to the organizers, I'll all due respect to the organizers, I'll all due respect to the organizers, I'll have to make a change. Uh and the actual have to make a change. Uh and the actual have to make a change. Uh and the actual topic that I want to focus here today is topic that I want to focus here today is topic that I want to focus here today is lessons learned uh from a team that is lessons learned uh from a team that is lessons learned uh from a team that is building proper generative uh UX and UI. building proper generative uh UX and UI. building proper generative uh UX and UI. And I was going to uh touch on agentic And I was going to uh touch on agentic And I was going to uh touch on agentic orchestration, but come on. Uh over the orchestration, but come on. Uh over the orchestration, but come on. Uh over the last 3 days, this is what we heard all last 3 days, this is what we heard all last 3 days, this is what we heard all the time. So, I'd rather focus on what I the time. So, I'd rather focus on what I the time. So, I'd rather focus on what I didn't hear enough about here in the didn't hear enough about here in the didn't hear enough about here in the conference, and hopefully you walk away, conference, and hopefully you walk away, conference, and hopefully you walk away, if not with something very tangible, but if not with something very tangible, but if not with something very tangible, but with a new mental model that can spark with a new mental model that can spark with a new mental model that can spark meaningful discussions down the road.

meaningful discussions down the road. meaningful discussions down the road. Sound good? Sound good? Sound good? All right. All right. All right. Very good. Very good. Very good. So, hi everybody. Uh I'm Gus. I'm a So, hi everybody. Uh I'm Gus. I'm a So, hi everybody. Uh I'm Gus. I'm a general manager at commercetools. I lead general manager at commercetools. I lead general manager at commercetools. I lead product, UX, and engineering for product, UX, and engineering for product, UX, and engineering for zero-to-one products. I tell people I'm zero-to-one products. I tell people I'm zero-to-one products. I tell people I'm in a very privileged position because we in a very privileged position because we in a very privileged position because we get to build really cool stuff. So, we get to build really cool stuff. So, we get to build really cool stuff. So, we cook really interesting stuff at cook really interesting stuff at cook really interesting stuff at commercetools. commercetools. commercetools. Fantastic. And this is what you guys can Fantastic. And this is what you guys can Fantastic. And this is what you guys can expect uh at least during the expect uh at least during the expect uh at least during the presentation. presentation. presentation. Uh I'd like to make sure that we're on Uh I'd like to make sure that we're on Uh I'd like to make sure that we're on the same page with respect to the the same page with respect to the the same page with respect to the problem uh the problem space uh followed problem uh the problem space uh followed problem uh the problem space uh followed by a quick demo of the product because by a quick demo of the product because by a quick demo of the product because I'm not sure if you guys agree, an image I'm not sure if you guys agree, an image I'm not sure if you guys agree, an image speaks more than 1,000 words. So, it speaks more than 1,000 words. So, it speaks more than 1,000 words. So, it would just uh make it more tangible for would just uh make it more tangible for would just uh make it more tangible for uh everybody here. Followed by a rapid uh everybody here. Followed by a rapid uh everybody here. Followed by a rapid discussion on the emergence of UI discussion on the emergence of UI discussion on the emergence of UI protocols, and I'm not sure if you guys protocols, and I'm not sure if you guys protocols, and I'm not sure if you guys joined maybe some of the talks here. joined maybe some of the talks here. joined maybe some of the talks here. Even the founder of some of these Even the founder of some of these Even the founder of some of these protocols were here uh this week, and protocols were here uh this week, and protocols were here uh this week, and that was really cool. Followed by and that was really cool. Followed by and that was really cool. Followed by and last but not least, the challenges that last but not least, the challenges that last but not least, the challenges that uh my team uh and I faced and we still uh my team uh and I faced and we still uh my team uh and I faced and we still face. And some of the mitigation tactics face. And some of the mitigation tactics face. And some of the mitigation tactics uh that we put in place to overcome some uh that we put in place to overcome some uh that we put in place to overcome some of these challenges.

of these challenges. of these challenges. So, with that So, with that So, with that let's continue. The problem space. let's continue. The problem space. let's continue. The problem space. This is This is more like a a a This is This is more like a a a This is This is more like a a a statement um statement um statement um and we're still adapt to the software and we're still adapt to the software and we're still adapt to the software that we ship, not the other way around. that we ship, not the other way around. that we ship, not the other way around. Right? And then uh although you could Right? And then uh although you could Right? And then uh although you could argue when GPT came out, this was argue when GPT came out, this was argue when GPT came out, this was November uh 2022, we had a really good November uh 2022, we had a really good November uh 2022, we had a really good glimpse of uh real personalization, but glimpse of uh real personalization, but glimpse of uh real personalization, but everything else uh remained static. And everything else uh remained static. And everything else uh remained static. And even with AI, we keep shipping a lot of even with AI, we keep shipping a lot of even with AI, we keep shipping a lot of stuff much faster, but to a significant stuff much faster, but to a significant stuff much faster, but to a significant extent, it is still static. And my extent, it is still static. And my extent, it is still static. And my question is, why? question is, why? question is, why? So, for the uh over the last 40 years, So, for the uh over the last 40 years, So, for the uh over the last 40 years, uh we kept shipping a static uh we kept shipping a static uh we kept shipping a static experiences. And if I put it myself in experiences. And if I put it myself in experiences. And if I put it myself in the shoes of uh some of uh my customers, the shoes of uh some of uh my customers, the shoes of uh some of uh my customers, they need several SaaS applications for they need several SaaS applications for they need several SaaS applications for uh the day-to-day work, and each with uh the day-to-day work, and each with uh the day-to-day work, and each with its own mental model and its own way to its own mental model and its own way to its own mental model and its own way to get anything done.

get anything done. get anything done. And over time, it just kept getting And over time, it just kept getting And over time, it just kept getting worse, just accruing uh debt. The worse, just accruing uh debt. The worse, just accruing uh debt. The cognitive load that we wanted to remove cognitive load that we wanted to remove cognitive load that we wanted to remove and that we wanted to transfer to the and that we wanted to transfer to the and that we wanted to transfer to the machine, uh it's on us. machine, uh it's on us. machine, uh it's on us. And now with AI, things can be And now with AI, things can be And now with AI, things can be different. different. different. I don't know if you guys agree, but this I don't know if you guys agree, but this I don't know if you guys agree, but this is uh what we what we think. is uh what we what we think. is uh what we what we think. And I do have a couple of examples. I'm And I do have a couple of examples. I'm And I do have a couple of examples. I'm not going to say out loud the name of not going to say out loud the name of not going to say out loud the name of these apps, but let's take a look. these apps, but let's take a look. these apps, but let's take a look. Here. Here. Here. First one. First one. First one. You know, that icon is very well known. You know, that icon is very well known. You know, that icon is very well known. This is probably the most famous CRM of This is probably the most famous CRM of This is probably the most famous CRM of all times. all times. all times. But when I look at this screen, there's But when I look at this screen, there's But when I look at this screen, there's a lot going on. I don't even know where a lot going on. I don't even know where a lot going on. I don't even know where to start. Right? Not only the to start. Right? Not only the to start. Right? Not only the information overload, how many features information overload, how many features information overload, how many features how many teams do you think are somewhat how many teams do you think are somewhat how many teams do you think are somewhat involved just to ship this? involved just to ship this? involved just to ship this? Many, probably. Right? And this is just Many, probably. Right? And this is just Many, probably. Right? And this is just one. one. one. Let's have a look. I have a couple of Let's have a look. I have a couple of Let's have a look. I have a couple of other examples. other examples. other examples. So, So, So, this here. this here. this here. Look at that. Look at that. Look at that. Oh my god, this is a fancy table. I Oh my god, this is a fancy table. I Oh my god, this is a fancy table. I don't even know where to start. But don't even know where to start. But don't even know where to start. But anyways, anyways, anyways, one more.

one more. one more. Does everybody know this one here? Does everybody know this one here? Does everybody know this one here? Beautiful. Beautiful UI. Right? It's a Beautiful. Beautiful UI. Right? It's a Beautiful. Beautiful UI. Right? It's a fantastic, super intuitive, and fantastic, super intuitive, and fantastic, super intuitive, and something else just to highlight. something else just to highlight. something else just to highlight. Do you guys know how much time these Do you guys know how much time these Do you guys know how much time these companies need to invest in onboarding companies need to invest in onboarding companies need to invest in onboarding people? people? people? So, that was the the trade-off. So, that was the the trade-off. So, that was the the trade-off. Right? So, you got to allocate a lot of Right? So, you got to allocate a lot of Right? So, you got to allocate a lot of time for a lot of people just to onboard time for a lot of people just to onboard time for a lot of people just to onboard newcomers. As a result of this newcomers. As a result of this newcomers. As a result of this complexity that has been introduced over complexity that has been introduced over complexity that has been introduced over time. time. time. So, this is just at least the hardcore So, this is just at least the hardcore So, this is just at least the hardcore evidence. So, different apps, different evidence. So, different apps, different evidence. So, different apps, different logic every single time. And then more logic every single time. And then more logic every single time. And then more apps are coming out. So, imagine just apps are coming out. So, imagine just apps are coming out. So, imagine just put yourself in the shoes of average put yourself in the shoes of average put yourself in the shoes of average user, and then oh, now I have five five user, and then oh, now I have five five user, and then oh, now I have five five apps. And then every single one I need apps. And then every single one I need apps. And then every single one I need to learn how to navigate, how to browse, to learn how to navigate, how to browse, to learn how to navigate, how to browse, and so on and so forth. And then this and so on and so forth. And then this and so on and so forth. And then this has been the history up until now. has been the history up until now. has been the history up until now. But then this was August last year, I But then this was August last year, I But then this was August last year, I sat down with my boss. It happens to be sat down with my boss. It happens to be sat down with my boss. It happens to be the founder of the company, so big shout the founder of the company, so big shout the founder of the company, so big shout out to my boss. And then we asked this out to my boss. And then we asked this out to my boss. And then we asked this uh question because at Commerce Tools, uh question because at Commerce Tools, uh question because at Commerce Tools, we are an API first company, 300 plus we are an API first company, 300 plus we are an API first company, 300 plus still counting. And we ask this posing still counting. And we ask this posing still counting. And we ask this posing question, through the lens of artificial question, through the lens of artificial question, through the lens of artificial intelligence, what are the foundational intelligence, what are the foundational intelligence, what are the foundational shifts that could be made if we could shifts that could be made if we could shifts that could be made if we could change drastically the way that we change drastically the way that we change drastically the way that we interact with software?

interact with software? interact with software? Not in a uh static uh fashion. And the Not in a uh static uh fashion. And the Not in a uh static uh fashion. And the answer to that question led to the answer to that question led to the answer to that question led to the product that I'm going to demo right product that I'm going to demo right product that I'm going to demo right now. now. now. Let's go. Let's go. Let's go. How about quick demo? Guys like the How about quick demo? Guys like the How about quick demo? Guys like the idea? idea? idea? Give me a thumbs up. Bye. I know Give me a thumbs up. Bye. I know Give me a thumbs up. Bye. I know everybody's tired. Let's go. Yeah. everybody's tired. Let's go. Yeah. everybody's tired. Let's go. Yeah. All right. Cool. All right. Cool. All right. Cool. Look at this. I know, I'm going to zoom Look at this. I know, I'm going to zoom Look at this. I know, I'm going to zoom in. No worries. So, in. No worries. So, in. No worries. So, I have this query here. Create a sales I have this query here. Create a sales I have this query here. Create a sales report for Q1. report for Q1. report for Q1. And the UX side of me, when I look at uh And the UX side of me, when I look at uh And the UX side of me, when I look at uh what was generated, and by the way, what was generated, and by the way, what was generated, and by the way, everything here on the right side has everything here on the right side has everything here on the right side has been auto-generated, guided by us, but been auto-generated, guided by us, but been auto-generated, guided by us, but this is AI, right? Deciding on the this is AI, right? Deciding on the this is AI, right? Deciding on the placement, on the information placement, on the information placement, on the information architecture, deciding which uh architecture, deciding which uh architecture, deciding which uh components had to be actually retrieved components had to be actually retrieved components had to be actually retrieved from the catalog, but I don't like it from the catalog, but I don't like it from the catalog, but I don't like it at all. Right? It's a Even if you don't at all. Right? It's a Even if you don't at all. Right? It's a Even if you don't know a lot uh about UX, just let's, you know a lot uh about UX, just let's, you know a lot uh about UX, just let's, you know, I have at least four different know, I have at least four different know, I have at least four different variations because those were four variations because those were four variations because those were four different terms for the same query.

different terms for the same query. different terms for the same query. Let's check it out. First one here, it Let's check it out. First one here, it Let's check it out. First one here, it was Q1 uh Urban Thread, whatever that was Q1 uh Urban Thread, whatever that was Q1 uh Urban Thread, whatever that is. Uh Q1, I see all of these KPI cards. is. Uh Q1, I see all of these KPI cards. is. Uh Q1, I see all of these KPI cards. There's a lot going on here. And then my There's a lot going on here. And then my There's a lot going on here. And then my intuition tells me, man, this doesn't intuition tells me, man, this doesn't intuition tells me, man, this doesn't add up. add up. add up. Okay. Second one, it's not Q1 anymore. Okay. Second one, it's not Q1 anymore. Okay. Second one, it's not Q1 anymore. Now this is January and March. There's Now this is January and March. There's Now this is January and March. There's no consistency. no consistency. no consistency. Does that help? Yes or no? Does that help? Yes or no? Does that help? Yes or no? No, right? This is This is You only No, right? This is This is You only No, right? This is This is You only create confusion. If this is a heavy create confusion. If this is a heavy create confusion. If this is a heavy personalized experience for the user, personalized experience for the user, personalized experience for the user, imagine if every single time you need to imagine if every single time you need to imagine if every single time you need to prompt, and then uh at least the model prompt, and then uh at least the model prompt, and then uh at least the model will output something different. This is will output something different. This is will output something different. This is not good, and this is just the second not good, and this is just the second not good, and this is just the second turn. Let's have a look at the third turn. Let's have a look at the third turn. Let's have a look at the third one. one. one. Oh my god, now there is even more stuff Oh my god, now there is even more stuff Oh my god, now there is even more stuff here on the right side. But so, this is here on the right side. But so, this is here on the right side. But so, this is I have this component, the KPI cards, a I have this component, the KPI cards, a I have this component, the KPI cards, a bunch of text, a bunch of charts, and bunch of text, a bunch of charts, and bunch of text, a bunch of charts, and then this this was the beginning of a then this this was the beginning of a then this this was the beginning of a journey. journey. journey. One more?

One more? One more? Yes. Okay, now it's still Q1, Yes. Okay, now it's still Q1, Yes. Okay, now it's still Q1, but still for me this still it's but still for me this still it's but still for me this still it's confusing. And because this has been a confusing. And because this has been a confusing. And because this has been a very experimental journey, very experimental journey, very experimental journey, at least okay, let's let's move on here. at least okay, let's let's move on here. at least okay, let's let's move on here. My feedback to the team and to myself My feedback to the team and to myself My feedback to the team and to myself was no, no, and no. There's no way that was no, no, and no. There's no way that was no, no, and no. There's no way that I would ship this to to prod whatsoever, I would ship this to to prod whatsoever, I would ship this to to prod whatsoever, right? And then I have my colleagues right? And then I have my colleagues right? And then I have my colleagues here just to confirm what I just said. here just to confirm what I just said. here just to confirm what I just said. But then things evolved, and I'll like But then things evolved, and I'll like But then things evolved, and I'll like to demo the current state of the to demo the current state of the to demo the current state of the product. It's much more sophisticated, product. It's much more sophisticated, product. It's much more sophisticated, and let's have a look. I and let's have a look. I and let's have a look. I I'll like to plan a campaign, and for I'll like to plan a campaign, and for I'll like to plan a campaign, and for what it's worth, I'm going to save you what it's worth, I'm going to save you what it's worth, I'm going to save you from all the nitty-gritty details for from all the nitty-gritty details for from all the nitty-gritty details for everything that is domain specific, but everything that is domain specific, but everything that is domain specific, but I'm going to pick this query here, I'm going to pick this query here, I'm going to pick this query here, and then let's see what happens. And and then let's see what happens. And and then let's see what happens. And this is the the agentic orchestration this is the the agentic orchestration this is the the agentic orchestration part that I was going to highlight. part that I was going to highlight. part that I was going to highlight. Underneath the hood we have the Underneath the hood we have the Underneath the hood we have the orchestrator, and the orchestrator can orchestrator, and the orchestrator can orchestrator, and the orchestrator can then just extract the intent of the then just extract the intent of the then just extract the intent of the query. Based off of the intent of the query. Based off of the intent of the query. Based off of the intent of the query, it can locate the tools, right?

query, it can locate the tools, right? query, it can locate the tools, right? Those can be first party, third party Those can be first party, third party Those can be first party, third party tools, and the outputs of this different tools, and the outputs of this different tools, and the outputs of this different it could be agents on MCP servers, it could be agents on MCP servers, it could be agents on MCP servers, combined will give combined will give combined will give enough what I call ammunition and enough what I call ammunition and enough what I call ammunition and context for the UX agent to eventually context for the UX agent to eventually context for the UX agent to eventually render something that we call render something that we call render something that we call meaningful. So, compared to the previous meaningful. So, compared to the previous meaningful. So, compared to the previous turns, turns, turns, this is this is decent, right? I want to this is this is decent, right? I want to this is this is decent, right? I want to just remove my bias, but the overall just remove my bias, but the overall just remove my bias, but the overall aesthetics, the look and feel of this aesthetics, the look and feel of this aesthetics, the look and feel of this query, query, query, uh it it resonates with me. Would you uh it it resonates with me. Would you uh it it resonates with me. Would you agree? Give me a thumbs up if you like agree? Give me a thumbs up if you like agree? Give me a thumbs up if you like if you agree. Okay, at least um the vast if you agree. Okay, at least um the vast if you agree. Okay, at least um the vast majority here. And then see, it is majority here. And then see, it is majority here. And then see, it is decent. And let me just continue here. decent. And let me just continue here. decent. And let me just continue here. And then once again, right? This was And then once again, right? This was And then once again, right? This was decided by AI guided by us. I would just decided by AI guided by us. I would just decided by AI guided by us. I would just want to make make that clear want to make make that clear want to make make that clear here. And then I'm going to touch on the here. And then I'm going to touch on the here. And then I'm going to touch on the on the UI protocols and then how you can on the UI protocols and then how you can on the UI protocols and then how you can make this happen. make this happen. make this happen. But okay, let's see. If I approve here, But okay, let's see. If I approve here, But okay, let's see. If I approve here, and then this is already live. And this and then this is already live. And this and then this is already live. And this is pre-prod, right? So, uh this is is pre-prod, right? So, uh this is is pre-prod, right? So, uh this is great. Let me go back to my great. Let me go back to my great. Let me go back to my presentation.

presentation. presentation. Perfect. I have one more question. Are Perfect. I have one more question. Are Perfect. I have one more question. Are you guys as skeptical you guys as skeptical you guys as skeptical that this is possible? Because I can that this is possible? Because I can that this is possible? Because I can tell you this it is possible. If you're tell you this it is possible. If you're tell you this it is possible. If you're still skeptical, don't worry. I have all still skeptical, don't worry. I have all still skeptical, don't worry. I have all of these guys uh here uh also uh every of these guys uh here uh also uh every of these guys uh here uh also uh every day just looking at me and then day just looking at me and then day just looking at me and then challenging whether uh this uh can be challenging whether uh this uh can be challenging whether uh this uh can be made possible at scale, right? And then made possible at scale, right? And then made possible at scale, right? And then uh okay. And then this is the part that uh okay. And then this is the part that uh okay. And then this is the part that I'll like just to touch uh touch base on I'll like just to touch uh touch base on I'll like just to touch uh touch base on the the the uh on the three ways that you can render uh on the three ways that you can render uh on the three ways that you can render what you just saw. And they're different what you just saw. And they're different what you just saw. And they're different UI protocols. Are you guys familiar with UI protocols. Are you guys familiar with UI protocols. Are you guys familiar with uh generative uh UI? Have you guys uh generative uh UI? Have you guys uh generative uh UI? Have you guys played with it? Let me see here. played with it? Let me see here. played with it? Let me see here. Okay, well, that's really cool. Uh okay. Okay, well, that's really cool. Uh okay. Okay, well, that's really cool. Uh okay. So, uh what I would like just to share So, uh what I would like just to share So, uh what I would like just to share with you guys it's all about how much with you guys it's all about how much with you guys it's all about how much control you want to exercise over the control you want to exercise over the control you want to exercise over the experience. And this matters a lot experience. And this matters a lot experience. And this matters a lot because uh you know, as a because uh you know, as a because uh you know, as a non-deterministic solution, uh you can non-deterministic solution, uh you can non-deterministic solution, uh you can decide if you want something I really um decide if you want something I really um decide if you want something I really um like this. So, let's have a look. like this. So, let's have a look. like this. So, let's have a look. Here, this is uh ChatGPT. Here, this is uh ChatGPT. Here, this is uh ChatGPT. And my query was help me find a Japanese And my query was help me find a Japanese And my query was help me find a Japanese restaurant uh in SF today.

restaurant uh in SF today. restaurant uh in SF today. If you guys see here, this component If you guys see here, this component If you guys see here, this component this component is very opinionated. this component is very opinionated. this component is very opinionated. Would you agree with that? Here? Would you agree with that? Here? Would you agree with that? Here? Right? So, uh you can have complete Right? So, uh you can have complete Right? So, uh you can have complete control over this uh component. So, control over this uh component. So, control over this uh component. So, depending upon the nature of your depending upon the nature of your depending upon the nature of your business, this works really well, right? business, this works really well, right? business, this works really well, right? And for that, let me just go back here. And for that, let me just go back here. And for that, let me just go back here. And then this is what I call a control. And then this is what I call a control. And then this is what I call a control. Essentially, you ship the component as Essentially, you ship the component as Essentially, you ship the component as it is. The agent will pick and it will it is. The agent will pick and it will it is. The agent will pick and it will display exactly the way that you display exactly the way that you display exactly the way that you describe. However, depending upon the describe. However, depending upon the describe. However, depending upon the nature of your business, at least for nature of your business, at least for nature of your business, at least for us, right, at my company, where a B2B us, right, at my company, where a B2B us, right, at my company, where a B2B SaaS, the there's so much configuration SaaS, the there's so much configuration SaaS, the there's so much configuration that we don't want to be over uh that we don't want to be over uh that we don't want to be over uh prescriptive because uh the feedback prescriptive because uh the feedback prescriptive because uh the feedback that I keep getting from my customers, that I keep getting from my customers, that I keep getting from my customers, "Oh, the flows are so confusing. There's "Oh, the flows are so confusing. There's "Oh, the flows are so confusing. There's so much configuration. How can you so much configuration. How can you so much configuration. How can you remove the cognitive load uh for me?" remove the cognitive load uh for me?" remove the cognitive load uh for me?" But uh if you're like Booking, for But uh if you're like Booking, for But uh if you're like Booking, for example, this approach uh works uh example, this approach uh works uh example, this approach uh works uh really well. And then let's see uh how really well. And then let's see uh how really well. And then let's see uh how it works. So, it works. So, it works. So, essentially, you have the agent. The essentially, you have the agent. The essentially, you have the agent. The agent will just pick the component from agent will just pick the component from agent will just pick the component from your catalog and then it will render uh your catalog and then it will render uh your catalog and then it will render uh as it is, right? And uh in the interest as it is, right? And uh in the interest as it is, right? And uh in the interest uh of time, I'm not going to touch base uh of time, I'm not going to touch base uh of time, I'm not going to touch base on the code snippets uh that I have for on the code snippets uh that I have for on the code snippets uh that I have for this uh three different types. But this uh three different types. But this uh three different types. But afterwards, if you guys are interested, afterwards, if you guys are interested, afterwards, if you guys are interested, uh I can share the presentation and then uh I can share the presentation and then uh I can share the presentation and then you can have a look, all right?

you can have a look, all right? you can have a look, all right? Very good. This is uh Very good. This is uh Very good. This is uh at least on the left side. Let's discuss at least on the left side. Let's discuss at least on the left side. Let's discuss a little bit on the right side because a little bit on the right side because a little bit on the right side because this is when you give full autonomy to this is when you give full autonomy to this is when you give full autonomy to the LLM. If you guys remember uh at the LLM. If you guys remember uh at the LLM. If you guys remember uh at least the previous attempts for my least the previous attempts for my least the previous attempts for my product, this exactly what we did. So, product, this exactly what we did. So, product, this exactly what we did. So, we just said, "So, hey LLM, how uh how we just said, "So, hey LLM, how uh how we just said, "So, hey LLM, how uh how would you compose this experience would you compose this experience would you compose this experience knowing that you have these components?" knowing that you have these components?" knowing that you have these components?" Right? But uh that was uh that was a Right? But uh that was uh that was a Right? But uh that was uh that was a little bit uh a little bit of our little bit uh a little bit of our little bit uh a little bit of our opinion because uh it has uh it had uh opinion because uh it has uh it had uh opinion because uh it has uh it had uh some of the components available. But it some of the components available. But it some of the components available. But it could happen that uh you can just could happen that uh you can just could happen that uh you can just delegate fully to the LLM. And then delegate fully to the LLM. And then delegate fully to the LLM. And then right now, I'm here on Claude and I right now, I'm here on Claude and I right now, I'm here on Claude and I asked Claude, "So, hey, create an org asked Claude, "So, hey, create an org asked Claude, "So, hey, create an org chart with three levels." chart with three levels." chart with three levels." That was it. And then Claude just That was it. And then Claude just That was it. And then Claude just rendered this diagram and it works rendered this diagram and it works rendered this diagram and it works really well. Right? But here, if I put really well. Right? But here, if I put really well. Right? But here, if I put myself in the shoes of a company, I'm myself in the shoes of a company, I'm myself in the shoes of a company, I'm not sure I would delegate fully to the not sure I would delegate fully to the not sure I would delegate fully to the LLM. LLM. LLM. Because I cannot control at least the Because I cannot control at least the Because I cannot control at least the app the output and the outcome. And me app the output and the outcome. And me app the output and the outcome. And me personally, me guys, as a UX leader, the personally, me guys, as a UX leader, the personally, me guys, as a UX leader, the UX side of me will always say no. You UX side of me will always say no. You UX side of me will always say no. You got to be in control. There's been a got to be in control. There's been a got to be in control. There's been a couple of talks here at least this week couple of talks here at least this week couple of talks here at least this week on design, on taste, and judgment. And on design, on taste, and judgment. And on design, on taste, and judgment. And this matters a lot. If you guys want to this matters a lot. If you guys want to this matters a lot. If you guys want to embark on this journey of leveraging embark on this journey of leveraging embark on this journey of leveraging these protocols, you don't want to these protocols, you don't want to these protocols, you don't want to delegate too much delegate too much delegate too much of the actual experience to the LLM. You of the actual experience to the LLM. You of the actual experience to the LLM. You got to find alternatives and I'm going got to find alternatives and I'm going got to find alternatives and I'm going to touch on that in just a little bit.

to touch on that in just a little bit. to touch on that in just a little bit. Okay, cool. And then this is how the Okay, cool. And then this is how the Okay, cool. And then this is how the open-ended open-ended open-ended approach works. So essentially there approach works. So essentially there approach works. So essentially there there's going to be an MCP tool and then there's going to be an MCP tool and then there's going to be an MCP tool and then this will literally ship the HTML and this will literally ship the HTML and this will literally ship the HTML and then in a sandbox I frame environment, then in a sandbox I frame environment, then in a sandbox I frame environment, this will be rendered in the host of this will be rendered in the host of this will be rendered in the host of your choice. But it can be a chat. It your choice. But it can be a chat. It your choice. But it can be a chat. It can be this can be this can be this cloud. It could be a chat GPT. It could cloud. It could be a chat GPT. It could cloud. It could be a chat GPT. It could be perplexity or it could be any other be perplexity or it could be any other be perplexity or it could be any other chat. If you're willing just to to give chat. If you're willing just to to give chat. If you're willing just to to give full control to the LLM, good luck. But full control to the LLM, good luck. But full control to the LLM, good luck. But the one that I would like to highlight the one that I would like to highlight the one that I would like to highlight is is is this here and this was our choice that this here and this was our choice that this here and this was our choice that we call the declarative. Declarative is we call the declarative. Declarative is we call the declarative. Declarative is in the middle. in the middle. in the middle. If you guys heard some of the protocols If you guys heard some of the protocols If you guys heard some of the protocols and I don't want to get into the and I don't want to get into the and I don't want to get into the specifics of each because they have specifics of each because they have specifics of each because they have different characteristics. But HTMX from different characteristics. But HTMX from different characteristics. But HTMX from Google, JSON render from Vercel, OpenUI Google, JSON render from Vercel, OpenUI Google, JSON render from Vercel, OpenUI by Thesis, by Thesis, by Thesis, those are some of the protocols that those are some of the protocols that those are some of the protocols that will give you this in between here. And will give you this in between here. And will give you this in between here. And let let's have a look, right? So at let let's have a look, right? So at let let's have a look, right? So at least for my product, the one that I least for my product, the one that I least for my product, the one that I just showed, the orchestrator agent will just showed, the orchestrator agent will just showed, the orchestrator agent will eventually, if you think of the whole eventually, if you think of the whole eventually, if you think of the whole traversal, the user will enter the query traversal, the user will enter the query traversal, the user will enter the query and then there's going to be the intent and then there's going to be the intent and then there's going to be the intent classification. Based off of the intent classification. Based off of the intent classification. Based off of the intent classification, then the tools will be classification, then the tools will be classification, then the tools will be invoked, the data will be retrieved, and invoked, the data will be retrieved, and invoked, the data will be retrieved, and somewhat somewhat in between Alice, somewhat somewhat in between Alice, somewhat somewhat in between Alice, there will be the mapping of the there will be the mapping of the there will be the mapping of the eligible components from your catalog to

eligible components from your catalog to eligible components from your catalog to the entities of of the tools, right? And the entities of of the tools, right? And the entities of of the tools, right? And then the orchestrator will just then the orchestrator will just then the orchestrator will just broadcast this UI description. It's like broadcast this UI description. It's like broadcast this UI description. It's like a UI spec. This UI spec will be also we a UI spec. This UI spec will be also we a UI spec. This UI spec will be also we have this component catalog here and we have this component catalog here and we have this component catalog here and we use the Zod schema and then you got to use the Zod schema and then you got to use the Zod schema and then you got to be compliant with this protocols. This be compliant with this protocols. This be compliant with this protocols. This is just one of the requirements and then is just one of the requirements and then is just one of the requirements and then you just render that. Right? And then you just render that. Right? And then you just render that. Right? And then their final output will be the native their final output will be the native their final output will be the native UI, in this case the React components. UI, in this case the React components. UI, in this case the React components. This is it. The good thing about the This is it. The good thing about the This is it. The good thing about the declarative approach declarative approach declarative approach is is is it will be compliant with your design it will be compliant with your design it will be compliant with your design system everywhere. This matters a lot. system everywhere. This matters a lot. system everywhere. This matters a lot. So, in our case, we did not want to So, in our case, we did not want to So, in our case, we did not want to delegate to the LLM because you guys saw delegate to the LLM because you guys saw delegate to the LLM because you guys saw over there you can't change the copy, over there you can't change the copy, over there you can't change the copy, right? So, it's not key one. Sometimes right? So, it's not key one. Sometimes right? So, it's not key one. Sometimes it's going to be March January to March. it's going to be March January to March. it's going to be March January to March. It matters a lot. So, within UX we have It matters a lot. So, within UX we have It matters a lot. So, within UX we have different factors, right? We have the different factors, right? We have the different factors, right? We have the actual UX's if you think of the overall actual UX's if you think of the overall actual UX's if you think of the overall experience. There is UI, there's also experience. There is UI, there's also experience. There is UI, there's also copy, UX writing, and so on and so copy, UX writing, and so on and so copy, UX writing, and so on and so forth. But this approach gives us this forth. But this approach gives us this forth. But this approach gives us this in between. It is less deterministic and in between. It is less deterministic and in between. It is less deterministic and I think this is a really good segue to I think this is a really good segue to I think this is a really good segue to some of the challenges because imagine, some of the challenges because imagine, some of the challenges because imagine, I'm going to use my example once again, I'm going to use my example once again, I'm going to use my example once again, the orchestrator will just fetch the the orchestrator will just fetch the the orchestrator will just fetch the eligible components for that query, but eligible components for that query, but eligible components for that query, but then it's up to the LLM how to place in then it's up to the LLM how to place in then it's up to the LLM how to place in the UI. And this can get really really the UI. And this can get really really the UI. And this can get really really messy, but messy, but messy, but and those are the challenges that I'll and those are the challenges that I'll and those are the challenges that I'll like to share with you guys here.

like to share with you guys here. like to share with you guys here. I got to be careful because I've got I got to be careful because I've got I got to be careful because I've got only 30 minutes, but let's go. only 30 minutes, but let's go. only 30 minutes, but let's go. So, the first challenge is if the agent So, the first challenge is if the agent So, the first challenge is if the agent picks the components, who's in charge or picks the components, who's in charge or picks the components, who's in charge or what entity is in charge of arranging what entity is in charge of arranging what entity is in charge of arranging them? And this is information them? And this is information them? And this is information architecture, right? This is a critical architecture, right? This is a critical architecture, right? This is a critical aspect of UX. And then once again, if aspect of UX. And then once again, if aspect of UX. And then once again, if you put yourself in the shoes of the you put yourself in the shoes of the you put yourself in the shoes of the average customer, it matters a lot. So, average customer, it matters a lot. So, average customer, it matters a lot. So, if you're just left alone, the placement if you're just left alone, the placement if you're just left alone, the placement can be totally random. So, at least in can be totally random. So, at least in can be totally random. So, at least in my team, we borrow this concept of my team, we borrow this concept of my team, we borrow this concept of atomic design. And atomic design, it atomic design. And atomic design, it atomic design. And atomic design, it goes like this. Let me just change here. goes like this. Let me just change here. goes like this. Let me just change here. Uh yeah. So, as you can see, atomic Uh yeah. So, as you can see, atomic Uh yeah. So, as you can see, atomic design, and then I have the definition, design, and then I have the definition, design, and then I have the definition, is a methodology composed of five is a methodology composed of five is a methodology composed of five distinct stages working together to distinct stages working together to distinct stages working together to create interface design systems in a create interface design systems in a create interface design systems in a more deliberate and hierarchical manner. more deliberate and hierarchical manner. more deliberate and hierarchical manner. This helps a lot. Why? Because those are This helps a lot. Why? Because those are This helps a lot. Why? Because those are the individual elements, and if you the individual elements, and if you the individual elements, and if you think of the overall structure of the think of the overall structure of the think of the overall structure of the page, it gives me the ability to steer page, it gives me the ability to steer page, it gives me the ability to steer it as I see fit. And what we've done in it as I see fit. And what we've done in it as I see fit. And what we've done in my team, so this UX agent, we harnessed my team, so this UX agent, we harnessed my team, so this UX agent, we harnessed this UX agent. So, we eventually taught this UX agent. So, we eventually taught this UX agent. So, we eventually taught this this UX agent what good looks like, this this UX agent what good looks like, this this UX agent what good looks like, what is the optimal layout for a given what is the optimal layout for a given what is the optimal layout for a given situation, and we have a catalog of situation, and we have a catalog of situation, and we have a catalog of different templates. But I'll like to different templates. But I'll like to different templates. But I'll like to show you at least this because this show you at least this because this show you at least this because this detail is very important.

detail is very important. detail is very important. Here, Here, Here, it is the overall hierarchy hierarchy. it is the overall hierarchy hierarchy. it is the overall hierarchy hierarchy. So, if you remember part of the So, if you remember part of the So, if you remember part of the orchestrator, the orchestrator will orchestrator, the orchestrator will orchestrator, the orchestrator will eventually just fetch the eligible eventually just fetch the eligible eventually just fetch the eligible components to accomplish the query of components to accomplish the query of components to accomplish the query of the user, but then the next big question the user, but then the next big question the user, but then the next big question is, how do we arrange that? The approach is, how do we arrange that? The approach is, how do we arrange that? The approach that we used was think of this that we used was think of this that we used was think of this hierarchy. So, you have the overall hierarchy. So, you have the overall hierarchy. So, you have the overall page, the layout. The layout will page, the layout. The layout will page, the layout. The layout will contain different slots. So, think of contain different slots. So, think of contain different slots. So, think of this one here, the header. You can have this one here, the header. You can have this one here, the header. You can have the main, and then you can have sub sub the main, and then you can have sub sub the main, and then you can have sub sub slots. Sub slots can have sub slots. slots. Sub slots can have sub slots. slots. Sub slots can have sub slots. And within the sub slots, you can have And within the sub slots, you can have And within the sub slots, you can have eligible component categories. And then eligible component categories. And then eligible component categories. And then this will allow us to steer eventually this will allow us to steer eventually this will allow us to steer eventually the the the optimal placement of the the the the optimal placement of the the the the optimal placement of the components that have been retrieved by components that have been retrieved by components that have been retrieved by the orchestrator. So this is literally the orchestrator. So this is literally the orchestrator. So this is literally us codifying us codifying us codifying our UX knowledge into this agent. So the our UX knowledge into this agent. So the our UX knowledge into this agent. So the next time, you know, it doesn't matter. next time, you know, it doesn't matter. next time, you know, it doesn't matter. We'll just follow the same approach. And We'll just follow the same approach. And We'll just follow the same approach. And then this is the hierarchy that we're then this is the hierarchy that we're then this is the hierarchy that we're using. So layout to slot to sub slot to using. So layout to slot to sub slot to using. So layout to slot to sub slot to components. However, because of the components. However, because of the components. However, because of the orchestrator, we we flipped the order.

orchestrator, we we flipped the order. orchestrator, we we flipped the order. So from components, components will map So from components, components will map So from components, components will map to sub slot, sub slots to slots, and to sub slot, sub slots to slots, and to sub slot, sub slots to slots, and slots to templates. And then we can just slots to templates. And then we can just slots to templates. And then we can just arrange as needed. arrange as needed. arrange as needed. So this was a hell of a challenge. It is So this was a hell of a challenge. It is So this was a hell of a challenge. It is still a challenge, by the way, and it's still a challenge, by the way, and it's still a challenge, by the way, and it's a really good segue to the second one, a really good segue to the second one, a really good segue to the second one, the second challenge, which is the second challenge, which is the second challenge, which is the actual design, your design system the actual design, your design system the actual design, your design system and your catalog. This becomes the and your catalog. This becomes the and your catalog. This becomes the heartbeat of the whole thing, right? I heartbeat of the whole thing, right? I heartbeat of the whole thing, right? I cannot stress enough, if you guys see cannot stress enough, if you guys see cannot stress enough, if you guys see the potential of leveraging this UI the potential of leveraging this UI the potential of leveraging this UI protocols for your product, this is protocols for your product, this is protocols for your product, this is going to be a big deal, right? So we're going to be a big deal, right? So we're going to be a big deal, right? So we're pushing the boundaries and then we're pushing the boundaries and then we're pushing the boundaries and then we're testing and we keep on testing these testing and we keep on testing these testing and we keep on testing these different protocols that I mentioned, different protocols that I mentioned, different protocols that I mentioned, ATUI, JSON Render, ATUI, JSON Render, ATUI, JSON Render, Open UI, and so on and so forth. But Open UI, and so on and so forth. But Open UI, and so on and so forth. But this has been a quite challenging this has been a quite challenging this has been a quite challenging because the the catalog is the contract because the the catalog is the contract because the the catalog is the contract between the agent and the UI. between the agent and the UI. between the agent and the UI. So every property mat- matters. And not So every property mat- matters. And not So every property mat- matters. And not only for the catalog, but for the layout only for the catalog, but for the layout only for the catalog, but for the layout as well. So if you remember, the layout as well. So if you remember, the layout as well. So if you remember, the layout has its own components, the slots and has its own components, the slots and has its own components, the slots and the sub slots. Each of those components the sub slots. Each of those components the sub slots. Each of those components will have its own attributes. And all of will have its own attributes. And all of will have its own attributes. And all of that, this curation, let me just that, this curation, let me just that, this curation, let me just encapsulate into curation. This curation encapsulate into curation. This curation encapsulate into curation. This curation is absolutely needed so that you can is absolutely needed so that you can is absolutely needed so that you can deliver something meaningful. Not as deliver something meaningful. Not as deliver something meaningful. Not as some demo that you will see out there some demo that you will see out there some demo that you will see out there for the sake of demo, right? So, this for the sake of demo, right? So, this for the sake of demo, right? So, this this this will give you a control will this this will give you a control will this this will give you a control will allow you to steer from UX perspective.

allow you to steer from UX perspective. allow you to steer from UX perspective. Very good. And last but not least, Very good. And last but not least, Very good. And last but not least, one significant challenge that we had is one significant challenge that we had is one significant challenge that we had is that my teams do not design the pixel that my teams do not design the pixel that my teams do not design the pixel anymore. I don't know if you could see anymore. I don't know if you could see anymore. I don't know if you could see that, right? So, we're not here that, right? So, we're not here that, right? So, we're not here designing at the entire flow. Now, AI designing at the entire flow. Now, AI designing at the entire flow. Now, AI can can can dictate that to a significant extent, dictate that to a significant extent, dictate that to a significant extent, but the nature of the work shifted quite but the nature of the work shifted quite but the nature of the work shifted quite a bit and it's been an interesting a bit and it's been an interesting a bit and it's been an interesting journey to say the least, a really good journey to say the least, a really good journey to say the least, a really good one, but even for the non-technical PMs one, but even for the non-technical PMs one, but even for the non-technical PMs and UX designers, it it was a big hit and UX designers, it it was a big hit and UX designers, it it was a big hit because right now we talk about the because right now we talk about the because right now we talk about the schema. Let's talk about this curation schema. Let's talk about this curation schema. Let's talk about this curation of the catalog. Let's talk about the of the catalog. Let's talk about the of the catalog. Let's talk about the rules. Let's talk about the synthetic rules. Let's talk about the synthetic rules. Let's talk about the synthetic data that we can generate. How can we data that we can generate. How can we data that we can generate. How can we generate the queries that will map to a generate the queries that will map to a generate the queries that will map to a given component as part of this mapping given component as part of this mapping given component as part of this mapping logic? Let's talk about interaction logic? Let's talk about interaction logic? Let's talk about interaction patterns. So, this patterns. So, this patterns. So, this if once again, if you guys are going to if once again, if you guys are going to if once again, if you guys are going to embark on this journey, be aware that embark on this journey, be aware that embark on this journey, be aware that the people element is very important. the people element is very important. the people element is very important. When I talk to other leaders, I I talk When I talk to other leaders, I I talk When I talk to other leaders, I I talk about the three P's: people, product, about the three P's: people, product, about the three P's: people, product, and process, right? And then a very and process, right? And then a very and process, right? And then a very lightweight process, but this matters a lightweight process, but this matters a lightweight process, but this matters a lot.

lot. lot. And with that, And with that, And with that, I'm going to leave a couple of resources I'm going to leave a couple of resources I'm going to leave a couple of resources here. So, and by the way, those are here. So, and by the way, those are here. So, and by the way, those are talks from AIE talks from AIE talks from AIE for what it's worth. So, yeah, people for what it's worth. So, yeah, people for what it's worth. So, yeah, people that have been talking about this that have been talking about this that have been talking about this protocols over and over and over. So, protocols over and over and over. So, protocols over and over and over. So, please just take advantage, take a please just take advantage, take a please just take advantage, take a screenshot. screenshot. screenshot. And if you guys want to connect with me And if you guys want to connect with me And if you guys want to connect with me here, here, here, yeah, my LinkedIn or just take a yeah, my LinkedIn or just take a yeah, my LinkedIn or just take a screenshot. I'll love to talk more about screenshot. I'll love to talk more about screenshot. I'll love to talk more about the topic. I can tell this is just a the topic. I can tell this is just a the topic. I can tell this is just a matter of time, right? So, this is matter of time, right? So, this is matter of time, right? So, this is coming. So, thank you so much.