Video Has No Memory. Here's How We Built One. — James Le, TwelveLabs

Thanks so much for having me and uh Thanks so much for having me and uh inviting me to to be a speaker at uh the inviting me to to be a speaker at uh the inviting me to to be a speaker at uh the warfare. You know I attended last year warfare. You know I attended last year warfare. You know I attended last year and was so impressed about the quality and was so impressed about the quality and was so impressed about the quality of presenters. So so glad to to have a of presenters. So so glad to to have a of presenters. So so glad to to have a chance to be here and present. Uh so the chance to be here and present. Uh so the chance to be here and present. Uh so the title of my talk is you know video has title of my talk is you know video has title of my talk is you know video has no memory, right? uh and you know uh no memory, right? uh and you know uh no memory, right? uh and you know uh this might sound strange because video this might sound strange because video this might sound strange because video is already like a preservation of the is already like a preservation of the is already like a preservation of the past right you think about like you have past right you think about like you have past right you think about like you have footage you preserve uh recording footage you preserve uh recording footage you preserve uh recording training data incident creative work uh training data incident creative work uh training data incident creative work uh history etc. But actually most of the history etc. But actually most of the history etc. But actually most of the video AI system these day do not have video AI system these day do not have video AI system these day do not have memory in the system sense. So actually memory in the system sense. So actually memory in the system sense. So actually for this talk I will try to answer the for this talk I will try to answer the for this talk I will try to answer the question like what could it take to question like what could it take to question like what could it take to build a memory layer for video build a memory layer for video build a memory layer for video intelligence. To start I want to be intelligence. To start I want to be intelligence. To start I want to be clear about what makes V different from clear about what makes V different from clear about what makes V different from other data type right. So this is the other data type right. So this is the other data type right. So this is the first mental model that I want to first mental model that I want to first mental model that I want to highlight which is that video is not a highlight which is that video is not a highlight which is that video is not a bag of frames. Um so in you know many of bag of frames. Um so in you know many of bag of frames. Um so in you know many of my conversation with like developers uh my conversation with like developers uh my conversation with like developers uh you know who using a product a lot of you know who using a product a lot of you know who using a product a lot of them still treat video as like a stack them still treat video as like a stack them still treat video as like a stack of images uh maybe a transcript being of images uh maybe a transcript being of images uh maybe a transcript being attached or you know um like you know attached or you know um like you know attached or you know um like you know but essentially like like a frame level but essentially like like a frame level but essentially like like a frame level right and that is useful approximation right and that is useful approximation right and that is useful approximation for some task but it throw away the for some task but it throw away the for some task but it throw away the thing that makes video very unique which thing that makes video very unique which thing that makes video very unique which is continuity right so meaning in video is continuity right so meaning in video is continuity right so meaning in video derives from space time modalities the derives from space time modalities the derives from space time modalities the sequence so a better mental sequence so a better mental sequence so a better mental for video is a spatial temporal volume.

for video is a spatial temporal volume. for video is a spatial temporal volume. So what do I mean that inside that So what do I mean that inside that So what do I mean that inside that volume you have visual information volume you have visual information volume you have visual information speech sound motion OCR camera changes speech sound motion OCR camera changes speech sound motion OCR camera changes scene transition metadata and time right scene transition metadata and time right scene transition metadata and time right so the hard part here is really about so the hard part here is really about so the hard part here is really about how can you preserving relationship how can you preserving relationship how can you preserving relationship across this volume so that later an across this volume so that later an across this volume so that later an application can traverse it and then you application can traverse it and then you application can traverse it and then you know especially at the enterprise scale know especially at the enterprise scale know especially at the enterprise scale you know across like industry like you know across like industry like you know across like industry like entertainment sport you know um short entertainment sport you know um short entertainment sport you know um short form content then you sitting on pabytes form content then you sitting on pabytes form content then you sitting on pabytes of footage right so fighting moment is of footage right so fighting moment is of footage right so fighting moment is already hard. So how can you preserve already hard. So how can you preserve already hard. So how can you preserve meaning across millions of moments in meaning across millions of moments in meaning across millions of moments in the deeper platform? So I work at Troll the deeper platform? So I work at Troll the deeper platform? So I work at Troll Lab which is a series B uh startup. Um Lab which is a series B uh startup. Um Lab which is a series B uh startup. Um we foundation models that understand you we foundation models that understand you we foundation models that understand you know video the way that human do. Um and know video the way that human do. Um and know video the way that human do. Um and the way we talk about our positioning is the way we talk about our positioning is the way we talk about our positioning is like the existing uh stack of of dealing like the existing uh stack of of dealing like the existing uh stack of of dealing with video is not equipped to do that. with video is not equipped to do that. with video is not equipped to do that. Right? Obviously language model are very Right? Obviously language model are very Right? Obviously language model are very powerful. They are good reasoning powerful. They are good reasoning powerful. They are good reasoning interfaces. They are increasingly interfaces. They are increasingly interfaces. They are increasingly multimodal as well. But the supporting multimodal as well. But the supporting multimodal as well. But the supporting stack right around that is very um I stack right around that is very um I stack right around that is very um I would say limited and that create three would say limited and that create three would say limited and that create three problem. Number one is wrong context problem. Number one is wrong context problem. Number one is wrong context right. So video is not naturally a right. So video is not naturally a right. So video is not naturally a sequence of text token. If we force it sequence of text token. If we force it sequence of text token. If we force it into that sequence by sampling frames by into that sequence by sampling frames by into that sequence by sampling frames by extracting a transcript uh by dumping extracting a transcript uh by dumping extracting a transcript uh by dumping everything into a prom you lose the everything into a prom you lose the everything into a prom you lose the spatial temporal relationships right spatial temporal relationships right spatial temporal relationships right that actually defy the event. Second uh that actually defy the event. Second uh that actually defy the event. Second uh wrong memory. So if you think about text wrong memory. So if you think about text wrong memory. So if you think about text system memory here is often mean system memory here is often mean system memory here is often mean reachable management generation vector reachable management generation vector reachable management generation vector search or probably like larger context

search or probably like larger context search or probably like larger context window uh those are very useful but window uh those are very useful but window uh those are very useful but video memory has a different requirement video memory has a different requirement video memory has a different requirement it needs to link today's scene for it needs to link today's scene for it needs to link today's scene for something that happened in another file something that happened in another file something that happened in another file another episode another camera angle another episode another camera angle another episode another camera angle another season another year so it another season another year so it another season another year so it actually need durable continuity right actually need durable continuity right actually need durable continuity right and the last part here is strong and the last part here is strong and the last part here is strong reasoning like I said you know text reasoning like I said you know text reasoning like I said you know text first system cannot reason over you know first system cannot reason over you know first system cannot reason over you know um natively over motion causality all um natively over motion causality all um natively over motion causality all that so uh you know they do not that so uh you know they do not that so uh you know they do not automatically build like a persistent automatically build like a persistent automatically build like a persistent structure uh on you know who appear what structure uh on you know who appear what structure uh on you know who appear what happened what changes etc and so my happened what changes etc and so my happened what changes etc and so my argument is that video intelligence need argument is that video intelligence need argument is that video intelligence need a memory layer that decide what to a memory layer that decide what to a memory layer that decide what to preverse how to connect it and how to preverse how to connect it and how to preverse how to connect it and how to reip so um I want to kind of gr it into reip so um I want to kind of gr it into reip so um I want to kind of gr it into the properties of video right to to make the properties of video right to to make the properties of video right to to make it even clearer there's five challenges it even clearer there's five challenges it even clearer there's five challenges dealing with video number one is dealing with video number one is dealing with video number one is temporal right so meaning depends on temporal right so meaning depends on temporal right so meaning depends on before and after so frame by itself can before and after so frame by itself can before and after so frame by itself can can be misleading right the same can be misleading right the same can be misleading right the same expression product shot physical action expression product shot physical action expression product shot physical action can mean different things depending on can mean different things depending on can mean different things depending on the sequence right second is that video the sequence right second is that video the sequence right second is that video is obviously multimodal I explained is obviously multimodal I explained is obviously multimodal I explained already um you know a transcript alone already um you know a transcript alone already um you know a transcript alone may miss you know the logo a frame alone may miss you know the logo a frame alone may miss you know the logo a frame alone may miss the spoken claim videos video may miss the spoken claim videos video may miss the spoken claim videos video is also very dense right so a few is also very dense right so a few is also very dense right so a few minutes can contain dozen of shots minutes can contain dozen of shots minutes can contain dozen of shots people objects action location claims people objects action location claims people objects action location claims the useful signal is uneven across the the useful signal is uneven across the the useful signal is uneven across the distribution on on the frame some are distribution on on the frame some are distribution on on the frame some are decisive, others are noisy. Fourth is decisive, others are noisy. Fourth is decisive, others are noisy. Fourth is that uh video is also ambiguous, right?

that uh video is also ambiguous, right? that uh video is also ambiguous, right? Um people reappear under different Um people reappear under different Um people reappear under different lighting and angles. Brands are lighting and angles. Brands are lighting and angles. Brands are partially visible. Location are imply partially visible. Location are imply partially visible. Location are imply concepts much over time rather than concepts much over time rather than concepts much over time rather than being named in a single moment. And being named in a single moment. And being named in a single moment. And lastly, it is uh expensive uh because in lastly, it is uh expensive uh because in lastly, it is uh expensive uh because in a lot of uh big enterprise and and a lot of uh big enterprise and and a lot of uh big enterprise and and complex workflow, you need to know point complex workflow, you need to know point complex workflow, you need to know point back to the source moment like where it back to the source moment like where it back to the source moment like where it come from, right? So these are the five come from, right? So these are the five come from, right? So these are the five properties explain why vo memory um is properties explain why vo memory um is properties explain why vo memory um is very complex. It need to preserve very complex. It need to preserve very complex. It need to preserve temporal span multimodal evidence temporal span multimodal evidence temporal span multimodal evidence continuity or uh these are very simple u continuity or uh these are very simple u continuity or uh these are very simple u stack of how we build things at 12 labs. stack of how we build things at 12 labs. stack of how we build things at 12 labs. Um at the bottom we have this semantic Um at the bottom we have this semantic Um at the bottom we have this semantic chunks that capture you know meaningful chunks that capture you know meaningful chunks that capture you know meaningful temporal units. A above that is our u temporal units. A above that is our u temporal units. A above that is our u multimodal embedding encoder called multimodal embedding encoder called multimodal embedding encoder called morango which essentially turn those morango which essentially turn those morango which essentially turn those spend into spatial temporal relations. spend into spatial temporal relations. spend into spatial temporal relations. um basically vector embeddings that um basically vector embeddings that um basically vector embeddings that represent video content and then we have represent video content and then we have represent video content and then we have a spatial spatial temporal context store a spatial spatial temporal context store a spatial spatial temporal context store which is where it preserve pre reusable which is where it preserve pre reusable which is where it preserve pre reusable structure like moment entities metadata structure like moment entities metadata structure like moment entities metadata all that uh we also build our own uh VLM all that uh we also build our own uh VLM all that uh we also build our own uh VLM video context aware language model video context aware language model video context aware language model called Pegasus that essentially serve as called Pegasus that essentially serve as called Pegasus that essentially serve as the the reasoning layer right that can the the reasoning layer right that can the the reasoning layer right that can preser prepare over video content so preser prepare over video content so preser prepare over video content so think about summaries like um metadata think about summaries like um metadata think about summaries like um metadata synthesis comparison and we expose our synthesis comparison and we expose our synthesis comparison and we expose our our models as API because you know we our models as API because you know we our models as API because you know we want to get develop velopers to use them want to get develop velopers to use them want to get develop velopers to use them as infrastructure.

as infrastructure. as infrastructure. Now moving beyond like kind of the stack Now moving beyond like kind of the stack Now moving beyond like kind of the stack right here I want to talk about the right here I want to talk about the right here I want to talk about the difference between search and memory difference between search and memory difference between search and memory right very quickly speaking uh search is right very quickly speaking uh search is right very quickly speaking uh search is obviously super important is how you obviously super important is how you obviously super important is how you recover relevant moments from large recover relevant moments from large recover relevant moments from large video library but then it give you video library but then it give you video library but then it give you candidate it actually not give you like candidate it actually not give you like candidate it actually not give you like any continuity so memory on the other any continuity so memory on the other any continuity so memory on the other hand is is all the um you know uh the hand is is all the um you know uh the hand is is all the um you know uh the things that enable the system to answer things that enable the system to answer things that enable the system to answer a different class of question as you see a different class of question as you see a different class of question as you see here on the right side of my screen. So here on the right side of my screen. So here on the right side of my screen. So these are not the single retrieval code these are not the single retrieval code these are not the single retrieval code right they require the system to right they require the system to right they require the system to preserve entities timeline evidence preserve entities timeline evidence preserve entities timeline evidence across an entire corus um and so like across an entire corus um and so like across an entire corus um and so like you can actually build product moving you can actually build product moving you can actually build product moving beyond from like show me something like beyond from like show me something like beyond from like show me something like this to you know tell me what this this to you know tell me what this this to you know tell me what this collection knows right and so that that collection knows right and so that that collection knows right and so that that might sounds you know simple and subtle might sounds you know simple and subtle might sounds you know simple and subtle but uh the the the output is completely but uh the the the output is completely but uh the the the output is completely different like with search you you get different like with search you you get different like with search you you get like an output like a time moment but like an output like a time moment but like an output like a time moment but with memory you actually return like with memory you actually return like with memory you actually return like structure knowledge timeline uh structure knowledge timeline uh structure knowledge timeline uh explanation explanation explanation composable output and that like you know composable output and that like you know composable output and that like you know is very important because we can now is very important because we can now is very important because we can now move um the the unit output from clip move um the the unit output from clip move um the the unit output from clip retrieval to corus memory right um there retrieval to corus memory right um there retrieval to corus memory right um there are two scaling dimensions shown here on are two scaling dimensions shown here on are two scaling dimensions shown here on the slide the first is time scaling so a the slide the first is time scaling so a the slide the first is time scaling so a real video system should be able to real video system should be able to real video system should be able to reason over years of footage without reason over years of footage without reason over years of footage without reprocessing the whole archive every reprocessing the whole archive every reprocessing the whole archive every time right that means memory first time right that means memory first time right that means memory first retrieval uh be reusable representation retrieval uh be reusable representation retrieval uh be reusable representation once and then support multiop timeline once and then support multiop timeline once and then support multiop timeline episodic recall follow-up question at episodic recall follow-up question at episodic recall follow-up question at lower latency and cost. And then the lower latency and cost. And then the lower latency and cost. And then the second uh dimension is in space, right?

second uh dimension is in space, right? second uh dimension is in space, right? So many real workflow actually um So many real workflow actually um So many real workflow actually um involve multiple perspective like involve multiple perspective like involve multiple perspective like different camera angles uh you know live different camera angles uh you know live different camera angles uh you know live stream creator broadcasting content body stream creator broadcasting content body stream creator broadcasting content body cam stock cameras uh event fit right. So cam stock cameras uh event fit right. So cam stock cameras uh event fit right. So how can you build a system that can fuse how can you build a system that can fuse how can you build a system that can fuse evident across all the sources and then evident across all the sources and then evident across all the sources and then maintain current understanding right and maintain current understanding right and maintain current understanding right and so that is the challenges here. How can so that is the challenges here. How can so that is the challenges here. How can you build a representation that let you build a representation that let you build a representation that let application traverse video across time application traverse video across time application traverse video across time and across sources? Um since this is um and across sources? Um since this is um and across sources? Um since this is um you know a track on on graph right. So you know a track on on graph right. So you know a track on on graph right. So uh the the best mental model that I can uh the the best mental model that I can uh the the best mental model that I can come up with is to represent you know come up with is to represent you know come up with is to represent you know video collection as a context graph. So video collection as a context graph. So video collection as a context graph. So a context graph is a durable queryable a context graph is a durable queryable a context graph is a durable queryable representation that connects video representation that connects video representation that connects video moment entities appearances relationship moment entities appearances relationship moment entities appearances relationship time stamp metadata and compost level time stamp metadata and compost level time stamp metadata and compost level context. Right? So if you take a look context. Right? So if you take a look context. Right? So if you take a look here on on the screen all the way in the here on on the screen all the way in the here on on the screen all the way in the bottom you got time bounded moment. bottom you got time bounded moment. bottom you got time bounded moment. These are like the the scene the shot These are like the the scene the shot These are like the the scene the shot right these are evidence unit. One one right these are evidence unit. One one right these are evidence unit. One one level up are the appearances where and level up are the appearances where and level up are the appearances where and when each entity show up and then you when each entity show up and then you when each entity show up and then you got the actual entity itself. So think got the actual entity itself. So think got the actual entity itself. So think about the people on the video the brand about the people on the video the brand about the people on the video the brand the places the concept. Next you have the places the concept. Next you have the places the concept. Next you have relationship uh core occurrences the relationship uh core occurrences the relationship uh core occurrences the same brand sequences between different same brand sequences between different same brand sequences between different places places places and timeline and finally at the top you and timeline and finally at the top you and timeline and finally at the top you have corpus level context. What are the have corpus level context. What are the have corpus level context. What are the main themes, the patterns, the gap, the main themes, the patterns, the gap, the main themes, the patterns, the gap, the coverage that this video collection coverage that this video collection coverage that this video collection cover, right? Uh this matter because cover, right? Uh this matter because cover, right? Uh this matter because different question travels different different question travels different different question travels different part of the graph. If you ask a simple part of the graph. If you ask a simple part of the graph. If you ask a simple search question then might that might go search question then might that might go search question then might that might go directly into the moment but like an directly into the moment but like an directly into the moment but like an entity workflow might start with a entity workflow might start with a entity workflow might start with a person and then it expand into person and then it expand into person and then it expand into appearances right and if you ask appearances right and if you ask appearances right and if you ask question like a story line like

question like a story line like question like a story line like narrative storytelling of certain uh you narrative storytelling of certain uh you narrative storytelling of certain uh you know uh you know person then it may know uh you know person then it may know uh you know person then it may follow relationship across time right. follow relationship across time right. follow relationship across time right. uh so the key idea here is that memory uh so the key idea here is that memory uh so the key idea here is that memory in the context of video understanding is in the context of video understanding is in the context of video understanding is a navigable structure over the entire a navigable structure over the entire a navigable structure over the entire video volume from that concept I come up video volume from that concept I come up video volume from that concept I come up with these five principles of building with these five principles of building with these five principles of building um you know a memory layer for video um you know a memory layer for video um you know a memory layer for video intelligence number one is to ingest intelligence number one is to ingest intelligence number one is to ingest once and reason many times so um you once and reason many times so um you once and reason many times so um you don't want to like do sing every single don't want to like do sing every single don't want to like do sing every single query uh from scratch like you want to query uh from scratch like you want to query uh from scratch like you want to pay the cost up front do one uh pay the cost up front do one uh pay the cost up front do one uh interpretation from the video content up interpretation from the video content up interpretation from the video content up u up front pay the cost and then you u up front pay the cost and then you u up front pay the cost and then you move expensive understanding into move expensive understanding into move expensive understanding into injection So this is the same mental injection So this is the same mental injection So this is the same mental model uh of database right you you do model uh of database right you you do model uh of database right you you do not repeatedly have to pass your entire not repeatedly have to pass your entire not repeatedly have to pass your entire source of data um for you know every source of data um for you know every source of data um for you know every application request. Um second principle application request. Um second principle application request. Um second principle is to store primitive not just answer. is to store primitive not just answer. is to store primitive not just answer. So uh you know moments entities So uh you know moments entities So uh you know moments entities appearances I already talked about that appearances I already talked about that appearances I already talked about that those are the the primitives right that those are the the primitives right that those are the the primitives right that allows you to uh do downstream workflow allows you to uh do downstream workflow allows you to uh do downstream workflow like search editing um you know like search editing um you know like search editing um you know analytics all of that. Third is to analytics all of that. Third is to analytics all of that. Third is to ground every claim like basically if you ground every claim like basically if you ground every claim like basically if you ask a question you need to site back ask a question you need to site back ask a question you need to site back into where that scene happening in the into where that scene happening in the into where that scene happening in the video. So evidence like you know should video. So evidence like you know should video. So evidence like you know should be grounded to a specific time stamp be grounded to a specific time stamp be grounded to a specific time stamp within the video right. Uh fourth is to within the video right. Uh fourth is to within the video right. Uh fourth is to let intent shape memory. Um this is let intent shape memory. Um this is let intent shape memory. Um this is important because the same footage mean important because the same footage mean important because the same footage mean different thing in different workflow.

different thing in different workflow. different thing in different workflow. We work across spots uh application We work across spots uh application We work across spots uh application brand safety compliance review creator brand safety compliance review creator brand safety compliance review creator analytics all of them require different analytics all of them require different analytics all of them require different primitives from the same video. So the primitives from the same video. So the primitives from the same video. So the memory layer should be configurable, memory layer should be configurable, memory layer should be configurable, right? Developers should uh should be right? Developers should uh should be right? Developers should uh should be able to tell the system what matters. able to tell the system what matters. able to tell the system what matters. And lastly, uh keep the layer And lastly, uh keep the layer And lastly, uh keep the layer composable. Um so basically being API composable. Um so basically being API composable. Um so basically being API first, you know, um it should provide first, you know, um it should provide first, you know, um it should provide the the layers that allows those the the layers that allows those the the layers that allows those application on top of that to to serve application on top of that to to serve application on top of that to to serve it uh structure graded metadata that can it uh structure graded metadata that can it uh structure graded metadata that can be plugged into any sort of application. be plugged into any sort of application. be plugged into any sort of application. Um so moving beyond these five Um so moving beyond these five Um so moving beyond these five principles, I want to talk about like principles, I want to talk about like principles, I want to talk about like kind of the the harnesses around kind of the the harnesses around kind of the the harnesses around building a memory layer, right? There's building a memory layer, right? There's building a memory layer, right? There's a lot of talk these day about um you a lot of talk these day about um you a lot of talk these day about um you know building the right harnesses for know building the right harnesses for know building the right harnesses for the conext of language model. So what the conext of language model. So what the conext of language model. So what does it look like for for video on a does it look like for for video on a does it look like for for video on a static model right um a model co a static model right um a model co a static model right um a model co a single answer it is stateless it start single answer it is stateless it start single answer it is stateless it start fresh time start fresh each time and fresh time start fresh each time and fresh time start fresh each time and doesn't have any constraint so the doesn't have any constraint so the doesn't have any constraint so the output is largely based on what the output is largely based on what the output is largely based on what the model decide to produce a video worker model decide to produce a video worker model decide to produce a video worker on the other hand operate inside a a on the other hand operate inside a a on the other hand operate inside a a very deterministic system understand very deterministic system understand very deterministic system understand what is available uh it can plan the what is available uh it can plan the what is available uh it can plan the task receive evidence inspect uh uh the task receive evidence inspect uh uh the task receive evidence inspect uh uh the relevant moments synthesize validate relevant moments synthesize validate relevant moments synthesize validate return output and then the entire return output and then the entire return output and then the entire workflow can be evaluated, right? Um so workflow can be evaluated, right? Um so workflow can be evaluated, right? Um so so for video understanding this is very so for video understanding this is very so for video understanding this is very important because the worker need to important because the worker need to important because the worker need to know what memory is available what know what memory is available what know what memory is available what evident matters and also like how deep evident matters and also like how deep evident matters and also like how deep to inspect because that will depend uh to inspect because that will depend uh to inspect because that will depend uh determine how much cost to spend what determine how much cost to spend what determine how much cost to spend what output contract to satisfy right um output contract to satisfy right um output contract to satisfy right um talking about harness engineering for talking about harness engineering for talking about harness engineering for for video understanding um I come up

for video understanding um I come up for video understanding um I come up with this like different capabilities with this like different capabilities with this like different capabilities for for like a video worker right um for for like a video worker right um for for like a video worker right um number one is memory I talked about that number one is memory I talked about that number one is memory I talked about that already number two is task planning so already number two is task planning so already number two is task planning so given given a query from from an user given given a query from from an user given given a query from from an user uh you have to decide like what task to uh you have to decide like what task to uh you have to decide like what task to execute is it like search or is it like execute is it like search or is it like execute is it like search or is it like summarization or like you know multi summarization or like you know multi summarization or like you know multi multi-step reasoning uh third is multi-step reasoning uh third is multi-step reasoning uh third is retrieval like every single system retrieval like every single system retrieval like every single system should be able to like select the right should be able to like select the right should be able to like select the right evidence from from your video corpus to evidence from from your video corpus to evidence from from your video corpus to read for specific task uh expert tools read for specific task uh expert tools read for specific task uh expert tools right so we work with customer where right so we work with customer where right so we work with customer where they require like you know zoom in zoom they require like you know zoom in zoom they require like you know zoom in zoom out uh comparing different uh uh you out uh comparing different uh uh you out uh comparing different uh uh you know frames and reaching uh content with know frames and reaching uh content with know frames and reaching uh content with like additional metadata so building like additional metadata so building like additional metadata so building expert tools inside like a like a video expert tools inside like a like a video expert tools inside like a like a video worker uh is very important operating worker uh is very important operating worker uh is very important operating envelope. So these are like explicit envelope. So these are like explicit envelope. So these are like explicit limit on time, cost, dep scope, limit on time, cost, dep scope, limit on time, cost, dep scope, autonomy. Uh an output contract. So autonomy. Uh an output contract. So autonomy. Uh an output contract. So sometimes natural language is snap, sometimes natural language is snap, sometimes natural language is snap, sometime the patient need structure data sometime the patient need structure data sometime the patient need structure data with references and time stamp and of with references and time stamp and of with references and time stamp and of course finally you have evaluation right course finally you have evaluation right course finally you have evaluation right uh like you know did the retrieval find uh like you know did the retrieval find uh like you know did the retrieval find the right evidence did the synthes the right evidence did the synthes the right evidence did the synthes synthesis preserve it right? Did the synthesis preserve it right? Did the synthesis preserve it right? Did the workers stay within the budget? All workers stay within the budget? All workers stay within the budget? All right. So, so that's a lot of like you right. So, so that's a lot of like you right. So, so that's a lot of like you know uh slide and and and talk. I want know uh slide and and and talk. I want know uh slide and and and talk. I want to quickly jump into some demos uh that to quickly jump into some demos uh that to quickly jump into some demos uh that I actually build using to labs uh you I actually build using to labs uh you I actually build using to labs uh you know video Asian product. So the uh know video Asian product. So the uh know video Asian product. So the uh there'll be three demos. Um the the there'll be three demos. Um the the there'll be three demos. Um the the video Asian product that we've been video Asian product that we've been video Asian product that we've been building is called jockey. So this first building is called jockey. So this first building is called jockey. So this first example here is for spot understanding.

example here is for spot understanding. example here is for spot understanding. Uh you know obviously everyone is super Uh you know obviously everyone is super Uh you know obviously everyone is super excited about the world cup that excited about the world cup that excited about the world cup that happening right now. So what I did is I happening right now. So what I did is I happening right now. So what I did is I ingest um 67 videos from the 2022 World ingest um 67 videos from the 2022 World ingest um 67 videos from the 2022 World Cup in Qatar. And you can see here I Cup in Qatar. And you can see here I Cup in Qatar. And you can see here I asked it to find the near misses uh the asked it to find the near misses uh the asked it to find the near misses uh the shot that almost become go but did not shot that almost become go but did not shot that almost become go but did not for each explain why it was not a goal for each explain why it was not a goal for each explain why it was not a goal uh but do not include actual goals. uh but do not include actual goals. uh but do not include actual goals. Right? So these are the the the top Right? So these are the the the top Right? So these are the the the top output it return. So that um that's hitting the good work. So that um that's hitting the good work. This is safe from the goalkeeper. I This is safe from the goalkeeper. I This is safe from the goalkeeper. I don't know if the sound is up, but like don't know if the sound is up, but like don't know if the sound is up, but like I'm playing the the video by the way. I'm playing the the video by the way. I'm playing the the video by the way. Um, right. This is another save from the right. This is another save from the goalkeeper. goalkeeper. goalkeeper. It even catch like you know the upside It even catch like you know the upside It even catch like you know the upside from one of the goals. And then I asked question. Okay. When is And then I asked question. Okay. When is the goal? Uh, five the most dramatic the goal? Uh, five the most dramatic the goal? Uh, five the most dramatic actual goals. show the build up play and actual goals. show the build up play and actual goals. show the build up play and the finish. For each goal, describe the the finish. For each goal, describe the the finish. For each goal, describe the sequence, right?

sequence, right? sequence, right? So, if you know this one, this is the So, if you know this one, this is the So, if you know this one, this is the um the first goal of the World Cup final um the first goal of the World Cup final um the first goal of the World Cup final like four years ago and it actually like like four years ago and it actually like like four years ago and it actually like returned like you know you don't returned like you know you don't returned like you know you don't understand who who are the passer like understand who who are the passer like understand who who are the passer like it was Alvarez passing to Maralista and it was Alvarez passing to Maralista and it was Alvarez passing to Maralista and passing to De Maria to score the goal. passing to De Maria to score the goal. passing to De Maria to score the goal. Um take a look at this one from Rich Um take a look at this one from Rich Um take a look at this one from Rich Allison. This is golden tournament uh Allison. This is golden tournament uh Allison. This is golden tournament uh from Brazil again. uh South Korea I from Brazil again. uh South Korea I from Brazil again. uh South Korea I believe right and it returned like you know u an area and it returned like you know u an area of skill in a buildup right the name the of skill in a buildup right the name the of skill in a buildup right the name the player who did the return pass player who did the return pass player who did the return pass you can even do player tracking so I you can even do player tracking so I you can even do player tracking so I asked it to track Leon Messi across this asked it to track Leon Messi across this asked it to track Leon Messi across this entire compost including the shot where entire compost including the shot where entire compost including the shot where he's one of the many figure on the he's one of the many figure on the he's one of the many figure on the screen describe the camera faming right screen describe the camera faming right screen describe the camera faming right uh so this is a hilarious all the uh so this is a hilarious all the uh so this is a hilarious all the important moment in the game and this is important moment in the game and this is important moment in the game and this is a scene where Messi He triple past a a scene where Messi He triple past a a scene where Messi He triple past a sliding defender. You can see here sliding defender. You can see here sliding defender. You can see here it picked up the scene where he scored it picked up the scene where he scored it picked up the scene where he scored the first goal against Australia in the the first goal against Australia in the the first goal against Australia in the round 16, I believe.

round 16, I believe. round 16, I believe. Right. This is another scene where he Right. This is another scene where he Right. This is another scene where he scored the third goal in the final. Yeah. So that one example on spot uh Yeah. So that one example on spot uh spot understanding. spot understanding. spot understanding. But then you can obviously build more But then you can obviously build more But then you can obviously build more interesting and more uh like real interesting and more uh like real interesting and more uh like real practical application of which uh practical application of which uh practical application of which uh security is one that we encounter a lot. security is one that we encounter a lot. security is one that we encounter a lot. So on on this example I ingest it um you So on on this example I ingest it um you So on on this example I ingest it um you know publicly available camera footage know publicly available camera footage know publicly available camera footage for context. These are the clip you have for context. These are the clip you have for context. These are the clip you have a traffic jam sub urban uh you know a traffic jam sub urban uh you know a traffic jam sub urban uh you know urban area uh and given this footage urban area uh and given this footage urban area uh and given this footage right I asked jockey to cow and classify right I asked jockey to cow and classify right I asked jockey to cow and classify every vehicle in the intersection break every vehicle in the intersection break every vehicle in the intersection break it down by tie plus pedestrian and it it down by tie plus pedestrian and it it down by tie plus pedestrian and it return the number of vehicles and the return the number of vehicles and the return the number of vehicles and the big foot traffic as well um it can big foot traffic as well um it can big foot traffic as well um it can detect safety events right so you see there a red SUV turn and so you see there a red SUV turn and almost gets struck Another almost gets struck Another almost gets struck Another scene here.

scene here. scene here. Turn left into an upcoming car. Yeah. So that a clearest red light Yeah. So that a clearest red light entry. Uh it works well in you know entry. Uh it works well in you know entry. Uh it works well in you know different scenario. This scene is uh different scenario. This scene is uh different scenario. This scene is uh very crowded uh area in Bangkok. very crowded uh area in Bangkok. very crowded uh area in Bangkok. Uh Uh Uh it as it I also asked it to work on the it as it I also asked it to work on the it as it I also asked it to work on the the you know the rain, right? So this is another scene where it So this is another scene where it understanding the um rainy condition. It identify the buses um intersection It identify the buses um intersection vehicle window. So yeah those are some example for for So yeah those are some example for for camera security surveillance footage. Um camera security surveillance footage. Um camera security surveillance footage. Um finally advertising. So um you probably finally advertising. So um you probably finally advertising. So um you probably seen this uh Adidas clip in all all the seen this uh Adidas clip in all all the seen this uh Adidas clip in all all the commercial leading up to World Cup commercial leading up to World Cup commercial leading up to World Cup recently. It's a five minutes Adidas recently. It's a five minutes Adidas recently. It's a five minutes Adidas footage and I asked it to classify all footage and I asked it to classify all footage and I asked it to classify all the point where you can put an add-on.

the point where you can put an add-on. the point where you can put an add-on. So it find the reveal the hard the So it find the reveal the hard the So it find the reveal the hard the impact energy pick right it a scene impact energy pick right it a scene impact energy pick right it a scene where certain player appear on the where certain player appear on the where certain player appear on the screen. It did identify like you know high It did identify like you know high impact action like this impact action like this impact action like this condition um the hard cut to knife condition um the hard cut to knife condition um the hard cut to knife football underlies and of course it it football underlies and of course it it football underlies and of course it it pointing to the logo um pointing to the logo um pointing to the logo um of Adidas. of Adidas. of Adidas. So you know uh from from perspective of So you know uh from from perspective of So you know uh from from perspective of an advertiser these are very important an advertiser these are very important an advertiser these are very important moments because they can you know fight moments because they can you know fight moments because they can you know fight the scene with slow motion hero in which they can advertise their brand in which they can advertise their brand content against this footage. Right. content against this footage. Right. content against this footage. Right. Um yeah so those are three sample demo Um yeah so those are three sample demo Um yeah so those are three sample demo application um that I want to highlight application um that I want to highlight application um that I want to highlight uh of using tool apps. Um and again um uh of using tool apps. Um and again um uh of using tool apps. Um and again um now what can you build with with this now what can you build with with this now what can you build with with this sort of video memory layer based on sort of video memory layer based on sort of video memory layer based on example these are the categories of of example these are the categories of of example these are the categories of of application that I believe developers application that I believe developers application that I believe developers can build you can discover things you can build you can discover things you can build you can discover things you can view reasoning experience you can can view reasoning experience you can can view reasoning experience you can organize your content across different organize your content across different organize your content across different video library and you can view action video library and you can view action video library and you can view action workflow assemble uh different scene workflow assemble uh different scene workflow assemble uh different scene together do compliance review data together do compliance review data together do compliance review data operation etc.

operation etc. operation etc. The same framework apply for different The same framework apply for different The same framework apply for different vehicles, email, entertainment, sport, vehicles, email, entertainment, sport, vehicles, email, entertainment, sport, segmentation, highlight generation, in segmentation, highlight generation, in segmentation, highlight generation, in commercial security, evidence review, commercial security, evidence review, commercial security, evidence review, contextual analysis, in advertising, uh contextual analysis, in advertising, uh contextual analysis, in advertising, uh brand safety, uh creative intelligence, brand safety, uh creative intelligence, brand safety, uh creative intelligence, right? right? right? And uh yeah, so this is our product uh And uh yeah, so this is our product uh And uh yeah, so this is our product uh that upcoming up. Um one quick highlight that upcoming up. Um one quick highlight that upcoming up. Um one quick highlight is that we we try to code as a video is that we we try to code as a video is that we we try to code as a video cognition infrastructure. So we have a cognition infrastructure. So we have a cognition infrastructure. So we have a knowledge store that basically become knowledge store that basically become knowledge store that basically become the video memory layer. Web configurable the video memory layer. Web configurable the video memory layer. Web configurable injection that let builders shape what injection that let builders shape what injection that let builders shape what the system can can extract copus digest the system can can extract copus digest the system can can extract copus digest so that you can understanding what is in so that you can understanding what is in so that you can understanding what is in the library and the resolution as you the library and the resolution as you the library and the resolution as you search responses API so the thing I want search responses API so the thing I want search responses API so the thing I want to highlight here is it's not an to highlight here is it's not an to highlight here is it's not an application layer it's not an editing application layer it's not an editing application layer it's not an editing platform not a compliance product it's platform not a compliance product it's platform not a compliance product it's the cognition infrastructure with the the cognition infrastructure with the the cognition infrastructure with the layer and the harnesses that enable like layer and the harnesses that enable like layer and the harnesses that enable like those product being to become available those product being to become available those product being to become available um and if you found the content of this um and if you found the content of this um and if you found the content of this talk interesting um definitely recommend talk interesting um definitely recommend talk interesting um definitely recommend you to to scan this QR you to to scan this QR you to to scan this QR Uh the product is currently in private Uh the product is currently in private Uh the product is currently in private beta right now. Um if you bring any sort beta right now. Um if you bring any sort beta right now. Um if you bring any sort of workflow that touch video content of workflow that touch video content of workflow that touch video content especially around content assembly, especially around content assembly, especially around content assembly, content organization, you know, think content organization, you know, think content organization, you know, think about media archive, content creator, about media archive, content creator, about media archive, content creator, YouTube, Tik Toker, uh spot analysis, uh YouTube, Tik Toker, uh spot analysis, uh YouTube, Tik Toker, uh spot analysis, uh media workflow. Uh definitely uh either media workflow. Uh definitely uh either media workflow. Uh definitely uh either scan this code and register for the scan this code and register for the scan this code and register for the interest or come talk to me after the interest or come talk to me after the interest or come talk to me after the talk. So that should be my time. Thanks talk. So that should be my time. Thanks talk. So that should be my time. Thanks a lot.