Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI

Hi everyone. Uh, sorry it's a bit loud Hi everyone. Uh, sorry it's a bit loud in here. Was not expecting this. Um, I'm in here. Was not expecting this. Um, I'm in here. Was not expecting this. Um, I'm Siman. I'm a principal scientist at Siman. I'm a principal scientist at Siman. I'm a principal scientist at Together AI. Um, I previously did my PhD Together AI. Um, I previously did my PhD Together AI. Um, I previously did my PhD in the Hazy Research Lab with Chris Ray in the Hazy Research Lab with Chris Ray in the Hazy Research Lab with Chris Ray at Stanford and I'm an incoming at Stanford and I'm an incoming at Stanford and I'm an incoming professor at Caltech. Um I lead the professor at Caltech. Um I lead the professor at Caltech. Um I lead the frontier performance research team at frontier performance research team at frontier performance research team at together where we develop systems, together where we develop systems, together where we develop systems, frameworks and algorithms to extract as frameworks and algorithms to extract as frameworks and algorithms to extract as much performance as possible out of much performance as possible out of much performance as possible out of modern um AI hardware. Today I want to modern um AI hardware. Today I want to modern um AI hardware. Today I want to share a little bit about our share a little bit about our share a little bit about our contributions towards simplifying the contributions towards simplifying the contributions towards simplifying the development of uh multi-GPU AI kernels. development of uh multi-GPU AI kernels. development of uh multi-GPU AI kernels. A few years ago, um, GPU utilization A few years ago, um, GPU utilization A few years ago, um, GPU utilization used to be limited by poor intraGPU used to be limited by poor intraGPU used to be limited by poor intraGPU memory access and single GPU kernels. memory access and single GPU kernels. memory access and single GPU kernels. But with significant investment in But with significant investment in But with significant investment in better kernels like flash attention, uh, better kernels like flash attention, uh, better kernels like flash attention, uh, memory efficient architectures like from memory efficient architectures like from memory efficient architectures like from deepseek, sparse attentions, mambas and deepseek, sparse attentions, mambas and deepseek, sparse attentions, mambas and so on, um, and better DSLs, we've sort so on, um, and better DSLs, we've sort so on, um, and better DSLs, we've sort of shifted the bottleneck to multi-GPU of shifted the bottleneck to multi-GPU of shifted the bottleneck to multi-GPU communication.

During this talk, I'll start by telling During this talk, I'll start by telling you a little bit about why now, why GPU you a little bit about why now, why GPU you a little bit about why now, why GPU networking now. Then I'll tell you about networking now. Then I'll tell you about networking now. Then I'll tell you about um the sort of problem space. So what um the sort of problem space. So what um the sort of problem space. So what are the challenges in maximizing are the challenges in maximizing are the challenges in maximizing hardware utilization and development hardware utilization and development hardware utilization and development simplicity for multiGPU kernels. Um simplicity for multiGPU kernels. Um simplicity for multiGPU kernels. Um three, we'll talk a little bit about the three, we'll talk a little bit about the three, we'll talk a little bit about the fundamentals be behind designing fundamentals be behind designing fundamentals be behind designing effective multiGGPU kernels. Four, we'll effective multiGGPU kernels. Four, we'll effective multiGGPU kernels. Four, we'll look at whether frontier AI models can look at whether frontier AI models can look at whether frontier AI models can uh leverage these fundamental uh leverage these fundamental uh leverage these fundamental principles. Do they understand them? Can principles. Do they understand them? Can principles. Do they understand them? Can they reason about them? Um, you know, in they reason about them? Um, you know, in they reason about them? Um, you know, in theory, these models are very good at theory, these models are very good at theory, these models are very good at reasoning. Um, and then five, we'll talk reasoning. Um, and then five, we'll talk reasoning. Um, and then five, we'll talk through the results of these frontier through the results of these frontier through the results of these frontier models on a benchmark that we've models on a benchmark that we've models on a benchmark that we've developed called um parallel kernel developed called um parallel kernel developed called um parallel kernel bench for multiGPU kernel generation bench for multiGPU kernel generation bench for multiGPU kernel generation evaluation. Okay, before we dive into those five Okay, before we dive into those five parts, just basic preliminaries. So, parts, just basic preliminaries. So, parts, just basic preliminaries. So, this is an Nvidia GPU, uh, an H100 GPU this is an Nvidia GPU, uh, an H100 GPU this is an Nvidia GPU, uh, an H100 GPU that you can see on the screen. Um, I that you can see on the screen. Um, I that you can see on the screen. Um, I always like to help ground people in GPU always like to help ground people in GPU always like to help ground people in GPU kernels via looking at the hardware. So, kernels via looking at the hardware. So, kernels via looking at the hardware. So, these rainbow colored dots are these rainbow colored dots are these rainbow colored dots are processors where actual compute is processors where actual compute is processors where actual compute is happening. um all of the you know happening. um all of the you know happening. um all of the you know parallel threads are operating within parallel threads are operating within parallel threads are operating within one of those colored dots and there's one of those colored dots and there's one of those colored dots and there's typically you know 100 200 of them on typically you know 100 200 of them on typically you know 100 200 of them on modern AI GPUs.

modern AI GPUs. modern AI GPUs. Um around those processors you can see Um around those processors you can see Um around those processors you can see um some of the memory that these um some of the memory that these um some of the memory that these processors retrieve data so large processors retrieve data so large processors retrieve data so large weights activations from. So these weights activations from. So these weights activations from. So these rectangles between the colored dots are rectangles between the colored dots are rectangles between the colored dots are an L2 cache slightly faster memory. um an L2 cache slightly faster memory. um an L2 cache slightly faster memory. um not a ter like crazy large amount of it. not a ter like crazy large amount of it. not a ter like crazy large amount of it. And then these black boxes are high And then these black boxes are high And then these black boxes are high bandwidth memory. So when you Nvidia Smi bandwidth memory. So when you Nvidia Smi bandwidth memory. So when you Nvidia Smi and see you know 80 gigabytes, 1008 and see you know 80 gigabytes, 1008 and see you know 80 gigabytes, 1008 gigabytes, whatever it is on your GPU, gigabytes, whatever it is on your GPU, gigabytes, whatever it is on your GPU, that's that memory. that's that memory. that's that memory. Um a GPU is operating a highly parallel Um a GPU is operating a highly parallel Um a GPU is operating a highly parallel program. So multiple threads are program. So multiple threads are program. So multiple threads are combined together in uh into larger combined together in uh into larger combined together in uh into larger coarser units. and we schedule these coarser units. and we schedule these coarser units. and we schedule these threads and and blocks onto these threads and and blocks onto these threads and and blocks onto these processors to perform our AI compute. processors to perform our AI compute. processors to perform our AI compute. Beyond the GPU, we'll have multiple GPUs Beyond the GPU, we'll have multiple GPUs Beyond the GPU, we'll have multiple GPUs and we'll also have, you know, CPUs um and we'll also have, you know, CPUs um and we'll also have, you know, CPUs um that have memory as well.

Um so to perform computation um the Um so to perform computation um the memory that these threads use is going memory that these threads use is going memory that these threads use is going to be stored in a really fast register to be stored in a really fast register to be stored in a really fast register memory that's right next to the memory that's right next to the memory that's right next to the computation units. um simple physics if computation units. um simple physics if computation units. um simple physics if I am pulling data from very very close I am pulling data from very very close I am pulling data from very very close to my proc my compute unit it's really to my proc my compute unit it's really to my proc my compute unit it's really fast to get to it because it you know fast to get to it because it you know fast to get to it because it you know that data is right next to me but that data is right next to me but that data is right next to me but there's not a large radius and not a there's not a large radius and not a there's not a large radius and not a large volume of space that's close by to large volume of space that's close by to large volume of space that's close by to my process uh my my compute units and so my process uh my my compute units and so my process uh my my compute units and so I don't have very much of it so you can I don't have very much of it so you can I don't have very much of it so you can see that the fastest memory here the see that the fastest memory here the see that the fastest memory here the registers is sup is you know 130 registers is sup is you know 130 registers is sup is you know 130 terabytes per second on an H100 but we terabytes per second on an H100 but we terabytes per second on an H100 but we don't have very much on of it and as we don't have very much on of it and as we don't have very much on of it and as we go to the further away memory we have a go to the further away memory we have a go to the further away memory we have a lot more of it but it takes longer to lot more of it but it takes longer to lot more of it but it takes longer to reach it. Uh again simple physics. reach it. Uh again simple physics. reach it. Uh again simple physics. So in multiGPU systems in particular um So in multiGPU systems in particular um So in multiGPU systems in particular um there is a hierarchy of interconnects. there is a hierarchy of interconnects. there is a hierarchy of interconnects. So we will have something called PCIe um So we will have something called PCIe um So we will have something called PCIe um as the channel for CPUGGPU as the channel for CPUGGPU as the channel for CPUGGPU communications. We'll have multiGPU or communications. We'll have multiGPU or communications. We'll have multiGPU or multi- uh node communications over multi- uh node communications over multi- uh node communications over infiniband TCP and then in the we're infiniband TCP and then in the we're infiniband TCP and then in the we're going to focus mostly on the Nvidia going to focus mostly on the Nvidia going to focus mostly on the Nvidia sphere here. Um in uh the intraGPU sphere here. Um in uh the intraGPU sphere here. Um in uh the intraGPU regime we'll have NVLink um providing regime we'll have NVLink um providing regime we'll have NVLink um providing point-to-point connections between GPUs point-to-point connections between GPUs point-to-point connections between GPUs and the NV switch. NV switch connects and the NV switch. NV switch connects and the NV switch. NV switch connects all NVLink in endpoints into a all NVLink in endpoints into a all NVLink in endpoints into a non-blocking fabric for full GPUGGPU non-blocking fabric for full GPUGGPU non-blocking fabric for full GPUGGPU communication and NV switch is exciting

communication and NV switch is exciting communication and NV switch is exciting because it also provides support for in because it also provides support for in because it also provides support for in network offdevice acceleration for oper network offdevice acceleration for oper network offdevice acceleration for oper like communication primitives like like communication primitives like like communication primitives like multiccast and reductions. Okay, so diving in with the Okay, so diving in with the preliminaries in mind. Why GPU preliminaries in mind. Why GPU preliminaries in mind. Why GPU networking now? networking now? networking now? So as I mentioned at the beginning, So as I mentioned at the beginning, So as I mentioned at the beginning, we've really put a lot of effort into we've really put a lot of effort into we've really put a lot of effort into making AIO uh more efficient over recent making AIO uh more efficient over recent making AIO uh more efficient over recent years. Um again, we have uh years. Um again, we have uh years. Um again, we have uh architectures that use less compute, architectures that use less compute, architectures that use less compute, less memory like Mamba or sparse less memory like Mamba or sparse less memory like Mamba or sparse attentions. We have algorithms that make attentions. We have algorithms that make attentions. We have algorithms that make AI more hardware affair aware like flash AI more hardware affair aware like flash AI more hardware affair aware like flash attention. We have tools to make it easy attention. We have tools to make it easy attention. We have tools to make it easy to map AI algorithms to the hardware to map AI algorithms to the hardware to map AI algorithms to the hardware like tileang, mojo, triton, gluon, like tileang, mojo, triton, gluon, like tileang, mojo, triton, gluon, thunder kittens. Um, and we have new thunder kittens. Um, and we have new thunder kittens. Um, and we have new techniques to overlap uh execution techniques to overlap uh execution techniques to overlap uh execution across many AI operators very tightly across many AI operators very tightly across many AI operators very tightly like mega kernels. And we also have like mega kernels. And we also have like mega kernels. And we also have tools to make it easy to run on multiple tools to make it easy to run on multiple tools to make it easy to run on multiple vendor and silicon platforms like vendor and silicon platforms like vendor and silicon platforms like thunder mittens for Apple silicon or or thunder mittens for Apple silicon or or thunder mittens for Apple silicon or or hipkittens for AMD and so on.

hipkittens for AMD and so on. hipkittens for AMD and so on. Um, at this point we really believe that Um, at this point we really believe that Um, at this point we really believe that GPU networking offers many new and GPU networking offers many new and GPU networking offers many new and exciting opportunities for AI exciting opportunities for AI exciting opportunities for AI efficiency. efficiency. efficiency. Modern AI workloads are getting very big Modern AI workloads are getting very big Modern AI workloads are getting very big and require kernels that span multiple and require kernels that span multiple and require kernels that span multiple um, you know, GPUs. So on many um, you know, GPUs. So on many um, you know, GPUs. So on many production um distributed training and production um distributed training and production um distributed training and inference workloads, communication is inference workloads, communication is inference workloads, communication is increasingly consuming the majority of increasingly consuming the majority of increasingly consuming the majority of the runtime and yields low model flop the runtime and yields low model flop the runtime and yields low model flop utilization at scale. So the uh pace of innovation and So the uh pace of innovation and diversity of approaches that different diversity of approaches that different diversity of approaches that different hardware providers are taking in their hardware providers are taking in their hardware providers are taking in their networking stacks is another reason why networking stacks is another reason why networking stacks is another reason why it's an exciting time to study um you it's an exciting time to study um you it's an exciting time to study um you know networking and communication. So we know networking and communication. So we know networking and communication. So we can see here um AMD hardware with uh can see here um AMD hardware with uh can see here um AMD hardware with uh what's called XGMI interconnects um what's called XGMI interconnects um what's called XGMI interconnects um providing pointto-point links between providing pointto-point links between providing pointto-point links between different GPUs in a scaleup domain. We different GPUs in a scaleup domain. We different GPUs in a scaleup domain. We can see here TPU and uh interconnect uh can see here TPU and uh interconnect uh can see here TPU and uh interconnect uh as well. So the TPU will use a 3D Taurus as well. So the TPU will use a 3D Taurus as well. So the TPU will use a 3D Taurus and also have optical wraparound links.

and also have optical wraparound links. and also have optical wraparound links. So yet another diverse form of the So yet another diverse form of the So yet another diverse form of the topology and links. Um and then again topology and links. Um and then again topology and links. Um and then again for Nvidia we'll have the INV um switch for Nvidia we'll have the INV um switch for Nvidia we'll have the INV um switch which integrates um compute capabilities which integrates um compute capabilities which integrates um compute capabilities directly into the interconnect fabric directly into the interconnect fabric directly into the interconnect fabric like for in network reductions and then like for in network reductions and then like for in network reductions and then we'll have our NV link providing up to we'll have our NV link providing up to we'll have our NV link providing up to you know 900 gigabytes of unidirectional you know 900 gigabytes of unidirectional you know 900 gigabytes of unidirectional bandwidth between any two remote GPUs bandwidth between any two remote GPUs bandwidth between any two remote GPUs high bandwidth memory um on you know high bandwidth memory um on you know high bandwidth memory um on you know particular generation of Nvidia hardware particular generation of Nvidia hardware particular generation of Nvidia hardware um Beyond the diversity in the um Beyond the diversity in the um Beyond the diversity in the networking stacks, there's also a lot of networking stacks, there's also a lot of networking stacks, there's also a lot of evolution in um how AI workloads are evolution in um how AI workloads are evolution in um how AI workloads are adapting to take advantage of this adapting to take advantage of this adapting to take advantage of this hardware and the diversity of types of hardware and the diversity of types of hardware and the diversity of types of um you know hardware that we're we're um you know hardware that we're we're um you know hardware that we're we're using simultaneously for one AI using simultaneously for one AI using simultaneously for one AI workload. So um KV cache memory in workload. So um KV cache memory in workload. So um KV cache memory in modern inference systems is going to modern inference systems is going to modern inference systems is going to beworked across GPU, CPU, disk and beworked across GPU, CPU, disk and beworked across GPU, CPU, disk and remote machines. um inference systems remote machines. um inference systems remote machines. um inference systems increasingly disagregate different steps increasingly disagregate different steps increasingly disagregate different steps of inference across different hardware of inference across different hardware of inference across different hardware backends. So you could run speculative backends. So you could run speculative backends. So you could run speculative decoding on some hardware, decode on decoding on some hardware, decode on decoding on some hardware, decode on different hardware, prefill on different different hardware, prefill on different different hardware, prefill on different hardware. Um hardware is also evolving hardware. Um hardware is also evolving hardware. Um hardware is also evolving to have larger scaleup domains than ever to have larger scaleup domains than ever to have larger scaleup domains than ever before with you know 72 GPUs and a before with you know 72 GPUs and a before with you know 72 GPUs and a scaleup domain in in the coming um chips scaleup domain in in the coming um chips scaleup domain in in the coming um chips and Nvidia planning on a single system and Nvidia planning on a single system and Nvidia planning on a single system in 2027 with 576 GPUs.

in 2027 with 576 GPUs. in 2027 with 576 GPUs. Um at the same time to take advantage of Um at the same time to take advantage of Um at the same time to take advantage of these more uh intensive scaleup domains these more uh intensive scaleup domains these more uh intensive scaleup domains we're getting richer uh primitives for we're getting richer uh primitives for we're getting richer uh primitives for fine grained control and kernel writing fine grained control and kernel writing fine grained control and kernel writing over these domains. So we have something over these domains. So we have something over these domains. So we have something called tensor memory acceleration where called tensor memory acceleration where called tensor memory acceleration where we can provide uh perform asynchronous we can provide uh perform asynchronous we can provide uh perform asynchronous network transfers from the device side network transfers from the device side network transfers from the device side um on these GPUs. um on these GPUs. um on these GPUs. So all of these changes are opening up So all of these changes are opening up So all of these changes are opening up new opportunities and challenges in um new opportunities and challenges in um new opportunities and challenges in um both AI you know how do we build models both AI you know how do we build models both AI you know how do we build models that take advantage of these trends and that take advantage of these trends and that take advantage of these trends and in systems. in systems. in systems. Okay, so the problems that we're going Okay, so the problems that we're going Okay, so the problems that we're going to go after um how do we get peak to go after um how do we get peak to go after um how do we get peak hardware utilization and also hardware utilization and also hardware utilization and also development simplicity um for these development simplicity um for these development simplicity um for these multiGPU kernels. multiGPU kernels. multiGPU kernels. So um it's been very difficult to write So um it's been very difficult to write So um it's been very difficult to write multiGGPU kernels and there's a lot of multiGGPU kernels and there's a lot of multiGGPU kernels and there's a lot of there are a lot of papers a lot of there are a lot of papers a lot of there are a lot of papers a lot of systems reports that document you know systems reports that document you know systems reports that document you know challenges here. Um, one of the things challenges here. Um, one of the things challenges here. Um, one of the things here is it's compounded by the fact that here is it's compounded by the fact that here is it's compounded by the fact that communication hardware around GPUs has communication hardware around GPUs has communication hardware around GPUs has progressed a lot more slowly relative to progressed a lot more slowly relative to progressed a lot more slowly relative to compute and memory. Um, so comparing compute and memory. Um, so comparing compute and memory. Um, so comparing NVIDIA A100's in 2020 to B200s in 2024, NVIDIA A100's in 2020 to B200s in 2024, NVIDIA A100's in 2020 to B200s in 2024, u BF16 tensor core speeds improved by u BF16 tensor core speeds improved by u BF16 tensor core speeds improved by 7.2x.

7.2x. 7.2x. um while intra node communication by um while intra node communication by um while intra node communication by just 3x and inter node communication by just 3x and inter node communication by just 3x and inter node communication by just 2x. Um and coming back to my points just 2x. Um and coming back to my points just 2x. Um and coming back to my points about how diverse networking is right about how diverse networking is right about how diverse networking is right now things like tensor cores that run now things like tensor cores that run now things like tensor cores that run map moles and our memory hierarchies are map moles and our memory hierarchies are map moles and our memory hierarchies are pretty consistent and resemble one pretty consistent and resemble one pretty consistent and resemble one another across diverse AI vendors and another across diverse AI vendors and another across diverse AI vendors and multisilicon. Um but again as I I multisilicon. Um but again as I I multisilicon. Um but again as I I mentioned the networking stack is mentioned the networking stack is mentioned the networking stack is something that is really different something that is really different something that is really different across vendors still across vendors still across vendors still um you know a first step as we went um you know a first step as we went um you know a first step as we went about all this work is to just study the about all this work is to just study the about all this work is to just study the baselines um that are out there. So um baselines um that are out there. So um baselines um that are out there. So um one of the popular tools for um one of the popular tools for um one of the popular tools for um communications is this nickel library or communications is this nickel library or communications is this nickel library or Rickle on AMD um that both you know Rickle on AMD um that both you know Rickle on AMD um that both you know companies respectively spend a lot of companies respectively spend a lot of companies respectively spend a lot of you know engineering investment into you know engineering investment into you know engineering investment into releasing to make it easy for people to releasing to make it easy for people to releasing to make it easy for people to do multiGPU work. Um but they're not do multiGPU work. Um but they're not do multiGPU work. Um but they're not very flexible. So they're tuned for bulk very flexible. So they're tuned for bulk very flexible. So they're tuned for bulk transfers for large contiguous chunks of transfers for large contiguous chunks of transfers for large contiguous chunks of data transfers. And the design really data transfers. And the design really data transfers. And the design really breaks down when you care about peak breaks down when you care about peak breaks down when you care about peak performance, fine grain communication, performance, fine grain communication, performance, fine grain communication, um, and sort of non-trivial collectives um, and sort of non-trivial collectives um, and sort of non-trivial collectives that you want to fuse together.

that you want to fuse together. that you want to fuse together. So as a result, you can achieve much So as a result, you can achieve much So as a result, you can achieve much higher performance by writing custom higher performance by writing custom higher performance by writing custom communication kernels that directly communication kernels that directly communication kernels that directly address these needs. address these needs. address these needs. If we look at a naive baseline that's If we look at a naive baseline that's If we look at a naive baseline that's representative of very popular libraries representative of very popular libraries representative of very popular libraries in machine learning stacking pietorch in machine learning stacking pietorch in machine learning stacking pietorch with nickel um we can we find that with nickel um we can we find that with nickel um we can we find that across um you know the many uh problems across um you know the many uh problems across um you know the many uh problems in our parallel kernel bench benchmark in our parallel kernel bench benchmark in our parallel kernel bench benchmark that the majority of these um simple that the majority of these um simple that the majority of these um simple baselines will fall below 50% of their baselines will fall below 50% of their baselines will fall below 50% of their communication aware roof line bound. So communication aware roof line bound. So communication aware roof line bound. So there's a lot of room for improvement there's a lot of room for improvement there's a lot of room for improvement here. here. here. Um the current frameworks beyond um Um the current frameworks beyond um Um the current frameworks beyond um nickel which is popular in like Megatron nickel which is popular in like Megatron nickel which is popular in like Megatron LM, Flex Flow, Nanoflow um you know all LM, Flex Flow, Nanoflow um you know all LM, Flex Flow, Nanoflow um you know all again these systems are primarily again these systems are primarily again these systems are primarily orchestrating bulk collectives via orchestrating bulk collectives via orchestrating bulk collectives via nickel um and require synchronization nickel um and require synchronization nickel um and require synchronization before and after data transfers. So before and after data transfers. So before and after data transfers. So beyond th those off-the-shelf libraries, beyond th those off-the-shelf libraries, beyond th those off-the-shelf libraries, we have compilers and DSLs that exist.

we have compilers and DSLs that exist. we have compilers and DSLs that exist. So there's Triton distributed is one of So there's Triton distributed is one of So there's Triton distributed is one of them. Um and uh you know tile link is them. Um and uh you know tile link is them. Um and uh you know tile link is another one. Um we have found that it's another one. Um we have found that it's another one. Um we have found that it's very difficult to support the rapid pace very difficult to support the rapid pace very difficult to support the rapid pace of networking improvements within these of networking improvements within these of networking improvements within these frameworks. So our benchmarks and our frameworks. So our benchmarks and our frameworks. So our benchmarks and our papers highlight results where Triton papers highlight results where Triton papers highlight results where Triton distributed originally tuned around 8 uh distributed originally tuned around 8 uh distributed originally tuned around 8 uh H800 GPUs fails to adapt efficiently to H800 GPUs fails to adapt efficiently to H800 GPUs fails to adapt efficiently to other architectures like H100s. other architectures like H100s. other architectures like H100s. And then the third category of how And then the third category of how And then the third category of how people can proceed here is to really people can proceed here is to really people can proceed here is to really handtune specific AI operators one by handtune specific AI operators one by handtune specific AI operators one by one. So there's a lot of popular work um one. So there's a lot of popular work um one. So there's a lot of popular work um DPP um comet ring attention um flux DPP um comet ring attention um flux DPP um comet ring attention um flux flashdoe flashdoe flashdoe um and then several distributed gem um and then several distributed gem um and then several distributed gem kernels from cutless and these methods kernels from cutless and these methods kernels from cutless and these methods achieve peak performance but often um achieve peak performance but often um achieve peak performance but often um they do not you know some of these they do not you know some of these they do not you know some of these methods have been designed in one methods have been designed in one methods have been designed in one precision and it takes five or six precision and it takes five or six precision and it takes five or six months to scale it to another precision months to scale it to another precision months to scale it to another precision and just the scalability of this hand and just the scalability of this hand and just the scalability of this hand tuning and um fine grain kernel writing tuning and um fine grain kernel writing tuning and um fine grain kernel writing is not very um effective.

is not very um effective. is not very um effective. So um with this landscape in mind, our So um with this landscape in mind, our So um with this landscape in mind, our research question was really about research question was really about research question was really about whether there is a small set of whether there is a small set of whether there is a small set of principles and fundamentals that really principles and fundamentals that really principles and fundamentals that really governs multiGPU kernel writing and governs multiGPU kernel writing and governs multiGPU kernel writing and whether there are methods that can whether there are methods that can whether there are methods that can leverage those principles if they exist leverage those principles if they exist leverage those principles if they exist to simplify the development of these to simplify the development of these to simplify the development of these kernels. kernels. kernels. Um I'll briefly highlight two works here Um I'll briefly highlight two works here Um I'll briefly highlight two works here that um govern like that that represent that um govern like that that represent that um govern like that that represent our approach. So first we think it's our approach. So first we think it's our approach. So first we think it's important to build our own fundamental important to build our own fundamental important to build our own fundamental understanding and to manually do the understanding and to manually do the understanding and to manually do the work to understand it rather than just work to understand it rather than just work to understand it rather than just throwing say an LLM at the problem. So throwing say an LLM at the problem. So throwing say an LLM at the problem. So we spent the time to build out parallel we spent the time to build out parallel we spent the time to build out parallel kittens which is a small set of minimal kittens which is a small set of minimal kittens which is a small set of minimal primitives and patterns for multiGPU primitives and patterns for multiGPU primitives and patterns for multiGPU kernels. Um we use this to understand kernels. Um we use this to understand kernels. Um we use this to understand the trade-offs of multiGPU kernels and the trade-offs of multiGPU kernels and the trade-offs of multiGPU kernels and to one write a large collection of um to one write a large collection of um to one write a large collection of um peak performance kernels for a variety peak performance kernels for a variety peak performance kernels for a variety of parallelism schemes. Um and this I of parallelism schemes. Um and this I of parallelism schemes. Um and this I will use to hopefully you know educate will use to hopefully you know educate will use to hopefully you know educate and bring us all on the same page on and bring us all on the same page on and bring us all on the same page on what patterns we figured out. Um and what patterns we figured out. Um and what patterns we figured out. Um and then once we found that there is indeed then once we found that there is indeed then once we found that there is indeed a small set of trade-offs governing this a small set of trade-offs governing this a small set of trade-offs governing this landscape, we were curious whether landscape, we were curious whether landscape, we were curious whether models, especially these models right models, especially these models right models, especially these models right now that claim to be very good at kernel now that claim to be very good at kernel now that claim to be very good at kernel writing and also reasoning um could writing and also reasoning um could writing and also reasoning um could reason about these trade-offs when we reason about these trade-offs when we reason about these trade-offs when we provide them in context to actually provide them in context to actually provide them in context to actually generate a bunch of net new multiGPU

generate a bunch of net new multiGPU generate a bunch of net new multiGPU kernels for us. Unfortunately, we found kernels for us. Unfortunately, we found kernels for us. Unfortunately, we found they were not very good, but we'll dive they were not very good, but we'll dive they were not very good, but we'll dive into more of that at the end. into more of that at the end. into more of that at the end. Um so just the fundamental section this Um so just the fundamental section this Um so just the fundamental section this is going to be more you know educational is going to be more you know educational is going to be more you know educational what are the trade-offs that go into what are the trade-offs that go into what are the trade-offs that go into these kernels. these kernels. these kernels. So there are three main ways to do um So there are three main ways to do um So there are three main ways to do um intraGPU data transfers. Um there's the intraGPU data transfers. Um there's the intraGPU data transfers. Um there's the per GPU what's called copy engine and per GPU what's called copy engine and per GPU what's called copy engine and this is host or CPU initiated work. It's this is host or CPU initiated work. It's this is host or CPU initiated work. It's really good for large message transfers. really good for large message transfers. really good for large message transfers. So when your message size, the amount of So when your message size, the amount of So when your message size, the amount of data being transferred is really big um data being transferred is really big um data being transferred is really big um and it can get to sort of like peak and it can get to sort of like peak and it can get to sort of like peak bandwidth um on on the communication bandwidth um on on the communication bandwidth um on on the communication side. In contrast, you can use device side. In contrast, you can use device side. In contrast, you can use device initiated or GPU initiated transfers via initiated or GPU initiated transfers via initiated or GPU initiated transfers via that tensor memory accelerator that I that tensor memory accelerator that I that tensor memory accelerator that I mentioned or via register level mentioned or via register level mentioned or via register level instructions called uh in sort of their instructions called uh in sort of their instructions called uh in sort of their PTX lingo like LDST red multime. Um, and PTX lingo like LDST red multime. Um, and PTX lingo like LDST red multime. Um, and the uh T TMA is really nice. These the uh T TMA is really nice. These the uh T TMA is really nice. These device initiated ones are really nice device initiated ones are really nice device initiated ones are really nice because they can saturate our NVLink because they can saturate our NVLink because they can saturate our NVLink bandwidth using relatively small message bandwidth using relatively small message bandwidth using relatively small message sizes. And this means that they can be sizes. And this means that they can be sizes. And this means that they can be really nice when we're trying to do fine really nice when we're trying to do fine really nice when we're trying to do fine grain communication rather than sending grain communication rather than sending grain communication rather than sending bulk amounts of data over the links all bulk amounts of data over the links all bulk amounts of data over the links all at once coarsely.

at once coarsely. at once coarsely. Um, there are some trade-offs here. So Um, there are some trade-offs here. So Um, there are some trade-offs here. So the copy engine is really nice because the copy engine is really nice because the copy engine is really nice because it doesn't take away or you know waste a it doesn't take away or you know waste a it doesn't take away or you know waste a lot of our precious registers that I lot of our precious registers that I lot of our precious registers that I mentioned are important for compute on mentioned are important for compute on mentioned are important for compute on the GPU. Um and it doesn't also use any the GPU. Um and it doesn't also use any the GPU. Um and it doesn't also use any of those rainbow colored dots the of those rainbow colored dots the of those rainbow colored dots the processors on our GPU allowing us to processors on our GPU allowing us to processors on our GPU allowing us to repurpose those for memory or um repurpose those for memory or um repurpose those for memory or um computation on our uh you know other computation on our uh you know other computation on our uh you know other parts of the AI pipeline. Um, TMA, the parts of the AI pipeline. Um, TMA, the parts of the AI pipeline. Um, TMA, the second option here, consumes very few second option here, consumes very few second option here, consumes very few registers, which is why it's nice. Um, registers, which is why it's nice. Um, registers, which is why it's nice. Um, and it also can achieve high utilization and it also can achieve high utilization and it also can achieve high utilization using very few of our processors. So, using very few of our processors. So, using very few of our processors. So, it's a nice useful tool for fine grain it's a nice useful tool for fine grain it's a nice useful tool for fine grain overlapping. Um, TMA does have overlapping. Um, TMA does have overlapping. Um, TMA does have limitations. It can't effectively take limitations. It can't effectively take limitations. It can't effectively take advantage of these in network um, advantage of these in network um, advantage of these in network um, computations that I mentioned are computations that I mentioned are computations that I mentioned are feasible with technologies like NV feasible with technologies like NV feasible with technologies like NV switch. And the register level switch. And the register level switch. And the register level instructions are really nice for being instructions are really nice for being instructions are really nice for being able to take advantage of um you know able to take advantage of um you know able to take advantage of um you know those those sort of in network those those sort of in network those those sort of in network reductions that NV switch offers.

reductions that NV switch offers. reductions that NV switch offers. So again there are different tradeoffs So again there are different tradeoffs So again there are different tradeoffs different functionalities that these different functionalities that these different functionalities that these transfer mechanisms offer and they face transfer mechanisms offer and they face transfer mechanisms offer and they face different trade-offs. different trade-offs. different trade-offs. Second um beyond transfer mechanism the Second um beyond transfer mechanism the Second um beyond transfer mechanism the trade-off is around how to overlap trade-off is around how to overlap trade-off is around how to overlap compute memory and uh communication in compute memory and uh communication in compute memory and uh communication in GPU kernels. So there's two main GPU kernels. So there's two main GPU kernels. So there's two main categories of schedules. The first is categories of schedules. The first is categories of schedules. The first is intraSM within one of those rainbow dots intraSM within one of those rainbow dots intraSM within one of those rainbow dots um where we'll we'll have different um where we'll we'll have different um where we'll we'll have different warps or threads within that processor warps or threads within that processor warps or threads within that processor specialized to handle either compute or specialized to handle either compute or specialized to handle either compute or one specialized for communication one specialized for communication one specialized for communication concurrently. Um we can dedicate you concurrently. Um we can dedicate you concurrently. Um we can dedicate you know different warps to each of these. know different warps to each of these. know different warps to each of these. The challenge with this intm overlapping The challenge with this intm overlapping The challenge with this intm overlapping is that the communication and is that the communication and is that the communication and computation pattern really need to like computation pattern really need to like computation pattern really need to like align and jive with one another. they align and jive with one another. they align and jive with one another. they need to use the same data as inputs for need to use the same data as inputs for need to use the same data as inputs for the computation and communication. When the computation and communication. When the computation and communication. When they don't align, you could use they don't align, you could use they don't align, you could use something like interm um schedules that something like interm um schedules that something like interm um schedules that are shown on your right here where we'll are shown on your right here where we'll are shown on your right here where we'll now have each of the different rainbow now have each of the different rainbow now have each of the different rainbow colored dots on our GPUs, those colored dots on our GPUs, those colored dots on our GPUs, those different processors specialized to different processors specialized to different processors specialized to compute communication and memory. Um and compute communication and memory. Um and compute communication and memory. Um and so this this is nice when the colonel so this this is nice when the colonel so this this is nice when the colonel would others wise need to split across would others wise need to split across would others wise need to split across resources like the register file or resources like the register file or resources like the register file or shared memory um across these different shared memory um across these different shared memory um across these different steps in misaligned ways.

steps in misaligned ways. steps in misaligned ways. Um this is also really nice when it's Um this is also really nice when it's Um this is also really nice when it's hard to maximize NVLink traversal with hard to maximize NVLink traversal with hard to maximize NVLink traversal with intram intram intram overlapping. overlapping. overlapping. So I just wanted to highlight one quick So I just wanted to highlight one quick So I just wanted to highlight one quick example here where each of the patterns example here where each of the patterns example here where each of the patterns excels in popular you know AI uh uh like excels in popular you know AI uh uh like excels in popular you know AI uh uh like kind of patterns that you'll see. So on kind of patterns that you'll see. So on kind of patterns that you'll see. So on a gem plus uh reduced uh scatter here we a gem plus uh reduced uh scatter here we a gem plus uh reduced uh scatter here we can see that the um intraSM overlapping can see that the um intraSM overlapping can see that the um intraSM overlapping scheduleuler schedule is very effective scheduleuler schedule is very effective scheduleuler schedule is very effective in the gem plus all reduce we can see in the gem plus all reduce we can see in the gem plus all reduce we can see that the interSM which again leverages that the interSM which again leverages that the interSM which again leverages the in network reductions of envy switch the in network reductions of envy switch the in network reductions of envy switch is very effective. So we face these is very effective. So we face these is very effective. So we face these trade-offs and you can read more about trade-offs and you can read more about trade-offs and you can read more about um the design decisions that go into um the design decisions that go into um the design decisions that go into them in our parallel kittens paper. them in our parallel kittens paper. them in our parallel kittens paper. Um, and then finally, ideally, Um, and then finally, ideally, Um, and then finally, ideally, abstraction should allow the developer abstraction should allow the developer abstraction should allow the developer flexibility to control how they're flexibility to control how they're flexibility to control how they're buffering and synchronizing between data buffering and synchronizing between data buffering and synchronizing between data senders and receivers.

senders and receivers. senders and receivers. So, we encapsulated these ideas into So, we encapsulated these ideas into So, we encapsulated these ideas into parallel kittens. Again, a simple set of parallel kittens. Again, a simple set of parallel kittens. Again, a simple set of programming primitives and templates for programming primitives and templates for programming primitives and templates for these multiGPU kernels. Parallel Kittens these multiGPU kernels. Parallel Kittens these multiGPU kernels. Parallel Kittens is used in production at Together AI as is used in production at Together AI as is used in production at Together AI as well as our partner um you know Cursor well as our partner um you know Cursor well as our partner um you know Cursor and other uh companies in the AI space. and other uh companies in the AI space. and other uh companies in the AI space. Um here's some sample code. I won't Um here's some sample code. I won't Um here's some sample code. I won't spend too much time here, but we usually spend too much time here, but we usually spend too much time here, but we usually add roughly a dozen lines of code over a add roughly a dozen lines of code over a add roughly a dozen lines of code over a single GPU kernel to insert these single GPU kernel to insert these single GPU kernel to insert these multiGPU primitives. multiGPU primitives. multiGPU primitives. Um, and you can see here across data Um, and you can see here across data Um, and you can see here across data sequence and expert parallelism how our sequence and expert parallelism how our sequence and expert parallelism how our parallel kittens kernels are achieving parallel kittens kernels are achieving parallel kittens kernels are achieving state-of-the-art results um, compared to state-of-the-art results um, compared to state-of-the-art results um, compared to strong reference baselines. And you can strong reference baselines. And you can strong reference baselines. And you can check out our repo to learn more. check out our repo to learn more. check out our repo to learn more. Okay, so we understand a little bit Okay, so we understand a little bit Okay, so we understand a little bit about um the trade-offs that underly about um the trade-offs that underly about um the trade-offs that underly these multiGPU kernels and there's just these multiGPU kernels and there's just these multiGPU kernels and there's just a couple main, you know, ones that a couple main, you know, ones that a couple main, you know, ones that exist. So can models reason through them exist. So can models reason through them exist. So can models reason through them um and give us these uh you know kernels um and give us these uh you know kernels um and give us these uh you know kernels models are getting better at reasoning models are getting better at reasoning models are getting better at reasoning today. Um do they generalize well to today. Um do they generalize well to today. Um do they generalize well to these problems or are we benchmaxed on these problems or are we benchmaxed on these problems or are we benchmaxed on you know benchmarks of the past which you know benchmarks of the past which you know benchmarks of the past which are more single GPU centric are more single GPU centric are more single GPU centric uh models right now are showing really uh models right now are showing really uh models right now are showing really promising results on single GPU uh promising results on single GPU uh promising results on single GPU uh benchmarks. So it's a ripe time to to benchmarks. So it's a ripe time to to benchmarks. So it's a ripe time to to extend it.

extend it. extend it. In our benchmark parallel kernel bench, In our benchmark parallel kernel bench, In our benchmark parallel kernel bench, each task presents the model with an each task presents the model with an each task presents the model with an unoptimized reference implementation unoptimized reference implementation unoptimized reference implementation written in PyTorch with torch written in PyTorch with torch written in PyTorch with torch distributed um nickel operations and distributed um nickel operations and distributed um nickel operations and then a system topology that specifies then a system topology that specifies then a system topology that specifies the number of ranks and intraode the number of ranks and intraode the number of ranks and intraode hardware configuration. And the model hardware configuration. And the model hardware configuration. And the model needs to rewrite the reference into a needs to rewrite the reference into a needs to rewrite the reference into a performance CUDA kernel that uh uses performance CUDA kernel that uh uses performance CUDA kernel that uh uses unified virtual addressing. The um multiGPU problem space expands The um multiGPU problem space expands combinatorally beyond single GPU um combinatorally beyond single GPU um combinatorally beyond single GPU um cases. So a standard transformer layer cases. So a standard transformer layer cases. So a standard transformer layer can be parallelized across data can be parallelized across data can be parallelized across data sequence, tensor, context, layer, sequence, tensor, context, layer, sequence, tensor, context, layer, pipeline and expert dimensions. and each pipeline and expert dimensions. and each pipeline and expert dimensions. and each composition um induces a different composition um induces a different composition um induces a different communication pass uh pattern. So to communication pass uh pattern. So to communication pass uh pattern. So to make sure that our benchmark has high make sure that our benchmark has high make sure that our benchmark has high coverage over the representative types coverage over the representative types coverage over the representative types of multiGPU problems um we created this of multiGPU problems um we created this of multiGPU problems um we created this taxonomy um that you can read more about taxonomy um that you can read more about taxonomy um that you can read more about in our paper and then picked in our paper and then picked in our paper and then picked representative problems for each part of representative problems for each part of representative problems for each part of the taxonomy.

the taxonomy. the taxonomy. Um these are all patterns that arise in Um these are all patterns that arise in Um these are all patterns that arise in real AI workloads from inference to RL real AI workloads from inference to RL real AI workloads from inference to RL to post-training. to post-training. to post-training. There are 87 problems overall um drawn There are 87 problems overall um drawn There are 87 problems overall um drawn from GitHub repositories that we found from GitHub repositories that we found from GitHub repositories that we found to be very informative. Um and we uh and to be very informative. Um and we uh and to be very informative. Um and we uh and like optimized library implementations like optimized library implementations like optimized library implementations and DSLs that people have written and DSLs that people have written and DSLs that people have written multiGPU kernels in. multiGPU kernels in. multiGPU kernels in. Um, we wanted to really make sure that Um, we wanted to really make sure that Um, we wanted to really make sure that solving PK this parallel kernel bench solving PK this parallel kernel bench solving PK this parallel kernel bench would lead to net new useful production would lead to net new useful production would lead to net new useful production kernels rather than artificial or kernels rather than artificial or kernels rather than artificial or useless kernels. useless kernels. useless kernels. Okay, so that's parallel kernel bench. Okay, so that's parallel kernel bench. Okay, so that's parallel kernel bench. How do models perform? So we measured to sorry that this is a So we measured to sorry that this is a bit small. We measured two main metrics. bit small. We measured two main metrics. bit small. We measured two main metrics. Um pass at K which is the number of Um pass at K which is the number of Um pass at K which is the number of correct kernels generated after K correct kernels generated after K correct kernels generated after K attempts and then fast um one at K which attempts and then fast um one at K which attempts and then fast um one at K which counts solutions that are both correct counts solutions that are both correct counts solutions that are both correct and outperform the speed of the pi torch and outperform the speed of the pi torch and outperform the speed of the pi torch plus nickel baseline.

plus nickel baseline. plus nickel baseline. Um so pass K just correctness fast one Um so pass K just correctness fast one Um so pass K just correctness fast one at K is whether you're getting a 1x or at K is whether you're getting a 1x or at K is whether you're getting a 1x or higher speed up over the reference. So higher speed up over the reference. So higher speed up over the reference. So performanceoriented performanceoriented performanceoriented um we found that in the zeroot setting um we found that in the zeroot setting um we found that in the zeroot setting the best of the frontier models we tried the best of the frontier models we tried the best of the frontier models we tried solves 28 out of 87 problems and 22 of solves 28 out of 87 problems and 22 of solves 28 out of 87 problems and 22 of those problems are faster than the those problems are faster than the those problems are faster than the pietorch plus nickel baseline. pietorch plus nickel baseline. pietorch plus nickel baseline. If we make multiple samples, you know, If we make multiple samples, you know, If we make multiple samples, you know, standard uh scaling up test time standard uh scaling up test time standard uh scaling up test time compute, we can uh get that number from compute, we can uh get that number from compute, we can uh get that number from say like to to 36 correct solutions, but say like to to 36 correct solutions, but say like to to 36 correct solutions, but the fast uh one performance still the fast uh one performance still the fast uh one performance still plateaus out at roughly 31%. So we don't plateaus out at roughly 31%. So we don't plateaus out at roughly 31%. So we don't see much room from continuing to scale see much room from continuing to scale see much room from continuing to scale there as we increase the number of there as we increase the number of there as we increase the number of parallel generations. parallel generations. parallel generations. Um we find that the correct once Um we find that the correct once Um we find that the correct once correctness is established um speedups correctness is established um speedups correctness is established um speedups naturally come from eliminating nickel naturally come from eliminating nickel naturally come from eliminating nickel staging overhead in favor of direct staging overhead in favor of direct staging overhead in favor of direct NVLink loads in stores.

NVLink loads in stores. NVLink loads in stores. Um the success patterns here are really Um the success patterns here are really Um the success patterns here are really concentrated into familiar patterns. So concentrated into familiar patterns. So concentrated into familiar patterns. So collective primitives, tensor parallel collective primitives, tensor parallel collective primitives, tensor parallel gems and Ulyses style context gems and Ulyses style context gems and Ulyses style context parallelism. So in other words, patterns parallelism. So in other words, patterns parallelism. So in other words, patterns that we see heavily represented um on that we see heavily represented um on that we see heavily represented um on the internet rather than necessarily the internet rather than necessarily the internet rather than necessarily patterns that the model has used its patterns that the model has used its patterns that the model has used its reasoning abilities to think through. Okay, even the best available model that Okay, even the best available model that we benchmarked here GPT 5.5 drops off we benchmarked here GPT 5.5 drops off we benchmarked here GPT 5.5 drops off very quickly as the speed up threshold very quickly as the speed up threshold very quickly as the speed up threshold increases. So on the x-axis here, we're increases. So on the x-axis here, we're increases. So on the x-axis here, we're increasing the speed up threshold over increasing the speed up threshold over increasing the speed up threshold over that pietor torch plus nickel baseline. that pietor torch plus nickel baseline. that pietor torch plus nickel baseline. And then we're showing the number of And then we're showing the number of And then we're showing the number of correct kernels that are faster than correct kernels that are faster than correct kernels that are faster than that baseline or this much faster than that baseline or this much faster than that baseline or this much faster than the baseline on the y-axis. the baseline on the y-axis. the baseline on the y-axis. So GPT 5.5 is this orange line here and So GPT 5.5 is this orange line here and So GPT 5.5 is this orange line here and then DeepSeek V4 Pro is the aqua line at then DeepSeek V4 Pro is the aqua line at then DeepSeek V4 Pro is the aqua line at the bottom.

um we found that there's deeper issues um we found that there's deeper issues than CUDA syntax. So we found that if than CUDA syntax. So we found that if than CUDA syntax. So we found that if you do multiple sampling or have the you do multiple sampling or have the you do multiple sampling or have the model kind of look at its errors and model kind of look at its errors and model kind of look at its errors and correct them, it can often compile the correct them, it can often compile the correct them, it can often compile the kernels. But the models really struggle kernels. But the models really struggle kernels. But the models really struggle to reason through the tradeoffs that we to reason through the tradeoffs that we to reason through the tradeoffs that we talked about in the prior section. um talked about in the prior section. um talked about in the prior section. um collective ordering, data partitioning, collective ordering, data partitioning, collective ordering, data partitioning, thinking about intra versus interm thinking about intra versus interm thinking about intra versus interm scheduling or deciding between the scheduling or deciding between the scheduling or deciding between the different transfer mechanisms. Um we different transfer mechanisms. Um we different transfer mechanisms. Um we find that they often do not use things find that they often do not use things find that they often do not use things like the register transfer instructions like the register transfer instructions like the register transfer instructions or tensor memory acceleration when or tensor memory acceleration when or tensor memory acceleration when writing the kernels. writing the kernels. writing the kernels. Um we wanted to try a pretty simple Um we wanted to try a pretty simple Um we wanted to try a pretty simple instantiation of something like a clawed instantiation of something like a clawed instantiation of something like a clawed code um coding agent. So, we took uh the code um coding agent. So, we took uh the code um coding agent. So, we took uh the mini sui agent multi-turn harness and mini sui agent multi-turn harness and mini sui agent multi-turn harness and one of the best performing models, one of the best performing models, one of the best performing models, Gemini 3 Pro, and gave it access to a Gemini 3 Pro, and gave it access to a Gemini 3 Pro, and gave it access to a local bash environment to sort of mimic local bash environment to sort of mimic local bash environment to sort of mimic the standard claude code setup. We found the standard claude code setup. We found the standard claude code setup. We found that this could help the agent um go that this could help the agent um go that this could help the agent um go from solving 24 problems to 35 of 87 from solving 24 problems to 35 of 87 from solving 24 problems to 35 of 87 problems um with um 26 achieving over a problems um with um 26 achieving over a problems um with um 26 achieving over a 1x speed up over the reference. But we 1x speed up over the reference. But we 1x speed up over the reference. But we found that as we scaled the amount of found that as we scaled the amount of found that as we scaled the amount of time um the performance plateaued as uh time um the performance plateaued as uh time um the performance plateaued as uh and we find that additional techniques and we find that additional techniques and we find that additional techniques would be required to continue seeing the would be required to continue seeing the would be required to continue seeing the scaling there. Again these results are scaling there. Again these results are scaling there. Again these results are discussed in more detail in our paper.

discussed in more detail in our paper. discussed in more detail in our paper. Um we think this is a really exciting Um we think this is a really exciting Um we think this is a really exciting you know just to wrap up here we hope you know just to wrap up here we hope you know just to wrap up here we hope that people out here can use both that people out here can use both that people out here can use both parallel kittens and parallel kernel parallel kittens and parallel kernel parallel kittens and parallel kernel bench. We think that the kernels bench. We think that the kernels bench. We think that the kernels generated from solving parallel kernel generated from solving parallel kernel generated from solving parallel kernel bench will lead to net new production bench will lead to net new production bench will lead to net new production kernels that are important bottlenecks kernels that are important bottlenecks kernels that are important bottlenecks for inference in RL right now. um for inference in RL right now. um for inference in RL right now. um they're you know we tried our best to they're you know we tried our best to they're you know we tried our best to make them you know non-artificial and we make them you know non-artificial and we make them you know non-artificial and we can already see signs of life and can already see signs of life and can already see signs of life and exciting results where people have not exciting results where people have not exciting results where people have not invested a bunch of time to handw write invested a bunch of time to handw write invested a bunch of time to handw write a multiGPU kernel and we've gotten some a multiGPU kernel and we've gotten some a multiGPU kernel and we've gotten some net new interesting ones like this Nemo net new interesting ones like this Nemo net new interesting ones like this Nemo vocab parallel um you know filtering vocab parallel um you know filtering vocab parallel um you know filtering kernel um a hyena architecture context kernel um a hyena architecture context kernel um a hyena architecture context parallelism kernel parallelism kernel parallelism kernel and uh the SAM 3 video segmentation and uh the SAM 3 video segmentation and uh the SAM 3 video segmentation model um IOU suppression kernel. model um IOU suppression kernel. model um IOU suppression kernel. Just to conclude here um we're really Just to conclude here um we're really Just to conclude here um we're really excited about um how uh well just excited about um how uh well just excited about um how uh well just talking about the lessons. First off, we talking about the lessons. First off, we talking about the lessons. First off, we think there aren't that many patterns think there aren't that many patterns think there aren't that many patterns that are involved in writing intragpu that are involved in writing intragpu that are involved in writing intragpu effective kernels again encapsulated by effective kernels again encapsulated by effective kernels again encapsulated by our small set of programming primitives.

our small set of programming primitives. our small set of programming primitives. Um, but unfortunately models do not Um, but unfortunately models do not Um, but unfortunately models do not currently understand how to reason currently understand how to reason currently understand how to reason through these trade-offs even when we through these trade-offs even when we through these trade-offs even when we provide them in context. Um, we're provide them in context. Um, we're provide them in context. Um, we're really excited about methods that can really excited about methods that can really excited about methods that can help attack this benchmark. We're help attack this benchmark. We're help attack this benchmark. We're excited about architectures that can excited about architectures that can excited about architectures that can grow with the trends of how networking grow with the trends of how networking grow with the trends of how networking stacks are evolving. Um, you know, stacks are evolving. Um, you know, stacks are evolving. Um, you know, larger scale up domains shift away from larger scale up domains shift away from larger scale up domains shift away from scale out um and massive onchip memory scale out um and massive onchip memory scale out um and massive onchip memory structures. And we hope that uh you know structures. And we hope that uh you know structures. And we hope that uh you know we can also extend and you you can feel we can also extend and you you can feel we can also extend and you you can feel free to reach out to me at my email. free to reach out to me at my email. free to reach out to me at my email. Thanks.