Your Mac is slowing you down (a rant about file systems)

I've been doing a lot of work with a lot I've been doing a lot of work with a lot of agents recently. And the more work I of agents recently. And the more work I of agents recently. And the more work I do, the more frustrated I get with a do, the more frustrated I get with a do, the more frustrated I get with a bunch of things that have always been bunch of things that have always been bunch of things that have always been problems, but now are problems more than problems, but now are problems more than problems, but now are problems more than ever. A lot of those are centered around ever. A lot of those are centered around ever. A lot of those are centered around this thing right here, my very this thing right here, my very this thing right here, my very unnecessarily overpriced MacBook. And as unnecessarily overpriced MacBook. And as unnecessarily overpriced MacBook. And as great as the hardware is, as nice as it great as the hardware is, as nice as it great as the hardware is, as nice as it is to use this computer, I've been is to use this computer, I've been is to use this computer, I've been finding a lot of problems when I try to finding a lot of problems when I try to finding a lot of problems when I try to run agents on it. I've talked a lot run agents on it. I've talked a lot run agents on it. I've talked a lot about those problems in previous videos about those problems in previous videos about those problems in previous videos about my move to Linux and my whole about my move to Linux and my whole about my move to Linux and my whole setup and how I've been coding, but I setup and how I've been coding, but I setup and how I've been coding, but I want to really fixate on one of them want to really fixate on one of them want to really fixate on one of them today because it's one that keeps coming today because it's one that keeps coming today because it's one that keeps coming up and I have not been able to find any up and I have not been able to find any up and I have not been able to find any definitive research about it. So, I definitive research about it. So, I definitive research about it. So, I decided to sit and do it myself. This decided to sit and do it myself. This decided to sit and do it myself. This might be a weird screen to cut to, but I might be a weird screen to cut to, but I might be a weird screen to cut to, but I want you to notice one particular thing. want you to notice one particular thing. want you to notice one particular thing. The numbers at the bottom. This run of The numbers at the bottom. This run of The numbers at the bottom. This run of deleting a bunch of folders from this deleting a bunch of folders from this deleting a bunch of folders from this project took over 35 seconds on my Mac. project took over 35 seconds on my Mac. project took over 35 seconds on my Mac. That same run took under 7 and 12 That same run took under 7 and 12 That same run took under 7 and 12 seconds on my Linux machine. You might seconds on my Linux machine. You might seconds on my Linux machine. You might think, "Oh yeah, that Linux machine is think, "Oh yeah, that Linux machine is think, "Oh yeah, that Linux machine is way more powerful then." Nope. Is drive way more powerful then." Nope. Is drive way more powerful then." Nope. Is drive faster? Absolutely not. The Mac drive is faster? Absolutely not. The Mac drive is faster? Absolutely not. The Mac drive is way faster. Is Linux's system just that way faster. Is Linux's system just that way faster. Is Linux's system just that much better than Mac OS? Kind of. But much better than Mac OS? Kind of. But much better than Mac OS? Kind of. But this is a measure of a very different this is a measure of a very different this is a measure of a very different thing. Let me show you with a real real thing. Let me show you with a real real thing. Let me show you with a real real brief demo before we go into the brief demo before we go into the brief demo before we go into the details. I'm going to time a PNPM details. I'm going to time a PNPM details. I'm going to time a PNPM install on this project on my Mac install on this project on my Mac install on this project on my Mac as well as on my Linux box.

as well as on my Linux box. as well as on my Linux box. You might already notice a difference You might already notice a difference You might already notice a difference between the two. The whole Linux box was between the two. The whole Linux box was between the two. The whole Linux box was able to finish the install and all of able to finish the install and all of able to finish the install and all of the weird post install stuff in under 10 the weird post install stuff in under 10 the weird post install stuff in under 10 seconds. And my Mac is under halfway seconds. And my Mac is under halfway seconds. And my Mac is under halfway through. through. through. This will be going for a bit. So, while This will be going for a bit. So, while This will be going for a bit. So, while I wait for this, I'll let you know that I wait for this, I'll let you know that I wait for this, I'll let you know that today's video is about file systems. One today's video is about file systems. One today's video is about file systems. One of my favorite topics I almost never get of my favorite topics I almost never get of my favorite topics I almost never get to talk about. And hopefully by the time to talk about. And hopefully by the time to talk about. And hopefully by the time this ad is done, this run will be done this ad is done, this run will be done this ad is done, this run will be done as well. I have to complain a little bit as well. I have to complain a little bit as well. I have to complain a little bit about AI code. It's crazy how powerful about AI code. It's crazy how powerful about AI code. It's crazy how powerful AI is at writing code when you're trying AI is at writing code when you're trying AI is at writing code when you're trying to actually get it merged is just to actually get it merged is just to actually get it merged is just miserable to deal with. I am so tired of miserable to deal with. I am so tired of miserable to deal with. I am so tired of these giant PRs. It feels like these giant PRs. It feels like these giant PRs. It feels like everything anyone's trying to merge is everything anyone's trying to merge is everything anyone's trying to merge is now over a thousand lines and getting now over a thousand lines and getting now over a thousand lines and getting through it is just it's miserable. It's through it is just it's miserable. It's through it is just it's miserable. It's not pleasant at all. Especially as not pleasant at all. Especially as not pleasant at all. Especially as things start to change, trying to find things start to change, trying to find things start to change, trying to find the things that matter and make sure the things that matter and make sure the things that matter and make sure they work properly. It's obnoxious. And they work properly. It's obnoxious. And they work properly. It's obnoxious. And sure, we do have AI code review with sure, we do have AI code review with sure, we do have AI code review with tools like Code Rabbit, and it helps a tools like Code Rabbit, and it helps a tools like Code Rabbit, and it helps a bit, but it doesn't help with the human bit, but it doesn't help with the human bit, but it doesn't help with the human problem here of actually having to look problem here of actually having to look problem here of actually having to look at the code if you want to have any at the code if you want to have any at the code if you want to have any confidence at all about what shipping. confidence at all about what shipping. confidence at all about what shipping. How do I even know what the state of How do I even know what the state of How do I even know what the state of this PR is with 500 bots updating the this PR is with 500 bots updating the this PR is with 500 bots updating the description and everything else in it description and everything else in it description and everything else in it constantly? It's just not a good surface constantly? It's just not a good surface constantly? It's just not a good surface for humans. Well, thankfully, today's for humans. Well, thankfully, today's for humans. Well, thankfully, today's sponsor is trying to solve this. You've sponsor is trying to solve this. You've sponsor is trying to solve this. You've already heard them. It's Code Rabbit.

already heard them. It's Code Rabbit. already heard them. It's Code Rabbit. What you might not know about is the new What you might not know about is the new What you might not know about is the new change stack that they just shipped. change stack that they just shipped. change stack that they just shipped. It's an allnew UI for dealing with your It's an allnew UI for dealing with your It's an allnew UI for dealing with your pull requests, and it's so much better. pull requests, and it's so much better. pull requests, and it's so much better. It all starts with the overview, which It all starts with the overview, which It all starts with the overview, which tells you exactly what's going on and tells you exactly what's going on and tells you exactly what's going on and what you should care about. In what you should care about. In what you should care about. In particular, it calls out what the particular, it calls out what the particular, it calls out what the blockers are currently. In this case, blockers are currently. In this case, blockers are currently. In this case, this PR is blocked as a human hasn't this PR is blocked as a human hasn't this PR is blocked as a human hasn't reviewed yet. Convenient, cuz I'm here reviewed yet. Convenient, cuz I'm here reviewed yet. Convenient, cuz I'm here now to review it. If you want to know now to review it. If you want to know now to review it. If you want to know how the PR got where it is, there's this how the PR got where it is, there's this how the PR got where it is, there's this awesome timeline view that shows awesome timeline view that shows awesome timeline view that shows everything that changed over time. everything that changed over time. everything that changed over time. incredibly helpful to see what made a incredibly helpful to see what made a incredibly helpful to see what made a change, why it made the change, what change, why it made the change, what change, why it made the change, what other things are going on. In this case, other things are going on. In this case, other things are going on. In this case, all the Verselb bot deploys as well. all the Verselb bot deploys as well. all the Verselb bot deploys as well. Super handy. And if anything's unclear, Super handy. And if anything's unclear, Super handy. And if anything's unclear, you can click this little agent button, you can click this little agent button, you can click this little agent button, which will pull you into a chat where which will pull you into a chat where which will pull you into a chat where you can ask questions about the PR. The you can ask questions about the PR. The you can ask questions about the PR. The thing that really makes this magic thing that really makes this magic thing that really makes this magic though is the stack itself. Instead of though is the stack itself. Instead of though is the stack itself. Instead of just listing all of the code in just listing all of the code in just listing all of the code in alphabetical order, it breaks it up into alphabetical order, it breaks it up into alphabetical order, it breaks it up into logical stacks that are categorized logical stacks that are categorized logical stacks that are categorized based on the specific things a PR based on the specific things a PR based on the specific things a PR changes. This PR is a huge change to changes. This PR is a huge change to changes. This PR is a huge change to navigation in T3 chat and actually navigation in T3 chat and actually navigation in T3 chat and actually reading through it is hellish when reading through it is hellish when reading through it is hellish when you're just going through an order you're just going through an order you're just going through an order splitting the back end, the front end splitting the back end, the front end splitting the back end, the front end and the actual parts that bind to the UI and the actual parts that bind to the UI and the actual parts that bind to the UI obnoxious. Now we just have the API and obnoxious. Now we just have the API and obnoxious. Now we just have the API and the pageionation piece, the navigation the pageionation piece, the navigation the pageionation piece, the navigation on the UI on the rail side, as well as on the UI on the rail side, as well as on the UI on the rail side, as well as the loading changes to the way the data the loading changes to the way the data the loading changes to the way the data is being fetched. This is so much easier is being fetched. This is so much easier is being fetched. This is so much easier to get through. It's insane. Stop guilt to get through. It's insane. Stop guilt to get through. It's insane. Stop guilt merging code and start understanding it merging code and start understanding it merging code and start understanding it again at soyb.link/codrabbit.

again at soyb.link/codrabbit. again at soyb.link/codrabbit. 35 seconds later, I've now finished my 35 seconds later, I've now finished my 35 seconds later, I've now finished my installs, which aren't even going over installs, which aren't even going over installs, which aren't even going over the network. By the way, all of those the network. By the way, all of those the network. By the way, all of those PNPM installs were hitting caches on PNPM installs were hitting caches on PNPM installs were hitting caches on this computer. It was just linking to this computer. It was just linking to this computer. It was just linking to existing files and folders on the existing files and folders on the existing files and folders on the machine. And somehow that took almost 35 machine. And somehow that took almost 35 machine. And somehow that took almost 35 seconds when on Linux it consistently seconds when on Linux it consistently seconds when on Linux it consistently takes under 10. And the majority of that takes under 10. And the majority of that takes under 10. And the majority of that time was running post install scripts, time was running post install scripts, time was running post install scripts, not even just doing the install itself. not even just doing the install itself. not even just doing the install itself. That is brutal. That is absolutely That is brutal. That is absolutely That is brutal. That is absolutely brutal. It almost all of this comes down brutal. It almost all of this comes down brutal. It almost all of this comes down to file systems. To Apple's credit, they to file systems. To Apple's credit, they to file systems. To Apple's credit, they have done incredible work with file have done incredible work with file have done incredible work with file system stuff in the past. Their move system stuff in the past. Their move system stuff in the past. Their move from the old journal file system to APFS from the old journal file system to APFS from the old journal file system to APFS is truly legendary. They were able to is truly legendary. They were able to is truly legendary. They were able to successfully port every iPhone's file successfully port every iPhone's file successfully port every iPhone's file system over with a random DOT update system over with a random DOT update system over with a random DOT update with iOS and nobody noticed. Nobody's with iOS and nobody noticed. Nobody's with iOS and nobody noticed. Nobody's files went corrupt. They just partially files went corrupt. They just partially files went corrupt. They just partially wrote the data from one partition to wrote the data from one partition to wrote the data from one partition to another until they could move everything another until they could move everything another until they could move everything over safely, quietly during a random over safely, quietly during a random over safely, quietly during a random iPhone update. Genuinely unbelievable.

iPhone update. Genuinely unbelievable. iPhone update. Genuinely unbelievable. Apple's file system team clearly knows Apple's file system team clearly knows Apple's file system team clearly knows what they're doing, but it does seem what they're doing, but it does seem what they're doing, but it does seem like their focus is not in the best like their focus is not in the best like their focus is not in the best place right now, especially for us as place right now, especially for us as place right now, especially for us as we've been doing more agentic dev work we've been doing more agentic dev work we've been doing more agentic dev work that involves lots of cloning of big that involves lots of cloning of big that involves lots of cloning of big directories with tons of small files, directories with tons of small files, directories with tons of small files, cloning our work trees all around the cloning our work trees all around the cloning our work trees all around the machine, having to do npm installs and machine, having to do npm installs and machine, having to do npm installs and copying all these modules to different copying all these modules to different copying all these modules to different places. The way that I work right now places. The way that I work right now places. The way that I work right now does not benefit meaningfully from the does not benefit meaningfully from the does not benefit meaningfully from the compromises that Mac OS made with APFS compromises that Mac OS made with APFS compromises that Mac OS made with APFS to the point where it is getting really to the point where it is getting really to the point where it is getting really really annoying. Novox popular made a really annoying. Novox popular made a really annoying. Novox popular made a bench that I've been showing in videos bench that I've been showing in videos bench that I've been showing in videos quite a bit now that really highlights quite a bit now that really highlights quite a bit now that really highlights the problem. This is a disk performance the problem. This is a disk performance the problem. This is a disk performance test with git and pnpm. There's a bunch test with git and pnpm. There's a bunch test with git and pnpm. There's a bunch of projects boiler plates for various of projects boiler plates for various of projects boiler plates for various frameworks in the web dev world just frameworks in the web dev world just frameworks in the web dev world just like the classic boiler plate for MJS like the classic boiler plate for MJS like the classic boiler plate for MJS classic Ember Gatsby next all these classic Ember Gatsby next all these classic Ember Gatsby next all these different things including T3 by the way different things including T3 by the way different things including T3 by the way and the repo has the boiler plate for and the repo has the boiler plate for and the repo has the boiler plate for all of these things and the bench is all of these things and the bench is all of these things and the bench is simple it's a monor repo so you install simple it's a monor repo so you install simple it's a monor repo so you install all the npm modules and then you also all the npm modules and then you also all the npm modules and then you also bench how long it takes to delete them bench how long it takes to delete them bench how long it takes to delete them all when you run the get clean command all when you run the get clean command all when you run the get clean command and you can immediately see some scary and you can immediately see some scary and you can immediately see some scary numbers here the left is the amount of numbers here the left is the amount of numbers here the left is the amount of time it takes to do the clean task where time it takes to do the clean task where time it takes to do the clean task where it deletes things and the right is the it deletes things and the right is the it deletes things and the right is the install task where it actually installs install task where it actually installs install task where it actually installs the modules remember from a cache so the modules remember from a cache so the modules remember from a cache so it's just copy paste and we could see it's just copy paste and we could see it's just copy paste and we could see here with a meaningfully beefy AMD Ryzen here with a meaningfully beefy AMD Ryzen here with a meaningfully beefy AMD Ryzen machine this 1224 machine running ext4

machine this 1224 machine running ext4 machine this 1224 machine running ext4 which is the standard file system in which is the standard file system in which is the standard file system in Ubuntu it took 6 seconds to clean and 4 Ubuntu it took 6 seconds to clean and 4 Ubuntu it took 6 seconds to clean and 4 and a half seconds not even to install and a half seconds not even to install and a half seconds not even to install an M1 Max took 31 seconds to clean and an M1 Max took 31 seconds to clean and an M1 Max took 31 seconds to clean and 44 to inst install. That is a 5x 44 to inst install. That is a 5x 44 to inst install. That is a 5x increase in time for the deletions and a increase in time for the deletions and a increase in time for the deletions and a 10x increase in time for the 10x increase in time for the 10x increase in time for the installation. So, what the [ __ ] is going installation. So, what the [ __ ] is going installation. So, what the [ __ ] is going on? Well, I'm already seeing some on? Well, I'm already seeing some on? Well, I'm already seeing some questions I feel like I have to address, questions I feel like I have to address, questions I feel like I have to address, but I want to open with a question. What but I want to open with a question. What but I want to open with a question. What is the best file system for working with is the best file system for working with is the best file system for working with agents? APFS is a file system. It's a agents? APFS is a file system. It's a agents? APFS is a file system. It's a way that files are actually stored and way that files are actually stored and way that files are actually stored and managed on your drive because your SSD managed on your drive because your SSD managed on your drive because your SSD doesn't know what a folder is. It just doesn't know what a folder is. It just doesn't know what a folder is. It just holds ones and zeros. So you need a holds ones and zeros. So you need a holds ones and zeros. So you need a protocol to track what's being stored, protocol to track what's being stored, protocol to track what's being stored, what it means, how it's managed, how what it means, how it's managed, how what it means, how it's managed, how it's rewritten, how it's linked, and all it's rewritten, how it's linked, and all it's rewritten, how it's linked, and all these different things. Your computer these different things. Your computer these different things. Your computer has all these files and folders, but I has all these files and folders, but I has all these files and folders, but I really need you guys to understand all really need you guys to understand all really need you guys to understand all of this [ __ ] is imaginary. These don't of this [ __ ] is imaginary. These don't of this [ __ ] is imaginary. These don't actually exist on your drive in the actually exist on your drive in the actually exist on your drive in the sense that there is a section of your sense that there is a section of your sense that there is a section of your drive that is your work folder or your drive that is your work folder or your drive that is your work folder or your play folder. There is just some metadata play folder. There is just some metadata play folder. There is just some metadata that tracks what is where. And then you that tracks what is where. And then you that tracks what is where. And then you have all the data just filling out the have all the data just filling out the have all the data just filling out the blocks in your drive and some metadata blocks in your drive and some metadata blocks in your drive and some metadata layer that tracks what files are where layer that tracks what files are where layer that tracks what files are where on that drive and gives you the illusion on that drive and gives you the illusion on that drive and gives you the illusion of folders on your machine. And as there of folders on your machine. And as there of folders on your machine. And as there are with all things in the software are with all things in the software are with all things in the software development world, there are a shitload development world, there are a shitload development world, there are a shitload of options for your file system.

of options for your file system. of options for your file system. Generally speaking, Windows uses NTFS. Generally speaking, Windows uses NTFS. Generally speaking, Windows uses NTFS. Mac OS used to use Journal, but now it Mac OS used to use Journal, but now it Mac OS used to use Journal, but now it uses APFS, which is the Apple file uses APFS, which is the Apple file uses APFS, which is the Apple file system. Currently, Apple considered ZFS system. Currently, Apple considered ZFS system. Currently, Apple considered ZFS for the file system in Mac at some for the file system in Mac at some for the file system in Mac at some point, but this never ended up point, but this never ended up point, but this never ended up happening. They ended up spinning in happening. They ended up spinning in happening. They ended up spinning in circles on their [ __ ] until circles on their [ __ ] until circles on their [ __ ] until eventually making APFS much much later. eventually making APFS much much later. eventually making APFS much much later. Also, the technical name for their old Also, the technical name for their old Also, the technical name for their old journal was HFS Plus, their old [ __ ] journal was HFS Plus, their old [ __ ] journal was HFS Plus, their old [ __ ] up annoying standard. APFS had up annoying standard. APFS had up annoying standard. APFS had meaningful improvements, but still. meaningful improvements, but still. meaningful improvements, but still. But then we have Linux, where, as you But then we have Linux, where, as you But then we have Linux, where, as you can guess, nearly infinite set of can guess, nearly infinite set of can guess, nearly infinite set of options. ext4, options. ext4, options. ext4, XFS, ZFS, XFS, ZFS, XFS, ZFS, butterfs, tons of others, too. These are butterfs, tons of others, too. These are butterfs, tons of others, too. These are the most important ones for sure. God the most important ones for sure. God the most important ones for sure. God damn, there's a lot. And remember, all damn, there's a lot. And remember, all damn, there's a lot. And remember, all of these are almost like different of these are almost like different of these are almost like different programming languages in a way where programming languages in a way where programming languages in a way where they are different ways to interpret the they are different ways to interpret the they are different ways to interpret the thing that your machine will do because thing that your machine will do because thing that your machine will do because in the end, all programming languages in the end, all programming languages in the end, all programming languages have to map to instructions in assembly.

have to map to instructions in assembly. have to map to instructions in assembly. Whether they're doing that through a Whether they're doing that through a Whether they're doing that through a runtime, a compiler, or many other runtime, a compiler, or many other runtime, a compiler, or many other things, that's how they do stuff. File things, that's how they do stuff. File things, that's how they do stuff. File systems are similar where in the end it systems are similar where in the end it systems are similar where in the end it all has to map to ones and zeros stored all has to map to ones and zeros stored all has to map to ones and zeros stored on your actual drive. It's a matter of on your actual drive. It's a matter of on your actual drive. It's a matter of how that mapping works and all of the how that mapping works and all of the how that mapping works and all of the layers between the actual content of the layers between the actual content of the layers between the actual content of the drive and the thing that you're drive and the thing that you're drive and the thing that you're interfacing with. These have to answer interfacing with. These have to answer interfacing with. These have to answer questions like what happens when you questions like what happens when you questions like what happens when you delete a file? What happens when you delete a file? What happens when you delete a file? What happens when you copy a file? when you move a file from copy a file? when you move a file from copy a file? when you move a file from one place to another, when you delete one place to another, when you delete one place to another, when you delete all but one copy of a file that's linked all but one copy of a file that's linked all but one copy of a file that's linked all over your drive. All these types of all over your drive. All these types of all over your drive. All these types of things are complex problems that your things are complex problems that your things are complex problems that your file system needs to handle. And then file system needs to handle. And then file system needs to handle. And then you get into the crazy world of RAID you get into the crazy world of RAID you get into the crazy world of RAID where you have drives redundantly where you have drives redundantly where you have drives redundantly storing the same data. So if one fails, storing the same data. So if one fails, storing the same data. So if one fails, you still have it. Or maybe you're just you still have it. Or maybe you're just you still have it. Or maybe you're just using RAID for performance or expanding using RAID for performance or expanding using RAID for performance or expanding your pool or all these other things. All your pool or all these other things. All your pool or all these other things. All these different file systems have these different file systems have these different file systems have different compromises around all of this different compromises around all of this different compromises around all of this stuff. And there's benefits and stuff. And there's benefits and stuff. And there's benefits and negatives to most of them. But I will negatives to most of them. But I will negatives to most of them. But I will just say out right now, Apple's file just say out right now, Apple's file just say out right now, Apple's file system does not have compromises that system does not have compromises that system does not have compromises that make sense for us as developers really make sense for us as developers really make sense for us as developers really at all. The performance is really bad.

at all. The performance is really bad. at all. The performance is really bad. And that's why we end up in this [ __ ] And that's why we end up in this [ __ ] And that's why we end up in this [ __ ] up, weird, annoying world where it takes up, weird, annoying world where it takes up, weird, annoying world where it takes over 30 seconds to do a few PNPM over 30 seconds to do a few PNPM over 30 seconds to do a few PNPM installs just copying files around my installs just copying files around my installs just copying files around my computer. And that same exact command computer. And that same exact command computer. And that same exact command happens in under 10 seconds on an okay happens in under 10 seconds on an okay happens in under 10 seconds on an okay Linux machine. And the clean is even Linux machine. And the clean is even Linux machine. And the clean is even more egregious. This clean command takes more egregious. This clean command takes more egregious. This clean command takes so long. It's really painful. There's a so long. It's really painful. There's a so long. It's really painful. There's a fun test that Novox did in his bench fun test that Novox did in his bench fun test that Novox did in his bench here where he actually throttled one of here where he actually throttled one of here where he actually throttled one of his Linux boxes to 550 megahertz and the his Linux boxes to 550 megahertz and the his Linux boxes to 550 megahertz and the resulting speeds were comparable to an resulting speeds were comparable to an resulting speeds were comparable to an M1 Max Apple chip. It was actually M1 Max Apple chip. It was actually M1 Max Apple chip. It was actually almost identical on the install speed. almost identical on the install speed. almost identical on the install speed. 550 megahertz throttle against an M1 Max 550 megahertz throttle against an M1 Max 550 megahertz throttle against an M1 Max chip and the results were similar chip and the results were similar chip and the results were similar speeds. That is terrible. This is very speeds. That is terrible. This is very speeds. That is terrible. This is very much my opinion from my day-to-day much my opinion from my day-to-day much my opinion from my day-to-day experience building with agents in experience building with agents in experience building with agents in parallel heavily across machines. parallel heavily across machines. parallel heavily across machines. There's a few things that I need my file There's a few things that I need my file There's a few things that I need my file systems to do to be good for the work systems to do to be good for the work systems to do to be good for the work that we are doing. The first thing that that we are doing. The first thing that that we are doing. The first thing that matters is of course speed. There's matters is of course speed. There's matters is of course speed. There's different types of speed to consider different types of speed to consider different types of speed to consider though. There is deletion speeds for though. There is deletion speeds for though. There is deletion speeds for cleaning up work. There is clone speeds cleaning up work. There is clone speeds cleaning up work. There is clone speeds for doing things like creating work for doing things like creating work for doing things like creating work trees. There's all the indexing work trees. There's all the indexing work trees. There's all the indexing work that has to be done to actually know that has to be done to actually know that has to be done to actually know what is where to make things like system what is where to make things like system what is where to make things like system search, rip grap, all those other tools search, rip grap, all those other tools search, rip grap, all those other tools work well. Quick lookup speeds and all work well. Quick lookup speeds and all work well. Quick lookup speeds and all of that general concept of how well it of that general concept of how well it of that general concept of how well it handles lots of small files which tends handles lots of small files which tends handles lots of small files which tends to be the place where a lot of these to be the place where a lot of these to be the place where a lot of these systems choke. On that note, another

systems choke. On that note, another systems choke. On that note, another category is features. things like category is features. things like category is features. things like metadata systems that make it easier to metadata systems that make it easier to metadata systems that make it easier to tag files, to know how old the file is, tag files, to know how old the file is, tag files, to know how old the file is, to know what should or shouldn't be to know what should or shouldn't be to know what should or shouldn't be overwritten, those types of things. overwritten, those types of things. overwritten, those types of things. There's deep links and copy on write There's deep links and copy on write There's deep links and copy on write behaviors so that you can clone a behaviors so that you can clone a behaviors so that you can clone a directory and not have the files be new directory and not have the files be new directory and not have the files be new files until you start editing them. Copy files until you start editing them. Copy files until you start editing them. Copy on write means instead of when I copy a on write means instead of when I copy a on write means instead of when I copy a thing, I make an identical version of thing, I make an identical version of thing, I make an identical version of the same file in two places. It means the same file in two places. It means the same file in two places. It means that when I copy it, it's just pointing that when I copy it, it's just pointing that when I copy it, it's just pointing at the old file until I edit it and then at the old file until I edit it and then at the old file until I edit it and then it makes the clone. I don't know if this it makes the clone. I don't know if this it makes the clone. I don't know if this really fits under features, but I think really fits under features, but I think really fits under features, but I think it's important. I'll say support in it's important. I'll say support in it's important. I'll say support in quotes. This is things like, is it in quotes. This is things like, is it in quotes. This is things like, is it in the Linux kernel? What OS is supported? the Linux kernel? What OS is supported? the Linux kernel? What OS is supported? If I plug this drive into a Windows or a If I plug this drive into a Windows or a If I plug this drive into a Windows or a Mac machine, will it handle it properly? Mac machine, will it handle it properly? Mac machine, will it handle it properly? And to this day, XFAT's pretty much the And to this day, XFAT's pretty much the And to this day, XFAT's pretty much the only thing supported by most platforms. only thing supported by most platforms. only thing supported by most platforms. So, if you want a flash drive you can So, if you want a flash drive you can So, if you want a flash drive you can plug into multiple things, you're kind plug into multiple things, you're kind plug into multiple things, you're kind of stuck with formats there. But for all of stuck with formats there. But for all of stuck with formats there. But for all these other fun formats, there are new these other fun formats, there are new these other fun formats, there are new benefits and new limitations. My dream benefits and new limitations. My dream benefits and new limitations. My dream was that I'd be able to partition some was that I'd be able to partition some was that I'd be able to partition some chunk of my SSD on my Mac to something chunk of my SSD on my Mac to something chunk of my SSD on my Mac to something else and it would work fine. Apple does else and it would work fine. Apple does else and it would work fine. Apple does not support other file systems well not support other file systems well not support other file systems well enough for that to make any sense at enough for that to make any sense at enough for that to make any sense at all, sadly. Then we get into section all, sadly. Then we get into section all, sadly. Then we get into section three. This is one that I did not expect three. This is one that I did not expect three. This is one that I did not expect to work as well as it does. Compression.

to work as well as it does. Compression. to work as well as it does. Compression. Compression has its own layers. Compression has its own layers. Compression has its own layers. Obviously, there is space savings. How Obviously, there is space savings. How Obviously, there is space savings. How well does this actually save space on well does this actually save space on well does this actually save space on your machine? But there's also your machine? But there's also your machine? But there's also performance as well here, which is performance as well here, which is performance as well here, which is separately. When you're compressing separately. When you're compressing separately. When you're compressing these files on the drive, how does that these files on the drive, how does that these files on the drive, how does that affect performance for the things that affect performance for the things that affect performance for the things that you actually do with those files on the you actually do with those files on the you actually do with those files on the drive? This is surprisingly hard to drive? This is surprisingly hard to drive? This is surprisingly hard to measure, especially alongside the space measure, especially alongside the space measure, especially alongside the space savings. Is when you run a command to savings. Is when you run a command to savings. Is when you run a command to see how big a file is, you get how big see how big a file is, you get how big see how big a file is, you get how big that file is. But if you have that file that file is. But if you have that file that file is. But if you have that file in four places, but you're only saving in four places, but you're only saving in four places, but you're only saving it on the SSD once, that number doesn't it on the SSD once, that number doesn't it on the SSD once, that number doesn't really mean anything. And I had a point really mean anything. And I had a point really mean anything. And I had a point where if you measured the files on my where if you measured the files on my where if you measured the files on my system, it was like 3,000 gigs, but the system, it was like 3,000 gigs, but the system, it was like 3,000 gigs, but the actual space being taken up on the drive actual space being taken up on the drive actual space being taken up on the drive was only a few hundred gigs. That only was only a few hundred gigs. That only was only a few hundred gigs. That only can happen if you have good compression can happen if you have good compression can happen if you have good compression and linking features on the file system. and linking features on the file system. and linking features on the file system. So that the amount of data that you're So that the amount of data that you're So that the amount of data that you're actually taking up is different from the actually taking up is different from the actually taking up is different from the amount of data that you have represented amount of data that you have represented amount of data that you have represented on your system. To put this list more on your system. To put this list more on your system. To put this list more simply, I want fast work trees, fast ppm simply, I want fast work trees, fast ppm simply, I want fast work trees, fast ppm installs, fast cleanups, and ideally, if installs, fast cleanups, and ideally, if installs, fast cleanups, and ideally, if we're lucky, less space wasted on we're lucky, less space wasted on we're lucky, less space wasted on clones. Because when I make a work tree, clones. Because when I make a work tree, clones. Because when I make a work tree, it shouldn't take all of my code and to it shouldn't take all of my code and to it shouldn't take all of my code and to duplicate it on the drive if I'm only duplicate it on the drive if I'm only duplicate it on the drive if I'm only touching one or two files. That's a touching one or two files. That's a touching one or two files. That's a huge, absurdly massive waste of space.

huge, absurdly massive waste of space. huge, absurdly massive waste of space. So, I was looking for something that is So, I was looking for something that is So, I was looking for something that is a good balance of all of this. And as a good balance of all of this. And as a good balance of all of this. And as nerdy as I am about file systems, I was nerdy as I am about file systems, I was nerdy as I am about file systems, I was hoping I wouldn't have to do this work hoping I wouldn't have to do this work hoping I wouldn't have to do this work myself. So I asked some friends. The myself. So I asked some friends. The myself. So I asked some friends. The first place I asked was a group chat I first place I asked was a group chat I first place I asked was a group chat I had with some other creators doing home had with some other creators doing home had with some other creators doing home systems, home lab type stuff, and I systems, home lab type stuff, and I systems, home lab type stuff, and I noticed my friend Wendell from Level One noticed my friend Wendell from Level One noticed my friend Wendell from Level One Tech was there. I've been meaning to DM Tech was there. I've been meaning to DM Tech was there. I've been meaning to DM him about this for a while, but I him about this for a while, but I him about this for a while, but I figured I'll just send it in the group figured I'll just send it in the group figured I'll just send it in the group chat. Why not? And I explained where I chat. Why not? And I explained where I chat. Why not? And I explained where I was at. I'm curious what file systems was at. I'm curious what file systems was at. I'm curious what file systems people are using on their heavily people are using on their heavily people are using on their heavily parallel agent dev boxes. Been trying to parallel agent dev boxes. Been trying to parallel agent dev boxes. Been trying to figure out what's ideal. Having lots of figure out what's ideal. Having lots of figure out what's ideal. Having lots of work trees for large repos is brutal on work trees for large repos is brutal on work trees for large repos is brutal on a lot of file systems. Node modules is a lot of file systems. Node modules is a lot of file systems. Node modules is rough, too. There's a good bench here rough, too. There's a good bench here rough, too. There's a good bench here where I link the nullbox bench. APFS is where I link the nullbox bench. APFS is where I link the nullbox bench. APFS is garbage to the point where I barely run garbage to the point where I barely run garbage to the point where I barely run code agents on my Mac anymore. Butterfs code agents on my Mac anymore. Butterfs code agents on my Mac anymore. Butterfs seems promising, but the large batch of seems promising, but the large batch of seems promising, but the large batch of small file copy speeds are pretty rough. small file copy speeds are pretty rough. small file copy speeds are pretty rough. You can see that in the bench above. You can see that in the bench above. You can see that in the bench above. Currently using ext4 just for the small Currently using ext4 just for the small Currently using ext4 just for the small file write performance, but lack of good file write performance, but lack of good file write performance, but lack of good copy on primitives is brutal. I also copy on primitives is brutal. I also copy on primitives is brutal. I also bring up Anomaly with uh Rift, which bring up Anomaly with uh Rift, which bring up Anomaly with uh Rift, which we'll talk about in a little bit. I got we'll talk about in a little bit. I got we'll talk about in a little bit. I got a nearly immediate response from a nearly immediate response from a nearly immediate response from Wendell. Try XFS plus LZ4.

Wendell. Try XFS plus LZ4. Wendell. Try XFS plus LZ4. Huh? I haven't mentioned LZ4 at all yet. Huh? I haven't mentioned LZ4 at all yet. Huh? I haven't mentioned LZ4 at all yet. Well, this is where all the pieces start Well, this is where all the pieces start Well, this is where all the pieces start to come together. If you think of your to come together. If you think of your to come together. If you think of your drive as a layer, this is the drive. drive as a layer, this is the drive. drive as a layer, this is the drive. This just has ones and zeros all up and This just has ones and zeros all up and This just has ones and zeros all up and down it. You could think of these all as down it. You could think of these all as down it. You could think of these all as being in chunks on the drive. They're being in chunks on the drive. They're being in chunks on the drive. They're usually referred to as blocks. Each usually referred to as blocks. Each usually referred to as blocks. Each block can hold a certain amount of data block can hold a certain amount of data block can hold a certain amount of data and you have a certain number of these and you have a certain number of these and you have a certain number of these blocks on the machine. Most file systems blocks on the machine. Most file systems blocks on the machine. Most file systems have 4 kilobyte block sizes because if have 4 kilobyte block sizes because if have 4 kilobyte block sizes because if you have a file that's 12 kilobytes, it you have a file that's 12 kilobytes, it you have a file that's 12 kilobytes, it can split up across those three blocks can split up across those three blocks can split up across those three blocks and it makes it easy to move things and it makes it easy to move things and it makes it easy to move things around because what happens if I delete around because what happens if I delete around because what happens if I delete something in the middle here? Like I something in the middle here? Like I something in the middle here? Like I just delete this middle chunk. Now I just delete this middle chunk. Now I just delete this middle chunk. Now I have this weird empty block in the have this weird empty block in the have this weird empty block in the middle of the drive. So when you do middle of the drive. So when you do middle of the drive. So when you do something like a def frag for example, something like a def frag for example, something like a def frag for example, what that does is it looks through and what that does is it looks through and what that does is it looks through and finds all the empty blocks and it moves finds all the empty blocks and it moves finds all the empty blocks and it moves everything up to the front. So all the everything up to the front. So all the everything up to the front. So all the empty blocks are left at the end and all empty blocks are left at the end and all empty blocks are left at the end and all the data that's related is back to being the data that's related is back to being the data that's related is back to being close together. But what happens if you close together. But what happens if you close together. But what happens if you have data that doesn't need a full have data that doesn't need a full have data that doesn't need a full block? Like if I have two files index ts block? Like if I have two files index ts block? Like if I have two files index ts and something.ts and something.ts and something.ts and these files together are under 4 and these files together are under 4 and these files together are under 4 kilobyt. How do you represent that on kilobyt. How do you represent that on kilobyt. How do you represent that on the file system? Or better, what if I the file system? Or better, what if I the file system? Or better, what if I had two copies of the same file? How do had two copies of the same file? How do had two copies of the same file? How do you represent that in the blocks? Well, you represent that in the blocks? Well, you represent that in the blocks? Well, first off, you need a layer before this first off, you need a layer before this first off, you need a layer before this because your computer operating system because your computer operating system because your computer operating system doesn't know where all of these things doesn't know where all of these things doesn't know where all of these things are. So, you need something in front are. So, you need something in front are. So, you need something in front that has these file names in it as well

that has these file names in it as well that has these file names in it as well as the pointers to where the different as the pointers to where the different as the pointers to where the different data is. So, in this case, we could data is. So, in this case, we could data is. So, in this case, we could pretend index TS points to this block pretend index TS points to this block pretend index TS points to this block and something TS points to that block. and something TS points to that block. and something TS points to that block. What happens if I copy index ts? Well, What happens if I copy index ts? Well, What happens if I copy index ts? Well, this layer probably just copies the data this layer probably just copies the data this layer probably just copies the data to whatever blocks are available. And to whatever blocks are available. And to whatever blocks are available. And hopefully this all makes sense for your hopefully this all makes sense for your hopefully this all makes sense for your use case. But again, what about all the use case. But again, what about all the use case. But again, what about all the data that I'm copying? What about the data that I'm copying? What about the data that I'm copying? What about the data that's duplicate? What about data data that's duplicate? What about data data that's duplicate? What about data that doesn't need a whole block to that doesn't need a whole block to that doesn't need a whole block to itself? That's where these compression itself? That's where these compression itself? That's where these compression layers can come in and be way more layers can come in and be way more layers can come in and be way more powerful. With a file system like XFS, powerful. With a file system like XFS, powerful. With a file system like XFS, you have the ability to put VDO between you have the ability to put VDO between you have the ability to put VDO between XFS and the actual drive. And this is a XFS and the actual drive. And this is a XFS and the actual drive. And this is a layer that will compress the data before layer that will compress the data before layer that will compress the data before it gets stored to the drive. This means it gets stored to the drive. This means it gets stored to the drive. This means that it can take multiple chunks of data that it can take multiple chunks of data that it can take multiple chunks of data and put them in the same block if it and put them in the same block if it and put them in the same block if it makes sense to. This means that it can makes sense to. This means that it can makes sense to. This means that it can clone things without actually cloning clone things without actually cloning clone things without actually cloning them by just representing them with them by just representing them with them by just representing them with different blocks. It can do a lot of different blocks. It can do a lot of different blocks. It can do a lot of different things, but it does come with different things, but it does come with different things, but it does come with a performance overhead because now your a performance overhead because now your a performance overhead because now your processor needs to do more work when you processor needs to do more work when you processor needs to do more work when you save or open files in order to get save or open files in order to get save or open files in order to get through this compression layer. That's through this compression layer. That's through this compression layer. That's what makes LZ4 particularly cool. It is what makes LZ4 particularly cool. It is what makes LZ4 particularly cool. It is a very very lightweight compressor that a very very lightweight compressor that a very very lightweight compressor that makes the rights and the reads super makes the rights and the reads super makes the rights and the reads super cheap while also saving meaningful cheap while also saving meaningful cheap while also saving meaningful amounts of space. Enough chitchat amounts of space. Enough chitchat amounts of space. Enough chitchat [snorts] about this. Let's go through [snorts] about this. Let's go through [snorts] about this. Let's go through the real candidates here. Obviously, as the real candidates here. Obviously, as the real candidates here. Obviously, as I said before, APFS is garbage. So, I said before, APFS is garbage. So, I said before, APFS is garbage. So, we're just going to ignore that for now.

we're just going to ignore that for now. we're just going to ignore that for now. Let's look at the Linux options that are Let's look at the Linux options that are Let's look at the Linux options that are meaningful. We have ext4, we have XFS, meaningful. We have ext4, we have XFS, meaningful. We have ext4, we have XFS, ZFS, butterfs. I would argue these are ZFS, butterfs. I would argue these are ZFS, butterfs. I would argue these are the the real candidates people will push the the real candidates people will push the the real candidates people will push for others too, but I want to just talk for others too, but I want to just talk for others too, but I want to just talk about these for now. So ext4 is the about these for now. So ext4 is the about these for now. So ext4 is the first. It is super simple and minimal. first. It is super simple and minimal. first. It is super simple and minimal. This means that it's really fast This means that it's really fast This means that it's really fast regardless of the file sizes, but it's regardless of the file sizes, but it's regardless of the file sizes, but it's also limited. I'll say it's not much also limited. I'll say it's not much also limited. I'll say it's not much metadata, no copy and write, etc. metadata, no copy and write, etc. metadata, no copy and write, etc. So every time you make a new directory So every time you make a new directory So every time you make a new directory in ext4, that directory is real. It in ext4, that directory is real. It in ext4, that directory is real. It takes up real space. XFS is a much more takes up real space. XFS is a much more takes up real space. XFS is a much more extensible alternative. And then ZFS and extensible alternative. And then ZFS and extensible alternative. And then ZFS and Butterfs are both incredible, cool, Butterfs are both incredible, cool, Butterfs are both incredible, cool, often debated technologies, too, but I often debated technologies, too, but I often debated technologies, too, but I will say these are both much more will say these are both much more will say these are both much more intended for RAID. I knew they wouldn't intended for RAID. I knew they wouldn't intended for RAID. I knew they wouldn't be great fits here, but enough people be great fits here, but enough people be great fits here, but enough people asked that I actually took the time to asked that I actually took the time to asked that I actually took the time to bench them as well. And you can see here bench them as well. And you can see here bench them as well. And you can see here for creating a work tree, ZFS ended up for creating a work tree, ZFS ended up for creating a work tree, ZFS ended up being in between APFS and Butterfs for a being in between APFS and Butterfs for a being in between APFS and Butterfs for a single work tree creation where APFS single work tree creation where APFS single work tree creation where APFS took 1.26 26 seconds for T3 code. But took 1.26 26 seconds for T3 code. But took 1.26 26 seconds for T3 code. But RFS took under.7 seconds. Ext4 was even RFS took under.7 seconds. Ext4 was even RFS took under.7 seconds. Ext4 was even faster at 63 seconds. XFS was similar faster at 63 seconds. XFS was similar faster at 63 seconds. XFS was similar speeds. And then ZFS meaningfully slower speeds. And then ZFS meaningfully slower speeds. And then ZFS meaningfully slower at almost8 seconds. And once you start at almost8 seconds. And once you start at almost8 seconds. And once you start creating in parallel, which is a real creating in parallel, which is a real creating in parallel, which is a real use case, creating eight work trees, not use case, creating eight work trees, not use case, creating eight work trees, not eight necessarily, but a lot of these eight necessarily, but a lot of these eight necessarily, but a lot of these things moving around at once on the same things moving around at once on the same things moving around at once on the same machine. Absolutely a use case. Mac OS machine. Absolutely a use case. Mac OS machine. Absolutely a use case. Mac OS sees very little improvement in parallel

sees very little improvement in parallel sees very little improvement in parallel versus sequential creations because one versus sequential creations because one versus sequential creations because one takes 1.26 seconds. 8 takes 7.1 seconds. takes 1.26 seconds. 8 takes 7.1 seconds. takes 1.26 seconds. 8 takes 7.1 seconds. Meanwhile, on ext4 and XFS, the Meanwhile, on ext4 and XFS, the Meanwhile, on ext4 and XFS, the difference is like 5%, not 7x. ZFS takes difference is like 5%, not 7x. ZFS takes difference is like 5%, not 7x. ZFS takes a bigger hit though, now over 1 second a bigger hit though, now over 1 second a bigger hit though, now over 1 second for the creations. Butterfs took a for the creations. Butterfs took a for the creations. Butterfs took a brutal hit here too though at 2.1 brutal hit here too though at 2.1 brutal hit here too though at 2.1 seconds from 6. So already we're seeing seconds from 6. So already we're seeing seconds from 6. So already we're seeing some of the problems with those layers. some of the problems with those layers. some of the problems with those layers. Not as fast as I'd want them to be. Not as fast as I'd want them to be. Not as fast as I'd want them to be. Where things get much more fun is the Where things get much more fun is the Where things get much more fun is the amount of storage being taken up here. amount of storage being taken up here. amount of storage being taken up here. When I had a clean work tree, so none of When I had a clean work tree, so none of When I had a clean work tree, so none of like the packages installed and whatnot. like the packages installed and whatnot. like the packages installed and whatnot. The amount of space it took up on APFS The amount of space it took up on APFS The amount of space it took up on APFS was 240 megs or so. On Butterf FS, it was 240 megs or so. On Butterf FS, it was 240 megs or so. On Butterf FS, it was only like 14 megs or so. Extral was only like 14 megs or so. Extral was only like 14 megs or so. Extral were both the same at like 250 or so were both the same at like 250 or so were both the same at like 250 or so megs, roughly the same as APFS. I don't megs, roughly the same as APFS. I don't megs, roughly the same as APFS. I don't know what Butterfest did to cut it in know what Butterfest did to cut it in know what Butterfest did to cut it in half, but good for it. I I'm assuming it half, but good for it. I I'm assuming it half, but good for it. I I'm assuming it had some compression on. XFS plus VDO had some compression on. XFS plus VDO had some compression on. XFS plus VDO compresses similarly, actually even more compresses similarly, actually even more compresses similarly, actually even more so at.128. So that's about 128 megs so at.128. So that's about 128 megs so at.128. So that's about 128 megs versus 240. And then ZFS killed it at versus 240. And then ZFS killed it at versus 240. And then ZFS killed it at allegedly 20 megs. I don't believe it.

allegedly 20 megs. I don't believe it. allegedly 20 megs. I don't believe it. I'm curious how it did that, but I'll I'm curious how it did that, but I'll I'm curious how it did that, but I'll take a look later. Then once you take a look later. Then once you take a look later. Then once you installed the modules, you end up with a installed the modules, you end up with a installed the modules, you end up with a much bigger difference here. Ext, as I much bigger difference here. Ext, as I much bigger difference here. Ext, as I mentioned before, doesn't do linking mentioned before, doesn't do linking mentioned before, doesn't do linking very much at all. So, it ended up having very much at all. So, it ended up having very much at all. So, it ended up having to actually copy all the modules, which to actually copy all the modules, which to actually copy all the modules, which made it almost a gig. It's well over made it almost a gig. It's well over made it almost a gig. It's well over half a gig for the T3 code project and half a gig for the T3 code project and half a gig for the T3 code project and node modules. APFS didn't have to clone node modules. APFS didn't have to clone node modules. APFS didn't have to clone quite as much, so it only bumped about quite as much, so it only bumped about quite as much, so it only bumped about 150 megs. But RFS bumped a massive 150 megs. But RFS bumped a massive 150 megs. But RFS bumped a massive amount here. But then we see XFS plus amount here. But then we see XFS plus amount here. But then we see XFS plus VDO here, which after the install still VDO here, which after the install still VDO here, which after the install still takes up less space than the clean takes up less space than the clean takes up less space than the clean version on APFS. Actually insane. Those version on APFS. Actually insane. Those version on APFS. Actually insane. Those are all tests that are on the T3 code are all tests that are on the T3 code are all tests that are on the T3 code repo, though. So, let's look at the Nvox repo, though. So, let's look at the Nvox repo, though. So, let's look at the Nvox bench and see how numbers went here. As bench and see how numbers went here. As bench and see how numbers went here. As I mentioned before, APFS is [ __ ] I mentioned before, APFS is [ __ ] I mentioned before, APFS is [ __ ] garbage. It took 39 seconds roughly for garbage. It took 39 seconds roughly for garbage. It took 39 seconds roughly for the installs and cleanups. Ext4 took 2.5 the installs and cleanups. Ext4 took 2.5 the installs and cleanups. Ext4 took 2.5 seconds for the cleanup and 9 seconds seconds for the cleanup and 9 seconds seconds for the cleanup and 9 seconds roughly for the install. XFS was about 7 roughly for the install. XFS was about 7 roughly for the install. XFS was about 7 seconds for the cleanup and a little seconds for the cleanup and a little seconds for the cleanup and a little under 10 or 9.5 for the install. The VDO under 10 or 9.5 for the install. The VDO under 10 or 9.5 for the install. The VDO version is actually slightly slower for version is actually slightly slower for version is actually slightly slower for that test. and then ZFS, which is not that test. and then ZFS, which is not that test. and then ZFS, which is not looking great here either. But if we use looking great here either. But if we use looking great here either. But if we use hard links instead, you see a lot of the hard links instead, you see a lot of the hard links instead, you see a lot of the benefits of these file systems as well benefits of these file systems as well benefits of these file systems as well as the worst parts of APFS because as the worst parts of APFS because as the worst parts of APFS because again, APFS does support links if you're again, APFS does support links if you're again, APFS does support links if you're willing to eat a huge overhead when you willing to eat a huge overhead when you willing to eat a huge overhead when you do it because now it went from taking 40 do it because now it went from taking 40 do it because now it went from taking 40 secondsish to 100 seconds when you're secondsish to 100 seconds when you're secondsish to 100 seconds when you're installing your node modules with hard installing your node modules with hard installing your node modules with hard links. insane because all of those links

links. insane because all of those links links. insane because all of those links being created takes a ton of overhead on being created takes a ton of overhead on being created takes a ton of overhead on Mac OS. This is a problem I'm somewhat Mac OS. This is a problem I'm somewhat Mac OS. This is a problem I'm somewhat familiar with because I've talked about familiar with because I've talked about familiar with because I've talked about it a bunch with Jared Sumner, the it a bunch with Jared Sumner, the it a bunch with Jared Sumner, the creator of Bun as he was trying to creator of Bun as he was trying to creator of Bun as he was trying to optimize bun installs on Mac. The optimize bun installs on Mac. The optimize bun installs on Mac. The problem here is APFS is [ __ ] garbage. problem here is APFS is [ __ ] garbage. problem here is APFS is [ __ ] garbage. You'll also see though that ZFS does not You'll also see though that ZFS does not You'll also see though that ZFS does not handle cleanups on hard links handle cleanups on hard links handle cleanups on hard links particularly well, taking over 5 seconds particularly well, taking over 5 seconds particularly well, taking over 5 seconds when XFS and VDO takes under three. So, when XFS and VDO takes under three. So, when XFS and VDO takes under three. So, this is mostly meant to rule out APFS, this is mostly meant to rule out APFS, this is mostly meant to rule out APFS, ZFS, and Butterfs ZFS, and Butterfs ZFS, and Butterfs because it just doesn't really benefit because it just doesn't really benefit because it just doesn't really benefit to use those file systems for this. I to use those file systems for this. I to use those file systems for this. I still use both of those heavily, still use both of those heavily, still use both of those heavily, especially Butterf. I use it for all my especially Butterf. I use it for all my especially Butterf. I use it for all my NAS and like file storage stuff. Like NAS and like file storage stuff. Like NAS and like file storage stuff. Like the files for this video were stored on the files for this video were stored on the files for this video were stored on Butterfest before being uploaded to Butterfest before being uploaded to Butterfest before being uploaded to Dropbox for my editor. But we're talking Dropbox for my editor. But we're talking Dropbox for my editor. But we're talking about a gap between 8 seconds and a 100 about a gap between 8 seconds and a 100 about a gap between 8 seconds and a 100 seconds here between just going from a seconds here between just going from a seconds here between just going from a Mac to a base tier Ubuntu install. These Mac to a base tier Ubuntu install. These Mac to a base tier Ubuntu install. These things hurt when you're doing real work. things hurt when you're doing real work. things hurt when you're doing real work. When I make a new thread in T3 code or When I make a new thread in T3 code or When I make a new thread in T3 code or codeex or quad or whatever and it does a codeex or quad or whatever and it does a codeex or quad or whatever and it does a work tree clone, this is the difference work tree clone, this is the difference work tree clone, this is the difference between that clone taking 100 seconds between that clone taking 100 seconds between that clone taking 100 seconds before it can start working and starting before it can start working and starting before it can start working and starting in 10 seconds. In real world use cases in 10 seconds. In real world use cases in 10 seconds. In real world use cases like mine with working on T3 code like mine with working on T3 code like mine with working on T3 code itself, the difference was about 10 to itself, the difference was about 10 to itself, the difference was about 10 to 12 seconds to under two seconds. Full 12 seconds to under two seconds. Full 12 seconds to under two seconds. Full transparency, I did most of my benchmark transparency, I did most of my benchmark transparency, I did most of my benchmark work here after ruling out Butterfs and work here after ruling out Butterfs and work here after ruling out Butterfs and CFS. So the numbers that matter here are CFS. So the numbers that matter here are CFS. So the numbers that matter here are just comparing Mac OS with APFS against

just comparing Mac OS with APFS against just comparing Mac OS with APFS against ext4, which was my stock install with ext4, which was my stock install with ext4, which was my stock install with Ubuntu that I was having storage issues Ubuntu that I was having storage issues Ubuntu that I was having storage issues with. and then the new VDO XFS with. and then the new VDO XFS with. and then the new VDO XFS partitions that I created as well. The partitions that I created as well. The partitions that I created as well. The biggest inspiration for this was that I biggest inspiration for this was that I biggest inspiration for this was that I had accidentally filled my drive with a had accidentally filled my drive with a had accidentally filled my drive with a bunch of logs on my Linux box and I bunch of logs on my Linux box and I bunch of logs on my Linux box and I wanted to clean it up which took forever wanted to clean it up which took forever wanted to clean it up which took forever to find and delete all the files. So I to find and delete all the files. So I to find and delete all the files. So I wanted to rethink everything and like do wanted to rethink everything and like do wanted to rethink everything and like do it right in quotes this time. These are it right in quotes this time. These are it right in quotes this time. These are the top level numbers. Switching to the the top level numbers. Switching to the the top level numbers. Switching to the XSVDL partition on my Linux machine XSVDL partition on my Linux machine XSVDL partition on my Linux machine resulted in 44% resulted in 44% resulted in 44% of my storage being saved. That is of my storage being saved. That is of my storage being saved. That is basically a 50% increase in my drive basically a 50% increase in my drive basically a 50% increase in my drive space for changing file systems. That's space for changing file systems. That's space for changing file systems. That's crazy already. Even plain XFS saw a 10% crazy already. Even plain XFS saw a 10% crazy already. Even plain XFS saw a 10% incre improvement here. When creating incre improvement here. When creating incre improvement here. When creating eight work trees, the Mac took 10 times eight work trees, the Mac took 10 times eight work trees, the Mac took 10 times longer on APFS. And these numbers are longer on APFS. And these numbers are longer on APFS. And these numbers are crazy. The default PNPM behavior crazy. The default PNPM behavior crazy. The default PNPM behavior massively favored all of my Linux massively favored all of my Linux massively favored all of my Linux options. But when I switched over to options. But when I switched over to options. But when I switched over to hard links, the video of XFS got even hard links, the video of XFS got even hard links, the video of XFS got even better, getting comparable to the ext4.

better, getting comparable to the ext4. better, getting comparable to the ext4. This is one of my favorite charts This is one of my favorite charts This is one of my favorite charts because it really shows how the space because it really shows how the space because it really shows how the space difference adds up over time when you're difference adds up over time when you're difference adds up over time when you're working on a lot of things at once. With working on a lot of things at once. With working on a lot of things at once. With 125 work trees all with PNPM install 125 work trees all with PNPM install 125 work trees all with PNPM install run, stock ext4 took up 75 gigs on my run, stock ext4 took up 75 gigs on my run, stock ext4 took up 75 gigs on my computer because again it doesn't have computer because again it doesn't have computer because again it doesn't have any way to like link multiple things to any way to like link multiple things to any way to like link multiple things to the same stuff and it doesn't have any the same stuff and it doesn't have any the same stuff and it doesn't have any system like file system level system like file system level system like file system level compression. Plain XFS was better compression. Plain XFS was better compression. Plain XFS was better because it could at least do the linking because it could at least do the linking because it could at least do the linking but it still had to clone all the files but it still had to clone all the files but it still had to clone all the files for the work trees. So, it wasn't great for the work trees. So, it wasn't great for the work trees. So, it wasn't great there. Max APFS system was able to stock there. Max APFS system was able to stock there. Max APFS system was able to stock save a meaningful amount of space here. save a meaningful amount of space here. save a meaningful amount of space here. Like 43 gigs versus 75.5. Like that's a Like 43 gigs versus 75.5. Like that's a Like 43 gigs versus 75.5. Like that's a meaningful decrease. But the VDO storage meaningful decrease. But the VDO storage meaningful decrease. But the VDO storage being 24 gigs. Yeah, a 3x decrease in being 24 gigs. Yeah, a 3x decrease in being 24 gigs. Yeah, a 3x decrease in the space being used for switching from the space being used for switching from the space being used for switching from ext4 to the XFS video setup. Pretty ext4 to the XFS video setup. Pretty ext4 to the XFS video setup. Pretty nuts. And the one performance gap that nuts. And the one performance gap that nuts. And the one performance gap that existed between EXT4 and XFS, which was existed between EXT4 and XFS, which was existed between EXT4 and XFS, which was the deletion and recreation one from the the deletion and recreation one from the the deletion and recreation one from the Novox bench, gets flattened immediately Novox bench, gets flattened immediately Novox bench, gets flattened immediately when you start using hard links for PNPM when you start using hard links for PNPM when you start using hard links for PNPM instead. It goes from 7 seconds to 2.85.

instead. It goes from 7 seconds to 2.85. instead. It goes from 7 seconds to 2.85. Huge. All of this is meant to say that Huge. All of this is meant to say that Huge. All of this is meant to say that surprise, Wendell was right. He hadn't surprise, Wendell was right. He hadn't surprise, Wendell was right. He hadn't done any measurements or benching or done any measurements or benching or done any measurements or benching or anything to prove it yet. He just knew anything to prove it yet. He just knew anything to prove it yet. He just knew because he knows these things so well. because he knows these things so well. because he knows these things so well. And after a shitload of benchmarking on And after a shitload of benchmarking on And after a shitload of benchmarking on my end, I confirmed as usual that my end, I confirmed as usual that my end, I confirmed as usual that Wendell is right and you can usually Wendell is right and you can usually Wendell is right and you can usually just blindly follow his advice, which I just blindly follow his advice, which I just blindly follow his advice, which I have since done and I have been very, have since done and I have been very, have since done and I have been very, very happy since I've done that. And if very happy since I've done that. And if very happy since I've done that. And if you're curious how this affects my you're curious how this affects my you're curious how this affects my day-to-day work, I will show you. I'm day-to-day work, I will show you. I'm day-to-day work, I will show you. I'm pressing command shift O in T3 Code. I'm pressing command shift O in T3 Code. I'm pressing command shift O in T3 Code. I'm picking a project. I am going to create picking a project. I am going to create picking a project. I am going to create a new work tree when I send off this a new work tree when I send off this a new work tree when I send off this prompt. And when I do this, it has to prompt. And when I do this, it has to prompt. And when I do this, it has to clone the repo into another directory clone the repo into another directory clone the repo into another directory pnpm install to get all the content pnpm install to get all the content pnpm install to get all the content there and then it can start working. So there and then it can start working. So there and then it can start working. So when I send this prompt, it's not when I send this prompt, it's not when I send this prompt, it's not actually sending initially. It is actually sending initially. It is actually sending initially. It is creating a directory and running a creating a directory and running a creating a directory and running a script and then it gets actually sent. I script and then it gets actually sent. I script and then it gets actually sent. I am sending it now am sending it now am sending it now and it was basically instantaneous. This and it was basically instantaneous. This and it was basically instantaneous. This is running on my Linux box. This is the is running on my Linux box. This is the is running on my Linux box. This is the framework that I did all the testing on.

framework that I did all the testing on. framework that I did all the testing on. And if I do the same exact request, but And if I do the same exact request, but And if I do the same exact request, but this time on this MacBook. So, no this time on this MacBook. So, no this time on this MacBook. So, no network hops. Waiting. Waiting. Waiting. Waiting. Waiting. Waiting. Waiting. Still cloning. Looks like it's done now. It's bad. So, Looks like it's done now. It's bad. So, you will feel the difference every you will feel the difference every you will feel the difference every single day. To be completely honest, single day. To be completely honest, single day. To be completely honest, ext4, just the stock Ubuntu install was ext4, just the stock Ubuntu install was ext4, just the stock Ubuntu install was already so much better than what I was already so much better than what I was already so much better than what I was used to with Mac OS that I wasn't used to with Mac OS that I wasn't used to with Mac OS that I wasn't planning on moving off. I was honestly planning on moving off. I was honestly planning on moving off. I was honestly planning on just sticking with it, but I planning on just sticking with it, but I planning on just sticking with it, but I was curious what others were using, so I was curious what others were using, so I was curious what others were using, so I asked. I was concerned that the asked. I was concerned that the asked. I was concerned that the compression overhead for CPU would be compression overhead for CPU would be compression overhead for CPU would be great enough that it would cancel out great enough that it would cancel out great enough that it would cancel out any benefit for performance there. I was any benefit for performance there. I was any benefit for performance there. I was wrong. It flies. And now I do all of my wrong. It flies. And now I do all of my wrong. It flies. And now I do all of my agentic dev work on my Linux box on a agentic dev work on my Linux box on a agentic dev work on my Linux box on a drive separate from the OS drive that is drive separate from the OS drive that is drive separate from the OS drive that is formatted this way. And the results are formatted this way. And the results are formatted this way. And the results are shockingly good. I have been very happy shockingly good. I have been very happy shockingly good. I have been very happy with the impact in my day-to-day with the impact in my day-to-day with the impact in my day-to-day experience both from the Linux move but experience both from the Linux move but experience both from the Linux move but now also from this file system move. Do now also from this file system move. Do now also from this file system move. Do I think we need to write a new file I think we need to write a new file I think we need to write a new file system from scratch just for agents? And system from scratch just for agents? And system from scratch just for agents? And no, but I wouldn't be opposed to it.

no, but I wouldn't be opposed to it. no, but I wouldn't be opposed to it. That said, the ones we have today have That said, the ones we have today have That said, the ones we have today have their own benefits and negatives, and I their own benefits and negatives, and I their own benefits and negatives, and I have been blown away with the benefits have been blown away with the benefits have been blown away with the benefits of making this move in my own day-to-day of making this move in my own day-to-day of making this move in my own day-to-day work. The majority of the code I work in work. The majority of the code I work in work. The majority of the code I work in every day is now running on a Linux box every day is now running on a Linux box every day is now running on a Linux box on my home network using XFS with video on my home network using XFS with video on my home network using XFS with video compression, and it feels awesome. It compression, and it feels awesome. It compression, and it feels awesome. It uses way less space. I don't have to uses way less space. I don't have to uses way less space. I don't have to worry about my drive filling up anymore. worry about my drive filling up anymore. worry about my drive filling up anymore. I also put a bigger drive in, so I never I also put a bigger drive in, so I never I also put a bigger drive in, so I never have to worry about it again. In fact, have to worry about it again. In fact, have to worry about it again. In fact, I'm using 10% of my 3.5 TBTE drive here. I'm using 10% of my 3.5 TBTE drive here. I'm using 10% of my 3.5 TBTE drive here. And I also have a small partition on And I also have a small partition on And I also have a small partition on this drive that is just used for testing this drive that is just used for testing this drive that is just used for testing for comparing other file systems. I'll for comparing other file systems. I'll for comparing other file systems. I'll never have to worry again. The data on never have to worry again. The data on never have to worry again. The data on this system was over a terabyte before I this system was over a terabyte before I this system was over a terabyte before I made the move. And the terabyte that I made the move. And the terabyte that I made the move. And the terabyte that I moved over only ended up about 360 gigs moved over only ended up about 360 gigs moved over only ended up about 360 gigs of real data when I did the move, which of real data when I did the move, which of real data when I did the move, which is crazy. The remaining 490 gigs are is crazy. The remaining 490 gigs are is crazy. The remaining 490 gigs are mostly like backed up compressed things mostly like backed up compressed things mostly like backed up compressed things I left on that drive before I had the I left on that drive before I had the I left on that drive before I had the full confidence. I can probably go nuke full confidence. I can probably go nuke full confidence. I can probably go nuke that now and be fine. I couldn't be that now and be fine. I couldn't be that now and be fine. I couldn't be happier. This was a move that I am blown happier. This was a move that I am blown happier. This was a move that I am blown away with the impact it's had on my away with the impact it's had on my away with the impact it's had on my day-to-day agentic dev experience and it day-to-day agentic dev experience and it day-to-day agentic dev experience and it is yet another huge set of points in is yet another huge set of points in is yet another huge set of points in favor of setting up a Linux machine and favor of setting up a Linux machine and favor of setting up a Linux machine and running the stuff yourself. I do running the stuff yourself. I do running the stuff yourself. I do genuinely think every dev that's trying genuinely think every dev that's trying genuinely think every dev that's trying to maximize their use of these AI tools to maximize their use of these AI tools to maximize their use of these AI tools should have an old computer somewhere at should have an old computer somewhere at should have an old computer somewhere at their house that they can install Linux their house that they can install Linux their house that they can install Linux on, set up a file system like this, set on, set up a file system like this, set on, set up a file system like this, set up something like T3 Code or Claude's up something like T3 Code or Claude's up something like T3 Code or Claude's remote features or whatever to do their remote features or whatever to do their remote features or whatever to do their dev work on. The result will just feel dev work on. The result will just feel dev work on. The result will just feel so much better. I did not want this to so much better. I did not want this to so much better. I did not want this to be this big a win. I am surprised that be this big a win. I am surprised that be this big a win. I am surprised that it was. But I do have one last note for

it was. But I do have one last note for it was. But I do have one last note for you Apple people. I do have to talk you Apple people. I do have to talk you Apple people. I do have to talk about Rift by Anomaly. Largely out of about Rift by Anomaly. Largely out of about Rift by Anomaly. Largely out of loyalty to Nexel, one of the OG people loyalty to Nexel, one of the OG people loyalty to Nexel, one of the OG people from this community, one of the original from this community, one of the original from this community, one of the original creators of the T3 stack. The point of creators of the T3 stack. The point of creators of the T3 stack. The point of Rift is to be a better alternative to Rift is to be a better alternative to Rift is to be a better alternative to get work trees, especially on Mac OS. get work trees, especially on Mac OS. get work trees, especially on Mac OS. So, it virtualizes the files so that So, it virtualizes the files so that So, it virtualizes the files so that your agent can work on them and do real your agent can work on them and do real your agent can work on them and do real work as it would expect. and it doesn't work as it would expect. and it doesn't work as it would expect. and it doesn't actually persist the new files until you actually persist the new files until you actually persist the new files until you write to them again. Copy on write. The write to them again. Copy on write. The write to them again. Copy on write. The result is that it's instantaneous on Mac result is that it's instantaneous on Mac result is that it's instantaneous on Mac OS. The CLI is super fast and you can OS. The CLI is super fast and you can OS. The CLI is super fast and you can use it with bun or node directly. Last I use it with bun or node directly. Last I use it with bun or node directly. Last I checked it was only Mac support. It checked it was only Mac support. It checked it was only Mac support. It seems like it works now on Linux with seems like it works now on Linux with seems like it works now on Linux with butterfs. So that's really cool. I am butterfs. So that's really cool. I am butterfs. So that's really cool. I am genuinely excited to see where this genuinely excited to see where this genuinely excited to see where this goes. We definitely should be goes. We definitely should be goes. We definitely should be considering options like this in the considering options like this in the considering options like this in the near future. And I hope that they near future. And I hope that they near future. And I hope that they continue to iterate on this. Actually, continue to iterate on this. Actually, continue to iterate on this. Actually, no, they haven't. Its last commit was 2 no, they haven't. Its last commit was 2 no, they haven't. Its last commit was 2 months ago and hasn't been touched months ago and hasn't been touched months ago and hasn't been touched since. So, this project might sadly be since. So, this project might sadly be since. So, this project might sadly be dead. But I am hopeful something like dead. But I am hopeful something like dead. But I am hopeful something like this will get far enough along that it this will get far enough along that it this will get far enough along that it can become the new default. But until can become the new default. But until can become the new default. But until then, no hacks. I'm just going to use a then, no hacks. I'm just going to use a then, no hacks. I'm just going to use a better file system. But for those who better file system. But for those who better file system. But for those who desperately want something better right desperately want something better right desperately want something better right now, this might be a good place to look.

now, this might be a good place to look. now, this might be a good place to look. And hopefully they get a little further And hopefully they get a little further And hopefully they get a little further with it or someone else comes in to with it or someone else comes in to with it or someone else comes in to build their own better solution. I'm not build their own better solution. I'm not build their own better solution. I'm not going to lie, I'm pretty pumped. I just going to lie, I'm pretty pumped. I just going to lie, I'm pretty pumped. I just conned y'all to watch a file system conned y'all to watch a file system conned y'all to watch a file system video. I love nerding out about these video. I love nerding out about these video. I love nerding out about these things and normally I don't get to talk things and normally I don't get to talk things and normally I don't get to talk about them so much, but uh I guess about them so much, but uh I guess about them so much, but uh I guess agents are finally good enough that I agents are finally good enough that I agents are finally good enough that I can use them as an excuse to talk about can use them as an excuse to talk about can use them as an excuse to talk about how files are stored on your computer. how files are stored on your computer. how files are stored on your computer. Yay. This is a fun one. I hope you Yay. This is a fun one. I hope you Yay. This is a fun one. I hope you learned something and that you might learned something and that you might learned something and that you might benefit from the stuff I just ranted benefit from the stuff I just ranted benefit from the stuff I just ranted about. I put way too much time into about. I put way too much time into about. I put way too much time into benchmarking all of this and I really benchmarking all of this and I really benchmarking all of this and I really hope it benefits some other people hope it benefits some other people hope it benefits some other people because this info just did not exist in because this info just did not exist in because this info just did not exist in the format that I needed it. Hopefully, the format that I needed it. Hopefully, the format that I needed it. Hopefully, this will help you guys build some this will help you guys build some this will help you guys build some cooler things and use systems on open cooler things and use systems on open cooler things and use systems on open source software like Linux to build way source software like Linux to build way source software like Linux to build way cooler stuff with your agents. I hope cooler stuff with your agents. I hope cooler stuff with your agents. I hope this was helpful and until next time, this was helpful and until next time, this was helpful and until next time, peace nerds.