While my guitar gently speaks — Todd Fisher, Philo Ventures

So I am Todd Fisher. I love the guitar. So I am Todd Fisher. I love the guitar. That's one of my passions in life. Uh That's one of my passions in life. Uh That's one of my passions in life. Uh today I want to talk about uh today I want to talk about uh today I want to talk about uh this project I've been working on for a this project I've been working on for a this project I've been working on for a while here, effectively making my guitar while here, effectively making my guitar while here, effectively making my guitar speak. Uh but of course I want to start speak. Uh but of course I want to start speak. Uh but of course I want to start out with kind of framing it under this out with kind of framing it under this out with kind of framing it under this awesome premise. You know, we've all awesome premise. You know, we've all awesome premise. You know, we've all been to live performances where like been to live performances where like been to live performances where like your mind was just blown and it was your mind was just blown and it was your mind was just blown and it was awesome. I you know, the the first one I awesome. I you know, the the first one I awesome. I you know, the the first one I remember way back I was in high school. remember way back I was in high school. remember way back I was in high school. I went to a Slipknot concert, so a I went to a Slipknot concert, so a I went to a Slipknot concert, so a little bit heavier music uh here in the little bit heavier music uh here in the little bit heavier music uh here in the Bay Area. And I remember at some point Bay Area. And I remember at some point Bay Area. And I remember at some point the drummer was drumming a cool drum the drummer was drumming a cool drum the drummer was drumming a cool drum solo and his drum set started raising up solo and his drum set started raising up solo and his drum set started raising up and I was like, "Whoa, that's cool." And and I was like, "Whoa, that's cool." And and I was like, "Whoa, that's cool." And everyone got kind of excited, right? And everyone got kind of excited, right? And everyone got kind of excited, right? And then halfway through the solo, his drum then halfway through the solo, his drum then halfway through the solo, his drum set started to go like this and tipping. set started to go like this and tipping. set started to go like this and tipping. And his whole drum set was like on the And his whole drum set was like on the And his whole drum set was like on the wall while he was upside down playing wall while he was upside down playing wall while he was upside down playing his drum solo. And everyone was just his drum solo. And everyone was just his drum solo. And everyone was just cheering like, "Whoa, this is so crazy, cheering like, "Whoa, this is so crazy, cheering like, "Whoa, this is so crazy, mind-blowing." Like, "Whoa." It was so mind-blowing." Like, "Whoa." It was so mind-blowing." Like, "Whoa." It was so so impactful, right? I still remember it so impactful, right? I still remember it so impactful, right? I still remember it today. Um fast forward probably about a today. Um fast forward probably about a today. Um fast forward probably about a year ago, year ago, year ago, uh I went to New York to see the uh I went to New York to see the uh I went to New York to see the Stranger Things on Broadway, uh the the Stranger Things on Broadway, uh the the Stranger Things on Broadway, uh the the first shadow. And it was really cool first shadow. And it was really cool first shadow. And it was really cool because the whole time I was there, I because the whole time I was there, I because the whole time I was there, I was watching it and it felt like it was was watching it and it felt like it was was watching it and it felt like it was actually like a a Netflix episode but in actually like a a Netflix episode but in actually like a a Netflix episode but in real life. And it was just like the real life. And it was just like the real life. And it was just like the effects that they did there was so so effects that they did there was so so effects that they did there was so so amazing. I remember even one time at amazing. I remember even one time at amazing. I remember even one time at near the end there was a scene where near the end there was a scene where near the end there was a scene where somebody somebody was falling backwards somebody somebody was falling backwards somebody somebody was falling backwards and it was like a slow-motion scene and and it was like a slow-motion scene and and it was like a slow-motion scene and it probably about a minute long the it probably about a minute long the it probably about a minute long the whole scene. But just like the fact that whole scene. But just like the fact that whole scene. But just like the fact that like they they were able to produce that like they they were able to produce that like they they were able to produce that in real life but it looked like it was a in real life but it looked like it was a in real life but it looked like it was a post-production slow-motion scene. Just post-production slow-motion scene. Just post-production slow-motion scene. Just just mind-blowing. So it's like, "Whoa,

just mind-blowing. So it's like, "Whoa, just mind-blowing. So it's like, "Whoa, that's cool." So given you uh everyone that's cool." So given you uh everyone that's cool." So given you uh everyone here has probably had some experiences here has probably had some experiences here has probably had some experiences uh with live performances in the past uh with live performances in the past uh with live performances in the past and I think it's really awesome just to and I think it's really awesome just to and I think it's really awesome just to like see the creativity uh that people like see the creativity uh that people like see the creativity uh that people can do with leveraging technology in can do with leveraging technology in can do with leveraging technology in some creative way. some creative way. some creative way. And so, you know, I'm thinking about all And so, you know, I'm thinking about all And so, you know, I'm thinking about all the projects that we have as engineers the projects that we have as engineers the projects that we have as engineers over the years where it's like, "Hey, over the years where it's like, "Hey, over the years where it's like, "Hey, that's a cool idea. I'm going to write that's a cool idea. I'm going to write that's a cool idea. I'm going to write this down." I have like a big giant this down." I have like a big giant this down." I have like a big giant list. Anyone have a list of projects list. Anyone have a list of projects list. Anyone have a list of projects that maybe they'll get to? Yeah, so it that maybe they'll get to? Yeah, so it that maybe they'll get to? Yeah, so it all happens, right? Uh and it's really all happens, right? Uh and it's really all happens, right? Uh and it's really awesome cuz the last 6 months or so with awesome cuz the last 6 months or so with awesome cuz the last 6 months or so with AI uh being able to, you know, push us AI uh being able to, you know, push us AI uh being able to, you know, push us forward with some of these projects, forward with some of these projects, forward with some of these projects, it's just really cool to see where we it's just really cool to see where we it's just really cool to see where we can take all these projects. So so so can take all these projects. So so so can take all these projects. So so so part of uh my goal today is to inspire part of uh my goal today is to inspire part of uh my goal today is to inspire you guys to find whatever project you're you guys to find whatever project you're you guys to find whatever project you're passionate about, go start building it passionate about, go start building it passionate about, go start building it because it's super easy now with AI it's because it's super easy now with AI it's because it's super easy now with AI it's easier with AI. I know sometimes there's easier with AI. I know sometimes there's easier with AI. I know sometimes there's caveats, but but in general uh I I want caveats, but but in general uh I I want caveats, but but in general uh I I want to have you guys leave this session to have you guys leave this session to have you guys leave this session being inspired to go build whatever that being inspired to go build whatever that being inspired to go build whatever that cool thing is for you personally that's cool thing is for you personally that's cool thing is for you personally that's going to, you know, help you to learn going to, you know, help you to learn going to, you know, help you to learn stuff or maybe make a difference in the stuff or maybe make a difference in the stuff or maybe make a difference in the things that you're you're trying to things that you're you're trying to things that you're you're trying to figure out.

figure out. figure out. Uh so I want to start off with framing Uh so I want to start off with framing Uh so I want to start off with framing this kind of in the realm of the guitar. this kind of in the realm of the guitar. this kind of in the realm of the guitar. So the guitar because the guitar So the guitar because the guitar So the guitar because the guitar guitar's been around for several years guitar's been around for several years guitar's been around for several years several thousands years or hundreds of several thousands years or hundreds of several thousands years or hundreds of years, sorry. Um and at some point years, sorry. Um and at some point years, sorry. Um and at some point somebody said, "Hey, I'm going to put a somebody said, "Hey, I'm going to put a somebody said, "Hey, I'm going to put a pickup on a guitar. I'm going to plug it pickup on a guitar. I'm going to plug it pickup on a guitar. I'm going to plug it into a speaker and now we have rock and into a speaker and now we have rock and into a speaker and now we have rock and roll. We're going to make it super roll. We're going to make it super roll. We're going to make it super loud." That was pretty awesome. Uh and loud." That was pretty awesome. Uh and loud." That was pretty awesome. Uh and then of course we have a lot of people then of course we have a lot of people then of course we have a lot of people getting these stomp boxes these these getting these stomp boxes these these getting these stomp boxes these these effects pedals. We put them all effects pedals. We put them all effects pedals. We put them all together. We have really awesome sounds. together. We have really awesome sounds. together. We have really awesome sounds. You know, really famous people have a You know, really famous people have a You know, really famous people have a lot of these. Uh and then at some point lot of these. Uh and then at some point lot of these. Uh and then at some point Peter Peter Frampton uh came along and Peter Peter Frampton uh came along and Peter Peter Frampton uh came along and said, "Hey, what if we get the sound of said, "Hey, what if we get the sound of said, "Hey, what if we get the sound of the guitar, put it through it an actual the guitar, put it through it an actual the guitar, put it through it an actual hose like a physical hose, put it in my hose like a physical hose, put it in my hose like a physical hose, put it in my mouth, and then basically play the mouth, and then basically play the mouth, and then basically play the guitar sound in my mouth, and inform guitar sound in my mouth, and inform guitar sound in my mouth, and inform some sort of words." So that's pretty some sort of words." So that's pretty some sort of words." So that's pretty awesome that that's known as the talk awesome that that's known as the talk awesome that that's known as the talk box. Uh and then in the last couple box. Uh and then in the last couple box. Uh and then in the last couple decades we have a lot of progress made decades we have a lot of progress made decades we have a lot of progress made with software emulation. Uh you think with software emulation. Uh you think with software emulation. Uh you think of, you know, Pro Tools, you think of of, you know, Pro Tools, you think of of, you know, Pro Tools, you think of Logic uh Logic uh Logic uh uh Fruity Loops. There's a lot of uh Fruity Loops. There's a lot of uh Fruity Loops. There's a lot of software even to the point where all the software even to the point where all the software even to the point where all the effects are in software now. There is effects are in software now. There is effects are in software now. There is the argument made hot take uh that maybe the argument made hot take uh that maybe the argument made hot take uh that maybe you don't need all those physical you don't need all those physical you don't need all those physical effects pedals anymore. So so there's effects pedals anymore. So so there's effects pedals anymore. So so there's that question, right? And then of course that question, right? And then of course that question, right? And then of course looking forward it's what is that next looking forward it's what is that next looking forward it's what is that next evolution of the guitar uh evolution of the guitar uh evolution of the guitar uh with AI in the picture?

with AI in the picture? with AI in the picture? So with that said, I want to take you So with that said, I want to take you So with that said, I want to take you back several years ago uh Halloween time back several years ago uh Halloween time back several years ago uh Halloween time I was passing out candy, bunch of I was passing out candy, bunch of I was passing out candy, bunch of trick-or-treaters, you know, it got kind trick-or-treaters, you know, it got kind trick-or-treaters, you know, it got kind of boring. So I was like, "Hey, what if of boring. So I was like, "Hey, what if of boring. So I was like, "Hey, what if I go and actually bring my guitar out I go and actually bring my guitar out I go and actually bring my guitar out with my amp and just play?" Uh so for with my amp and just play?" Uh so for with my amp and just play?" Uh so for the past what 10 or plus years or so, the past what 10 or plus years or so, the past what 10 or plus years or so, I've actually been playing guitar on I've actually been playing guitar on I've actually been playing guitar on Halloween passing out candy and it's Halloween passing out candy and it's Halloween passing out candy and it's kind of fun. Uh and then fast forward kind of fun. Uh and then fast forward kind of fun. Uh and then fast forward probably 3 years ago, I decided to dress probably 3 years ago, I decided to dress probably 3 years ago, I decided to dress up as Eddie Munson. Anyone know Eddie up as Eddie Munson. Anyone know Eddie up as Eddie Munson. Anyone know Eddie Munson? Stranger Things guitar player Munson? Stranger Things guitar player Munson? Stranger Things guitar player guy. guy. guy. So pretty awesome guy. So I dressed up So pretty awesome guy. So I dressed up So pretty awesome guy. So I dressed up as him and I was like, "Yeah, this is as him and I was like, "Yeah, this is as him and I was like, "Yeah, this is going to be so fun. I can play the music going to be so fun. I can play the music going to be so fun. I can play the music with him, whatever." And you kind of go with him, whatever." And you kind of go with him, whatever." And you kind of go the Oh, hold on. So a bunch of little Stranger Things So a bunch of little Stranger Things awesomeness, right? Uh but figured out, awesomeness, right? Uh but figured out, awesomeness, right? Uh but figured out, "Hey, what if there was like some more "Hey, what if there was like some more "Hey, what if there was like some more stuff I could put in this whole stuff I could put in this whole stuff I could put in this whole experience?" And so I decided to go and experience?" And so I decided to go and experience?" And so I decided to go and build a little app that would actually build a little app that would actually build a little app that would actually paint the Stranger Things alphabet on my paint the Stranger Things alphabet on my paint the Stranger Things alphabet on my garage door just because it's fun, garage door just because it's fun, garage door just because it's fun, right? Uh and so let me just show you a right? Uh and so let me just show you a right? Uh and so let me just show you a quick example of how this worked. So quick example of how this worked. So quick example of how this worked. So effectively, make sure I have the mic effectively, make sure I have the mic effectively, make sure I have the mic permission. Whenever I play a note, it would go and communicate whatever it would go and communicate whatever lights. And I even had it to where I lights. And I even had it to where I lights. And I even had it to where I could actually uh set custom messages could actually uh set custom messages could actually uh set custom messages such as Happy Halloween, all that fun such as Happy Halloween, all that fun such as Happy Halloween, all that fun stuff. It was it was overall pretty fun stuff. It was it was overall pretty fun stuff. It was it was overall pretty fun uh just to kind of mess with that,

uh just to kind of mess with that, uh just to kind of mess with that, right? So once again, you know, whatever right? So once again, you know, whatever right? So once again, you know, whatever I type in there, I could actually spell I type in there, I could actually spell I type in there, I could actually spell whatever and and very much uh a cool nod whatever and and very much uh a cool nod whatever and and very much uh a cool nod to Stranger Things. But it kind of got to Stranger Things. But it kind of got to Stranger Things. But it kind of got me thinking like what is the next me thinking like what is the next me thinking like what is the next evolution of this project? And I settled evolution of this project? And I settled evolution of this project? And I settled on this idea of like, "Hey, how hard on this idea of like, "Hey, how hard on this idea of like, "Hey, how hard would it be to make my guitar speak?" would it be to make my guitar speak?" would it be to make my guitar speak?" It sounds easy, maybe, maybe not. So, so It sounds easy, maybe, maybe not. So, so It sounds easy, maybe, maybe not. So, so today I want to share kind of my journey today I want to share kind of my journey today I want to share kind of my journey in this process and where kind of I'm at in this process and where kind of I'm at in this process and where kind of I'm at today. today. today. So, looking around the different tools, So, looking around the different tools, So, looking around the different tools, there's a framework out there called there's a framework out there called there's a framework out there called JUCE. Really awesome for anyone building JUCE. Really awesome for anyone building JUCE. Really awesome for anyone building audio software, look into JUCE. It's audio software, look into JUCE. It's audio software, look into JUCE. It's it's pretty good. it's pretty good. it's pretty good. And then of course there's a a number of And then of course there's a a number of And then of course there's a a number of plugins plugins plugins sorry, plugin formats out there sorry, plugin formats out there sorry, plugin formats out there for your your digital audio workstation. for your your digital audio workstation. for your your digital audio workstation. And for those that are not aware, your And for those that are not aware, your And for those that are not aware, your DAW is effectively your IDE but for DAW is effectively your IDE but for DAW is effectively your IDE but for musicians and and music producers. And musicians and and music producers. And musicians and and music producers. And then I then I then I started with some text to speech stuff started with some text to speech stuff started with some text to speech stuff with Piper and some built-in Apple with Piper and some built-in Apple with Piper and some built-in Apple stuff. And then some other really fun stuff. And then some other really fun stuff. And then some other really fun digital signal processing. digital signal processing. digital signal processing. So, with that said, my first stab here So, with that said, my first stab here So, with that said, my first stab here was I want to get raw text so I could was I want to get raw text so I could was I want to get raw text so I could just type in whatever text I want push just type in whatever text I want push just type in whatever text I want push it through the text to speech, get the it through the text to speech, get the it through the text to speech, get the audio clip, and then whenever I play a audio clip, and then whenever I play a audio clip, and then whenever I play a note on the guitar, I want to go and note on the guitar, I want to go and note on the guitar, I want to go and play that back. So, let's go see how play that back. So, let's go see how play that back. So, let's go see how that works.

that works. that works. So, switching over to my Logic Pro here. So, switching over to my Logic Pro here. So, switching over to my Logic Pro here. And this is the plugin I made. So, it's And this is the plugin I made. So, it's And this is the plugin I made. So, it's just like any other plugin in Logic just like any other plugin in Logic just like any other plugin in Logic where you just plop it in there. It's where you just plop it in there. It's where you just plop it in there. It's just, you know, chaining all the effects just, you know, chaining all the effects just, you know, chaining all the effects together. And so, this is what I came up together. And so, this is what I came up together. And so, this is what I came up with. with. with. >> Developers. >> Developers. >> Developers. >> So, it's playing >> So, it's playing >> So, it's playing >> Developers. >> Developers. >> Developers. >> Pretty awesome, right? >> Pretty awesome, right? >> Pretty awesome, right? >> Developers. >> Developers. >> Developers. >> Yeah, kind of kind of reminds me of >> Yeah, kind of kind of reminds me of >> Yeah, kind of kind of reminds me of something, right? something, right? something, right? >> Developers. >> Developers. >> Developers. >> Start clapping. Everyone, start >> Start clapping. Everyone, start >> Start clapping. Everyone, start clapping. Developers. Developers. clapping. Developers. Developers. clapping. Developers. Developers. Developers. Developers. Developers. Developers. Developers. Developers. Developers. Developers. Developers. Awesome, thank you. That's pretty Awesome, thank you. That's pretty Awesome, thank you. That's pretty awesome. You guys are great. Um awesome. You guys are great. Um awesome. You guys are great. Um So, I got it to where it's playing an So, I got it to where it's playing an So, I got it to where it's playing an actual audio file. That's pretty actual audio file. That's pretty actual audio file. That's pretty awesome. But it turns out, you know, in awesome. But it turns out, you know, in awesome. But it turns out, you know, in English or in any languages for that English or in any languages for that English or in any languages for that matter, there's more than one word. So, matter, there's more than one word. So, matter, there's more than one word. So, the next kind of evolution Oh, that's a the next kind of evolution Oh, that's a the next kind of evolution Oh, that's a little bit chatty. The next evolution little bit chatty. The next evolution little bit chatty. The next evolution there is let's actually go and slice it there is let's actually go and slice it there is let's actually go and slice it per word now. per word now. per word now. So, I got it to the point where it's So, I got it to the point where it's So, I got it to the point where it's playing. Now it's going to slice per playing. Now it's going to slice per playing. Now it's going to slice per word. word. word. >> Look at me. >> Look at me. >> Look at me. I can speak. Look I can speak. Look I can speak. Look >> Oh.

>> Oh. >> Oh. >> at >> at >> at >> Oh. >> Oh. >> Oh. >> me. >> me. >> me. I can speak. I can speak. I can speak. >> So, now it's speaking words, and that's >> So, now it's speaking words, and that's >> So, now it's speaking words, and that's pretty awesome, right? Uh but it turns pretty awesome, right? Uh but it turns pretty awesome, right? Uh but it turns out that uh out that uh out that uh as we speak, there's a number of as we speak, there's a number of as we speak, there's a number of challenges in how we automatically slice challenges in how we automatically slice challenges in how we automatically slice words. And so, I looked into this thing words. And so, I looked into this thing words. And so, I looked into this thing called energy gap segmentation. The called energy gap segmentation. The called energy gap segmentation. The general idea here is if you look in a general idea here is if you look in a general idea here is if you look in a any waveform over here, we see that any waveform over here, we see that any waveform over here, we see that here's a bunch of of words that we're here's a bunch of of words that we're here's a bunch of of words that we're speaking, right? Uh the idea there is speaking, right? Uh the idea there is speaking, right? Uh the idea there is there's typically silence in between there's typically silence in between there's typically silence in between words. So, let's just cut it whenever words. So, let's just cut it whenever words. So, let's just cut it whenever the decibels are are very much close to the decibels are are very much close to the decibels are are very much close to zero, right? Uh but the issue with that zero, right? Uh but the issue with that zero, right? Uh but the issue with that is there's actually sometimes when you is there's actually sometimes when you is there's actually sometimes when you know as I'm speaking right now, for know as I'm speaking right now, for know as I'm speaking right now, for example, uh there's actually no silence example, uh there's actually no silence example, uh there's actually no silence in between some of my words. So, it gets in between some of my words. So, it gets in between some of my words. So, it gets a little bit challenging to where it's a little bit challenging to where it's a little bit challenging to where it's not 100% foolproof, right? not 100% foolproof, right? not 100% foolproof, right? So, beyond that, I looked into this So, beyond that, I looked into this So, beyond that, I looked into this thing called sonority peak syllaba thing called sonority peak syllaba thing called sonority peak syllaba syllabifier. That's a hard word to say. syllabifier. That's a hard word to say. syllabifier. That's a hard word to say. I effectively identifying the syllables I effectively identifying the syllables I effectively identifying the syllables of the audio signal and of the audio signal and of the audio signal and identifying that there's vowels in here. identifying that there's vowels in here. identifying that there's vowels in here. Vowels typically lead to syllables. Vowels typically lead to syllables. Vowels typically lead to syllables. That's kind of the idea. So, I said, That's kind of the idea. So, I said, That's kind of the idea. So, I said, "Okay, let's take the sonority peak, add "Okay, let's take the sonority peak, add "Okay, let's take the sonority peak, add it to the the energy gap, and figure out it to the the energy gap, and figure out it to the the energy gap, and figure out if we could just make that work if we could just make that work if we could just make that work automatically."

automatically." automatically." So, with that said, uh made it kind of So, with that said, uh made it kind of So, with that said, uh made it kind of work. So, let's just play this one. work. So, let's just play this one. work. So, let's just play this one. >> Thank you for letting me >> Thank you for letting me >> Thank you for letting me be be be >> Nope. Oh. >> Nope. Oh. >> Nope. Oh. >> here. >> here. >> here. It It It feels feels feels so so so good good good >> Oh. >> Oh. >> Oh. >> to get out of >> to get out of >> to get out of my my my big big big tiny tiny tiny cage cage cage once once once in a in a in a while. while. while. >> So, there you go. It's working pretty >> So, there you go. It's working pretty >> So, there you go. It's working pretty well. Not quite as good as I want to. well. Not quite as good as I want to. well. Not quite as good as I want to. So, long story short, uh I settled on So, long story short, uh I settled on So, long story short, uh I settled on just the ability to go and actually I just the ability to go and actually I just the ability to go and actually I could drag this and manually edit some could drag this and manually edit some could drag this and manually edit some of these of these of these uh uh uh uh segments in here. So, worked okay, uh segments in here. So, worked okay, uh segments in here. So, worked okay, right? Um right? Um right? Um but moving on, you know, we with the but moving on, you know, we with the but moving on, you know, we with the spirit of evolving the thought, evolving spirit of evolving the thought, evolving spirit of evolving the thought, evolving the project, right? It's like, "Okay, the project, right? It's like, "Okay, the project, right? It's like, "Okay, we're we're having the AI say stuff, we're we're having the AI say stuff, we're we're having the AI say stuff, that's great. But what if we could that's great. But what if we could that's great. But what if we could actually make it sing?" Let's take it to actually make it sing?" Let's take it to actually make it sing?" Let's take it to the next step cuz this is music, why the next step cuz this is music, why the next step cuz this is music, why not, right? Uh so, looked into pitch not, right? Uh so, looked into pitch not, right? Uh so, looked into pitch detection. And for those that are not detection. And for those that are not detection. And for those that are not aware, uh as you hear any any noise out aware, uh as you hear any any noise out aware, uh as you hear any any noise out there, there's typically multiple there, there's typically multiple there, there's typically multiple frequencies going on at any given time.

frequencies going on at any given time. frequencies going on at any given time. You know, you think of when you play the You know, you think of when you play the You know, you think of when you play the the C key on the piano, uh there is the C key on the piano, uh there is the C key on the piano, uh there is definitely a fundamental frequency or definitely a fundamental frequency or definitely a fundamental frequency or the the kind of the the one that we the the kind of the the one that we the the kind of the the one that we identify as the note, but there's a identify as the note, but there's a identify as the note, but there's a bunch of other frequencies. So, we bunch of other frequencies. So, we bunch of other frequencies. So, we needed a way to go and figure out like, needed a way to go and figure out like, needed a way to go and figure out like, "How do we detect that fundamental "How do we detect that fundamental "How do we detect that fundamental frequency? So, when I press something on frequency? So, when I press something on frequency? So, when I press something on the guitar, how do I translate that into the guitar, how do I translate that into the guitar, how do I translate that into an actual note?" Uh and then I found an actual note?" Uh and then I found an actual note?" Uh and then I found this uh Yin pitch algorithm. Basically, this uh Yin pitch algorithm. Basically, this uh Yin pitch algorithm. Basically, what it does is it detects the pitch. Uh what it does is it detects the pitch. Uh what it does is it detects the pitch. Uh I won't get into all the details, but I won't get into all the details, but I won't get into all the details, but look it up, it's kind of a fun uh really look it up, it's kind of a fun uh really look it up, it's kind of a fun uh really cool way of detecting the pitch. Uh but cool way of detecting the pitch. Uh but cool way of detecting the pitch. Uh but effectively, what I do is I play the effectively, what I do is I play the effectively, what I do is I play the guitar, uh I detect the the pitch, I I guitar, uh I detect the the pitch, I I guitar, uh I detect the the pitch, I I make what's effectively a pitch sawtooth make what's effectively a pitch sawtooth make what's effectively a pitch sawtooth or aka a synthesized note. Uh so, if for or aka a synthesized note. Uh so, if for or aka a synthesized note. Uh so, if for those that are not familiar with how those that are not familiar with how those that are not familiar with how audio works on the the computer, you audio works on the the computer, you audio works on the the computer, you think of all the electronic music, all think of all the electronic music, all think of all the electronic music, all that stuff, uh that is basically uh a that stuff, uh that is basically uh a that stuff, uh that is basically uh a synthesized note. We have ADSR, which synthesized note. We have ADSR, which synthesized note. We have ADSR, which effectively are the levers to figure out effectively are the levers to figure out effectively are the levers to figure out how to actually make the note sound in how to actually make the note sound in how to actually make the note sound in different ways. Uh and then so, different ways. Uh and then so, different ways. Uh and then so, basically, we get the pitch notes or the basically, we get the pitch notes or the basically, we get the pitch notes or the synthesized note. Uh we then push it synthesized note. Uh we then push it synthesized note. Uh we then push it through uh the the voice clip through uh the the voice clip through uh the the voice clip effectively. So, think of the talk box.

effectively. So, think of the talk box. effectively. So, think of the talk box. Uh we're kind of filling up the cavity Uh we're kind of filling up the cavity Uh we're kind of filling up the cavity of the talk of the voice, that is. And of the talk of the voice, that is. And of the talk of the voice, that is. And we push it through a vocoder, and it we push it through a vocoder, and it we push it through a vocoder, and it should sing. So, that's kind of the should sing. So, that's kind of the should sing. So, that's kind of the idea, right? idea, right? idea, right? So, with that said, let's go ahead and So, with that said, let's go ahead and So, with that said, let's go ahead and jam out a little bit because I have a jam out a little bit because I have a jam out a little bit because I have a guitar, and it's fun. So. guitar, and it's fun. So. guitar, and it's fun. So. Let me get back to Logic here. Uh so, I'm going to play some chords. Uh Uh so, I'm going to play some chords. Uh was going to uh play a song that is very was going to uh play a song that is very was going to uh play a song that is very much related to the talk or the the much related to the talk or the the much related to the talk or the the title of my talk, uh while my guitar title of my talk, uh while my guitar title of my talk, uh while my guitar gently speaks, but because it's going to gently speaks, but because it's going to gently speaks, but because it's going to be posted online, I don't want to muddy be posted online, I don't want to muddy be posted online, I don't want to muddy up the waters with any copyright things. up the waters with any copyright things. up the waters with any copyright things. So, I will play some chords that may or So, I will play some chords that may or So, I will play some chords that may or may not sound similar to a famous song. may not sound similar to a famous song. may not sound similar to a famous song. So, So, that's the backing track. So, let's So, that's the backing track. So, let's go ahead go ahead go ahead and have some and have some and have some AI speaking on top of it.

>> I'm >> I'm here. here. here. This This This >> No. >> No. >> No. All right, technical difficulties. Let's All right, technical difficulties. Let's All right, technical difficulties. Let's try that again. Live demos always the best. Live demos always the best. >> I'm here. >> I'm here. >> I'm here. This This This can't >> Uh technical difficulties again, so >> Uh technical difficulties again, so sorry for that, but let's just run sorry for that, but let's just run sorry for that, but let's just run through it and see what happens. Thanks. That's peak. Thanks. That's peak. I'm the guitar that can't speak. I'm the guitar that can't speak. I'm the guitar that can't speak. So. So. So. >> Awesome. So, there you go. So, some bugs >> Awesome. So, there you go. So, some bugs >> Awesome. So, there you go. So, some bugs to work out, but overall we are able to to work out, but overall we are able to to work out, but overall we are able to to say words on top of this and for what to say words on top of this and for what to say words on top of this and for what it's worth I'm trying to mix the uh the it's worth I'm trying to mix the uh the it's worth I'm trying to mix the uh the the synthesized note uh with this the synthesized note uh with this the synthesized note uh with this clarity lever right here. So, balance or clarity lever right here. So, balance or clarity lever right here. So, balance or mixing the the synthesized note with the mixing the the synthesized note with the mixing the the synthesized note with the actual voice that the AI is giving. Uh actual voice that the AI is giving. Uh actual voice that the AI is giving. Uh so so, that worked pretty good. Um But, there's also this other question, But, there's also this other question, you know, given that uh you know, given that uh you know, given that uh when you speak it's typically when you speak it's typically when you speak it's typically conversational, what if I could take conversational, what if I could take conversational, what if I could take this to the next level? And what if we this to the next level? And what if we this to the next level? And what if we actually had this microphone right here actually had this microphone right here actually had this microphone right here where I could speak into the mic, it where I could speak into the mic, it where I could speak into the mic, it would then respond on the guitar. Uh so, would then respond on the guitar. Uh so, would then respond on the guitar. Uh so, the way that I I accomplished this is the way that I I accomplished this is the way that I I accomplished this is speak in the mic, use uh speech to text

speak in the mic, use uh speech to text speak in the mic, use uh speech to text so, Whisper, uh put that into raw text, so, Whisper, uh put that into raw text, so, Whisper, uh put that into raw text, run a local model on my computer. Uh it run a local model on my computer. Uh it run a local model on my computer. Uh it doesn't really matter what LLM, just any doesn't really matter what LLM, just any doesn't really matter what LLM, just any local model. I have a conversation local model. I have a conversation local model. I have a conversation coming and then from that output go and coming and then from that output go and coming and then from that output go and plop it on the guitar. So, let's see how plop it on the guitar. So, let's see how plop it on the guitar. So, let's see how this works. Anyone have a question you want to ask Anyone have a question you want to ask my guitar? Anybody? Anybody? >> What is reality? >> What is reality? >> What is reality? >> Let's do it. So, it's thinking, going through the LLM So, it's thinking, going through the LLM things. things. things. And let's see what it says. And let's see what it says. And let's see what it says. >> That's quite a question >> That's quite a question >> That's quite a question to to to start with. How How does your music help you understand does your music help you understand does your music help you understand and and and that elusive concept of reality?

that elusive concept of reality? that elusive concept of reality? What do you write in What do you write in What do you write in that? that? that? >> So, yeah, very existential like that. >> So, yeah, very existential like that. >> So, yeah, very existential like that. So, thank you for that suggestion there. So, thank you for that suggestion there. So, thank you for that suggestion there. So, so there you go. We have a choppy So, so there you go. We have a choppy So, so there you go. We have a choppy version, but but it is working. So, so version, but but it is working. So, so version, but but it is working. So, so that's a win, right? Uh that's a win, right? Uh that's a win, right? Uh Awesome, and thanks for the clap, sir. Awesome, and thanks for the clap, sir. Awesome, and thanks for the clap, sir. Um Um Um But really like it's not quite singing But really like it's not quite singing But really like it's not quite singing yet, right? So, moving on in this yet, right? So, moving on in this yet, right? So, moving on in this project is like how do I actually make project is like how do I actually make project is like how do I actually make it sing, right? Uh I found a lot of open it sing, right? Uh I found a lot of open it sing, right? Uh I found a lot of open source options out there as far as source options out there as far as source options out there as far as recordings or samples, if you will. Uh recordings or samples, if you will. Uh recordings or samples, if you will. Uh there's a thing called vocal set out there's a thing called vocal set out there's a thing called vocal set out there. They had basically recorded a there. They had basically recorded a there. They had basically recorded a bunch of singers, and I could use those bunch of singers, and I could use those bunch of singers, and I could use those audio files to go and do some fun stuff audio files to go and do some fun stuff audio files to go and do some fun stuff with them. And effectively I took those with them. And effectively I took those with them. And effectively I took those samples, I used a project called World, samples, I used a project called World, samples, I used a project called World, which helps with things like pitch which helps with things like pitch which helps with things like pitch shifting and some other things, and I'm shifting and some other things, and I'm shifting and some other things, and I'm able to then shift the the pitches of able to then shift the the pitches of able to then shift the the pitches of everyone singing on those audio clips everyone singing on those audio clips everyone singing on those audio clips and map it to the guitar. Uh this is a and map it to the guitar. Uh this is a and map it to the guitar. Uh this is a very heavy process, and so I can't very heavy process, and so I can't very heavy process, and so I can't really do that live. I have to really do that live. I have to really do that live. I have to effectively pre-bake that. And then once effectively pre-bake that. And then once effectively pre-bake that. And then once it's already pre-baked, I could then go it's already pre-baked, I could then go it's already pre-baked, I could then go and jam out with it. So, with that said, and jam out with it. So, with that said, and jam out with it. So, with that said, uh let me go ahead and show that example uh let me go ahead and show that example uh let me go ahead and show that example here.

here. here. Um. So, uh because it takes so long, I So, uh because it takes so long, I actually just started with the vowel actually just started with the vowel actually just started with the vowel sounds. So, there's somebody singing all sounds. So, there's somebody singing all sounds. So, there's somebody singing all each of the five vowels. And so, we'll each of the five vowels. And so, we'll each of the five vowels. And so, we'll see how that sounds, right? Kind of fun, kind of weird, but overall Kind of fun, kind of weird, but overall it's working. It's it's closer to it's working. It's it's closer to it's working. It's it's closer to singing, right? Uh singing, right? Uh singing, right? Uh So, let's go ahead and throw that on top So, let's go ahead and throw that on top So, let's go ahead and throw that on top of all the chords over here. of all the chords over here. of all the chords over here. And let's see if we can make it sound And let's see if we can make it sound And let's see if we can make it sound decently well. decently well. decently well. Uh Uh Uh So, we'll just play these chords again. So, not quite your opera singer, but So, not quite your opera singer, but getting closer and and so I I think getting closer and and so I I think getting closer and and so I I think there's something awesome there. Uh so, there's something awesome there. Uh so, there's something awesome there. Uh so, effectively once again, that is uh going effectively once again, that is uh going effectively once again, that is uh going through uh the whole synthesized through uh the whole synthesized through uh the whole synthesized process, uh putting a sample, shifting process, uh putting a sample, shifting process, uh putting a sample, shifting the the pitch shift of the sample, and the the pitch shift of the sample, and the the pitch shift of the sample, and then effectively mapping each fret or then effectively mapping each fret or then effectively mapping each fret or each note on the guitar to one of those each note on the guitar to one of those each note on the guitar to one of those samples that are pre-baked effectively, samples that are pre-baked effectively, samples that are pre-baked effectively, right? Uh uh right? Uh uh right? Uh uh So, with that said, So, with that said, So, with that said, you know, where do I want to take this you know, where do I want to take this you know, where do I want to take this next? Uh very much want to get into more next? Uh very much want to get into more next? Uh very much want to get into more of of the, you know, AI heavier options.

of of the, you know, AI heavier options. of of the, you know, AI heavier options. Uh if anyone has any other ideas of how Uh if anyone has any other ideas of how Uh if anyone has any other ideas of how to make the guitar sing, feel free to to make the guitar sing, feel free to to make the guitar sing, feel free to come up to me after. I think it's pretty come up to me after. I think it's pretty come up to me after. I think it's pretty awesome. Uh but more importantly, you awesome. Uh but more importantly, you awesome. Uh but more importantly, you know, everyone here, uh going back to know, everyone here, uh going back to know, everyone here, uh going back to the the charge to go and and build some the the charge to go and and build some the the charge to go and and build some awesome, you know, side projects, some awesome, you know, side projects, some awesome, you know, side projects, some passion projects of yours, like what is passion projects of yours, like what is passion projects of yours, like what is that for you? Go and build awesome that for you? Go and build awesome that for you? Go and build awesome things because nowadays with AI, we things because nowadays with AI, we things because nowadays with AI, we could build so many really cool things, could build so many really cool things, could build so many really cool things, and time is typically not the the big and time is typically not the the big and time is typically not the the big time suck that that it once was, right? time suck that that it once was, right? time suck that that it once was, right? So, with that said, uh be awesome uh or So, with that said, uh be awesome uh or So, with that said, uh be awesome uh or or be good to each other, and thank you or be good to each other, and thank you or be good to each other, and thank you very much. very much. very much. >> Woo!