EPISODE 23 - Refchecker, a Useful Tool Coded by Mark and AI

So, here's the GitHub repo for it. So, here's the GitHub repo for it. >> Could you have a larger monitor? >> Could you have a larger monitor? >> Could you have a larger monitor? >> No. >> No. >> No. >> Okay. Could you at least, for God's >> Okay. Could you at least, for God's >> Okay. Could you at least, for God's sake, zoom in because we can't see a sake, zoom in because we can't see a sake, zoom in because we can't see a damn thing. damn thing. damn thing. >> Does the live zoom work? >> Does the live zoom work? >> Does the live zoom work? >> The live zoom is working, but it it it >> The live zoom is working, but it it it >> The live zoom is working, but it it it makes it barely tolerable, but okay. Ref makes it barely tolerable, but okay. Ref makes it barely tolerable, but okay. Ref checker. I see. There we go. This will checker. I see. There we go. This will checker. I see. There we go. This will work for the toilet crowd. work for the toilet crowd. work for the toilet crowd. >> So, here's a GitHub repo for it. Yeah. >> So, here's a GitHub repo for it. Yeah. >> So, here's a GitHub repo for it. Yeah. So, ref checker. um academic paper So, ref checker. um academic paper So, ref checker. um academic paper reference checker. Uh here's some reference checker. Uh here's some reference checker. Uh here's some example output, but I'll show a demo of example output, but I'll show a demo of example output, but I'll show a demo of it. You can see some of the kind of it. You can see some of the kind of it. You can see some of the kind of warnings or errors. warnings or errors. warnings or errors. >> But see, archive has all these URLs, but >> But see, archive has all these URLs, but >> But see, archive has all these URLs, but like none of these has decided I'm like none of these has decided I'm like none of these has decided I'm looking at these DOI, archive, and and looking at these DOI, archive, and and looking at these DOI, archive, and and the semantic scholar all have different the semantic scholar all have different the semantic scholar all have different there's no one has reconciled any of there's no one has reconciled any of there's no one has reconciled any of these ID ids. There's no Dewey decimal these ID ids. There's no Dewey decimal these ID ids. There's no Dewey decimal system for these things. I can find a system for these things. I can find a system for these things. I can find a book in the library but book in the library but book in the library but >> it's partially applied. >> it's partially applied. >> it's partially applied. >> Okay. >> Okay. >> Okay. >> Um so this this part is the actual >> Um so this this part is the actual >> Um so this this part is the actual reference from the paper. This part is reference from the paper. This part is reference from the paper. This part is what the tool spits out. Like here's what the tool spits out. Like here's what the tool spits out. Like here's this this the URL that is actually this this the URL that is actually this this the URL that is actually verifying this the information on verifying this the information on verifying this the information on >> and by the way semantic scholar has >> and by the way semantic scholar has >> and by the way semantic scholar has errors in it as well.

errors in it as well. errors in it as well. >> Okay. But um but you can give it an >> Okay. But um but you can give it an >> Okay. But um but you can give it an archive paper, a local PDF, an archive archive paper, a local PDF, an archive archive paper, a local PDF, an archive ID, a local file, a latte file, a text ID, a local file, a latte file, a text ID, a local file, a latte file, a text document that just has free form document that just has free form document that just has free form references in it. references in it. references in it. >> Um now the bibliography itself and the >> Um now the bibliography itself and the >> Um now the bibliography itself and the paper uh so there's no standard for you paper uh so there's no standard for you paper uh so there's no standard for you know how the bibliography in a paper is know how the bibliography in a paper is know how the bibliography in a paper is identified. So some papers have a identified. So some papers have a identified. So some papers have a references but if you're taking a PDF references but if you're taking a PDF references but if you're taking a PDF because you have no other source for the because you have no other source for the because you have no other source for the bibliography bibliography bibliography the tool translate and translate into the tool translate and translate into the tool translate and translate into text and then has to figure out where text and then has to figure out where text and then has to figure out where the bibliography is. So it uses so it the bibliography is. So it uses so it the bibliography is. So it uses so it uses reax to go try to find out of the uses reax to go try to find out of the uses reax to go try to find out of the min out of the zillion you know ways the min out of the zillion you know ways the min out of the zillion you know ways the beginning and end of the of the beginning and end of the of the beginning and end of the of the bibliography bibliography bibliography >> because I don't want to give it the >> because I don't want to give it the >> because I don't want to give it the whole paper to an LLM which could be you whole paper to an LLM which could be you whole paper to an LLM which could be you know the paper could be know the paper could be know the paper could be >> so you called that intelligent pattern >> so you called that intelligent pattern >> so you called that intelligent pattern matching using matching using matching using >> a red this yeah yeah this is intelligent >> a red this yeah yeah this is intelligent >> a red this yeah yeah this is intelligent pattern matching right pattern matching right pattern matching right >> yeah yeah >> yeah yeah >> yeah yeah >> now this part once you have the section >> now this part once you have the section >> now this part once you have the section that you think is the references that you think is the references that you think is the references >> Mhm. again given first of all the PDF to >> Mhm. again given first of all the PDF to >> Mhm. again given first of all the PDF to text is full of crap annotation crap text is full of crap annotation crap text is full of crap annotation crap from PDF from PDF from PDF >> right >> right >> right >> that's some PDFs have text embedded and >> that's some PDFs have text embedded and >> that's some PDFs have text embedded and some PDFs are literally images some PDFs are literally images some PDFs are literally images >> yeah well on these references though >> yeah well on these references though >> yeah well on these references though there's like PDF formatting characters there's like PDF formatting characters there's like PDF formatting characters >> interled inside the references >> interled inside the references >> interled inside the references >> no >> no >> no >> so it's incredibly messy >> so it's incredibly messy >> so it's incredibly messy >> but even with without that again the

>> but even with without that again the >> but even with without that again the given there's given there's given there's any number of permutations for how the any number of permutations for how the any number of permutations for how the ref you know it's some kind of put the ref you know it's some kind of put the ref you know it's some kind of put the author first the author last the venue author first the author last the venue author first the author last the venue first the year first you know first the year first you know first the year first you know >> and commas in between no com periods in >> and commas in between no com periods in >> and commas in between no com periods in between like between like between like infinite number of permutations and what infinite number of permutations and what infinite number of permutations and what I started with was I'm going to make I started with was I'm going to make I started with was I'm going to make this thing do reax 2 to pull these out this thing do reax 2 to pull these out this thing do reax 2 to pull these out >> and then that became exception nightmare >> and then that became exception nightmare >> and then that became exception nightmare like another and and where the trying to like another and and where the trying to like another and and where the trying to fix the reax to address this case fix the reax to address this case fix the reax to address this case regresses this case and I just became I regresses this case and I just became I regresses this case and I just became I was like this is impossible this is was like this is impossible this is was like this is impossible this is literally impossible to solve this with literally impossible to solve this with literally impossible to solve this with reject what if we use LLM reject what if we use LLM reject what if we use LLM what if I can get like so then what I what if I can get like so then what I what if I can get like so then what I tried was just take the bibliography and tried was just take the bibliography and tried was just take the bibliography and throw it at the LLM and say pull out the throw it at the LLM and say pull out the throw it at the LLM and say pull out the references and format in this way the references and format in this way the references and format in this way the metapar prompt is a little bit more metapar prompt is a little bit more metapar prompt is a little bit more involved in that than that but involved in that than that but involved in that than that but >> voila that solved it LLM are really >> voila that solved it LLM are really >> voila that solved it LLM are really really good at taking this unstructured really good at taking this unstructured really good at taking this unstructured data and structuring it data and structuring it data and structuring it >> so with Yeah, without that this would >> so with Yeah, without that this would >> so with Yeah, without that this would have completely been a failure.

have completely been a failure. have completely been a failure. >> So LLMs can take unstructured data and >> So LLMs can take unstructured data and >> So LLMs can take unstructured data and structure it, but they still but this is structure it, but they still but this is structure it, but they still but this is a problem that you are having where LLMs a problem that you are having where LLMs a problem that you are having where LLMs are making up references. are making up references. are making up references. >> In this case, they're not well let me >> In this case, they're not well let me >> In this case, they're not well let me say this. say this. say this. >> You see what I'm saying? Like the >> You see what I'm saying? Like the >> You see what I'm saying? Like the problem you're using a thing that we problem you're using a thing that we problem you're using a thing that we can't rely on to check a thing we can't can't rely on to check a thing we can't can't rely on to check a thing we can't rely on. rely on. rely on. >> But at least there's a a formal check >> But at least there's a a formal check >> But at least there's a a formal check >> as the back stop against the you know >> as the back stop against the you know >> as the back stop against the you know the source of truth. It's not like the the source of truth. It's not like the the source of truth. It's not like the MLM is saying this reference is correct. MLM is saying this reference is correct. MLM is saying this reference is correct. >> I'm just getting the reference and then >> I'm just getting the reference and then >> I'm just getting the reference and then looking it up. looking it up. looking it up. >> There we go. Okay. So, you're This is a >> There we go. Okay. So, you're This is a >> There we go. Okay. So, you're This is a great example of a multi-disciplinary great example of a multi-disciplinary great example of a multi-disciplinary application that knows when reaxes are application that knows when reaxes are application that knows when reaxes are good, when LLMs are good. good, when LLMs are good. good, when LLMs are good. >> Exactly. >> Exactly. >> Exactly. >> And when an HTTP get is good. >> And when an HTTP get is good. >> And when an HTTP get is good. [music] >> Um, so I've vibe coded over the last >> Um, so I've vibe coded over the last month, mostly vibecoded. So I'd say 90% month, mostly vibecoded. So I'd say 90% month, mostly vibecoded. So I'd say 90% and I've had to inter 95% of the code is and I've had to inter 95% of the code is and I've had to inter 95% of the code is completely generated where I did not completely generated where I did not completely generated where I did not touch it. And about 5% is me either touch it. And about 5% is me either touch it. And about 5% is me either making minor tweaks or actually having making minor tweaks or actually having making minor tweaks or actually having to to debug so I understand the problem to to debug so I understand the problem to to debug so I understand the problem so I can say this is the no this is the so I can say this is the no this is the so I can say this is the no this is the problem because the LLM's problem because the LLM's problem because the LLM's >> were too obtuse to figure it out and got >> were too obtuse to figure it out and got >> were too obtuse to figure it out and got stuck in the I fixed it. No you didn't.

stuck in the I fixed it. No you didn't. stuck in the I fixed it. No you didn't. I fixed it. No you didn't. I fixed it. No you didn't. I fixed it. No you didn't. Try again cuz it's still broken. And do Try again cuz it's still broken. And do Try again cuz it's still broken. And do the like I did. I verified it and it's the like I did. I verified it and it's the like I did. I verified it and it's fixed. I'm like what what are you fixed. I'm like what what are you fixed. I'm like what what are you verifying it on? because I just ran the verifying it on? because I just ran the verifying it on? because I just ran the verification that you said you ran and verification that you said you ran and verification that you said you ran and it fails. Um, so I've run into that m it fails. Um, so I've run into that m it fails. Um, so I've run into that m multiple times. multiple times. multiple times. >> Uhhuh. So we've talked about the dangers >> Uhhuh. So we've talked about the dangers >> Uhhuh. So we've talked about the dangers of vibe coding into production, but for of vibe coding into production, but for of vibe coding into production, but for you, we use the term vibes just because you, we use the term vibes just because you, we use the term vibes just because that's the term that people use, but you that's the term that people use, but you that's the term that people use, but you knew what you wanted, right? knew what you wanted, right? knew what you wanted, right? >> Yeah. Yeah. I mean, I've been very >> Yeah. Yeah. I mean, I've been very >> Yeah. Yeah. I mean, I've been very specific about the way that I want specific about the way that I want specific about the way that I want things to work, things to work, things to work, >> right? >> right? >> right? >> And then um >> And then um >> And then um but I've I've let a long leash on how it but I've I've let a long leash on how it but I've I've let a long leash on how it does it. does it. does it. >> Okay. Um, and where I've gotten involved >> Okay. Um, and where I've gotten involved >> Okay. Um, and where I've gotten involved is I'll periodically go look to see what is I'll periodically go look to see what is I'll periodically go look to see what it's doing. Like I'll look at the diffs it's doing. Like I'll look at the diffs it's doing. Like I'll look at the diffs >> or I'll go look at the code to to see. >> or I'll go look at the code to to see. >> or I'll go look at the code to to see. And I've run into some interesting And I've run into some interesting And I've run into some interesting things. things. things. >> Um, >> Um, >> Um, like one like one like one one it um it basically wanted to make a one it um it basically wanted to make a one it um it basically wanted to make a monolith monolithic file everything in monolith monolithic file everything in monolith monolithic file everything in one file. So I had to be like no start one file. So I had to be like no start one file. So I had to be like no start break it apart and put yeah break it apart and put yeah break it apart and put yeah >> different pieces. The other thing that >> different pieces. The other thing that >> different pieces. The other thing that it did it did it did >> was um repetitive code duplic. So, I've >> was um repetitive code duplic. So, I've >> was um repetitive code duplic. So, I've got like it's a plug-in architecture for got like it's a plug-in architecture for got like it's a plug-in architecture for this tool where you can plug in this tool where you can plug in this tool where you can plug in different reference validators. So, let different reference validators. So, let different reference validators. So, let me give you a two second spiel on what me give you a two second spiel on what me give you a two second spiel on what the tool is.

the tool is. the tool is. >> Mhm. >> Mhm. >> Mhm. >> Uh, and the tool is inspired by the rise >> Uh, and the tool is inspired by the rise >> Uh, and the tool is inspired by the rise of hallucinated references in academic of hallucinated references in academic of hallucinated references in academic publications. And I'm like, should be publications. And I'm like, should be publications. And I'm like, should be possible to or there should be some possible to or there should be some possible to or there should be some reference validator out there. reference validator out there. reference validator out there. >> Well, turns out there's some >> Well, turns out there's some >> Well, turns out there's some professional companies you can pay for professional companies you can pay for professional companies you can pay for some service that go validate some service that go validate some service that go validate references, but I I don't even know the references, but I I don't even know the references, but I I don't even know the way they work. there's nothing off the way they work. there's nothing off the way they work. there's nothing off the shelf, no tool that you can say, here shelf, no tool that you can say, here shelf, no tool that you can say, here are references, go and check them. Um, are references, go and check them. Um, are references, go and check them. Um, so I was like, I let me go about writing so I was like, I let me go about writing so I was like, I let me go about writing a tool. Surely can't be that hard. a tool. Surely can't be that hard. a tool. Surely can't be that hard. >> Famous last words of a developer. It's >> Famous last words of a developer. It's >> Famous last words of a developer. It's way harder than it seems, way harder than it seems, way harder than it seems, >> of course. >> of course. >> of course. >> But, uh, one of the things that I found, >> But, uh, one of the things that I found, >> But, uh, one of the things that I found, um, is I have plugins for sources, um, is I have plugins for sources, um, is I have plugins for sources, online database, databases that have online database, databases that have online database, databases that have APIs where you can check a reference and APIs where you can check a reference and APIs where you can check a reference and it's pluggable and it's pluggable and it's pluggable and >> yeah, that's what you're thinking like >> yeah, that's what you're thinking like >> yeah, that's what you're thinking like identify all the links, look all the identify all the links, look all the identify all the links, look all the links up in n number of references links up in n number of references links up in n number of references >> and if you don't find it in this place, >> and if you don't find it in this place, >> and if you don't find it in this place, go to the next API and go so so on. So go to the next API and go so so on. So go to the next API and go so so on. So there'd be a thing where the author there'd be a thing where the author there'd be a thing where the author lists come back from these APIs in uh in lists come back from these APIs in uh in lists come back from these APIs in uh in a format um in in some format and then a format um in in some format and then a format um in in some format and then need to get normalized and I go look at need to get normalized and I go look at need to get normalized and I go look at cuz there's I like here's another paper cuz there's I like here's another paper cuz there's I like here's another paper and uh the author lists or matching is and uh the author lists or matching is and uh the author lists or matching is broken. So go do normalization and fix broken. So go do normalization and fix broken. So go do normalization and fix it. So I go look at after a few rounds it. So I go look at after a few rounds it. So I go look at after a few rounds of this and what it's done is do author of this and what it's done is do author of this and what it's done is do author normalization for normalization for normalization for API 1, do the same normalization for API

API 1, do the same normalization for API API 1, do the same normalization for API 2, do the name and it's like duplicated 2, do the name and it's like duplicated 2, do the name and it's like duplicated the code. It's like now I need to go put the code. It's like now I need to go put the code. It's like now I need to go put the updates in this code now I need to the updates in this code now I need to the updates in this code now I need to go put it in that code and I'm like no go put it in that code and I'm like no go put it in that code and I'm like no make a shared function that all of it so make a shared function that all of it so make a shared function that all of it so you can do it once. Oh you're absolutely you can do it once. Oh you're absolutely you can do it once. Oh you're absolutely right which is famous you know throw right which is famous you know throw right which is famous you know throw >> the oh you're absolutely right is so >> the oh you're absolutely right is so >> the oh you're absolutely right is so obnoxious. [laughter] obnoxious. [laughter] obnoxious. [laughter] What was I I did I think I I I told you What was I I did I think I I I told you What was I I did I think I I I told you I did the back end for my podcast and uh I did the back end for my podcast and uh I did the back end for my podcast and uh there's some arbitrary numbers, magic there's some arbitrary numbers, magic there's some arbitrary numbers, magic numbers for horrible historical reasons numbers for horrible historical reasons numbers for horrible historical reasons like the right now you're supposed to like the right now you're supposed to like the right now you're supposed to make the podcast uh album art 1400 by400 make the podcast uh album art 1400 by400 make the podcast uh album art 1400 by400 but because my podcast is 20 years old but because my podcast is 20 years old but because my podcast is 20 years old it's 636 x 636. So, I mentioned that and it's 636 x 636. So, I mentioned that and it's 636 x 636. So, I mentioned that and I forgot about it. And days later, I I forgot about it. And days later, I I forgot about it. And days later, I discover that it's hard-coded 636 x 636 discover that it's hard-coded 636 x 636 discover that it's hard-coded 636 x 636 in like nine [laughter] places. in like nine [laughter] places. in like nine [laughter] places. >> But now I'm afraid that if I tell it >> But now I'm afraid that if I tell it >> But now I'm afraid that if I tell it make that a const, make that a const, make that a const, >> it's going to destroy the world. Oh, I >> it's going to destroy the world. Oh, I >> it's going to destroy the world. Oh, I ran into the exact same thing with a ran into the exact same thing with a ran into the exact same thing with a threshold for threshold for threshold for >> title matching because >> title matching because >> title matching because >> one of the things that I mean I the way >> one of the things that I mean I the way >> one of the things that I mean I the way that references are put into papers, that references are put into papers, that references are put into papers, it's the literally the wild west. It's it's the literally the wild west. It's it's the literally the wild west. It's every crazy format, every crazy every crazy format, every crazy every crazy format, every crazy >> I thought it was just all MLA, right?

>> I thought it was just all MLA, right? >> I thought it was just all MLA, right? >> Well, yeah. And people make subtle >> Well, yeah. And people make subtle >> Well, yeah. And people make subtle mistakes on titles for papers that mistakes on titles for papers that mistakes on titles for papers that they're they're they're >> citing. >> citing. >> citing. >> Like they might leave, you know, might >> Like they might leave, you know, might >> Like they might leave, you know, might leave off part of the title. They might leave off part of the title. They might leave off part of the title. They might >> There's no fluid for title for for >> There's no fluid for title for for >> There's no fluid for title for for academic papers. So I've got to do to academic papers. So I've got to do to academic papers. So I've got to do to like to know which paper like the goal like to know which paper like the goal like to know which paper like the goal is there's an authoritative database of is there's an authoritative database of is there's an authoritative database of the papers where the paper name really the papers where the paper name really the papers where the paper name really is but the reference might be off that a is but the reference might be off that a is but the reference might be off that a little bit and I need to do a soft match little bit and I need to do a soft match little bit and I need to do a soft match or otherwise the answer is going to be or otherwise the answer is going to be or otherwise the answer is going to be like I have no idea what paper this is. like I have no idea what paper this is. like I have no idea what paper this is. I can't verify it. I can't verify it. I can't verify it. >> Yeah, it's just hell. This whole thing >> Yeah, it's just hell. This whole thing >> Yeah, it's just hell. This whole thing this whole thing is hell. Like it sucked this whole thing is hell. Like it sucked this whole thing is hell. Like it sucked so many hours of my life beyond way so many hours of my life beyond way so many hours of my life beyond way beyond I thought it would take. beyond I thought it would take. beyond I thought it would take. >> Why is this not a solved problem? Like I >> Why is this not a solved problem? Like I >> Why is this not a solved problem? Like I understand understand understand >> it's a very hard problem. No, but why >> it's a very hard problem. No, but why >> it's a very hard problem. No, but why but why is it a problem in the first but why is it a problem in the first but why is it a problem in the first place? Why isn't there just a GID that place? Why isn't there just a GID that place? Why isn't there just a GID that represents that's the thing? I assume represents that's the thing? I assume represents that's the thing? I assume the same thing like patents have patent the same thing like patents have patent the same thing like patents have patent numbers, but academic papers don't have numbers, but academic papers don't have numbers, but academic papers don't have gooids. gooids. gooids. >> They I it's because there's no tooling. >> They I it's because there's no tooling. >> They I it's because there's no tooling. Like even if you go to archive and Like even if you go to archive and Like even if you go to archive and you'll see a here's a here's a citation you'll see a here's a here's a citation you'll see a here's a here's a citation block, here's how you site my paper.

block, here's how you site my paper. block, here's how you site my paper. That's like not required that you do That's like not required that you do That's like not required that you do that that you just grab it and slap it that that you just grab it and slap it that that you just grab it and slap it into your paper. into your paper. into your paper. So people, some people do that. Some So people, some people do that. Some So people, some people do that. Some people just go look at the archive paper people just go look at the archive paper people just go look at the archive paper title, you know, the PDF and go, "Oh, title, you know, the PDF and go, "Oh, title, you know, the PDF and go, "Oh, here's the title. Here's the author here's the title. Here's the author here's the title. Here's the author list." And then they make mistakes. list." And then they make mistakes. list." And then they make mistakes. >> And um so there's first of all, there's >> And um so there's first of all, there's >> And um so there's first of all, there's really no formal standard for how you really no formal standard for how you really no formal standard for how you specify it even in text. And then specify it even in text. And then specify it even in text. And then second, it's just the wild west for how second, it's just the wild west for how second, it's just the wild west for how people reference things. Like even on people reference things. Like even on people reference things. Like even on the names, I've run into every single the names, I've run into every single the names, I've run into every single combination of first name, last name, combination of first name, last name, combination of first name, last name, every single permutation. every single permutation. every single permutation. >> Isn't it just M.Reneavich? >> Isn't it just M.Reneavich? >> Isn't it just M.Reneavich? >> Well, it could be Mark Reinovich. It >> Well, it could be Mark Reinovich. It >> Well, it could be Mark Reinovich. It could be Rinovich Mark with no comma. It could be Rinovich Mark with no comma. It could be Rinovich Mark with no comma. It can be Rinovich, Mark. It can be RM. It can be Rinovich, Mark. It can be RM. It can be Rinovich, Mark. It can be RM. It can be can be can be uh it can just be Rinovich. uh it can just be Rinovich. uh it can just be Rinovich. It can be um It can be um It can be um yeah mama reinovich yeah mama reinovich yeah mama reinovich >> and period recinovich mop period >> and period recinovich mop period >> and period recinovich mop period reinovich.

reinovich. reinovich. >> I mean it's just and then if you run >> I mean it's just and then if you run >> I mean it's just and then if you run into compound names into compound names into compound names then it can be then it can be then it can be >> like vanals you know Scott vanals. It >> like vanals you know Scott vanals. It >> like vanals you know Scott vanals. It can be s vouse can be s vouse can be s vouse svs. svs. svs. I mean it's just insanity. Your goal is I mean it's just insanity. Your goal is I mean it's just insanity. Your goal is to discover, let me here's the question. to discover, let me here's the question. to discover, let me here's the question. Is your goal to validate with confidence Is your goal to validate with confidence Is your goal to validate with confidence beyond a reasonable doubt or invalidate beyond a reasonable doubt or invalidate beyond a reasonable doubt or invalidate possibly possibly possibly um hallucinated ones which is a very um hallucinated ones which is a very um hallucinated ones which is a very different goal because the confidence different goal because the confidence different goal because the confidence >> I go switch from looking at >> I go switch from looking at >> I go switch from looking at hallucination to just can I verify that hallucination to just can I verify that hallucination to just can I verify that there's no mistakes and what I'll do and there's no mistakes and what I'll do and there's no mistakes and what I'll do and and just assume that the paper's and just assume that the paper's and just assume that the paper's references reference format is the way references reference format is the way references reference format is the way they want to do it. So if they want to they want to do it. So if they want to they want to do it. So if they want to do you know SVS then I have to say well do you know SVS then I have to say well do you know SVS then I have to say well the database says it's Scott Van South the database says it's Scott Van South the database says it's Scott Van South is that a match for SVS? is that a match for SVS? is that a match for SVS? Oh, Oh, Oh, >> and the same by the way and that's just >> and the same by the way and that's just >> and the same by the way and that's just every component has its own craziness.

every component has its own craziness. every component has its own craziness. Take venues for example like Take venues for example like Take venues for example like publications publications publications >> like um the journal of computer science. >> like um the journal of computer science. >> like um the journal of computer science. So it can say journal of computer So it can say journal of computer So it can say journal of computer science. It can say J computer science. science. It can say J computer science. science. It can say J computer science. It can say JCS. It can say J It can say JCS. It can say J It can say JCS. It can say J comp period SI period. It can say like comp period SI period. It can say like comp period SI period. It can say like I'm telling you every freaking I'm telling you every freaking I'm telling you every freaking permutation in the world is out there. permutation in the world is out there. permutation in the world is out there. >> Okay. Is there an 8020 rule here? Like >> Okay. Is there an 8020 rule here? Like >> Okay. Is there an 8020 rule here? Like did you get to 80 very quickly and did you get to 80 very quickly and did you get to 80 very quickly and you're just spending a lot? you're just spending a lot? you're just spending a lot? >> Oh yeah. I mean, I get to 80 and then >> Oh yeah. I mean, I get to 80 and then >> Oh yeah. I mean, I get to 80 and then it's like, but then it's like, let me it's like, but then it's like, let me it's like, but then it's like, let me try this paper because I just go pick try this paper because I just go pick try this paper because I just go pick random papers and it's like, oh, another random papers and it's like, oh, another random papers and it's like, oh, another edge case. Another I might like the last edge case. Another I might like the last edge case. Another I might like the last few weeks have just been edge case after few weeks have just been edge case after few weeks have just been edge case after edge case after. edge case after. edge case after. >> So, there's more edge case. So, edge >> So, there's more edge case. So, edge >> So, there's more edge case. So, edge cases are the standard, not the not the cases are the standard, not the not the cases are the standard, not the not the norm. Y norm. Y norm. Y >> do you have a demo? >> do you have a demo? >> do you have a demo? >> Yeah, I can show a demo. >> Yeah, I can show a demo. >> Yeah, I can show a demo. >> Let's see it. >> Let's see it. >> Let's see it. >> So, first it's on GitHub. Um, >> So, first it's on GitHub. Um, >> So, first it's on GitHub. Um, so here's the GitHub repo for it. Yeah. so here's the GitHub repo for it. Yeah. so here's the GitHub repo for it. Yeah. So, ref checker um academic paper So, ref checker um academic paper So, ref checker um academic paper reference checker. Uh here's some reference checker. Uh here's some reference checker. Uh here's some example output but I'll show a demo of example output but I'll show a demo of example output but I'll show a demo of it. You can see some of the kind of it. You can see some of the kind of it. You can see some of the kind of warnings or errors.

warnings or errors. warnings or errors. >> But see, archive has all these URLs, but >> But see, archive has all these URLs, but >> But see, archive has all these URLs, but like none of these has decided. I'm like none of these has decided. I'm like none of these has decided. I'm looking at these DOI, archive, and and looking at these DOI, archive, and and looking at these DOI, archive, and and the semantic scholar all have different the semantic scholar all have different the semantic scholar all have different >> there's no one has reconciled any of >> there's no one has reconciled any of >> there's no one has reconciled any of these ID ids. There's no Dewey decimal these ID ids. There's no Dewey decimal these ID ids. There's no Dewey decimal system for these things. system for these things. system for these things. >> I can find a book in the library, but >> I can find a book in the library, but >> I can find a book in the library, but >> yeah, >> yeah, >> yeah, >> it's partially applied. >> it's partially applied. >> it's partially applied. >> Okay. Um, so this this part is the >> Okay. Um, so this this part is the >> Okay. Um, so this this part is the actual reference from the paper. This actual reference from the paper. This actual reference from the paper. This part is what the tool spits out. Like part is what the tool spits out. Like part is what the tool spits out. Like here's this [snorts] here's this [snorts] here's this [snorts] this the URL that is actually verifying this the URL that is actually verifying this the URL that is actually verifying this the information on this the information on this the information on >> and by the way, Semantic Scholar has >> and by the way, Semantic Scholar has >> and by the way, Semantic Scholar has errors in it as well. errors in it as well. errors in it as well. >> Okay. >> Okay. >> Okay. >> But um >> But um >> But um >> hang on, go back for a second really >> hang on, go back for a second really >> hang on, go back for a second really briefly. briefly. briefly. >> Yeah. >> Yeah. >> Yeah. >> On those last two there, the nine the 90 >> On those last two there, the nine the 90 >> On those last two there, the nine the 90 and the 9 that right there. and the 9 that right there. and the 9 that right there. >> Yeah. I'm looking at that as a human >> Yeah. I'm looking at that as a human >> Yeah. I'm looking at that as a human thinking, is it more likely that it was thinking, is it more likely that it was thinking, is it more likely that it was refined or a new version came out or is refined or a new version came out or is refined or a new version came out or is it literally that they typed a one it literally that they typed a one it literally that they typed a one instead of a 90? instead of a 90? instead of a 90? >> I think it's that they by the way that >> I think it's that they by the way that >> I think it's that they by the way that one might not be a real one. That one one might not be a real one. That one one might not be a real one. That one might be an example one, but I'll go might be an example one, but I'll go might be an example one, but I'll go show you real ones. Um, okay. So, show you real ones. Um, okay. So, show you real ones. Um, okay. So, >> that don't fixate on that one.

>> that don't fixate on that one. >> that don't fixate on that one. >> Okay. Sorry. But you can give it an >> Okay. Sorry. But you can give it an >> Okay. Sorry. But you can give it an archive paper, a local PDF, an archive archive paper, a local PDF, an archive archive paper, a local PDF, an archive ID, a local file, a latte file, a text ID, a local file, a latte file, a text ID, a local file, a latte file, a text document that just has free form document that just has free form document that just has free form references in it. references in it. references in it. >> Um, now the bibliography itself and the >> Um, now the bibliography itself and the >> Um, now the bibliography itself and the paper uh so there's no standard for you paper uh so there's no standard for you paper uh so there's no standard for you know how the bibliography in a paper is know how the bibliography in a paper is know how the bibliography in a paper is identified. So some papers have a identified. So some papers have a identified. So some papers have a references but if you're taking a PDF references but if you're taking a PDF references but if you're taking a PDF because you have no other source for the because you have no other source for the because you have no other source for the bibliography bibliography bibliography the tool translate and translate into the tool translate and translate into the tool translate and translate into text and then has to figure out where text and then has to figure out where text and then has to figure out where the bibliography is. So it uses so it the bibliography is. So it uses so it the bibliography is. So it uses so it uses reax to go try to find out the out uses reax to go try to find out the out uses reax to go try to find out the out of the zillion you know ways the of the zillion you know ways the of the zillion you know ways the beginning and end of the of the beginning and end of the of the beginning and end of the of the bibliography bibliography bibliography >> because I don't want to give it the >> because I don't want to give it the >> because I don't want to give it the whole paper to an LLM which could be you whole paper to an LLM which could be you whole paper to an LLM which could be you know the paper could be know the paper could be know the paper could be >> so you called that intelligent pattern >> so you called that intelligent pattern >> so you called that intelligent pattern matching using a matching using a matching using a >> red this Yeah. Yeah. This is intelligent >> red this Yeah. Yeah. This is intelligent >> red this Yeah. Yeah. This is intelligent pattern matching right? pattern matching right? pattern matching right? >> Yeah. Yeah. >> Yeah. Yeah. >> Yeah. Yeah. >> Now this part once you have the section >> Now this part once you have the section >> Now this part once you have the section that you think is the references that you think is the references that you think is the references >> Mhm. again given first of all the PDF to >> Mhm. again given first of all the PDF to >> Mhm. again given first of all the PDF to text is full of crap annotation crap text is full of crap annotation crap text is full of crap annotation crap from PDF from PDF from PDF >> right >> right >> right >> that's some PDFs have text embedded and >> that's some PDFs have text embedded and >> that's some PDFs have text embedded and some PDFs are literally images some PDFs are literally images some PDFs are literally images >> yeah well the on these references though >> yeah well the on these references though >> yeah well the on these references though there's like PDF formatting characters there's like PDF formatting characters there's like PDF formatting characters >> interled inside the references >> interled inside the references >> interled inside the references >> no >> no >> no >> so it's incredibly messy >> so it's incredibly messy >> so it's incredibly messy >> but even with without that again there

>> but even with without that again there >> but even with without that again there given there's given there's given there's any number of permutations for how the any number of permutations for how the any number of permutations for how the ref, you know, it's some kind of put the ref, you know, it's some kind of put the ref, you know, it's some kind of put the author first, the author last, the venue author first, the author last, the venue author first, the author last, the venue first, the year first, you know, first, the year first, you know, first, the year first, you know, >> and commas in between, no commods in >> and commas in between, no commods in >> and commas in between, no commods in between, like between, like between, like infinite number of permutations. And infinite number of permutations. And infinite number of permutations. And what I started with was I'm going to what I started with was I'm going to what I started with was I'm going to make this thing do reax 2 to pull these make this thing do reax 2 to pull these make this thing do reax 2 to pull these out. out. out. >> Mhm. And then that became exception >> Mhm. And then that became exception >> Mhm. And then that became exception nightmare like another and and where the nightmare like another and and where the nightmare like another and and where the trying to fix the reax to address this trying to fix the reax to address this trying to fix the reax to address this case regresses this other case and I case regresses this other case and I case regresses this other case and I just became I was like this is just became I was like this is just became I was like this is impossible. This is literally impossible impossible. This is literally impossible impossible. This is literally impossible to solve this with reax. What if we use to solve this with reax. What if we use to solve this with reax. What if we use LLMs? LLMs? LLMs? What if I can get like so then what I What if I can get like so then what I What if I can get like so then what I tried was just take the bibliography and tried was just take the bibliography and tried was just take the bibliography and throw it at the LLM and say pull out the throw it at the LLM and say pull out the throw it at the LLM and say pull out the references and format them this way. The references and format them this way. The references and format them this way. The meta prompt is a little bit more meta prompt is a little bit more meta prompt is a little bit more involved in that than that. But involved in that than that. But involved in that than that. But >> voila, that solved it. LLMs are really >> voila, that solved it. LLMs are really >> voila, that solved it. LLMs are really really good at taking this unstructured really good at taking this unstructured really good at taking this unstructured data and structuring it. data and structuring it. data and structuring it. >> So without that, yeah, without that this >> So without that, yeah, without that this >> So without that, yeah, without that this would have completely been a failure.

would have completely been a failure. would have completely been a failure. >> So LLMs can take unstructured data and >> So LLMs can take unstructured data and >> So LLMs can take unstructured data and structure it, but they still But this is structure it, but they still But this is structure it, but they still But this is a problem that you are having where LLMs a problem that you are having where LLMs a problem that you are having where LLMs are making up references. In this case, are making up references. In this case, are making up references. In this case, there are not. Well, let me just say there are not. Well, let me just say there are not. Well, let me just say this. this. this. >> You see what I'm saying? Like the >> You see what I'm saying? Like the >> You see what I'm saying? Like the problem. You're using a thing that we problem. You're using a thing that we problem. You're using a thing that we can't rely on to check a thing we can't can't rely on to check a thing we can't can't rely on to check a thing we can't rely on. rely on. rely on. >> But at least there's a a formal check >> But at least there's a a formal check >> But at least there's a a formal check >> as the back stop against the you know >> as the back stop against the you know >> as the back stop against the you know the source of truth. It's not like the the source of truth. It's not like the the source of truth. It's not like the LM is saying this reference is correct. LM is saying this reference is correct. LM is saying this reference is correct. >> I'm just getting the reference and then >> I'm just getting the reference and then >> I'm just getting the reference and then looking it up. looking it up. looking it up. >> And for this I found that they're >> And for this I found that they're >> And for this I found that they're >> like Claude Sonnet >> like Claude Sonnet >> like Claude Sonnet >> um and GPT45 very reliable at this. >> um and GPT45 very reliable at this. >> um and GPT45 very reliable at this. >> There we go. Okay. So you're this is a >> There we go. Okay. So you're this is a >> There we go. Okay. So you're this is a great example of a multi-disciplinary great example of a multi-disciplinary great example of a multi-disciplinary application that knows when reaxes are application that knows when reaxes are application that knows when reaxes are good, when LLMs are good. good, when LLMs are good. good, when LLMs are good. >> Exactly. >> Exactly. >> Exactly. >> And when an HTTP get is good. >> And when an HTTP get is good. >> And when an HTTP get is good. >> Exactly. And what I found speaking of >> Exactly. And what I found speaking of >> Exactly. And what I found speaking of hallucinations hallucinations hallucinations >> and this set me down a rabbit hole for >> and this set me down a rabbit hole for >> and this set me down a rabbit hole for about an hour one day is I was started about an hour one day is I was started about an hour one day is I was started by using GPT40 by using GPT40 by using GPT40 >> and that seemed to be working great. And >> and that seemed to be working great. And >> and that seemed to be working great. And then one day I'm having I'm having it then one day I'm having I'm having it then one day I'm having I'm having it pulled the the references out of a paper pulled the the references out of a paper pulled the the references out of a paper and I do the it goes through the and I do the it goes through the and I do the it goes through the reference validation and it's like the reference validation and it's like the reference validation and it's like the DOI on this reference is wrong. The DOI DOI on this reference is wrong. The DOI DOI on this reference is wrong. The DOI on this reference is wrong. The DOI. So on this reference is wrong. The DOI. So on this reference is wrong. The DOI. So I'm like what the hell? So I start I'm like what the hell? So I start I'm like what the hell? So I start debugging it and an hour in well first debugging it and an hour in well first debugging it and an hour in well first of all I give it to the AI assistants of all I give it to the AI assistants of all I give it to the AI assistants and they're like we can't figure this and they're like we can't figure this and they're like we can't figure this out you know or it's fixed and it's not out you know or it's fixed and it's not out you know or it's fixed and it's not fixed. And so I this one of those cases

fixed. And so I this one of those cases fixed. And so I this one of those cases I had to start debugging I had to start debugging I had to start debugging and what I found was GPT40 had and what I found was GPT40 had and what I found was GPT40 had hallucinated the DOIs. hallucinated the DOIs. hallucinated the DOIs. >> Oh no. >> Oh no. >> Oh no. >> And sent me down this wild rat hole. So >> And sent me down this wild rat hole. So >> And sent me down this wild rat hole. So that was that was that was >> and wasted your time and wasted tokens >> and wasted your time and wasted tokens >> and wasted your time and wasted tokens as well. as well. as well. >> Yeah. But u like I said 4 is very >> Yeah. But u like I said 4 is very >> Yeah. But u like I said 4 is very reliable and GPT45 is very reliable at reliable and GPT45 is very reliable at reliable and GPT45 is very reliable at this. this. this. >> So not trying to over capitalize >> So not trying to over capitalize >> So not trying to over capitalize capitalism things but this feels like it capitalism things but this feels like it capitalism things but this feels like it should be a function or like you could should be a function or like you could should be a function or like you could turn this into a whole website and turn this into a whole website and turn this into a whole website and people could drag their stuff in there people could drag their stuff in there people could drag their stuff in there like this. Yeah. like this. Yeah. like this. Yeah. >> Are you going to make us run this at the >> Are you going to make us run this at the >> Are you going to make us run this at the command line or is this going to turn command line or is this going to turn command line or is this going to turn into like an Azure function? into like an Azure function? into like an Azure function? >> Well, I'm So, right now it's still a >> Well, I'm So, right now it's still a >> Well, I'm So, right now it's still a command line tool, but we can talk about command line tool, but we can talk about command line tool, but we can talk about where it can go next. Um, but here's an where it can go next. Um, but here's an where it can go next. Um, but here's an example of, you know, how easy it is. example of, you know, how easy it is. example of, you know, how easy it is. But this, if you do this, it's just But this, if you do this, it's just But this, if you do this, it's just going to use Reax to try to parse the going to use Reax to try to parse the going to use Reax to try to parse the references, too. references, too. references, too. >> Mhm. >> Mhm. >> Mhm. >> Um, so what you want to do is, and >> Um, so what you want to do is, and >> Um, so what you want to do is, and here's the providers that it supports. here's the providers that it supports. here's the providers that it supports. >> Okay. Bring your own key. >> Okay. Bring your own key. >> Okay. Bring your own key. >> Yep. Bring your own key. >> Yep. Bring your own key. >> Yep. Bring your own key. >> Yeah. Yeah. >> Yeah. Yeah. >> Yeah. Yeah. >> Um, here's you set up. You can set them >> Um, here's you set up. You can set them >> Um, here's you set up. You can set them up the whole thing as environment up the whole thing as environment up the whole thing as environment variables or use the command line.

variables or use the command line. variables or use the command line. >> Nice. >> Nice. >> Nice. >> Um, >> Um, >> Um, >> now are you going to start running this >> now are you going to start running this >> now are you going to start running this yourself whenever you feel like it? When yourself whenever you feel like it? When yourself whenever you feel like it? When you bump into stuff, are you checking you bump into stuff, are you checking you bump into stuff, are you checking all your papers? all your papers? all your papers? >> Yeah, I do. In fact, um, this is >> Yeah, I do. In fact, um, this is >> Yeah, I do. In fact, um, this is >> zoom in. >> So, one of these is I've got these debug >> So, one of these is I've got these debug here and Crescendo. here and Crescendo. here and Crescendo. >> Oh, well, that's nice. Hang on. Are >> Oh, well, that's nice. Hang on. Are >> Oh, well, that's nice. Hang on. Are those debug configurations? Yeah. those debug configurations? Yeah. those debug configurations? Yeah. So, here's um one of the papers I wrote, So, here's um one of the papers I wrote, So, here's um one of the papers I wrote, Crescendo, which I'm actually presenting Crescendo, which I'm actually presenting Crescendo, which I'm actually presenting at Usenix on a jailbreaking technique. at Usenix on a jailbreaking technique. at Usenix on a jailbreaking technique. >> Uh-huh. >> Uh-huh. >> Uh-huh. >> In two weeks. So, you can see I'm using >> In two weeks. So, you can see I'm using >> In two weeks. So, you can see I'm using GPD41. Sorry, it's not 451. Oh, GPD41. Sorry, it's not 451. Oh, GPD41. Sorry, it's not 451. Oh, >> okay. [laughter] Can you see that? >> okay. [laughter] Can you see that? >> okay. [laughter] Can you see that? >> I can see that. Thank you. >> I can see that. Thank you. >> I can see that. Thank you. >> Okay. GPD41. So, ref checker and I I >> Okay. GPD41. So, ref checker and I I >> Okay. GPD41. So, ref checker and I I just like timing it to see how long it just like timing it to see how long it just like timing it to see how long it takes. uh provider OpenAI GP241 paper takes. uh provider OpenAI GP241 paper takes. uh provider OpenAI GP241 paper this this this and um if you have a key to semantic and um if you have a key to semantic and um if you have a key to semantic scholar which is the primary source for scholar which is the primary source for scholar which is the primary source for validation it takes about a second per validation it takes about a second per validation it takes about a second per reference if you don't have a key you reference if you don't have a key you reference if you don't have a key you can semantic scholar rate limits public can semantic scholar rate limits public can semantic scholar rate limits public access got a certain quota access got a certain quota access got a certain quota >> so that can take longer but you can see >> so that can take longer but you can see >> so that can take longer but you can see it's verifying here it's verifying here it's verifying here >> okay now what are all these warnings >> okay now what are all these warnings >> okay now what are all these warnings >> so the warning this is so this one is >> so the warning this is so this one is >> so the warning this is so this one is just a kind of a tip since the the ver just a kind of a tip since the the ver just a kind of a tip since the the ver since the verific ification found an since the verific ification found an since the verific ification found an archive URL for this. The suggestion is archive URL for this. The suggestion is archive URL for this. The suggestion is go ahead and add a pointer to the go ahead and add a pointer to the go ahead and add a pointer to the archive paper because the reference is archive paper because the reference is archive paper because the reference is just has no pointer to any paper or ID.

just has no pointer to any paper or ID. just has no pointer to any paper or ID. >> So of these of these three things >> So of these of these three things >> So of these of these three things semantic scholar archive and DOI what is semantic scholar archive and DOI what is semantic scholar archive and DOI what is these all just like these all just like these all just like >> three of the most popular and then you >> three of the most popular and then you >> three of the most popular and then you should include or is archive the one? should include or is archive the one? should include or is archive the one? >> So this is using found this using >> So this is using found this using >> So this is using found this using semantic scholar API um which is why semantic scholar API um which is why semantic scholar API um which is why this this is the verified URL. Now, this this is the verified URL. Now, this this is the verified URL. Now, semantic scholar API in its database can semantic scholar API in its database can semantic scholar API in its database can have external ids have external ids have external ids >> that are like archive URLs or DOI URLs. >> that are like archive URLs or DOI URLs. >> that are like archive URLs or DOI URLs. Not always, but if they do, then the Not always, but if they do, then the Not always, but if they do, then the tool prints them. tool prints them. tool prints them. >> And has archive become the standard >> And has archive become the standard >> And has archive become the standard >> for computer science? It is. >> for computer science? It is. >> for computer science? It is. >> Yeah. Have you tried this on healthcare >> Yeah. Have you tried this on healthcare >> Yeah. Have you tried this on healthcare or is this only for or is this only for or is this only for >> I've tried it on everything. And and by >> I've tried it on everything. And and by >> I've tried it on everything. And and by the way, computer science is the most or the way, computer science is the most or the way, computer science is the most or most uh kind of most uh kind of most uh kind of structured of them all. If you get into structured of them all. If you get into structured of them all. If you get into physics, it's the even more of a wild physics, it's the even more of a wild physics, it's the even more of a wild west and math. west and math. west and math. >> Well, so this is great because I've got >> Well, so this is great because I've got >> Well, so this is great because I've got a buddy of mine. I think I told you a buddy of mine. I think I told you a buddy of mine. I think I told you about this. I've got a buddy who's got about this. I've got a buddy who's got about this. I've got a buddy who's got like 60 gigs of PDFs on diabetes like 60 gigs of PDFs on diabetes like 60 gigs of PDFs on diabetes research research research >> and he's been making a chatbot over that >> and he's been making a chatbot over that >> and he's been making a chatbot over that that uh you rag over that that's that that uh you rag over that that's that that uh you rag over that that's that thing. But he's never gone and had any thing. But he's never gone and had any thing. But he's never gone and had any tool like this tool like this tool like this >> to make sure that these things are real >> to make sure that these things are real >> to make sure that these things are real and legitimate and to plug all the stuff and legitimate and to plug all the stuff and legitimate and to plug all the stuff in. So with this result here, can I in. So with this result here, can I in. So with this result here, can I output it as a CSV or give it?

output it as a CSV or give it? output it as a CSV or give it? >> Yeah. Yeah, it output. You can output I >> Yeah. Yeah, it output. You can output I >> Yeah. Yeah, it output. You can output I mean you can pipe this to a file or you mean you can pipe this to a file or you mean you can pipe this to a file or you can you can um can you can um can you can um uh add a uh uh add a uh uh add a uh there's a option to save the errors and there's a option to save the errors and there's a option to save the errors and warnings to a file. warnings to a file. warnings to a file. >> Yeah, it'd be interesting if you could >> Yeah, it'd be interesting if you could >> Yeah, it'd be interesting if you could use this to use this to use this to >> right here >> right here >> right here >> output a correction to your application. >> output a correction to your application. >> output a correction to your application. >> So this does output fix your PDF. Yeah, >> So this does output fix your PDF. Yeah, >> So this does output fix your PDF. Yeah, this does output a correction like a this does output a correction like a this does output a correction like a correct bibliography for it. Um, but correct bibliography for it. Um, but correct bibliography for it. Um, but again, I don't know what format, you again, I don't know what format, you again, I don't know what format, you know, so you've got to know, so you've got to know, so you've got to >> turn. But I'm just wondering out loud if >> turn. But I'm just wondering out loud if >> turn. But I'm just wondering out loud if you could create the correct format and you could create the correct format and you could create the correct format and it would, you know, you've identified it would, you know, you've identified it would, you know, you've identified their bibliography. You could torch their bibliography. You could torch their bibliography. You could torch their bibliography and replace it with a their bibliography and replace it with a their bibliography and replace it with a corrected authoritative. corrected authoritative. corrected authoritative. >> The problem is I don't know the format. >> The problem is I don't know the format. >> The problem is I don't know the format. The LLM is extracting. So the LLM might The LLM is extracting. So the LLM might The LLM is extracting. So the LLM might know the format. I don't know if it can know the format. I don't know if it can know the format. I don't know if it can see where I'm going with this, right? see where I'm going with this, right? see where I'm going with this, right? Check it. But I think it's it's even Check it. But I think it's it's even Check it. But I think it's it's even it's hard. it's hard. it's hard. >> Yeah. >> Yeah. >> Yeah. >> Um I think >> Um I think >> Um I think >> the easy >> the easy >> the easy the way to do it I think would be for the way to do it I think would be for the way to do it I think would be for the user to provide here's a template of the user to provide here's a template of the user to provide here's a template of the way that I want author whatever and the way that I want author whatever and the way that I want author whatever and then it could spit it out that way.

then it could spit it out that way. then it could spit it out that way. >> I guess I just feel like why should they >> I guess I just feel like why should they >> I guess I just feel like why should they have a template? You should just decide have a template? You should just decide have a template? You should just decide that we've just we we as a community that we've just we we as a community that we've just we we as a community have decided that this is what the have decided that this is what the have decided that this is what the biblioraphy biblioraphy biblioraphy >> well then they'd have to update all >> well then they'd have to update all >> well then they'd have to update all their um references. their um references. their um references. >> Then you run this on the entire world. >> Then you run this on the entire world. >> Then you run this on the entire world. >> Yeah. >> Yeah. >> Yeah. >> This is great. Uh, >> This is great. Uh, >> This is great. Uh, >> is it fun or was it annoying? >> is it fun or was it annoying? >> is it fun or was it annoying? >> Exactly. >> Exactly. >> Exactly. >> It's it's it's annoying because even now >> It's it's it's annoying because even now >> It's it's it's annoying because even now I found another edge case like I found another edge case like I found another edge case like >> just now >> just now >> just now >> this morning. This morning. >> this morning. This morning. >> this morning. This morning. >> Oh no. >> Oh no. >> Oh no. >> Like every morning I'm like, "Oh, let me >> Like every morning I'm like, "Oh, let me >> Like every morning I'm like, "Oh, let me try this paper." And then it's like, try this paper." And then it's like, try this paper." And then it's like, "Oh, okay." "Oh, okay." "Oh, okay." >> You know, damn it, here's another edge >> You know, damn it, here's another edge >> You know, damn it, here's another edge case. case. case. >> You I assume you have robust test cases. >> You I assume you have robust test cases. >> You I assume you have robust test cases. >> Yeah. In fact, um >> Yeah. In fact, um >> Yeah. In fact, um >> Yeah. You got to zoom in, brother. You >> Yeah. You got to zoom in, brother. You >> Yeah. You got to zoom in, brother. You got to do something. You're killing me, got to do something. You're killing me, got to do something. You're killing me, man. man. man. >> Sorry. That's a lot. That's good. That's a lot. That's good. >> Yeah. And of course, >> Yeah. And of course, >> Yeah. And of course, >> I noticed that you're are you're are you >> I noticed that you're are you're are you >> I noticed that you're are you're are you using WSL here? using WSL here? using WSL here? >> No, this is um connected to a Linux >> No, this is um connected to a Linux >> No, this is um connected to a Linux machine in Azure.

machine in Azure. machine in Azure. >> Okay. So, you're SSH. You do that a lot. >> Okay. So, you're SSH. You do that a lot. >> Okay. So, you're SSH. You do that a lot. I've noticed that virtually all of your I've noticed that virtually all of your I've noticed that virtually all of your uh work is done on a bigger machine in uh work is done on a bigger machine in uh work is done on a bigger machine in the in the cloud somewhere. the in the cloud somewhere. the in the cloud somewhere. Um, these machines have GPUs on them, so Um, these machines have GPUs on them, so Um, these machines have GPUs on them, so I can do the VLM testing, for example. I can do the VLM testing, for example. I can do the VLM testing, for example. >> Gotcha. Gotcha. >> Gotcha. Gotcha. >> Gotcha. Gotcha. >> Um, >> Um, >> Um, but but but >> any case, so you you get the idea for >> any case, so you you get the idea for >> any case, so you you get the idea for what the tool does. Um, what the tool does. Um, what the tool does. Um, and it's been like you like I said, um, and it's been like you like I said, um, and it's been like you like I said, um, without despite the frustrations with without despite the frustrations with without despite the frustrations with the AI assistance, this would have been the AI assistance, this would have been the AI assistance, this would have been I I could not have I had the time to do I I could not have I had the time to do I I could not have I had the time to do this. this. this. >> So that there you go. So this here's a >> So that there you go. So this here's a >> So that there you go. So this here's a question. Let's end on this. Would it question. Let's end on this. Would it question. Let's end on this. Would it have been impossible or just have been impossible or just have been impossible or just unreasonably unreasonably unreasonably tedious? So first like I mentioned tedious? So first like I mentioned tedious? So first like I mentioned because with the uh unstructured because with the uh unstructured because with the uh unstructured bibliographies bibliographies bibliographies I think it would have been impossible I think it would have been impossible I think it would have been impossible to to to >> you be playing whack-a-ole with rejax >> you be playing whack-a-ole with rejax >> you be playing whack-a-ole with rejax for for for >> for years. >> for years. >> for years. >> Yeah.

>> Yeah. >> Yeah. >> Um and then as so that's just for the >> Um and then as so that's just for the >> Um and then as so that's just for the tool to even exist tool to even exist tool to even exist for the coding of it. It would have been for the coding of it. It would have been for the coding of it. It would have been possible without it but it would have possible without it but it would have possible without it but it would have taken me 3 months instead of one month taken me 3 months instead of one month taken me 3 months instead of one month or more and it would have been a slog. or more and it would have been a slog. or more and it would have been a slog. This is where I think people who are, This is where I think people who are, This is where I think people who are, you know, like AI haters, like I'm kind you know, like AI haters, like I'm kind you know, like AI haters, like I'm kind of like I said, I'm in the middle of like I said, I'm in the middle of like I said, I'm in the middle somewhere. Uh, once they experience somewhere. Uh, once they experience somewhere. Uh, once they experience something like that where it's like, something like that where it's like, something like that where it's like, wow, I couldn't have done that in the wow, I couldn't have done that in the wow, I couldn't have done that in the amount of time. amount of time. amount of time. >> I think I told you about the carbon to >> I think I told you about the carbon to >> I think I told you about the carbon to C64 thing. I could have done it. It C64 thing. I could have done it. It C64 thing. I could have done it. It would have been a week. I did it in an would have been a week. I did it in an would have been a week. I did it in an evening. evening. evening. >> Yeah. And and if it if something like >> Yeah. And and if it if something like >> Yeah. And and if it if something like that, you know, would take a week, that, you know, would take a week, that, you know, would take a week, you're like, I just can't. you're like, I just can't. you're like, I just can't. >> I wouldn't have done it. And that's the >> I wouldn't have done it. And that's the >> I wouldn't have done it. And that's the thing, like I've talked about my admin thing, like I've talked about my admin thing, like I've talked about my admin thing that I vibe coded. I just never thing that I vibe coded. I just never thing that I vibe coded. I just never wanted to burn a weekend. wanted to burn a weekend. wanted to burn a weekend. >> Yeah. >> Yeah. >> Yeah. >> Cuz it just wasn't that big of a deal. >> Cuz it just wasn't that big of a deal. >> Cuz it just wasn't that big of a deal. But the problem remains interesting. So But the problem remains interesting. So But the problem remains interesting. So it's the how interesting is the problem it's the how interesting is the problem it's the how interesting is the problem versus how much do I have patience for versus how much do I have patience for versus how much do I have patience for it. What you just showed us is a really it. What you just showed us is a really it. What you just showed us is a really interesting problem, but not worth three interesting problem, but not worth three interesting problem, but not worth three months of your life.

months of your life. months of your life. >> No, I wouldn't have done it for three >> No, I wouldn't have done it for three >> No, I wouldn't have done it for three months. actually and know you know for a months. actually and know you know for a months. actually and know you know for a month um and as I was recovering I spent month um and as I was recovering I spent month um and as I was recovering I spent a lot of time doing this B and it's a lot of time doing this B and it's a lot of time doing this B and it's basically uh my you know managing an basically uh my you know managing an basically uh my you know managing an intern writing the code for you. intern writing the code for you. intern writing the code for you. >> Yeah. Yeah. Yeah. >> Yeah. Yeah. Yeah. >> Yeah. Yeah. Yeah. >> And going you in like you overlooked >> And going you in like you overlooked >> And going you in like you overlooked this stupid thing like go back and fix this stupid thing like go back and fix this stupid thing like go back and fix it. it. it. >> Uh >> Uh >> Uh >> and was it enjoyable though? Like isn't >> and was it enjoyable though? Like isn't >> and was it enjoyable though? Like isn't it making you like enjoy coding again? it making you like enjoy coding again? it making you like enjoy coding again? >> It's a different very different kind of >> It's a different very different kind of >> It's a different very different kind of coding. Yeah, coding. Yeah, coding. Yeah, >> like I really, you know, >> like I really, you know, >> like I really, you know, >> um the actual coding of typing in >> um the actual coding of typing in >> um the actual coding of typing in everything in which I was doing up until everything in which I was doing up until everything in which I was doing up until roughly two years ago. roughly two years ago. roughly two years ago. >> Yeah. >> Yeah. >> Yeah. >> That um that was fun because it's like >> That um that was fun because it's like >> That um that was fun because it's like puzzle solving like a you know you're puzzle solving like a you know you're puzzle solving like a you know you're intricate like fitting this piece into intricate like fitting this piece into intricate like fitting this piece into this piece and now it's not that this piece and now it's not that this piece and now it's not that anymore. It's a totally different way of anymore. It's a totally different way of anymore. It's a totally different way of coding. coding. coding. >> It's a different part of your brain. It >> It's a different part of your brain. It >> It's a different part of your brain. It scratches different parts. scratches different parts. scratches different parts. >> Yeah. And I like the puzzle piecing part >> Yeah. And I like the puzzle piecing part >> Yeah. And I like the puzzle piecing part of real, but now I'm like, it's it's not of real, but now I'm like, it's it's not of real, but now I'm like, it's it's not worth the time.

>> Some people are saying that vibe coding >> Some people are saying that vibe coding is getting them fall in love with is getting them fall in love with is getting them fall in love with computers again cuz it's giving them computers again cuz it's giving them computers again cuz it's giving them time to do the things that they never time to do the things that they never time to do the things that they never really wanted to invest the time in. really wanted to invest the time in. really wanted to invest the time in. Like I'm doing more toy stuff, Like I'm doing more toy stuff, Like I'm doing more toy stuff, >> which I find enjoyable. depends on what >> which I find enjoyable. depends on what >> which I find enjoyable. depends on what the um the um the um you know where you're getting your you know where you're getting your you know where you're getting your enjoyment out of the use of the tool, enjoyment out of the use of the tool, enjoyment out of the use of the tool, you know, the creation of it. The the you know, the creation of it. The the you know, the creation of it. The the fact that it exists versus the process fact that it exists versus the process fact that it exists versus the process of creating it. of creating it. of creating it. >> Ooh, that is I love that you said that >> Ooh, that is I love that you said that >> Ooh, that is I love that you said that because there's the making of it. because there's the making of it. because there's the making of it. >> Yeah. >> Yeah. >> Yeah. >> There's simply the it exists and I put >> There's simply the it exists and I put >> There's simply the it exists and I put it into the universe and I'll never use it into the universe and I'll never use it into the universe and I'll never use it again. I also like just the like now it again. I also like just the like now it again. I also like just the like now I can rest. I can rest. I can rest. >> Yeah, >> Yeah, >> Yeah, >> it exists. >> it exists. >> it exists. >> But then there's also the regular I I >> But then there's also the regular I I >> But then there's also the regular I I enjoy tools that I'm going to use every enjoy tools that I'm going to use every enjoy tools that I'm going to use every single day. single day. single day. >> Yeah. Like you have to have joy in with >> Yeah. Like you have to have joy in with >> Yeah. Like you have to have joy in with in zoom because you're using in zoom because you're using in zoom because you're using >> the system internals tools were tools >> the system internals tools were tools >> the system internals tools were tools that I could use all the time. Um and so that I could use all the time. Um and so that I could use all the time. Um and so like we talked about before doing things like we talked about before doing things like we talked about before doing things that have multiple benefits like this that have multiple benefits like this that have multiple benefits like this tool has multiple benefits for me. tool has multiple benefits for me. tool has multiple benefits for me. >> For one which which like system >> For one which which like system >> For one which which like system internals do tools do which I love internals do tools do which I love internals do tools do which I love because for one I learned how to write a because for one I learned how to write a because for one I learned how to write a hybrid program one that is AI has AI hybrid program one that is AI has AI hybrid program one that is AI has AI components to it.

components to it. components to it. Um, two I learned, you know, I played Um, two I learned, you know, I played Um, two I learned, you know, I played with all the assistants, cursor, claude, with all the assistants, cursor, claude, with all the assistants, cursor, claude, code, GitHub, agent, code, GitHub, agent, code, GitHub, agent, >> um, and >> um, and >> um, and I'm missing one. Claude, [snorts] code, I'm missing one. Claude, [snorts] code, I'm missing one. Claude, [snorts] code, curser. curser. curser. >> Open AI has a coding one. >> Open AI has a coding one. >> Open AI has a coding one. >> No. Uh, Klein, I use I mean, I played >> No. Uh, Klein, I use I mean, I played >> No. Uh, Klein, I use I mean, I played with all of them. with all of them. with all of them. >> Um, so I learned what the state of the >> Um, so I learned what the state of the >> Um, so I learned what the state of the art is for at least for the, you know, art is for at least for the, you know, art is for at least for the, you know, few weeks what the state of the art is. few weeks what the state of the art is. few weeks what the state of the art is. >> Yeah. For that Tuesday. >> Yeah. For that Tuesday. >> Yeah. For that Tuesday. >> For that Tuesday. Uh and then the second >> For that Tuesday. Uh and then the second >> For that Tuesday. Uh and then the second thing, the third thing is here's a tool thing, the third thing is here's a tool thing, the third thing is here's a tool that actually is useful to for the that actually is useful to for the that actually is useful to for the academic community, both conference academic community, both conference academic community, both conference reviewers as well as people writing reviewers as well as people writing reviewers as well as people writing papers and submitting them to make sure papers and submitting them to make sure papers and submitting them to make sure that their references are correct. So that their references are correct. So that their references are correct. So it'll have some value beyond this. it'll have some value beyond this. it'll have some value beyond this. >> Well, and everybody wins when you make a >> Well, and everybody wins when you make a >> Well, and everybody wins when you make a tool like that. There's a talk. tool like that. There's a talk. tool like that. There's a talk. >> Yeah. >> Yeah. >> Yeah. >> Like I was going to do this talk at the >> Like I was going to do this talk at the >> Like I was going to do this talk at the C++ conference and then I did this thing C++ conference and then I did this thing C++ conference and then I did this thing in the hotel and I'm like, you know, in the hotel and I'm like, you know, in the hotel and I'm like, you know, that's a more interesting talk. that's a more interesting talk. that's a more interesting talk. >> So it it it changes what I want to talk >> So it it it changes what I want to talk >> So it it it changes what I want to talk about. about. about. >> Yeah. and I just gave a talk on it here >> Yeah. and I just gave a talk on it here >> Yeah. and I just gave a talk on it here at this podcast. at this podcast. at this podcast. >> Yeah. Yeah. Yeah. Very cool. Um I don't >> Yeah. Yeah. Yeah. Very cool. Um I don't >> Yeah. Yeah. Yeah. Very cool. Um I don't know what the name of this show is then.

know what the name of this show is then. know what the name of this show is then. >> Um >> Um >> Um >> Mark and Scott learn to what? >> Mark and Scott learn to what? >> Mark and Scott learn to what? >> Learn to >> Learn to >> Learn to >> learn to enjoy coding again. >> learn to enjoy coding again. >> learn to enjoy coding again. >> Write an AI hybrid app. >> It's uh it's not catchy, but it is uh it >> It's uh it's not catchy, but it is uh it is reality. is reality. is reality. >> All right. Um >> All right. Um >> All right. Um >> All right. >> All right. >> All right. >> We'll figure out the title for this one, >> We'll figure out the title for this one, >> We'll figure out the title for this one, but uh that's really cool. those kind of but uh that's really cool. those kind of but uh that's really cool. those kind of little utilities, the world's better little utilities, the world's better little utilities, the world's better because they're out there and now it's a because they're out there and now it's a because they're out there and now it's a thing. thing. thing. >> And by the way, the number of lines of >> And by the way, the number of lines of >> And by the way, the number of lines of uh Python, this is like multiple uh Python, this is like multiple uh Python, this is like multiple thousands of lines of Python. It's like thousands of lines of Python. It's like thousands of lines of Python. It's like a monstrous project. >> 9,000? >> 9,000? >> No, it's like 3,000 4,000. >> No, it's like 3,000 4,000. >> No, it's like 3,000 4,000. >> That's still a lot. >> That's still a lot. >> That's still a lot. >> Yeah, >> Yeah, >> Yeah, >> that's work. >> that's work. >> that's work. And do you look at it and you're like, And do you look at it and you're like, And do you look at it and you're like, "This is idiomatic. I would write it "This is idiomatic. I would write it "This is idiomatic. I would write it like this." like this." like this." >> Yeah. That's the part that freaks me >> Yeah. That's the part that freaks me >> Yeah. That's the part that freaks me out. out. out. Do you give it credit though? Does you Do you give it credit though? Does you Do you give it credit though? Does you put its name first in the put its name first in the put its name first in the >> Actually, go look. Oh, >> Actually, go look. Oh, >> Actually, go look. Oh, >> I will go look.

Ref checker. Ref checker. You got 87 stars already, bro. with various assistants. Very nice. with various assistants. Very nice. Okay, we need a name though. I think we're both getting pulled in to I think we're both getting pulled in to work now. work now. work now. Um, the joy of creation, something like the joy of creation, something like that. that. that. All right, friends. As you can see, Mark All right, friends. As you can see, Mark All right, friends. As you can see, Mark is off deleting email. He's lost all is off deleting email. He's lost all is off deleting email. He's lost all focus and he's [music] gone back to focus and he's [music] gone back to focus and he's [music] gone back to work. Like and subscribe. uh thanks for work. Like and subscribe. uh thanks for work. Like and subscribe. uh thanks for spending time with us and uh [music] and spending time with us and uh [music] and spending time with us and uh [music] and and help us out by sharing uh the and help us out by sharing uh the and help us out by sharing uh the podcast episode by linking to the podcast episode by linking to the podcast episode by linking to the YouTube directly from your LinkedIn post YouTube directly from your LinkedIn post YouTube directly from your LinkedIn post and we always read your comments [music] and we always read your comments [music] and we always read your comments [music] so please keep up uh the comments.