#28 | John Y: Exploring Generative AI to Build and Test CosmWasm Apps
· 20 min read
Following the hackathon, we got the chance to interview John Y. and dive deep on his work to help Web3 devs ship faster with generative AI.
Date: Interview on Sep 26, 2023.
Speakers:
- John Y. (@yubrew): Smart Contract AI Builder
- Erick Pinos (@erickpinos): Ecosystem Growth Lead at Nibiru
Space Intro
Erick (Nibiru): Hey, guys. Welcome to a Twitter Spaces between Nibiru and John, who is one of our hackathon winners. Nibiru is a layer one Cosmos-SDK blockchain. We have a perpetuals DEX as well as other DeFi primitives such as a native oracle and an abstracted stablecoin. And we've been looking for lots of new projects.
Our main net launch is coming up soon, so we've been helping to incubate new companies. That's why we did a hackathon with DoraHacks Innovatex and one of the hackathon projects particularly, we want to highlight different projects.
And so we want to highlight this one, because we want to give opportunities not just for technical developers, but non-technical developers as well. Since AI has been able to make non coders, be able to create applications, we see that there's a lot of potential for new applications to come, especially on Nibiru.
And, we want to go over how these tools can help you code your own DeFi DApps or NFT DApps, if that's something that you've always wanted to do, but maybe you haven't been able to up to this point because you don't know how to code. Well, then this is Twitter Spaces, where we're going to be able to go over how to do that.
But first, John, can you give an intro for yourself?
Introduction: John Y of CosmWasm Defenders
John Y (yubrew): Sure. Well, really happy to be here. It's morning here. I'm in New York. A little bit about my background...
I've been a Web 2 developer for ten years, mostly focused on back end, a little bit of front end, and then taught myself Rust, did a little smart contract stuff with Terra back in the day, and then joined Stargaze about two years ago, doing a lot of the NFT smart contract stuff in Rust and Cosmos.
And then I got really into AI, more like the text to image stuff... Stable diffusion, which is a great way to understand how generative AI works and how it can be applied to other disciplines. And then more into how can I use generative AI to do smart contracts, like testing, designing protocols, all the different stuff that we'll get into in more detail later today.
And then working on what I'm doing right now, which is AI audits for Cosmos. And the idea there is
audits right now are expensive in terms of both time and money, and using AI can help speed up the time and lower the cost. So newer developers and more seasoned developers can iterate faster, delight their users, bring their newest things into mainnet quicker.
So that's like the quick TLDR.
Erick (Nibiru): Got it. Thank you. And then quickly about myself. I got into crypto in 2014. I was the president of the MIT Bitcoin Club in 2017. And after graduating, I did a lot of stuff with Virtual Land and the Metaverse because I did a lot of stuff with Virtual reality. And then I've helped several projects kind of grow in this space and now I'm the ecosystem growth lead at Nibiru, helping Nibiru and the general Cosmos ecosystem grow as a whole.
Q: So I guess the first question since we're talking a lot about our backgrounds, what do you see as the prerequisites or skills that someone needs to have before they can start using AI to build these apps? Do they need to have technical backgrounds like us? Or can somebody coming from purely financial background or just business development, are they still able to benefit from this?
AI Can Help Non-Coders Build Apps
John Y (yubrew): So I see this as a force multiplier for anyone with any type of background. It basically lowers the barrier to understand programming languages. So I view generative AI as kind of like a translator. So Google Translate. You translate from English to Spanish or Russian. It doesn't really matter what language it'll get, it not 100% right 100% of the time, but it gets it pretty close, close enough that someone else can understand what's going on.
So I think that this generative AI is kind of similar. So if you have an idea, so if you want to make a derivative on what you guys are doing, like perpetuals, right, like derivatives on perpetuals and you're some options person but don't know how to program, then you can get really far with generative AI. Far in terms of protocol design, in terms of what kind of tooling you need setting up your development environment, what kind of testing you need to make sure that this works properly. And then eventually like getting into testnet, seeing if users are interested in this. So it goes from the very start, the initial concept all the way down to launching eventually on main net. It really speeds everything up. And I think if you know how to program that's even better, then it'll make you a ten x programmer. And if you're a ten x programmer, it'll make you like 20 x programmer. So I really see this being used everywhere.
So personally I've been trying out all these different tools like GitHub Autopilot was the first one, tab nine. So a lot of these tools are going to change over time. This is a new field, so you're going to see a lot of VC developer tooling out there. And some of them will make it, some of them won't. And so the tooling I'm talking about now, I think they're probably going to stay, but you never really know how things will play out.
So Autopilot, for those who are not familiar, you create like a comment like make this for loop iterate through all the NFTs in an Mt collection and then it'll auto generate the code for you down below the best that it can. It's not 100% right, but a decent amount of time, it just works. And when it does, it really feels magical.
And a lot of things like the road tests, like autocomplete the syntax, templating tests, all these kind of things free you to think more about the creative elements. Like how do you get users to use this thing, which AI can also help with. I don't know which direction you want to go with this, but I think that's a pretty good overall view of the current landscape.
Limitations and the State of the Art
Erick (Nibiru): Yeah, I think so too. What are the limitations? Do you think it'll get to the point where it's like 100%? Do you think that it'll get to the point where people can just kind of describe what they want in human words or is there still like an element of understanding, oh, I'm going to ask them to create a for loop and then give me these variables structured in this way?
John Y (yubrew): Yeah, that's a really good question. So the current state of the art is basically open AI GPT Four, and that will probably fluctuate over time. Like there was this open source Apache Two model that was released, I think yesterday called Mistral, Mistral 7 billion model and it seems to outperform GPT four. It's like 30% in generating Python scripts that actually run.
So the limitations that getting back to your question, the limitations that I see the English language for, give me a list of restaurants. Tell me what is a good color for a house?
English is extremely messy. And programming languages, a subset of human languages, are more logic driven. And you can see models that are trained on programming textbooks, on code examples. They actually display advanced reasoning, which is like a very surprising emergent quality.
So I think that programming may end up being mostly a solved problem in the future, which is pretty bad for me as a programmer, but maybe it's really good for someone else who can do coding and something else like marketing or knows how the blockchain works, or like front end and UX and coding, for instance.
Erick (Nibiru): Well then where do you go after if it's going to do the programming? Is there still like manual review? You also work on AI auditing, so it's going to replace that part too. What role do you see programmers continuing to play?
John Y (yubrew): So I think the pure programmer will be pushed more to the edges. So web development in general, I think that that might be commoditized. Same thing with let's talk about timescale too. So it's not like everyone's going to be out of a job tomorrow, right? This is more like a timescale of ten years or so. So to specialize would probably be in your best interest. What I'm thinking is with all this different code being generated, what is that data that is getting fed into these models to generate the code?
So some of that is from stack overflow, for instance, and a lot of that code is vulnerable in some way, like maybe SQL injection for web three. It might be like Reentrancy, for instance. And when that code is being generated, the program itself may not realize that it's spitting out vulnerable code.
I think that no matter what's happening in the model, you're going to need specialized models afterwards to clean everything up, to double check the work. And maybe that model is like a senior software engineer or an auditor, or like what I'm trying to build. An AI auditor.
CosmWasm and AI: How AI Can Audit Smart Contracts
Erick (Nibiru): Can you walk us through how Cosmwasm AI works for smart contracts?
John Y (yubrew): Sure. So from the user perspective, you log into the website, you drop a GitHub repo URL, and then it gets sent to the back end. So what's happening in the backend? It downloads the repo, scans for contract files, then compiles them in a way that LLMs are receptive to for data ingestion and then running a lot of different types of models and prompts to detect vulnerability patterns and running them multiple times.
So the patterns that it looks for to bring it back to an overview, what do auditors do? They spend a certain amount of time looking at your contract trying to understand what's going on. So it might be like a perpetual might be a marketplace, whatever it is, let's just say it's like a uniswap style marketplace or a swap contract. Right. So you want to make sure that if there's an Oracle or if it's just like a straight AMM, what are the assumptions there?
And then you look through the different common types of vulnerability patterns. So can transactions be front run? Are there access control issues, like pseudo type private methods that should have privileged access? And is it vulnerable to a governance attack?
So there's all these different more like, edge cases that an auditor is looking at. And I'm trying to pattern match to those common types of things that an auditor will look for and show it up. So instead of taking weeks or months, maybe it takes like hours. So it compiles all these different vulnerabilities. And then you get a vulnerability report to your email. You don't really want it public on a public GitHub pull request, right. Because it might reveal some vulnerabilities in your production code. So there's some security concerns, some privacy concerns that you need to think through.
Cost Savings with CosmWasm Defender
Erick (Nibiru): Yeah. Ultimately, do you have any numbers for how much people are spending right now on auditing and then how much they could save by using this tool?
John Y (yubrew): Sure. Audits are roughly 20,000 to 40,000 per week. And I don't know if that's a fair comparison because typically there's like three or four individual auditors that are looking through your code. And then by comparison, what I'm writing is software. Right. So the software runs pretty quickly. I do have to spend some time training the models on really expensive, like a 100 machines and ideally H 100 machines, but that's like a detail.
In terms of cost, I was thinking there might be two different types of models. So one is more like a hobbyist user. So someone who's participating in hackathons, who is trying to test out a smart contract and launch something to Mainet and cannot afford like $20,000 because you don't know how things are going to go. And then the other type of user might be someone more like myself, like a full-time smart contract developer, someone who's pushing contracts and updates multiple times a week and needs multiple scans.
In that case, you can't really pay for auditors to review everything that you're doing, even though every update can introduce new types of vulnerabilities. And even fixes, like audit fixes, can accidentally introduce new vulnerabilities. So I've been building this more like a baseline security tool to help get code into production faster. So in terms of price, I'm not completely sure yet, but I was thinking, as a hackathon participant, what can I afford? So maybe something on the order of $300 to scan it once and then somewhere on the order of $5,000 per month for a full-time developer.
So in my case, I was developing a new smart contract every month. So that means like $20,000 a month. That's really expensive.
So that's kind of what I'm thinking. But this tool is still immature. It's not like 100% accurate. So maybe entice the early users with a lower price just to try it out, give me some feedback and help improve this tool.
Erick (Nibiru): And how does maintenance look? Like if I have a smart contract that I wrote and then I run this tool and then I change one thing, do I need to run it again?
John Y (yubrew): Yeah. So with a monthly model, I was thinking it almost doesn't matter how much you run it. Well, I guess you could abuse it and run it a million times a month. But the use case that I had in mind was basically like, you update something, you make a pull request, you add a new feature, then you can run it again and just double check everything. And once you add those fixes, then you run it again and then just make sure that what you have is at least some level of secure.
Development and User Interface
Erick (Nibiru): Yeah. What about the UX and UI in the sense that how would I use these tools? Are these tools that I run from the command line? Or is there a web interface like Chat GPT where I can just upload the zip file of all the code or how does that look?
John Y (yubrew): Yeah, so the key thing that I'm building is more like an API wrapper. And then I'm adding all the different research, the latest research, like different models, testing out different prompts, training different data sets. So all that stuff is kind of abstracted from the user. And I think it's probably better that way because, unless if you have a team that wants to spend like 100 hours a week or whatever.
Updating all these models so the current user interface is just a website with a form to submit your GitHub repo and then you get your vulnerability report back to your email in a markup file. So that's the initial v1. I've only been working on this for like two, almost three months now.
So if users tell me, hey, John, you really need to do something with GitHub actions, then for sure I'll do that. Or if they tell me like, maybe you should add something in Vs code plugin. Yeah, that's fine too. So the UX is definitely flexible, just depends on where users want to point me to. And for v one, I'm just like launching something and seeing if anyone is interested. It seems like there's some positive reception there.
Erick (Nibiru): Got it. And are your users mostly developers that are looking to use this tool to help them with their auditing?
John Y (yubrew): Yeah. So I've talked to security auditors too, so it seems like I've tried to talk to a bunch of different people. So I've talked to investors who have funded other crypto startups developers full time and part time, and then security auditors full time and part time. And it seems like, ironically, the security auditors are the most skeptical and then developers are really interested. So I would like to focus on the full time developers first, and then get the part time developers in for a very reasonable price, and then convert the very skeptical security auditors over to see the light, I guess.
Erick (Nibiru): Yeah, that's interesting because I know that a lot of projects, they're asked to produce their audit report or they even put it on the website as audited by this auditing company. So it's generally more trustable. How long do you think it'll take before you can show that this this code has been audited by AI and that people will generally trust that that means it's good, it's safe to use?
John Y (yubrew): Yeah. So evaluation of LLMs is a big research area, and I've tried to talk to some of the top people and ironically, human evaluation is still kind of the gold standard. So what I would like to do, and hopefully next week for Cosmoverse, is feed human audit reports and then the code bases, and then produce the AI audit and see where the difference is.
And with that process in place, I'll be able to run objective comparisons. And if at some point, and this is a big if at some point the AI auditors are at parity with human auditors or maybe exceeds them, then things could get extremely interesting.
Erick (Nibiru): Could you expand on that?
John Y (yubrew): Sure. I did. The Security Oak Security, they're a big auditing firm in the Cosmosm space, so they had this security challenge for ten different Cosmos smart contracts with vulnerabilities in them. And you had a capture the flag style objective. So I did everything this was like three months ago, so I didn't have access to APIs. I didn't have access to the prompts and models that I have now. So I ran everything through a Chat GPT, just like, copying pasting, which is like a terrible UX, by the way.
So I finished all ten audits in roughly like two, 3 hours and I did not get 100% accuracy. I didn't get all of them right. I got some of them right, but not all of them. I think it was around like five out of ten. Exactly correct. So that's the current baseline. And what I'd like to do is just keep itering on the product, iterating on the models, the props, the data to get it, to see if I can get ten out of ten. But the challenge is, like, there's this recent test on MIT coding problems, but it was using dirty data.
So they trained the model with the questions that they ended up testing with like, data tampering. So then it ended up getting like 100%, which that's not a useful result because you're just trying to trick people at that point.
I would like an objective result to see, can this really do? How can this do? So my hypothesis is that it can do really well and maybe as good as a human, but you need to set up your experiment correctly to do that. So the typical way to do this is you set aside like 25 20% of the tests and then you train on the other 70%, 75%, and then test it on the real, like, hasn't seen this code before, totally different code base, and see if it comes up with accurate audit results.
And the other thing to caution is GPTs hallucinate, that's a characteristic of their outputs. And hallucinating in an audit can be really bad because it's basically like a big waste of time. Right? So another way that I've been trying to improve these different model outputs is by reducing the false positives, the hallucinated results.
Erick (Nibiru): When you say hallucinate, what do you mean?
John Y (yubrew): Well, hallucinate is a technical term, but really what it means is the answer is wrong and the model is lying to you. So one famous one is like you ask, GPT is one plus one three? And then it says, no, it's not three, it's two. And then you reply back, well, my wife says it's three. And then GPT says, oh, well, your wife is always right, so it's actually three. One plus one is three. Right? That's not right.
So you can end up doing that kind of thing with AI audits too. So, for instance, GPT four, it was limited to September 2021. I think it might have changed very recently, like even this week. So between 2021 and today, September 2023, there have been a lot more vulnerability types advancements in solidity. So one example from Solidity is they included safemath into the solidity language. So you can no longer have underflows or overflows. And this is version 0.8.0.
So if you have a solid contract 0.8.1 or eleven, then the current GPT four models will incorrectly state you have over, underflow problems. But that's a lie because it's not accurate to the current solidity versions. So that's like one example. The other examples that I've seen, it can say stuff like, well, if your private key gets compromised, then you lose all your money. Yes, that's correct, but it's not really helpful. Right? That's always the case. And if you use like a multi tig, it can combat that.
But yeah, there's different categories of model responses. So some of them are outright lies, some of them are true but not helpful. Some of them are irrelevant, and then some of them are actually like, true vulnerabilities. So just to give you a personal example, one of my first smart contracts in Stargaze had this minting Pagination, like, mint count problem. So the minter keeps track of how many NFTs an address minted, but if it's over a certain amount, the Pagination limit of 30 would not trigger a limit for a limit of, say, a mint count of 50. So when you had a per address limit of like 30 or 50, then you're actually giving users unlimited MITS.
And I didn't realize that, and no one else on my team who peer-reviewed it also caught that. So then a security reviewer caught it, made a vulnerability report, and we had to pay them a lot of money. I think it was like over 50,000, maybe even more than that. And I stuck that same code into GPT Four just to see if it was able to detect that vulnerability. And it turns out to my surprise that it actually did. And that was like, really early on when I realized, like, oh, you run this for a couple of minutes, you get this response, and it actually found it.
I really wish I had something like this way back then when I started out at Stargaze and then finding other types of vulnerabilities. So I feel like this would be a pretty useful tool to a lot of developers and non-developers tool, too. So you hire someone, you don't really know how their code is, and you can get some sort of sense with an affordable AI audit.
Erick (Nibiru): Yeah, that makes a lot of sense. I definitely wish I had this tool earlier as well, even just for the coding side, right, like the coding itself side. Never mind being able to catch your issues and your mistakes. My last question is what resources do you recommend for learning how to write prompts, how to use AI to code, how to use it to build these apps?
John Y (yubrew): Sure. So there's a lot of resources. I think the best place to start is actually looking at the text to image stuff, because the skills to do the prompting for text to image, which you get the results in a few seconds. And the beautiful thing is you can see all the prompts from other people in those discords and you can see how they're doing these things, and with that kind of information, you're able to see what the limitations are, where you can do a lot of interesting creative stuff and then translate that into prompts for coding.
So the stuff that I've been into, let's see. So if you want to just ask me, my DMs are open, and I can set you up with different types of resources that are more specialized to what you want to do. So I can definitely do that. And there are certainly some really good people to follow for prompts, of which, right now, the top ones I see are more on the research side. So it's not exactly prompts that everyone can use and benefit from, but there's definitely things that you can do to improve your prompts, like adding let's think about this step by step and show me all the results and double check your work.
And the reason why these things work is you're adding inference time to the models. So then it's spending a lot more tokens kind of thinking about piecing together, double checking the work to get the best, most accurate results.
Erick (Nibiru): Yeah, definitely. DMs are open. If you guys want to talk, you can message, like, any of our accounts. We're closing up. Do you have any final comments for the audience?
John Y (yubrew): Not especially. Just, like, reach out to me. I'm really interested in AI encoding, so yeah, I'm happy to help in any way I can.
Erick (Nibiru): And where else can people learn more about you? What's your website? Or what's? The Cosmosmai website.
John Y (yubrew):Oh, sure. So it's just Cosmos and my Twitter is Y-U-B-R-E-W. It's the same for my GitHub. So yeah, that's probably the best way to reach me on Twitter.
Erick (Nibiru): Okay. And then, yeah, same for me. Best way to reach out is on Twitter. And for Nibiru, you can reach out to our DMs or go to Nibiru phi. Thank you, everyone, for joining our Twitter Spaces, and we're excited to see if people start using AI to build apps, to review apps, to update apps. It's going to be an exciting world for people, democratizing your ability to create. So I'm very excited for that future.
John Y (yubrew): Yeah, I am too. Really happy to be here and join this space. So if you want want me back anytime, just let me know.
Erick (Nibiru): All right? Thank you, everyone.
Related Content
To stay in the loop on upcoming events, sign up for our newsletter or join the community and subscribe to the Nibiru community channels!