Adventures in Vibe Coding

Ai it's everywhere these days and more and more our corporate overlords are saying it will replace anyone and everyone who does anything creative or technical or in some case legal (which ironically is usually illegal).

More and more pressure to "explore the tools" and "come up with agentic work flows" and "put Instagram style duck face filters on all games"

So this week so I could at least say I'd installed the tools and played with them should that become a mandatory rather than heavily advised requirement.

I installed the various plugins for vscode I tried cline and codex which were then hooked up to one of the models in this case chatgpt 5.4.

First I tried to use it with our source and it wasn't initally promising codex refused to work over our remote tunnel to the build servers only able to function locally so that was out right away. I switched it to cline and I asked it where in the code various debug lines were and what they meant. It just spat out a grep command with the parameters I'd already specified which wasn't really what I was after.

We've had various engineers throwing patch at cline to check them and almost always it has reported bugs that arent bugs or advises changes that would actively break things. When its right its so far been somethign very trivial like an extra charige return on the end of a string that would automatically put one in due to the type of string handling they were using. Its been really pissing off some of our senior devs as they submit a code drop for review someone runs cline on it and copy pastes the results into the review and then they spend a few hours trying to work out what the hell the thing is trying to say find it was completely wrong or some minor thing and they've wasted half a day on that.

So not that promising, I wondered if our code base is just too large for it to fit in the context window it is hundreds of thousands of files and a lot of code when it pulls it in you get one or two questions before it runs out of tokens and then has to compress the context and loses most of its capabilities and really goes to shit.

So maybe I needed something smaller something it can start from scratch with. So I decided to throw a simple project at it and vibe code it as the ai boosters want you to. So I decided to ask it to make me a ray casting engine in html5 so essentially doom or wolfenstien. With just that prompt it spat out a game with solid colored walls generated from a simple array map with WASD controls and collision detection and a rendered minimap. from there I had it add mouse look and then change the solid coloured walls for procedural textures. The textures had issues when the camera was very close a quick sentance and it fixed that. It had a collision issue with corners it fixed that. I had it add enemies health and health pickups basic hit scan guns. I had it add simple AI to the enemies with patrol routes and chasing mechanics based on the last position they saw the player. I had it add a debug mode where it would show paths and states as well as positions and such initially it put that under the other gui elements and it took a few goes to move that somewhere you could see it.

Then I had it change the map to be more procedural rather than fixed it did that but that fucked up the textures on the walls and put all the enemies in the same room. I got it to fix that but more and more issues began to pile up. Since it changed the map generation to make rooms connected by corridors rather than a big open room with a few fixed walls when the patrol routes overlapped in a corridor and two or more enemies were in there at the same time they would collide and get stuck. I asked it to fix that it made them try and shuffle past each other but they still got stuck. I asked it again it tried some sort of queuing mechanism which also didn't work. I asked it a third time and it claimed it had it but again it just didn't work.

At this point it had burned through 200k tokens and increasingly it was taking more and more to try and fix things and it was just failing. It also began breaking other parts of it as it added to it pathing on things that had worked began to fail. Attacks stopped working and things that had functioned didn't anymore and each of these things was then a loop that you had to go through trying to get it to fix it hoping that didn't break something else.

I gave it one more attempt and then gave up.

So it was an interesting experiment. What it managed to throw together with me just saying things like "add enemies" "add health pickups" "the health pickups look weird when you are further away" "the health pickups look like they are sliding into the floor when further away" and so on.

It's an impressive magic trick what it can throw together just with plain English and produce something quite polished at least on the surface but then as you go further and the code gets more complex it starts to break down it takes more and more goes to get it to fix things and increasingly the more complex issues seemed to be too much for it to fix. Then I could go in and try and fix it myself but unlike a normal coding project where I've put it together piece by piece and understand its systems already and have some idea of what I need to even look at with this code I've not looked at any of it I don't understand what it is doing or why. I could go through it and work that out but I would basically need to try and reverse engineer the whole thing from top to bottom to get to the point I understand it enough to make changes and fix it.

And this seems to me to be the fundamental problem with the ai coding model. The more it is used the less you understand what it has developed the more of crutch it becomes and the more you can't fix it yourself and need to hope it can be persuaded to fix the thing by asking it over and over and over in slightly different ways each one potentially breaking something else as the context window slowly balloons and it burns more and more tokens to try and fix things.

I can also see why non technical people are fooled by it. That first pass that first 90% that it can generate from a simple prompt is pretty impressive but the further you go the more it breaks down and requires someone that actually understands the code to come in and fix it and that's not even considering what ever crazy security holes it creates, or code maintainability / reuse, or compatibility with other projects outside its limited context, or performance or memory considerations. When what it produces is a on the fly sort of design thrown together with bits it grabbed from a whole host of sources its a bit of a black box in terms of what its doing under the hood.

Back in the day when I were a youngun and I wanted to make games I bought this book on coding doomlikes. It was probably already obselete by that point as it used video hardware interrupt based frame buffers and manual bliting peeking and poking memory and so on which had already started to go out of fashion for more hardware based rendering and the early 3d stuff but it gave me a glimpse into how all of that worked and I took a few stabs at writing my own before giving up and trying to learn glide the voodoo 3d propriety opengl esq api. If I'd had this tool back then I could have gotten a lot further but without having to learn the basics of how code is put together of how hardware works how you get it to comply to your whims and how it fails most of the time and what to do if it does.

I can see a kid these days with the same interests just throwing prompts at chatgpt and then not having the tools to get beyond a certain point and by then the code would be sufficiently weird and complicated even if they have the interest to learn what is going on under the hood they have a much more complex and difficult task than if they'd started from scratch themselves and worked their way up. I can easily see people thinking the magic trick gets them close enough and then when they hit that wall they get discouraged and can't do anything more but repeatedly prod chatgpt till they run out of tokens or patience.

This is also in a case where a lot of these models are currently offering vastly more tokens that you would get if you actually had to pay for them. Its definatley in the first taste is free part of the marketing and when that goes away either you are paying hundreds or thousands or even in some cases tens of thousands of dollars in compute for the same service or you try and get by on the lower tier rate limited or lower token context modes. And if it only just barely works with the floodgates wide open what is going to be like when they turn that down to a trickle.

Still I feel like the magic trick of how it can get you most of the way there in the first go for managers and people that never actually have touched this stuff will easilly convince a lot of them they can just fire all their engineers and replace them with one guy and a chatgpt sub and for a while that might work but in the long run its going to stack up a lot of problems although by the time those chickens come home to roost maybe it will be too late or they'll hire us all back at a fraction of our previous salaries since we'll be desperate. I worry for the next generation as I said having this when I was a kid I could easilly see I might have used it and never learned the actual coding and the only way to really get good at these things is to write and review a lot of code and if AI is doing all of that grunt work how do young kids even get a foot in the door to start down the path to becoming cynical broken grizzled veterans like wot I am.