Art by Koi
AI has become too big to ignore. In the last several months, several prominent figures have come out to say that AI is "suddenly good now". People are starting to trust AI more than ever. "Grok, is this true?" is now being asked seriously and not as a joke. And so, I thought I ought to give this whole vibe coding thing a shot. Last time I tried agentic AI, which was last summer, it was unusable garbage. So let's see if it suddenly got better in the last six months. Before I get into my results and my thoughts, let me first explain the design of my experiment.
In general, I already use AI for work. I use GitHub copilot to autocomplete my code. I experience some degree of the notorious "copilot pause", but I don't feel dependent on AI. Copilot helps me speed through writing boilerplate and reduces the time I spend navigating poorly-documented APIs, and that's about it. Which is perfect, because those are my least favorite parts about writing code, and the output of the AI is very easy to double-check. So, in my mind, AI as a productivity tool for an otherwise human development process is already "there" and has been for quite some time. What I'm hearing that is different now is that agentic AI is good. Letting AI write the code has historically been a bad idea, but what everyone is saying is supposedly different now is that you can actually do it. And so, what I did is based on that perspective.
The experiment is as follows: Take an app idea I've been meaning to make for years, and use Claude Code (the best programming agent currently available) to make it. I will avoid writing code as much as possible and defer everything to Claude. If I notice the AI going down an unproductive rabbit hole, I will intervene and hint at another solution, but in general I will let the AI drive. Someone who was trying to sell me on this whole "AI is already here" idea gave me with a 1-week Claude Pro referral. So, the experiment will end either when I use up all my tokens for the week or when the app reaches a satisfactory state. The goal would be to make a music player app. There was a feature I wanted to add, so it would be more than just a music player, but spoiler alert: I didn't get far enough to actually add it before running out of tokens for the week.
For some additional context, I did not spend any time setting up MCP servers, sub-agents (also called "swarms"), or writing skills for Claude. I also did not make any great efforts to minimize my token usage, so if the AI decided to waste a bunch of tokens doing internet searches then so be it. I did start using /compact in between tasks starting on the 3rd day, but this didn't seem to make much of a difference.
I have a few reasons for not doing these things. First of all, I didn't feel like it. If the point of AI is to eliminate effort, then it follows that I shouldn't have to put in a bunch of effort just to use it. There were some other reasons too. A big selling point of AI is that it can allow less-than-technical people to create software. If that's true, I shouldn't need to be fucking with multi-agent setups or have specialized knowledge about how to structure a project "for AI". Another reason is that for these optimizations, deciding which to use, and how to set them up properly, is still a matter of intense debate. Any setup I go with is guaranteed to be non-standard and wrong in some way. The experiment would, in effect, become a test of my AI workflow rather than a test of the tool itself. So, everything is out of the box for this test. Another reason not to do this is that I only have the AI for a week, and I have relatively limited tokens for that week, so there isn't enough time to nail down a concrete setup. I'm not paying another $200 so I can get enough tokens to work through the weaknesses in their default configuration.
Finally, I began this experiment on a Thursday, with the expectation that I would spend most of my time on the project over the weekend, hopefully managing to consume all my tokens by Monday. This basically worked out. All my tokens for the week were gone by Sunday afternoon. I will point out that means I used a week's worth of tokens in basically three days. So, let's recap how the experiement went and what my takeaways were.
The first day was relatively uneventful. I just wanted to get it started and see if I'd need to do any setup to convince the AI to start working. This proved very easy: I just made an empty folder, installed Claude, and told it I wanted to build a music player app for Android. It "thought" for a few minutes on how to do this, recommended a plan, and I accepted. It then scaffolded a project. This seemed to go okay. The tech stack it chose was Expo on top of React Native. This seemed like a reasonable choice. Then it came time to test the bootstrapped app.
Immediately, the AI ran into issues. I didn't have Node installed. No matter, I simply installed it via nvm for Windows. This proved to be a costly mistake. While using NVM to install Node is basically the only sensible way to do it, it does make your installation somewhat non-standard. And for some reason, even though my shell was fine, whatever environment Claude used was simply unable to find node in my path. The only way it could execute Node was by manually invoking node.exe via its full path. A full path which it couldn't figure out, because it lived inside NVM rather than the default install directory. Anytime Claude wanted to run node or npm it first decided it had to run export PATH="/c/nvm4w/nodejs:$PATH". Not great. It was unable to solve this in any kind of long-term way during the entire experiment.
I think what happened here is that Claude was convinced it was running inside a regular Bash terminal, in a regular Linux environment, when it was in fact inside of some fucked-up pseudo-Bash environment that was neither Linux nor Windows. Paths on my computer were formatted like WSL, but basic commands like tail were missing and it was executing Windows binaries. Utterly bizarre. Even the AI thought it was strange, because it would periodically forget it had to do this and I would have to remind it. Not sure why.
The second day went a lot smoother. With the app now scaffolded, and the Expo environment set up, I had the extremely barebones music player up and running pretty quickly. I could see it on my phone and hot reloading pretty much worked. This is more of an Expo win than an AI win, but there was an app and it did play music. Allegedly. Because, you see, that takes us to the next issue. The app couldn't find any of my music.
Fixing this issue basically boiled down to the following. I tell Claude "Hey, there are no songs". Claude says "Hmm, the MediaLibrary API is to blame", then does something random, then tells me to test again. I test again, still doesn't work. After three or four round trips of this I get a bit tired of constantly reloading the app on my phone and tell Claude to just let me pick a folder on my device to load music from. This also requires several stages of back and forth to actually get working. And when it does, it's bad.
A common theme throughout this experience was that the AI would struggle so much to get a basic feature working that, by the time the AI got it working, it would basically be out of tokens for the session. Claude sessions are 5 hours long, and I was usually running out of tokens around the 40 minute mark. Then, because the AI spent all its time trying to get the feature to work, there wasn't any time left to iterate on the feature to fix all the bizarre design choices it made. For example, the AI decided the correct way to let you add a folder was by adding a folder icon to the top right corner of the screen. Fine. But then, it decided to add a number in a circle next to the folder for each folder you added. This sucks, because it looks like a Discord ping notification. But it's not. It's just an irrelevant status. I never fixed this during the entire development of the app because there were always bigger issues to deal with. But these odd design choices kept accumulating over time and eventually made the app basically unusable for anyone except me.
Anyway, when I say it was bad, I mean it. The song list was populated, but it was just filenames. No metadata. So you couldn't actually see the real song titles. And all the songs had a length of "0:00". And there was a blank placeholder where the album art should go, but it wasn't populated.
I spent the entire weekend basically just convincing the AI to implement these obviously-missing features. It struggled massively. A big problem it ran into was performance. Scanning ID3 metadata takes a nontrivial amount of time, and so good music players will do this once when songs are added and then cache the results. Usually updating the cache only when you manually ask, or when you start playing a song (and then, just updating that song). Instead, this was happening at app startup. Which meant all 3,000 songs' metadata and album art were being loaded into memory. It didn't help that its implementation for loading this stuff sucked and it took around 20-30 minutes to finish loading the metadata for my collection, either.
But, aha! I'll just ask the AI to make it faster. And tell it to add caching. Well, it did add caching. But it wasn't faster. I told the AI it was still slow. It did something. Didn't fix it. Then again, this time it actually broke something. Finally, I told the AI something it should have known to do- write benchmarks so you can find out what's actually slow about it. Then make it faster. This slightly helped but not much.
On day three, which was the first full day, I admitted defeat on making this app any good, and focused on having it implement features. Perhaps this could at least serve as a proof of concept before I make a real app on my own later. I gave it a fairly simple task: Integrate support for the Android system miniplayer. That way I can see what's playing on my lockscreen and stuff. This turned out to be a critical mistake. First, it completely broke the Expo preview. Second, it turned out to require manifest changes, which meant I abruptly had to switch to testing via ADB and a "real" app instead. Which also meant I abruptly had to get the Android app build working. I tinkered around installing Android studio for a bit, then came back and tried to build. Of course, it didn't work.
So, I did what any self-respecting vibe coder would do, and I pasted the build errors into Claude. It would try stuff, which wouldn't work, and then ask me to test it again. Then I'd paste more logs and we'd go back and forth like this repeatedly. Meanwhile, I was taking notes on what it was actually doing. As it got deeper and deeper into this problem, it became more and more aggressive in its solutions. First, it tried to troubleshoot the environment. Then it looked at the code. Then it decided to give up entirely on some dependencies in the project to see if that helped. Finally, the AI resorted to writing a script that would manually edit an erroring package in my node_modules into something more agreeable. This worked... but holy shit. It's programming like a human. Or at least, programming like some humans, I guess. Maybe for Claude 5 they can make it program like humans who know how to maintain software.
Regardless, it did work. So I moved on. This wasn't meant to be a long-term effort anyway, so this type of noise is acceptable for now. Still, Claude loses a fuck ton of points in the evaluation for that. It's good to know that even Claude struggles to navigate the mess that is the Android ecosystem.
I took a brief break to have the AI write some tests. I've been cautioned against doing this, because supposedly the AI has a tendency to "cheat" by writing incorrect tests. I didn't observe this, though. Instead it wrote around 120 tests that seemed correct but were completely useless. Despite being a music player app, the AI decided it was a good use of tokens to write over a hundred string formatting tests. Zero tests written on the actual metadata parsing, which it regressed several times throughout development. No UI tests. Nothing to mock the results of various system calls to ensure that the app side remains sane, and certainly nothing that I could run on-device to ensure that the system-level assumptions remained correct. Instead we got this.
it('handles spaces in filenames', () => {
expect(filenameFromSafUri('content://tree/Music%2FMy%20Song.flac'))
.toBe('My Song.flac');
});Great test to write if the codebase is a freshman-level CS assignment. But in a real production codebase, what are the chances of anyone ever changing the code this tests, much less unwittingly regressing it? It also mostly validated the other app functionality by passing callbacks to random functions and then checking if the callbacks got called. Which like, sure. Great. But what does that prove, exactly?
The answer is "even less than you think" because this function does not exist in the codebase. Instead, filenameFromSafUri was implemented inside the test. And that function just shelled out to an ES5 built-in function. So apparently we're just testing the standard library now.
By Sunday evening, I had completely consumed my weekly usage. So, I decided to call it there. We had just started on playlist support, but ran out of tokens before it was actually working. I never actually made it to the part where I got it to implement my new features, because it struggled so much on just getting a music player together.
Obviously, from a technical standpoint, this was a complete and utter failure. I am unconvinced that the amount of progress I made with Claude over the four days I used it was more than what I could've achieved with a similar amount of effort on my own. And with my own effort, I would have learned a lot more about the Android ecosystem, music file formats, and app development. Instead I'm coming away from this project mostly just knowing half a dozen random tips for using Claude slightly better and a really vague understanding of the challenges involved in making a music player.
The app it developed sucked. A review of the codebase at the end shows a React Native app that is simultaneously too simple to be interesting and too complex to be maintainable. There is a huge amount of dead code, errors get alert popups with basically no further handling, which means the app almost always ends up crashing whenever anything unexpected happens. Lots of brain-dead simple best practices that everyone should be following, like putting units in variable names, were not followed. There is no logging whatsoever. The UI is somehow both oversimplified and confusing, with the AI basically never adding any information to the screen unless specifically asked. Even showing progress for file imports, a feature any human would've thought of within 10 seconds, was something the AI specifically had to be prompted to add.
So, do I feel threatened by this? No. Great apps that are worth developing, especially worth developing for money, take time. And this was not only not faster, but it also failed to scale even through a single week of usage. I can only imagine the insane gymnastics you would have to perform to make this usable on a year-long project. And you'd probably spend more on tokens than what a decent software engineer cost in the end, anyway. This stuff isn't exactly cheap when you're constantly running out of tokens and being told you need to buy more. Perhaps that's the real innovation here. Pay-as-you-go software development. Bringing the Freemium business model to coding.
You may (correctly) point out that my environment was weird and that likely had an impact on the performance of the agent. My counterargument to that is: If my environment was broken, the AI should have fixed it or suggested that I fix it. My biggest problem with the AI is that it provides no pushback whatsoever. If you suggest an approach that is bad, it will just do it and leave you to deal with the consequences later. It lacks the intuition and cynicism that all great software developers have. Plus, since it's trained on what ultimately is a lot of very mediocre code, it unsurprisingly spits out a lot of mediocre code itself and struggles with even very easy-to-grasp problems like "Node is installed to a non-default location", likely because none of the examples it's trained on account for this. In the real world, you don't need to explain how to run Node because everyone already knows how to do it on their machine.
I feel like a lot of vibe-coding proponents lean heavily on this idea that brute-forcing your way through vibe-coding an app will get you to the right result eventually, just so long as you do enough iterations and testing. But that's not good enough. You know what else gets you to the right result eventually? Learning how to do it yourself. That's a hell of a lot cheaper than spending unbounded amounts of money on tokens, and in my opinion, way more rewarding and less frustrating.
I will give credit where it's due. The overall project scaffolding was reasonable. The tests were garbage, but the test harness the AI set up was not bad and could be filled out by a human pretty sensibly. Even though the code sucked, the high-level organization of the code was fairly standard and if I was looking for the code to a specific part of the app, it was generally pretty easy to find. The CSS was a complete mess, but most humans I've worked are happy to waste time doing anything other than actually learn how to write it, so that's no surprise. The AI CSS was bad, but not awful.
There is one thing that the vibe coding approach was undeniably successful at. The amount of effort I expended was far, far lower over the weekend than it would've been doing this on my own. I barely had to think most of the time. In fact, I usually felt like a machine, blindly pasting console outputs into a chat window and awaiting my next instructions from the agent on what to test and how. I was Stanley, from The Stanley Parable, sitting at my computer monitor, waiting to be told what buttons to push and for how long.
At the end of the project, I looked over what I'd done, and how I spent my weekend, and I felt... bad, honestly. Normally, when I burn a weekend doing some kind of poorly-conceived side project, I have a great time and feel fantastic heading into the work week. But after this, I just felt spent.
Working on software is very energizing for me. It's my art. It's how I express myself. I pour my values into the software I produce, and everything I make has my own personal style written all over it. I am an artisan when it comes to software and I take my craft seriously. I know my website here isn't perfect, but it's undeniably mine. Vibe coding didn't do it for me. I didn't feel like I made an app at the end. I felt like I watched somebody make an app. It didn't feel the way I would've made it feel, it wasn't written the way I'd write it, it wasn't even the color I would've made it. Sure, I could've told the AI to change it, but I never got around to it because the fact that the app took 26 minutes to start seemed more important.
The feeling I got on Sunday when my tokens ran out was not a warm glow of accomplishment. It was a bitter apathy, the same sort of feeling I get after I waste an entire afternoon watching YouTube videos. Vibe coding is not a creative process. To argue it is would be like arguing it's creative to watch TV because you have to pick what channel to watch and what to set the volume to.
There is a certain degree of belief and ego that is needed to create truly great software. You have to be someone who nitpicks absolutely everything, someone who will spend three hours fixing that one edge case that makes the screen flash white sometimes. Someone who understands the importance of making your game respond correctly to someone pressing left and right at the same time. You don't need to prioritize everything, but if your only priority is "make it work" then you are never going to make something good. Maybe vibe coding will prove good enough for many uses in the industry. I'm sure nobody gives a shit if Xbox.com feels shoddy and uninspired. But I just cannot imagine trying to create something new, or making something for fun this way.
I did not enjoy vibe coding. I don't think I want to do it again. If this is the future of software development, it's a very dark future indeed, and it's one I am completely uninterested in being a part of. We are already experiencing a crisis of software quality. Everything we use is getting simultaneously worse and more expensive to make. The industry has been flooded with and is now run by people who don't give a shit about software. Generative AI is not a revolution, it is a commitment to the slippery slope we have been sliding down for the past 15 years.
I've noticed a disturbing trend recently of depicting vibe coding as a sort of liberation. There is this idea that software engineers are this predatory group of middlemen who prey on the skill gap that exists between the visionaries and the real world. Vibe coding solves this by eliminating them. This is a powerful narrative. However, I think it's based on a deliberately misleading assumption: You are not the visionary in this scenario. This definitely represents a significant transfer of power away from developers, but don't be fooled about whose hands that power will end up in.
When you make an app, the most interesting thing about it is that it's made by you. But when you vibe code an app, the most interesting thing about it is that it's not made by you. You should think about that.