What I learned building an AI coding agent for a year
我在打造 AI 程式碼代理人一年的學習心得
We thought we'd be the best within months. Here's what went wrong — and why I'm more excited than ever.
我們原以為幾個月內就能成為最棒的。事情出了什麼錯——以及為什麼我比以往更興奮。
It’s been a full year of trying to build the best coding agent!
這是一整年致力於打造最佳程式碼代理人的旅程！
I didn’t know that my world was about to change last July 4th, at hackathon where I first prototyped a CLI coding tool that became Codebuff. What a ride it’s been!
我不知道去年七月四日那天，我的世界即將改變，那是在一場黑客松中，我首次原型設計出一個 CLI 程式碼工具，後來成為了 Codebuff。這真是一段精彩的旅程！
From leaving Manifold, to doing YC F24, to hiring, to competing with Claude Code, all the while averaging ~70 hours weekly by working most weekends — it’s been a lot!
從離開 Manifold，到參加 YC F24，再到招聘，與 Claude Code 競爭，同時平均每週工作約 70 小時，大多數週末都在工作——真是充滿挑戰！
We may not have won the first round, but I’m more fired up and excited for the future than ever.
我們或許沒有贏得第一輪，但我比以往任何時候都更熱血、更期待未來。
Our bet 我們的賭注
We got so many things right initially:
我們一開始就做對了很多事情：
CLI first. Scoping down to just a command line tool helped us focus on the core of a coding agent.
先做 CLI。將範圍縮小到僅限命令列工具，幫助我們專注於編碼代理的核心。Inject more context. Immediately reading a dozen files related to the user prompt gave a huge advantage over competitors.
注入更多上下文。立即讀取與使用者提示相關的十幾個檔案，讓我們在競爭者中獲得巨大優勢。No permissions checks. We were full YOLO mode from the very beginning which was positively heretical then.
沒有權限檢查。我們從一開始就是全然 YOLO 模式，當時這在業界可說是異端。Premium tool. It makes sense to spend more when developer salaries are the alternative.
高級工具。當開發者薪資是替代方案時，花更多錢是合理的。Knowledge files. We came up with the idea of knowledge.md files that are checked in to your codebase. Codebuff would automatically update these files as it learned.
知識檔案。我們想出了 knowledge.md 檔案的概念，這些檔案會被加入你的程式碼庫。Codebuff 會在學習過程中自動更新這些檔案。
Most of these are standard or becoming standard in coding agents today!
這些大多是現在編碼代理中標準或正在成為標準的功能！
What didn’t work out
無法達成的部分
For the first 10 months, we always thought we were weeks away from breaking out and growing exponentially. During YC, we even did grow exponentially, to $5k MRR.
在最初的十個月裡，我們一直認為自己只差幾週就能突破並實現指數成長。在 YC 期間，我們甚至真的實現了指數成長，達到每月經常性收入 5,000 美元。
We regularly got people saying it was the best coding agent. But, it wasn’t always as reliable.
Our file editing strategy was flaky for months, much worse than Cursor’s with its custom model to rewrite a file.
Even after we adopted Relace’s fast rewriter model, our product still had a long tail of issues that made ~5-10% of tasks fail. Some of these issues just take time to isolate and fix, but we could have prioritized better.
Without reliability, we could not have high retention. Without high retention, Codebuff could not grow.
What we should have done
Here’s what I’d do differently after an extensive retrospective.
Build end-to-end evals and run them nightly
This would get us regular quantified feedback on how Codebuff performs as a coding agent. It would help solve reliability issues AND allow us to test hypotheses on how to further improve our product.
Because we did not have this, we spent way too much time manually testing Codebuff after every change or when evaluating whether to switch models.
Cut every feature that is not core
We thought we scoped down a lot by sticking to the CLI, but we should have cut even more. Elon Musk was right when he said you must first “delete the part!”.
Here are a few features we should have cut earlier:
Magic detection of whether the input is supposed to be a terminal command or prompt
Automatic knowledge file updates, which we tweaked for months before largely scrapping
A pseudo-terminal library (node-pty) for color output & aliases, which was recently named our biggest blackhole feature ever
Get the whole team improving the core product
I took on too much of the core system and left my cofounder to deal with other tasks which may not have been as impactful. It helps focus and morale to get all hands in the game.
Live in the future
Never stop thinking about how to disrupt your current product. What is the next thing? What experiments can we try today to make it work?
Monthly retrospectives
One bit of process that could have helped us achieve the above is monthly retrospective meetings. Schedule these on your calendar and set aside an hour for everyone to answer these questions and discuss them:
What should we double down on?
What should we cut?
What should we explore next?
Next steps for Codebuff
In the last couple months, we’ve done more reflection and exploration as competitors such as Claude Code have entered the market with similar ideas.
(Incidentally, I believe Claude Code succeeded in part by having a more focused bet: client-side only, search-replace file editing only, agentic-RAG only.)
We’ve been dreaming of the next thing, and now I’m confident we know what it is.
Our new multi-agent product is live!
I’m happy to say that, as of two days ago, we’ve soft-launched our multi-agent architecture, where agents spawn other agents with different roles.
The reception so far has been overwhelmingly positive even though this is the very beginning. My cofounder says we’re just scratching the surface of what is possible in this framework: “it feels like an infinite world of possibilities,” he says.
I agree — check it out! And stay tuned for a bigger launch soon!
Predictions for the next year
If we got so many things right about what was coming for coding agents last year, can we do it again? I think so!
Here are my forecasts:
The multi-agent paradigm will win. Our experience is that it’s possible to rapidly improve capabilities by delegating tasks to specialized agents.
“Live learning” will be standard. Having the coding agent learn as it does tasks is extremely powerful.
Coding agents will flip the initiative. We’ll see a shift from the user always initiating prompts, to the coding agent more often coming up with tasks for the user, e.g. to review key decisions.
Coding agents will close the loop. Instead of just proposing code changes, they will also use the product itself to perform QA and evals, and commit the changes autonomously.
Recursively improving coding agents will start working. And all the top coding agents will be a flavor of this.
xAI will gain a sizable lead. The multi-polar era will recede as xAI gains a decisive lead in model quality and intelligence.
The best model will not matter as much as today. Instead, it will be the network of agents that distinguishes the best product.
It’s been a blast
Thanks for reading, and cheers to another year of:
Big ideas, grinding, new employees, office snacks, customers that want to acquire us, offsites in Tokyo, afternoon breaks for running or basketball, and late night coding sessions.
May the best coding agent win!
James
P.S. Come help us build the world’s best coding agent!
You can join as a founding engineer and possibly have a stake in the first 10 trillion dollar startup once agents rule the world. Email james@codebuff.com. We also offer referral bonuses!
