Boris Cherny, an engineer close to the Claude ecosystem, attempted to use Claude Code to rewrite the Claude app's own codebase. The one-line verdict: it works better than you expect on isolated tasks, and worse than you need on anything requiring systemic judgment.
The setup
Cherny pointed Claude Code at the Claude app's own source, essentially asking the model to reason about and improve the codebase it powers. This is a high-signal stress test because the target is not a toy repo. It is a production application with real architectural decisions, dependencies, and implicit conventions. The task sits at the intersection of two hard problems: understanding existing intent and generating net-new structure that respects it.
This kind of self-referential experiment matters beyond the novelty. It is a proxy for what happens when you drop a coding agent onto any mature internal codebase, which is the exact scenario most engineering teams are now navigating.
What worked
Claude Code handled bounded, well-scoped refactors cleanly. When the task was something like "extract this logic into a utility function" or "rename and consolidate these types," the agent produced correct, mergeable output. It also showed competence at reading unfamiliar code and generating plausible explanations of what a module does, which is useful for onboarding-style tasks.
The model's ability to follow implicit style conventions within a file was notably strong. Given enough local context, it matched patterns without being told to.
Where it struggled
The failure modes clustered around cross-file architectural reasoning. When a rewrite required understanding how a decision in one module propagates to three others, the agent made locally reasonable choices that were globally inconsistent. It did not hallucinate wildly; it made the kind of plausible-but-wrong calls a new junior engineer makes when they do not yet have the full mental model.
A second failure mode: the agent did not know what it did not know. It would proceed confidently on tasks that required implicit product context (why a certain tradeoff was made, what a legacy pattern is protecting against) rather than pausing to ask. This is the classic agentic overconfidence problem, and it surfaces hard on any codebase with history.
This connects to a broader evaluation problem flagged separately in the community: standard benchmarks and LLM-judge metrics miss the distortions that show up in aggregate scoring, meaning your eval suite may not catch this class of failure until it hits production.
The verdict
Use Claude Code on mature codebases, but constrain the blast radius. Teams shipping agentic coding features should:
- Scope prompts to single modules or clearly bounded subsystems, not whole-app rewrites.
- Build a pre-execution review step, either human or a secondary agent, before merging agent output. Tools like Wolfpack-style control interfaces and company-policy agent checks before action exist precisely for this reason and are worth evaluating.
- Instrument your evals to catch cross-file consistency failures, not just per-file correctness. A test suite that only validates individual functions will miss the category of error that matters most here.
Who should run this kind of experiment now: teams building internal developer tools, platform engineers evaluating agentic IDE integrations, and anyone writing prompts that hand an agent a large unfamiliar repo. Who should wait: teams without a review layer in place. The agent will ship confident, wrong architectural decisions without one.
The hardest part of agentic coding is not generating correct code, it is knowing which decisions require a human in the loop.
Scope your agent tasks like you scope a junior engineer's first PR: small, reviewable, and with explicit guardrails on what they are not allowed to touch alone.
READY TO ASCEND
Get AI news that respects your time
The signal, distilled. Curated AI news and prompt-engineering insight. No noise.