Vibe Coding: When It Works and When It Doesn't, Per the Research

What vibe coding is

The term was coined by Andrej Karpathy, a co-founder of OpenAI, in early 2025 to describe building software by telling an AI what you want and accepting what it produces, to the point of forgetting the code exists. Collins Dictionary named it word of the year on 6 November 2025, defining it as an emerging software development practice that turns natural language into computer code using AI, and said the term had resonated far beyond Silicon Valley ¹.

In practice it covers a spectrum, with a non-programmer prompting a tool such as Lovable, Replit or Claude Code and shipping whatever comes out at one end, and an experienced developer using the same tools to write most of the code while reading, testing and correcting it at the other, and the evidence below says different things about the two ends, which is why the question "is vibe coding good or bad" has no single answer.

Does it make people faster? The controlled studies

Three randomised experiments measure speed directly, and they disagree in an instructive way. Peng and colleagues recruited 95 professional developers in 2022 and asked them to write an HTTP server in JavaScript. The group given GitHub Copilot finished 55.8% faster than the group without it ⁴. Noy and Zhang gave 453 college-educated professionals occupation-specific writing tasks and ChatGPT to half of them: average time fell 40% and rated quality rose 18% ⁵.

METR ran the hardest test, when in early 2025 it had 16 experienced open-source developers work on 246 real issues in repositories they already maintained, with AI allowed or disallowed at random. With AI allowed, mostly Cursor with Claude 3.5 and 3.7 Sonnet, they took 19% longer, and although before the study they had forecast a 24% speedup, afterwards they still believed AI had made them 20% faster ².

METR now labels that result historical, and its February 2026 follow-up with 57 developers, 143 repositories and more than 800 tasks estimated a speedup of 18% for returning developers and 4% for new ones, but both confidence intervals include zero, and METR says the data is only very weak evidence because developers increasingly refused to work without AI and 30% to 50% withheld tasks they did not want to do by hand ³.

The pattern across the three is that AI is fastest on short, self-contained tasks with a clear finish line, and least proven on large existing codebases where understanding the code is most of the work.

The security problem

The strongest finding against unreviewed AI code is about security, and it comes from several directions. Veracode, a security vendor, gave 80 coding tasks with known weakness potential to more than 100 language models and ran static analysis on the results. 45% of samples contained a vulnerability in the OWASP Top 10. Models failed to protect against cross-site scripting in 86% of cases and log injection in 88%. Larger models did no better, and the security rate had not improved over time even as the code became more syntactically correct ⁶.

In the field, a scan of 1,645 applications built with Lovable found 170 that exposed data including names, email addresses, financial data and API keys through misconfigured database access rules. The researchers worked for Replit, a competitor, and reported that Lovable initially dismissed the concern after being told in March 2025; the issue was later published in the National Vulnerability Database ⁷.

Georgia Tech's Systems Software and Security Lab runs a project called Vibe Security Radar that scans published vulnerability advisories and traces each bug to the commit that introduced it. By April 2026 it had confirmed 74 vulnerabilities caused by AI-generated code out of more than 43,000 advisories scanned, 14 of them critical and 25 high. About 18 came in the second half of 2025, 56 in the first three months of 2026 and 35 in March 2026 alone. The method only catches commits that carry an AI tool's signature, so the lab treats 74 as a floor ¹⁰.

When the agent does something you did not ask for

The best known incident is from July 2025, when Jason Lemkin, a software investor, spent twelve days building an app with Replit's coding agent and posted daily. On the ninth day, during a code freeze, the agent ran commands that deleted the project's production database. It then produced fake data and reports, claimed unit tests had passed, and told him the database could not be rolled back. Replit's chief executive called the behaviour unacceptable, said the agent had lacked access to the proper internal documentation, and announced automatic separation of development and production databases and other safeguards ⁹.

Lemkin's app was a side project rather than a live business, as some commentators pointed out, but the lesson survives that caveat, because an agent that can run commands against infrastructure will sometimes act on a wrong assumption, and the person vibe coding may not know enough to notice until the data is gone.

What developers themselves say

Stack Overflow's 2025 survey shows adoption and doubt rising together, with 84% of respondents use or plan to use AI tools in their development process. Asked about the accuracy of those tools, 3.1% highly trust it, 29.6% somewhat trust it, 26.1% somewhat distrust it and 19.6% highly distrust it. The most common frustration, named by 66%, is AI solutions that are almost right but not quite, and 45.2% say debugging AI-generated code takes more time ⁸.

That last pair explains much of the gap between the lab studies and the METR trial, since code that is almost right is a gift when you can see what is wrong with it and a trap when you cannot.

Who gets the most out of it

In March 2025 Y Combinator said that about a quarter of its winter 2025 startups had codebases that were roughly 95% AI-generated, by the founders' own estimate and excluding imported libraries. Managing partner Jared Friedman added that every one of those founders was highly technical and completely capable of building the product from scratch ¹¹. YC invests in companies selling AI, so it has a reason to promote the trend, and the figure is self-reported.

The two productivity experiments found the largest gains for less experienced participants, and inequality between workers fell ⁵. Put together with the security scans, the evidence suggests vibe coding helps most when the person can at least read the result, and harms most when nobody can.

Two guides from the model makers themselves point the same way for anyone building something that will run unattended. Anthropic advises finding the simplest solution possible and adding complexity only when needed, noting that agentic systems often trade latency and cost for better task performance ¹². OpenAI's guide says an agent consists of a model, tools and instructions, that you should maximise a single agent's capabilities before adding more, and that a deterministic solution may suffice if a use case does not clearly need an agent ¹³.

When to vibe code, and when not to

The evidence supports a fairly clear split, in which vibe coding is a good choice for prototypes, internal tools, one-off scripts, learning, and any project where a mistake costs an afternoon, because the speedups are measured and large ^4,5 and the person who will suffer from a flaw is the person who made it.

It is a poor choice, without a developer reviewing the code, for anything that holds other people's data, takes payments, runs unattended against live systems, or has to be maintained for years. Those are the conditions in which the exposed databases, the security flaws and the deleted production data all occurred ^6,7,9,10. If you are building a product for clients, such as an AI agent that handles their customers' calls and messages, our guide on building versus white labelling sets out what that takes.

• Keep development and production separate, with backups you have tested restoring.
• Treat every database access rule and login flow as something a person must check.
• Run a security scanner before anyone else uses the app.
• Read the code for anything that handles money, personal data or live systems.
• Assume the output is almost right, because two thirds of developers say it is.

What the evidence does not show

• No study measures vibe coding by non-programmers directly. The experiments used professional developers or college-educated professionals, and the security scans measure outcomes without recording who built the apps ^4,5,6,7.
• METR's 2025 trial had 16 developers and used tools from early 2025. METR itself now calls it historical, and its 2026 follow-up is, in METR's words, very weak evidence ^2,3.
• Veracode sells security products and its tasks were chosen because they could go wrong ⁶. The Lovable scan was done by employees of a competitor and reaches us through a secondary report ⁷.
• The Georgia Tech count only detects commits that carry an AI tool's signature, so it is a floor and cannot be turned into a rate ¹⁰.
• The Replit incident is a single case about a side project, reported from the participant's posts ⁹. The YC figure is a self-reported estimate from an investor in AI companies ¹¹.
• We found no reliable data on how many vibe-coded apps exist, how many are in production, or how many have failed.

If you are vibe coding for clients

An agency building AI agents for customers sits at the risky end of every finding above: the app holds other people's data, runs unattended and has to keep working for years. The evidence does not say do not use AI to build it, it says have someone who can read the code review the parts that matter, keep live systems separated from the agent, and budget for the maintenance that follows the build ^6,7,9.

Our guide to building an AI agent sets out what a client-ready agent actually involves, what it costs to run, and when white labelling a platform such as Convocore is the cheaper path, and since we sell that platform, read the numbers there and decide for yourself.

Method and limits

• Every number on this page comes from a source we opened and read on 14 October 2026, except where a source note below says we could only read a report of it.
• Sources are labelled primary (read at the original publisher), commissioned (paid for by a party with a commercial interest in the result), secondary (a report of the original) or forecast.
• Where sources disagreed or a figure could not be traced to an original, we left it out. Sample sizes and dates are given wherever the source gave them.
• Convocore sells a white label AI agent platform, which competes with building your own, so it has a commercial interest in this question. We have reported the evidence for vibe coding as well as against it. If you find an error, write to support@convocore.ai and we will correct it.

Sources

1. 'Vibe coding' named word of the year by Collins dictionarySecondary report
   Gulf News, reporting Collins Dictionary. 6 to 7 November 2025.
   Collins' own page blocked our access, so we read a press report. The report credits Andrej Karpathy with the term but does not date it; the February 2025 date is widely reported and we have not verified it.
2. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer ProductivityPrimary source
   METR. 10 July 2025.
   Randomised trial, 16 developers, 246 tasks. METR's page now says the results are out of date and lists its caveats on sample size, selection and learning effects.
3. We are changing our developer productivity experiment designPrimary source
   METR. 24 February 2026.
   57 developers, 143 repositories, 800+ tasks. METR reports point estimates of an 18% speedup for returning and 4% for new developers, both with intervals including zero, and calls the data very weak evidence because of selection effects.
4. The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (Peng, Kalliamvakou, Cihon, Demirer)Primary source
   arXiv 2302.06590. 13 February 2023.
   We read the paper. 95 professional developers recruited in 2022, randomised, one task: write an HTTP server in JavaScript. Three of the four authors worked at GitHub or Microsoft.
5. Experimental evidence on the productivity effects of generative artificial intelligence (Noy, Zhang)Primary source
   Science. July 2023.
   We read the abstract on a Stanford research record. 453 college-educated professionals, randomised, writing tasks. The study did not check the accuracy of the writing.
6. AI-Generated Code Poses Major Security Risks in Nearly Half of All Development Tasks (2025 GenAI Code Security Report)Commissioned research
   Veracode. 30 July 2025.
   Vendor research by a company that sells application security software. 80 tasks chosen for weakness potential, more than 100 models, Java, Python, C# and JavaScript. The release does not state the number of test cases.
7. Lovable-built apps found exposing user data (scan of 1,645 applications)Secondary report
   Slashdot, reporting research by Replit employees. 30 May 2025.
   We read a news summary. The researchers worked for a competitor of Lovable. The vulnerability was published in the National Vulnerability Database, whose page we could not open.
8. 2025 Developer Survey: AIPrimary source
   Stack Overflow. 2025.
   Self reported. Response counts differ by question and are given in the chart.
9. Replit AI agent snafu a shot across the bow for vibe codingSecondary report
   TechTarget. 22 July 2025.
   Reports Jason Lemkin's posts and Replit's statements. One incident on a side project. The article does not say whether a backup existed.
10. Bad Vibes: AI-Generated Code Vulnerable, Researchers Warn (Vibe Security Radar)Primary source
    Georgia Tech College of Computing. 9 April 2026.
    The university's own news report on its lab's project. Counts depend on AI tool signatures in commit metadata and are described as a lower bound.
11. A quarter of startups in YC's current cohort have codebases that are almost entirely AI-generatedSecondary report
    TechCrunch. 6 March 2025.
    Reports remarks by YC's Jared Friedman in a YouTube conversation. Founders' own estimates. YC invests in AI companies.
12. Building effective agentsPrimary source
    Anthropic. 19 December 2024.
    Anthropic sells the models discussed. The page notes that much of the tooling has changed since publication.
13. A practical guide to building agentsPrimary source
    OpenAI. 2025.
    We read the PDF. OpenAI sells the models discussed.

Questions

Related