VinvAI

Vinv runs, benchmarks and optimizes your agent-written Python code until it's production-ready.

Coding agents know your code. They have never understood how it behaves when it runs. Vinv's context bandits build one context graph — your code, your traces, and the metrics derived from them — and serve it to your agent.

_{Judgement comes from what the code actually did, not from what the agent claims.}

_{One command starts it. Vinv drives the other eight stages — every arrow is evidence, not a guess.}

Install: Open VSX · one-click, pick your editor

Or straight from your editor's CLI:

_{First run builds the engines (~4 min: compiles the Rust index, fetches a one-time ~500 MB local embedding model). Needs uv and Rust. First trace about a minute after that.}

git clone https://github.com/VinvAI/VinvAI ~/.vinv/engines && cd ~/.vinv/engines && ./install.sh

_{The whole loop on Vinv's own repo: install → discover → trace → catch a real bug → dispatch → verified fix, zero clicks.}

The problem

84% of developers now use or plan to use AI coding tools. More of them actively distrust the output (46%) than trust it (33%) — and distrust nearly doubled in a year (Stack Overflow 2025, 49k developers). You know why: the agent edits the wrong handler, invents return shapes, then grades its own homework while the server won't even start.

Or it enters the doom loop — test fails, agent edits the same function, test fails the same way, agent edits it again, burning your context window on "let me verify." Anthropic's own research documents agents "stuck in loops, repeating the same failed approach" when they lack codebase context.

Both failures have one root cause: the agent has never watched your code run. It argues from static text.

Case study: commodity models out-fix frontier ones

Vinv found four bugs and one performance problem in fastapi/full-stack-fastapi-template (44k★). We handed all five to each setup — same issues, same prompts, one trial per condition, Vinv grading every run:

This is a demonstration, not a benchmark — five issues, one repo, one trial per condition. We're publishing it because it's checkable, not because n=5 settles anything.

The claim isn't a model ranking — blind, the commodity model scored zero. The claim is that a model holding the failing frame, the caller chain, and the real argument values beats a stronger model guessing from static code. The evidence is what moved, not the weights.

Why the pass rate means something: Vinv grades, and it doesn't take the agent's word. Acceptance tests are written before the fix and the agent never sees them. Any change that alters observable output is reverted automatically, even when it's faster: the behavior replay has to come back byte-identical, and the paired-bootstrap 95% CI on a speedup has to exclude zero.

And the loop keeps finding real ones. On the same pristine template, the optimization loop later surfaced — and statistically proved — a fix nobody planted: the app's default database pool (SQLAlchemy's 5+10) makes requests queue for connection checkouts under concurrent load, so a 7-row indexed lookup measured 22× the typical symbol's cost. Pool sized to the worker concurrency: sustained-load median 75.6ms → 41.2ms — 45.4% faster, 95% CI [36.3%, 45.8%] — responses byte-identical. The same engine auto-reverted two earlier attempts whose measurement windows couldn't certify the win; the accept only landed when the evidence did.

If any of these is your open tab

What Vinv does

Give your coding agent runtime context — ten capabilities, one loop:

• Semantic code search — ask by meaning, get ranked symbols with def bodies and line numbers, embedded by a local model (no cloud keys).
  
• Code Graph — a persistent map of every symbol and call edge, updated incrementally on save, with a live runtime overlay.
  
• Runtime tracing — zero-edit runtime tracing for AI coding agents: timing, memory, args, returns, errors — per call, joined to source.
  
• Rank suspects — on any failure, symbols ranked by fault-localization score over real pass/fail requests, error messages attached.
  
• Verified fixes — verify AI-generated code actually works: replayed start, live port, acceptance tests the agent never sees. One click reverts everything an episode touched.
  
• Ask Vinv — ask anything about your running system in plain English; every answer cites the exact trace spans and source lines it came from, and a deterministic critic blocks any claim the evidence can't back — grounded Q&A, not confident guessing.
• Behavior exerciser (new) — Vinv doesn't wait for traffic: it drives every endpoint itself — schema-derived valid/boundary/negative inputs, values mined from real traces, multi-step auth scenarios — picks strategies with a Thompson-sampling bandit rewarded by oracle violations first and new coverage only as a bonus, and turns every response into a permanent regression case.
• Journey (new) — one walkthrough of everything verified: every service, then every endpoint's call tree, latency flamegraph, and the exact inputs → outputs exercised — with a form to add your own test inputs that the engine replays forever after.
  
• Auto-Pilot & the red ring — one click drives discover → set up → trace → exercise → fix → verify until green or budget; when new trace errors land, the fix episode is already dispatched by the time you see the red ring in the graph. The budget is yours: set attempts per service in Configure, and when a run exhausts them Vinv asks whether to grant more instead of quietly giving up.
• Agent babysitting — a doom-loop guard (token-set self-similarity) catches a repeating agent, an adaptive silence watchdog catches a hung one, and "Dispute a Verified Fix" keeps even the verifier accountable.
• Findings (new) — what Vinv found and what it fixed, with the statistical evidence: issue clusters, optimization episodes with paired-bootstrap confidence intervals, regression diff kinds, and a machine-readable findings.json your agent can consume directly.
  

Honest scope: Python backends first — other stacks get the index, graph, and QnA, but no runtime evidence yet (TS & Go next).

Why agents don't reward-hack under Vinv

Vinv ties every runtime trace to the exact code segment that produced it and hands your agent a context graph built from that join — so the agent argues from evidence, not vibes. And when the agent claims victory, Vinv doesn't take its word:

• Acceptance tests are authored before the fix and never shown to the agent — it can't train to the test.
• A "faster" fix that changes any observable output is auto-reverted — the behavior suite must replay byte-identical, and the speedup's paired-bootstrap 95% CI must exclude zero. Faster-but-wrong never lands.
• Deliberate 4xx rejections aren't "errors" to fix — the defect classifier knows the difference between a service saying no correctly and a service breaking, so the agent is never handed a fake goal it can only game.
• When two attempts stop making progress, a Nash-bargaining stall judge decides — continue only if both an explorer stance and an auditor stance strictly prefer it to asking you. Otherwise you get a judgment panel, not a token bonfire.

The same run, in detail

Everything above came from one all-local pass on that template, on an M-series MacBook:

• Indexed 855 symbols across 151 files with 516 call edges in 27.6s — cold, from clone.
• Semantic search: 5/6 natural questions hit the right symbol in the top 5, p50 64ms:

• The backend then ran under Vinv's zero-edit tracer inside Cursor desktop, extension live — no code changes to the template.

The actual run, captured frame by frame: install → 855-symbol graph → search hits → trace hotspots → the failing frame named → verified.

Then we ran its backend under Vinv's zero-edit tracer (inside Cursor desktop, DB deliberately down) and hit it with real traffic. From one run, Vinv produced:

Your agent sees "500". Vinv hands it the exact failing function, the error type, and the chain that led there — before it opens a single file.

Bonus: this very demo caught a real Vinv bug (Python 3.14 broke OTel's contrib loader; the error was being swallowed). We fixed it the same day — that's the loop working on ourselves.

Then we let the exerciser loose on the same template

Traffic only shows you the code paths users happen to hit. The behavior exerciser drives the rest — same repo, same laptop, one run:

The authenticated sweep (every endpoint replayed under credentials the login scenario captured, with freshly created resource IDs fed to the by-id endpoints) surfaced four real bugs that anonymous traffic can never reach:

1. GET /api/v1/users/ → HTTP 500 — an invalid email stored by an unvalidated private endpoint poisons response serialization
2. POST /api/v1/private/users/ → IntegrityError escapes as a 500 — email: str instead of EmailStr, no duplicate guard
3. POST /api/v1/utils/test-email/ → HTTP 500 — assert settings.emails_enabled crashes instead of degrading
4. POST /api/v1/password-recovery-html-content/{email} → connection killed — unsanitized header rendering

The harness then fixed all four, and the regression suite now distinguishes your code regressed from the test engine's own leftover data changed the world (the state ledger) — so a re-run doesn't cry wolf. Phantom perf regressions are gone too: a latency diff must survive a median of 5 replays before it's reported.

_{One endpoint after the run: call tree with live runtime, latency flamegraph, and every input Vinv drove with the output it got back.}

Works with your agent

Vinv is an MCP server for Claude Code and Cursor — and every other MCP client you already use. One command (Register Vinv MCP in Agent Tools) writes both servers into every agent it detects:

Agent without Vinv vs with Vinv

Proven on itself

Vinv's release gate is Vinv — these numbers come from running the loop on this repository:

How it works

flowchart LR
  T[Trace] --> I[Index] --> S[Serve MCP] --> V[Verify] --> L[Learn] --> T

1. Trace — run your Python service under the bundled tracer: no SDK, no code changes.
2. Index — every function embedded locally into a semantic index + call graph.
3. Serve — two MCP servers hand the evidence to your agent.
4. Verify — replayed start, live port, acceptance tests generated before the fix.
5. Learn — propensity-logged decisions; retrieval updates only on off-policy-evaluation wins.

After install: the five things to try

1. Exercise your API — exerciser plan <repo> && exerciser run <repo> --base-url http://127.0.0.1:PORT (or let Auto-Pilot's exercise phase do it). An environment canary first dry-runs your login chains and tells you loudly if the database was reset or credentials unseeded — no more silently-401 runs.
2. Walk everything — Command Palette → "Vinv: Open Journey". Overview first (services, coverage, open issues), then Next/→ through every endpoint: call tree with live runtime, flamegraph, and the exact inputs → outputs driven. Hover anything cryptic — every marker explains itself in plain language.
3. Add your own test input — on any Journey endpoint step, fill body/params/expected status and hit Add input. It lands in the same plan layer the AI-authored scenarios use, runs with the endpoint's auth setup on the next exercise, and becomes a permanent regression case.
4. See what got fixed — Command Palette → "Vinv: Open Findings": issue clusters, optimization episodes with their confidence intervals, regression diffs by kind, latency profile, cleanup ledger. The tab's backing file .vinv/reports/findings.json is the same data, machine-readable — point your agent at it.
5. Regress after any change — exerciser regress <repo> --base-url … replays all banked cases (re-capturing fresh credentials itself) and reports behavior / contract / perf / environment diffs separately, so environment drift never masquerades as a code regression.
6. Hunt waste on demand — "Vinv: Optimize Latency Hotspots", "Analyze Memory Trends" (Theil–Sen leak suspects), and "Analyze Cache Opportunities" (recomputed-work finder) each turn one command into an evidence-seeded fix episode — accepted only if the paired-bootstrap CI clears and behavior stays byte-identical.

MCP tools reference

FAQ

Privacy

• Everything on your machine — per-repo state in .vinv/ (auto-gitignored), per-machine in ~/.vinv/. No account, no API keys, no telemetry — none.
• The only download is the embedding model (Hugging Face, once, ~500 MB); everything else builds from this repo.
• Traces store bounded summaries, not raw values; sensitive parameter names (password, token, api_key, …) are redacted, never captured.
• The only LLM Vinv talks to is the coding-agent CLI you configured, through its own auth.

Contributing & license

See CONTRIBUTING.md — uv sync, cargo build in index/, npm install && npm run check in extension/, keep tests/e2e/planted_bug_golden/run.py green. Good first issues are labeled. By taking part you agree to our Code of Conduct; to report a vulnerability, see SECURITY.md. Apache License 2.0 © 2026 VinvAI.