AutoFyn

Run Claude in self-improving loops to optimize measurable goals.

built the #1 Spider 2.0 DBT agent · found 197 vulnerabilities across popular OSS · improved Caveman compression from 44% to 54%

Getting Started · CLI · Remote Sandboxes · Config · Custom Subagents · FAQ

Give it a repo, a goal, and a time limit. Walk away. Come back to a PR.

Each round runs Claude in a sandboxed Docker container with fresh context. A persistent run state tracks the goal, eval history, and learned rules across rounds — the agent measures progress, learns from failures, and improves instead of degrading.

Results

Security audits

• Next.js — 8 vulnerabilities (1 High, 4 Medium, 3 Low). Responsibly disclosed via HackerOne. CVEs
• pnpm — 7 vulnerabilities (1 High, 5 Medium, 1 Low), 3 exploit chains. Responsibly disclosed. CVEs
• MetaMask Extension — 12 vulnerabilities (3 High, 7 Medium, 2 Low), 3 exploit chains. Responsibly disclosed via HackerOne. CVEs
• Warp — 30 vulnerabilities (6 Critical, 7 High, 8 Medium, 9 Low), 3 exploit chains. Responsibly disclosed. CVEs
• Langflow — 22 vulnerabilities (3 Critical, 13 High, 6 Medium), 4 exploit chains. Responsibly disclosed. CVEs
• Hermes Agent — 36 vulnerabilities (13 Critical, 22 High, 1 Medium), 18 exploit chains. Responsibly disclosed. CVEs
• Agent TARS — 25 vulnerabilities (4 Critical, 18 High, 3 Medium), 20 exploit chains. Responsibly disclosed. CVEs
• RAGFlow — 17 vulnerabilities (5 Critical, 11 High, 1 Medium), 5 exploit chains. Responsibly disclosed. CVEs
• LiteLLM — 14 vulnerabilities (3 Critical, 4 High, 4 Medium, 3 Low), 2 exploit chains. Responsibly disclosed. CVEs · Report
• Open WebUI — 12 vulnerabilities (4 Critical, 5 High, 3 Medium), 4 exploit chains. Responsibly disclosed. CVEs · Report
• Twenty — 2 vulnerabilities (1 High, 1 Low), 1 exploit chain. Responsibly disclosed. CVEs
• Supermemory — 4 vulnerabilities (1 High, 3 Medium), 4 exploit chains. Responsibly disclosed via email to Supermemory.
• Phantom Connect SDK — 8 vulnerabilities (5 Medium, 3 Low), 1 exploit chain. Responsibly disclosed. CVEs

Software engineering

• SignalPilot — built a data analysis agent from scratch, #1 on the Spider 2.0 dbt benchmark.
• Caveman — optimized the prompt compression skill by +10% without quality loss (write-up).

Quick start

git clone https://github.com/SignalPilot-Labs/AutoFyn.git ~/.autofyn
pip install ~/.autofyn/cli
autofyn update && autofyn start

If your agent needs docker access, run

autofyn start --allow-docker

Two release channels:

• autofyn update --branch production — stable (recommended)
• autofyn update --branch main — nightly (latest features)

Open localhost:3400 for the dashboard. AutoFyn auto-detects your Claude token, GitHub token, and repo from your local git remote.

Pick a starter preset — Security hardening, Bug sweep, Code quality, or Test coverage — or write your own goal:

autofyn run new -p "Optimize the algorithm to hit 60% compression ratio without further quality loss" -d 120

To configure manually:

autofyn settings set --claude-token YOUR_KEY --git-token YOUR_TOKEN --github-repo owner/repo

How it works

LLM agents that run in a loop hit three failure modes:

• Context rot — context grows until the model loses track.
• No learning — mistakes repeat because nothing carries between iterations.
• No compass — the agent can't tell whether it's making progress or going in circles.

AutoFyn's round loop addresses each one by simulating an expert iteration in LLM context space, not the weight space. Each round the LLM proposes a plan and a build, and an expert grades the raw proposal; the verified outcome and the learnings are distilled into context with a persistent run_state.md, and subagent specific memory files. Unlike AlphaZero, the expert isn't a search: it's a metric that reviewer subagents run, such as a test suite or a benchmarking script. And it's objective — a test suite passes or it doesn't, a live exploit fires or it doesn't, the benchmark score moves or it doesn't — so the LLM can't talk itself into false progress or spin in a loop. The key components of the system are:

• State, not context. Each round gets a clean context window. Cross-round knowledge lives in /tmp/memory/ — run_state.md (goal, eval history, rules) and per-subagent rule files. Context never degrades because it never accumulates.
• Objective reward signal. Every round ends with a real eval: run the benchmark, execute the exploit, check the test suite. The signal is sparse and binary — a bound improves or it doesn't, an exploit fires or it doesn't — but it's grounded, not a model's opinion, and that's what the loop optimizes against. The result is appended to eval history so the orchestrator can track progress across rounds.
• Policy updates from failures. Reviewer findings and repeated mistakes become persistent Rules: ALWAYS: run migrations before tests (because round 4 broke prod, round 4). Global rules are injected into every subagent. Per-subagent rules (e.g. architect.md, code-reviewer.md) let each subagent accumulate domain-specific knowledge across rounds.
• Honest feedback loop. Reviewers are independent. A round that improves the metric but violates a constraint is rejected. The agent corrects course instead of reinforcing bad decisions.
• Time-locked episodes. end_session is denied until the budget expires. It iterates toward the target for the full duration.
• Custom subagents. A repo can drop a .autofyn/subagents.json to bring its own subagents — adding new ones or overriding the shipped team by name — to tailor the pipeline to a domain (ML research, formal proofs, data work). See Custom Subagents.

CLI reference

# Services
autofyn start                          # start services
autofyn start --allow-docker           # start with Docker access for sandbox
autofyn stop                           # stop all services
autofyn update                         # pull latest code + images
autofyn update --branch main           # switch to nightly channel
autofyn update --image-tag abc1234     # pin to a specific version
autofyn update --build                 # force local build (for dev)
autofyn logs                           # stream container logs
autofyn kill                           # remove all containers
autofyn uninstall                      # remove everything (containers, images, ~/.autofyn)

# Runs
autofyn run                            # interactive run selector
autofyn run new -p "Fix auth bugs"     # start a new run
autofyn run list                       # list recent runs
autofyn run get <run_id>               # run details + action menu

# Settings
autofyn settings status                # check config
autofyn settings get                   # show all settings
autofyn settings set --claude-token TOKEN --git-token TOKEN --github-repo owner/repo

# Repos
autofyn repos list                     # list repos
autofyn repos set-active owner/repo    # set active repo

Use --json on any command for machine-readable output.

Remote sandboxes

Runs can execute on remote machines (HPC clusters, GPU servers) instead of local Docker. AutoFyn SSH-tunnels to the remote, streams logs back, and manages the lifecycle automatically.

See docs/user/remote-sandboxes.md for setup, start command examples, GPU access, and troubleshooting.

Responsible disclosure

All vulnerabilities were privately disclosed to maintainers before any public mention. Full reports are withheld until patches are available.

Built with the Claude Agent SDK. Apache 2.0 License.