Flock

Self-hosted AI for your team. One endpoint. Your hardware.

flockllm.com · GitHub · Maintained by Hadi Honarvar Nazari · Apache-2.0

Flock is the self-hosted control plane for LLMs. One Go binary turns your Macs and Linux boxes into a private inference cluster — multi-machine routing, per-user keys, daily quotas, full audit log, and a built-in admin dashboard, behind one endpoint that speaks both the OpenAI and Anthropic APIs.

Engine-agnostic: bring Ollama, vLLM, MLX-LM, or llama.cpp-RPC. Run open-weight models (Qwen, Llama, DeepSeek, …) on your own hardware, shard a giant model across several machines via llama.cpp-RPC, and transparently fall back to paid Claude / GPT only when you choose.

Point Cursor, Claude Code, Aider, Continue, or any OpenAI/Anthropic SDK at Flock. It just works.

🗺️ Where Flock sits

           ┌──────────────────────────────────────────────────────────────┐
           │                       YOUR USE CASES                         │
           │             (the tools your team already uses)               │
           └──────────────────────────────────────────────────────────────┘
                  │           │          │             │            │
                  ▼           ▼          ▼             ▼            ▼
            ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
            │  Cursor  │ │  Claude  │ │  Aider   │ │  Custom  │ │   curl   │
            │          │ │   Code   │ │          │ │ Python   │ │  scripts │
            │          │ │          │ │          │ │   SDK    │ │          │
            └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
                 │  OpenAI    │ Anthropic  │  OpenAI    │  Either    │  HTTP
                 └────────────┴────────────┴────────────┴────────────┘
                                          │
                                          │   ONE URL · ONE API KEY
                                          ▼
      ╔══════════════════════════════════════════════════════════════════════╗
      ║                  ⬢ ⬢ ⬢   FLOCK   ⬢ ⬢ ⬢                              ║
      ║                  (this is what we built)                             ║
      ║  ════════════════════════════════════════════════════════════════    ║
      ║  Gateway     OpenAI + Anthropic on /v1/chat/completions              ║
      ║              per-user keys · daily quotas · full audit log           ║
      ║              admin dashboard at :8080                                ║
      ║                                                                      ║
      ║  Router      Same model on N nodes  → load-balance                   ║
      ║              Different models per node → route by placement          ║
      ║              Model bigger than any node → split via llama.cpp-RPC    ║
      ║              Claude / GPT requested → proxy to vendor                ║
      ║              Engine error or timeout  → retry catalog fallback chain ║
      ╚═════════════════════════════╤════════════════════════════════════════╝
                                    │
              ┌─────────────────────┼─────────────────────┐
              ▼                     ▼                     ▼
       ┌─────────────┐       ┌─────────────┐       ┌─────────────┐
       │   Engines   │       │   Engines   │       │   Egress    │
       │  (any mix)  │       │  (any mix)  │       │   proxy     │
       │  • Ollama   │       │  • Ollama   │       │             │
       │  • vLLM     │       │  • vLLM     │       │ api.anthro- │
       │  • MLX-LM   │       │  • MLX-LM   │       │ pic.com     │
       │  • llama.cpp│       │  • llama.cpp│       │ api.openai  │
       └──────┬──────┘       └──────┬──────┘       │ .com        │
              │                     │              └──────┬──────┘
              ▼                     ▼                     ▼
      ┌──────────────────────────────────────────────────────────────────────┐
      │                    UNDERLYING LLMs / WEIGHTS                         │
      │                                                                      │
      │   YOUR HARDWARE                              VENDOR APIs             │
      │   • Mac Studio · Mac Mini                    • Claude (Anthropic)    │
      │   • Linux + RTX GPU                          • GPT, o3, o4 (OpenAI)  │
      │                                                                      │
      │   41 curated catalog models (Qwen 3.6, GLM,   Each request routed   │
      │   gpt-oss, Llama 4, Gemma 4, DeepSeek V4,     to EITHER your hard-  │
      │   Kimi K2.6, Nemotron 3 Ultra, vision +       ware OR a vendor —    │
      │   embedding models)                           you pay vendors only  │
      │   + any HuggingFace or Ollama model.          when YOU chose to.    │
      └──────────────────────────────────────────────────────────────────────┘

One-sentence version: Flock is the layer that lets your tools talk to any LLM — open-weight on your hardware, or hosted Claude / GPT — through one URL and one API key, with the team controls (quotas, audit, per-user keys) that the raw vendor APIs don't give you.

🚀 Try it in 60 seconds

Flock is engine-agnostic. The quickest path uses Ollama as the local engine — but vLLM, MLX-LM, and llama.cpp-RPC all work. See Prerequisites — read first below for the alternatives.

🍎 macOS (Apple Silicon — M1/M2/M3/M4)

# 1. install Flock
curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"   # if the installer says so

# 2. install an engine (pick one) — Ollama is the simplest default
brew install --cask ollama && open -a Ollama
# alternatives: pip install mlx-lm  ·  or run llama.cpp's llama-server  ·  or run vLLM in Docker

# 3. start Flock with a tiny model (~1 GB, fast download)
FLOCK_DEFAULT_MODEL=llama-3.2-1b flock up

🐧 Linux (x86_64 or arm64) — including Raspberry Pi, NAS, edge boxes

Option A — .deb / .rpm package (recommended for Debian / Ubuntu / Raspbian / QNAP / Asustor / Fedora / RHEL):

# Debian / Ubuntu / Raspbian (arm64 example — also amd64)
curl -LO https://github.com/hadihonarvar/flock/releases/latest/download/flock_VERSION_linux_arm64.deb
sudo dpkg -i flock_VERSION_linux_arm64.deb
# Binary at /usr/bin/flock, catalog at /usr/share/flock/catalog
# Recommends llama.cpp for sharding — install via apt if you want it.

# Fedora / RHEL / CentOS
sudo rpm -i https://github.com/hadihonarvar/flock/releases/latest/download/flock_VERSION_linux_amd64.rpm

(Replace VERSION with the latest from Releases. The package version stays current via your distro's normal upgrade path — flock update also works as an in-place binary swap for non-package installs.)

Option B — install.sh (works everywhere; drops binary in ~/.local/bin/ and catalog in ~/.flock/catalog/):

# 1. install Flock
curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh | sh
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

# 2. install an engine (pick one) — Ollama is the simplest default
curl -fsSL https://ollama.com/install.sh | sh && sudo systemctl enable --now ollama
# alternatives: vLLM in Docker for NVIDIA  ·  llama.cpp's llama-server  ·  MLX-LM (Apple Silicon only)

# 3. start Flock with a tiny model (~1 GB, fast download)
FLOCK_DEFAULT_MODEL=llama-3.2-1b flock up

💡 Not sure which engine to install? Run flock doctor after step 1 — it inspects your hardware and tells you the single command to run.

What you should see (both platforms)

Flock prints something like:

✔ default model: llama-3.2-1b
✔ engine: ollama at http://127.0.0.1:11434
  Flock is ready.
  API:    http://localhost:8080/v1
  Admin API key:   sk-orc-xK9p…

Every command supports --help — flock <cmd> --help prints usage, flags, and examples.

Copy that admin key. In another terminal:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-orc-xK9p…" \
  -d '{"model":"auto","messages":[{"role":"user","content":"hi in 5 words"}]}'

You should see a JSON response with a 5-word reply. 🎉

Or use the web dashboard: open http://localhost:8080 and paste the admin key.

Or wire up Claude Code: in any terminal where you use Claude Code, set:

export ANTHROPIC_BASE_URL=http://localhost:8080
export ANTHROPIC_AUTH_TOKEN=sk-orc-xK9p…
claude

…and Claude Code talks to your local model instead of paying for the API.

If something breaks, run flock doctor — it tells you exactly what to fix. Common issues are in the Troubleshooting installation section.

What's shipped

See CHANGELOG.md for the full feature inventory, grouped by area (core, CLI ergonomics, multi-node + sharding, routing intelligence, multi-tenancy, observability, web UI, connect snippets, release + ops). For the per-release diff see Releases — every feat: / fix: commit on main cuts a new tag automatically.

For new users: see QUICKSTART.md — 3-minute install + first chat completion. For full usage docs: keep reading this file. For contributors: see ARCHITECTURE.md. For the dev team's roadmap: see TASKS.md.

Why Flock?

AI coding tools are the new dev tax. Cursor, Claude Code, Copilot, custom agents — every team uses them, and the bill grows with usage. A single engineer running modern agentic tools heavily can burn $200–500/month in API tokens. For a team of 10 that's $30–60k a year, and rising. Every request also sends proprietary code to a third party.

There are excellent open-weight models now — Qwen3-Coder, Llama 3.3, DeepSeek-V3 — that match or exceed paid APIs for most coding work. But running them across a few machines, exposing them through one API, routing traffic intelligently, and making it all feel as easy as pip install is not solved.

Flock is the orchestration layer. It does for self-hosted LLMs what Kubernetes did for web services — minus the YAML. One binary. One install command. Auto-discovery. Auto-placement. Drop-in compatibility with every tool you already use.

Design principles

1. One binary, zero dependencies. Static Go executable. No Python, no Docker (unless you want it), no virtualenv. Curl it down and run.
2. Zero config to first response. Smart defaults everywhere. Hardware auto-detected. Model auto-picked. Network auto-meshed.
3. The UI tells you the next step. Every state in the web UI has a clear, copy-pasteable next action. Juniors should never stare at a blank prompt.
4. Heterogeneous is invisible. Mac, NVIDIA, AMD — the user picks models, not hardware.
5. OpenAI- and Anthropic-compatible from day one. Same endpoint serves both protocols.
6. Permissive open source. Apache 2.0. No open-core gotchas.
7. The CLI is the source of truth. Every user-facing capability ships as a flock CLI command first. The web UI is a thin wrapper — it invokes the same Go functions the CLI invokes, never reimplements logic. If you can do it in the UI, you can do it in CI / scripts / SSH sessions, and vice versa.
8. Adding or switching a model is one action. No hand-written YAML, no manual GGUF downloads, no separate worker-side setup. flock model add hf:owner/repo does the rest — picks engine, picks quant, shards if needed, distributes weights, warms the model. The default model is auto-picked from hardware on first flock up; to change it later, set router.default_model in ~/.flock/config.yaml and restart, or FLOCK_DEFAULT_MODEL=<id> flock up.

60-second quick start

On the first machine (becomes the leader)

curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh | sh
flock up

You'll see:

▶ detected darwin/arm64 · 24 GB RAM · 8 cores
✔ default model: qwen-coder-7b
✔ engine: ollama at http://127.0.0.1:11434
▶ pulling qwen-coder-7b · downloading [████████████████████] 4.7/4.7 GB · 85 MB/s · ETA 0:00
✔ model ready: qwen-coder-7b

  Flock is ready.

  Dashboard: http://localhost:8080
  API:    http://localhost:8080/v1
  Key:    sk-orc-xK9p…  (also in UI)

  Add another machine:
    curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh | sh -s -- join flock-7f3a.ts.net?token=…

On any additional machine

curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh | sh -s -- join flock-7f3a.ts.net?token=…

The agent auto-joins the mesh, registers its capabilities, and the leader assigns it a model. You don't pick anything; you don't open any firewall ports.

Test it from your terminal

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-orc-xK9p…" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role":"user","content":"write fizzbuzz in rust"}]
  }'

Use it from Claude Code

export ANTHROPIC_BASE_URL=http://localhost:8080
export ANTHROPIC_AUTH_TOKEN=sk-orc-xK9p…
claude

Claude Code is now talking to your local Qwen-Coder. Same UX, your hardware.

Who is this for?

Non-goals

• Training or fine-tuning — Flock serves inference. Use Axolotl / Unsloth / torchtune for training, import the adapter.
• Replacing real Claude Opus — open models won't match Anthropic's frontier for long agentic runs. Flock makes the hybrid clean, not the choice unnecessary.
• A SaaS product — Flock is the software you run. The OSS is always complete.

Architecture overview

   CLIENTS  (Cursor · Claude Code · Aider · SDKs · curl)
                       │
                       ▼  one endpoint, one key
   ┌──────────────────────────────────────────────────┐
   │  GATEWAY      OpenAI + Anthropic compatible      │
   │               auth · routing · streaming · log   │
   └────────────────────┬─────────────────────────────┘
                        │
        ┌───────────────┼──────────────────┐
        ▼               ▼                  ▼
   ┌────────────┐ ┌────────────┐    ┌──────────────────┐
   │ Worker A   │ │ Worker B   │    │ External APIs    │
   │ Linux+GPU  │ │ Mac Mini   │    │ (Claude, GPT…    │
   │ vLLM       │ │ MLX-LM     │    │  fallback)       │
   └────────────┘ └────────────┘    └──────────────────┘
        ▲               ▲
        │               │  heartbeats, assignments
   ┌────┴───────────────┴──────────────────────────────┐
   │  CONTROL PLANE                                    │
   │  node registry · model registry · scheduler · UI  │
   └───────────────────────────────────────────────────┘
                        ▲
                        │ embedded Tailscale mesh
                        │ (mTLS, NAT-traversed)

See ARCHITECTURE.md for the full design.

Features

Inference

• OpenAI-compatible API (/v1/chat/completions, /v1/embeddings, /v1/models, /v1/rerank)
• Anthropic-compatible API (/v1/messages, /v1/messages/count_tokens)
• Audio endpoints (/v1/audio/transcriptions, /v1/audio/speech) — proxies to optional FLOCK_WHISPER_ENDPOINT / FLOCK_PIPER_ENDPOINT; returns HTTP 501 with setup hint when unconfigured
• Rerank endpoint passes through to llama-server's native /v1/rerank (b3580+); Cohere-shape response
• SSE streaming with proper client-disconnect handling (no goroutine leaks; bounded drain on cancel)
• Tool / function calling (pass-through for capable models)
• Vision (image input) on multimodal models — image_url content blocks on /v1/chat/completions route through the Ollama engine path
• Structured output (JSON schema)
• model=auto smart routing
• Response cache — embeddings cached against a sha256 of the canonicalized request body (object keys sorted; ephemeral fields stripped). Two drivers: in-memory LRU (default) and SQLite-backed (persists across leader restart). Per-request opt-out via Cache-Control: no-cache / no-store; per-tenant scoping via flock.cache.namespace body field. X-Flock-Cache: hit | miss response header.
• Typed engine_unreachable errors with engine name, endpoint, and start-hint (e.g. ollama serve) when the upstream engine isn't responding
• Engine health watchdog on auto-spawned engines (force-restart after 3 consecutive failures, covers hung llama-server)
• LoRA adapter hot-loading (planned)
• Chat completion caching with streaming replay + semantic cache (planned)

Cluster

• Auto-discovery — a node joins by running one command with a token
• Auto-placement — scheduler picks which node(s) host which model
• Memory lifecycle — admission control against live engine residency (a machine is never overcommitted), flock model load --swap with LRU evict-and-drain, --pin to protect a model, desired placements restored on restart, flock down releases engine memory by default, --exclusive for one-model-per-machine
• Heterogeneous sharding via llama.cpp RPC for models larger than any single node — flock shard create <model> <N> orchestrates the coordinator + every rpc-server end-to-end
• Live model migration (planned)
• Cross-platform workers: Mac (MLX), Linux+NVIDIA (vLLM), Linux+AMD (vLLM ROCm — planned), CPU (llama.cpp fallback)
• HA leader (planned)

Multi-tenancy

• Per-user API keys with revocation, scopes (admin / user / node), and TTL expiry (--ttl 7d, --expires-at 2026-07-01, flock token renew/expire)
• Daily token quotas per key with usage metering
• Per-key RPM + TPM rate limits — leaky-bucket admission control; HTTP 429 with Retry-After + X-RateLimit-Limit/Remaining/Reset-* headers (OpenAI shape). Reconciles upfront token estimate against actual completion tokens after the response.
• Per-key dollar + token budgets — multiple budgets compose with AND semantics ($10/day AND $100/month AND 1M tokens/day). Windows: day / week / month (UTC). HTTP 429 budget_exceeded with X-Flock-Budget-Reset-At + audit row.
• Per-call $ cost tracking — every usage row stores a cost_usd snapshot computed at write time from a built-in vendor pricing table (current Claude + OpenAI rates) or catalog-override fields. flock usage --summary shows $ spent; /admin/v1/usage/breakdown aggregates by user/model/protocol.
• Per-key model allowlist — pin a key to specific model ids (or vendor families via claude-* / gpt-* globs); unauthorized models return 403 model_not_allowed and the refusal is audit-logged
• Standard X-RateLimit-* headers on every /v1/* response + always-on X-Flock-Request-Id correlation token (also embedded in audit rows for traceability)
• Audit log of every admin mutation + middleware refusals (model_not_allowed, budget_exceeded, router.override, guardrail.block)
• OIDC / SSO login for the web UI — not planned (explicitly out of scope; see ROADMAP.md). The UI uses a pasted admin key; per-user API keys + quotas + audit cover accountability

flock token create alice --models qwen-coder-7b,qwen3-14b   # restrict at creation
flock token create bob   --models 'claude-*,gpt-*'          # vendor families via glob
flock token create dave  --rpm 60 --tpm 100000 --ttl 30d    # rate-limited + expiring
flock token budget add k_abc --window month --limit 100 --unit usd  # $100/month cap
flock token edit k_abc --add-model gpt-4o-mini              # extend
flock token edit k_abc --remove-model qwen3-14b             # tighten
flock token renew k_abc --ttl 30d                           # extend expiry

Hybrid local + cloud

• Built-in egress adapters for Anthropic + OpenAI; vendor model IDs (claude-*, gpt-*) transparently proxy upstream when ANTHROPIC_API_KEY / OPENAI_API_KEY is set

• OpenAI-compatible hosted gateways (20+) — openrouter/<model>, groq/<model>, together/<model>, fireworks/<model>, cohere/<model>, mistral/<model>, perplexity/<model> plus the registry providers deepseek/, cerebras/, nvidia/, gemini/ (OpenAI-compat), huggingface/, and zai/, ollama-cloud/, github/, cloudflare/, ovh/, kilo/, pollinations/, llm7/, opencode-zen/. Set the matching *_API_KEY; the slash prefix is stripped before forwarding so the upstream sees its native id. Providers with stable endpoints ship a default URL; the rest require <NAME>_BASE_URL (Flock won't ship a guessed endpoint). Adding a provider is one row in internal/api/providers.go.

• Multi-key rotation + 429 failover — stack several keys for one provider with numbered env vars (GROQ_API_KEY, GROQ_API_KEY_2, … _N). Flock rotates across them round-robin and, when a key returns 429 / 5xx / a transport error, parks it (honoring Retry-After) and retries the request on your next key — transparently, so the client never sees the rate limit. The last key always streams through, so a real error still surfaces once every key is exhausted.

• Routing chain (model="auto") — an ordered, persisted list of model ids Flock walks for model="auto": it tries each top-to-bottom, advances to the next on a rate-limit / transient failure, and commits the first success. Key rotation handles within a provider; the chain handles across providers; a local model pinned last is the always-available floor (never rate-limited, $0). Manage it with flock route ls/set/add/mv/rm/reset or the dashboard's Routing tab (drag to reorder) — both write the same audited /admin/v1/route endpoint. A sensible free → cheap → paid → local default is computed from whichever providers you've configured.

• Failure-based fallback chain: any catalog entry can declare fallback: [next-id, …] and the router will try the chain in order on engine errors, 503s, or timeouts (transparent to the client)

• Typed fallback chains — catalog entries can declare fallback_on_context_length (prompt too long → long-context variant) and fallback_on_content_policy (vendor refused → permissive open-weight). The router classifies the primary's error (sentinel errors.Is then heuristic substring) and switches the rest of the chain to the matching typed list. Generic fallback: is the default when no typed list matches.

• Per-request overrides — clients can override the catalog chain for a single call. Body block (flock.fallbacks, flock.num_retries, flock.retry_backoff_ms, flock.hedge) or X-Flock-* headers; the router walks the request chain instead of the catalog one and retries each candidate with exponential backoff (cap 5 retries, 5 s backoff). Traces tag flock.fallback.source = catalog | request so operators can see who's overriding policy.

• Request hedging — opt-in per-request (router.hedge_replicas: 2 in config + flock.hedge: true or X-Flock-Hedge: 1 per call) fires the request to the top-N least-loaded workers concurrently and returns whichever stream opens first; losers are cancelled. Tail-latency win at the cost of ~2× engine load.

• Sticky sessions — when router.sticky_session_ttl_seconds > 0, the router pins (user_id, model) to its last worker so multi-turn chats reuse the same node's KV cache. Falls through when the pinned node is in cooldown or stale.

• Placement cooldown (circuit breaker) — after router.placement_allowed_fails consecutive engine errors, a worker is parked for placement_cooldown_seconds. pick() skips it until expiry; a single success after expiry resets the counter. Dashboard's Nodes tab shows a 🚫 cooldown badge with seconds remaining.

  copy

  curl -s http://localhost:8080/v1/chat/completions \
    -H "Authorization: Bearer sk-orc-..." \
    -H "X-Flock-Num-Retries: 3" \
    -H "X-Flock-Hedge: 1" \
    -d '{
      "model": "qwen3-14b",
      "messages": [{"role":"user","content":"hi"}],
      "flock": {"fallbacks": ["qwen3-8b", "llama-3.2-3b"], "retry_backoff_ms": 250}
    }'
  

• AWS Bedrock: SigV4 signing for anthropic.* models (non-streaming). Streaming body translation for other families pending.

• GCP Vertex: ADC auth probe wired. Body translation for generateContent pending.

Policy + content checks

• Guardrails framework — observability.guardrails in config.yaml chains synchronous content checks against external services before the engine sees the request. Drivers: webhook (today; works as a thin shim for Presidio + Bedrock Guardrails + custom in-house policy). Modes: pre (block / rewrite / flag), logging_only (observe). On block: HTTP 403 guardrail_blocked with the guardrail name + reason; audit row recorded. fail_open: true|false chooses Allow vs Block on guardrail unreachable.

  copy

  observability:
    guardrails:
      - name: redact-pii
        kind: webhook
        mode: pre
        url: "http://presidio.lan:8080/v1/check"
        fail_open: false
  

• Observability callbacks — usage + audit events fan out to external sinks. Drivers: webhook (HMAC-SHA256 signed payloads), langfuse (maps usage to generation-create against /api/public/ingestion), and s3 (batched NDJSON to an S3 or S3-compatible bucket — MinIO / Cloudflare R2 / GCS-interop). Each sink runs on its own goroutine with a bounded queue — a slow receiver is non-blocking on the hot path; overflow events are dropped and counted on flock_callback_sent_total{outcome=dropped}. Admin GET /admin/v1/callbacks lists sinks; POST /admin/v1/callbacks/test[?sink=name] fires a synthetic event for wiring verification.

  copy

  observability:
    callbacks:
      - kind: webhook
        url: "https://hooks.example.com/flock"
        events: [usage, audit]
        secret: "${WEBHOOK_SECRET}"
      - kind: langfuse
        public_key: "${LANGFUSE_PUBLIC_KEY}"
        secret_key: "${LANGFUSE_SECRET_KEY}"
      - kind: s3
        bucket: "flock-logs"
        region: "us-east-1"
        prefix: "usage/"            # objects land at usage/YYYY/MM/DD/flock-<ts>.jsonl
        events: [usage, audit]      # omit for all kinds
        batch_size: 100             # events per object (default 100)
        flush_seconds: 30           # max batch age before upload (default 30)
        # endpoint: "https://<accountid>.r2.cloudflarestorage.com"  # S3-compatible
        # access_key_id: "${AWS_ACCESS_KEY_ID}"      # omit to use the default AWS chain
        # secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
  

  The s3 sink buffers events and writes one date-partitioned NDJSON object per flush (whichever of batch_size / flush_seconds comes first; a final flush runs on shutdown). Credentials come from access_key_id / secret_access_key when set, otherwise the standard AWS chain (env, shared config, IMDS) — the same chain Bedrock egress uses. Set endpoint (and the SDK switches to path-style addressing) for any S3-compatible store.

Observability

• Prometheus metrics endpoint (/metrics) — per-model RPS, latency, tokens, errors
• Per-call usage records (model, protocol, tokens, latency, outcome) via flock usage and the Usage tab
• Admin audit log via flock audit and the Audit tab
• Reference Grafana dashboards in dashboards/ — cluster-overview.json, per-model.json, per-node.json. Import any of them into Grafana 10+ and point at your Prometheus scrape of Flock's /metrics.
• OpenTelemetry / OTLP traces. Set observability.otlp_endpoint (or FLOCK_OTLP_ENDPOINT) to your collector — e.g. http://localhost:4318 — and Flock emits a full span hierarchy per request: http.request → router.Chat (covers the whole stream) → router.Chat.attempt (one per fallback retry) → <engine>.Chat (engine call with prompt/completion token counts). All four engine drivers (ollama, vllm, mlx, llamacpp) export the same span shape. W3C traceparent propagation is always on so Flock participates correctly between two services that both export. Empty endpoint = no-op (zero overhead beyond the NoopTracerProvider).

Developer experience

• One-line install (curl | sh)
• One-line model add (flock model add qwen3.6-27b) with a real progress bar and --dry-run preview
• One-line client config (UI generates per-tool snippets)
• Interactive picker for flock model add|info|remove and flock connect — no need to memorize IDs
• Shell completion for bash / zsh / fish (flock completion <shell>)
• Sensible defaults, no required flags
• Embedded web UI — no separate frontend to deploy

Supported models

For the complete per-model walkthrough (system requirements, performance per platform, install + use snippets for every client) see MODELS.md.

Flock ships a curated catalog of 41 open-weight models in catalog/*.yaml, spanning everything from 1 B edge models to 1 T-parameter sharded frontier MoE. Any other model also works via flock model add hf:<owner>/<repo> (HuggingFace direct) or flock model add ollama:<name> (any Ollama-pullable tag). See catalog/README.md for the YAML schema if you want to PR an entry.

📋 Picker table — what to install — full table with size, RAM, chat/code/reasoning/vision/audio/context ratings and license per model: MODELS.md → Picker table.

Shipped catalog at a glance

⭐ = current top picks (June 2026). All 41 catalog entries are listed; MODELS.md has the full picker table with sizes, RAM floors, and licenses.

Run flock model search to list everything live with sizes and capabilities, or flock model info <id> for one model's full spec. Add --sort=released for newest-first, --since 2026-01-01 to filter by date, or --json for machine-readable output. flock model ls, flock status, flock usage, and flock audit also accept --json. Running any flock model add|info|remove or flock connect with no ID launches an interactive picker (type to filter; arrow keys to navigate). Output is colored when stdout is a TTY; set NO_COLOR=1 (or FLOCK_NO_COLOR=1) to disable.

The dashboard at http://localhost:8080 mirrors the CLI: persistent top-bar chips show role + engine reachability + node/model counts (polled every 5 s); the Home tab summarizes traffic (requests-per-minute sparkline, p50/p95/p99, error rate, top model, recent activity); the Models tab includes a filterable catalog browser with per-row install; Nodes / Models / Usage / Audit refresh live while their tab is active; and "Add a worker" generates a one-time join token with copy-pasteable install-and-join snippets.

The same aggregates are available from the CLI: flock usage --summary and flock audit --summary print the top-models / p50-p95-p99 / error-rate / sparkline view that the dashboard renders. Both also accept --json.

Engine reliability: when Flock auto-spawned the engine itself (flock up with FLOCK_ENGINE=llamacpp), a health watchdog polls every 30 s and force-restarts the process after three consecutive failures — so a hung llama-server no longer requires manual intervention. For user-managed engines (Ollama, vLLM) Flock leaves the process alone but /v1/chat/completions now returns a typed engine_unreachable error with the engine name, endpoint, and the exact command to start it (ollama serve, mlx_lm.server …, etc.) when the engine isn't responding.

Proxied (paid APIs — shipped, works today)

When a request's model name matches one of these, Flock proxies to the upstream vendor with your API key (env-configured) and logs the call as usage like any other request:

• Anthropic upstream: any claude-* model id
• OpenAI upstream: gpt-*, o1*, o3*, o4* model ids

Routing logic lives in internal/api/egress.go; vendor detection in internal/router/router.go.

Roadmap — model families not yet in catalog

These work today via flock model add hf:owner/repo but don't have curated YAML entries with hardware specs:

• Larger general / agent models — Qwen3-235B, MiniMax-M2.7, MiMo-V2 sharded variants — pending sharded YAML entries.

Shipped recently (don't fall in this list):

• Speech / transcription — /v1/audio/transcriptions (and /v1/audio/speech) proxy to an optional Whisper / Piper endpoint (engine.whisper_endpoint / engine.piper_endpoint, or FLOCK_WHISPER_ENDPOINT / FLOCK_PIPER_ENDPOINT); HTTP 501 with a setup hint when unconfigured.
• Rerank — /v1/rerank passes through to llama-server's native rerank endpoint (b3580+); Cohere-shape response.
• Vision (image input) — gemma4-12b, gemma4-26b, gemma4-31b, gemma4-e2b, gemma4-e4b, qwen3-vl-8b, qwen3-vl-32b, pixtral-12b, moondream3, mimo-vl-7b, llama-4-scout all serve through /v1/chat/completions with image_url content blocks.
• Embeddings (for RAG) — /v1/embeddings is live; install nomic-embed-text and call it from any OpenAI-shape embedding client.
• Audio (input) — mimo-audio, gemma4-e2b, gemma4-e4b declare audio capability for future routing; today they serve as chat models.

Supported clients

The web UI generates a copy-pasteable config snippet for each tool.

Hardware recommendations

Solo / dev (1 node)

Team of ~10 (recommended)

Serves ~10 heavy users with headroom. Power draw ~300 W idle, ~900 W peak. Fits one 20 A circuit. Breaks even vs. typical Claude/GPT spend in ~5 months.

Larger team / production

• 1× H100 80 GB or 2× A100 80 GB for the flagship model
• 2× Mac Mini for completion
• 1× dedicated control box

Serves 25–50 users comfortably.

Installation

Prerequisites — read first

Flock is a gateway — it doesn't include an LLM engine. You need one of:

• Ollama (recommended for most users; works on Mac + Linux + NVIDIA + CPU)
• vLLM (for NVIDIA GPUs at scale — Linux only)
• MLX-LM (for fastest perf on Apple Silicon)

⚠️ Apple Silicon heads-up: the Homebrew ollama formula is currently missing the internal llama-server binary — model inference fails with 500: llama-server binary not found. Use the cask (brew install --cask ollama) or the official installer instead. The Flock installer detects this and warns you.

macOS (Apple Silicon)

# 1. install Ollama (use cask, NOT plain `brew install ollama`)
brew install --cask ollama
open -a Ollama                      # starts the daemon

# 2. install Flock
curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh | sh

# 3. add the install dir to PATH if the installer says so, e.g.:
export PATH="$HOME/.local/bin:$PATH"

# 4. start Flock
flock up

Linux (x86_64 or arm64)

# 1. install Ollama
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama   # or just: ollama serve &

# 2. install Flock
curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh | sh

# 3. add install dir to PATH if needed
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

# 4. start Flock
flock up

What the installer does

1. Detects your OS + architecture (must be macOS/arm64, Linux/x86_64, or Linux/arm64)
2. Checks for required shell tools (curl, tar)
3. Checks whether Ollama is installed and warns with the install command if not
4. Detects the broken-Homebrew-ollama case on macOS and tells you how to fix it
5. Fetches the latest release binary from GitHub Releases
6. Verifies SHA-256 against checksums.txt
7. Installs to ~/.local/bin/flock (or /usr/local/bin/flock with sudo)
8. Drops the bundled model catalog (*.yaml) into ~/.flock/catalog/ so flock up works without further setup
9. Prints next steps + tells you if PATH needs updating

Installer flags (after | sh -s --)

--help                  show usage
--version <vX.Y.Z>      install a specific version
--install-dir <path>    install to a specific dir
--no-engine             skip the Ollama check
--dry-run               show what would happen, no writes

Installer env vars (alternative to flags)

# pin a specific version (skips the GH API lookup — also avoids the 60/hr rate limit)
curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh \
  | FLOCK_VERSION=v1.14.0 sh

# install to a custom dir
curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh \
  | FLOCK_INSTALL_DIR=/opt/flock/bin sh

# skip the Ollama check (CI, custom engine setups)
curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh \
  | FLOCK_SKIP_ENGINE=1 sh

Install and join a cluster in one command:

curl -fsSL https://raw.githubusercontent.com/hadihonarvar/flock/main/installer/install.sh | \
    sh -s -- join https://leader.local:8080?token=<TOKEN>

Upgrade / uninstall

# upgrade in place (no need to re-run the installer)
flock update              # downloads latest release, verifies SHA-256, swaps binary
flock update --check      # just check, don't install

# uninstall — remove binary, catalog, and data dir
rm -f ~/.local/bin/flock       # (sudo-installed? then /usr/local/bin/flock)
rm -rf ~/.flock                 # catalog + data + config (destructive)

Build from source

git clone https://github.com/hadihonarvar/flock
cd flock
go build -o flock ./cmd/flock
./flock version

Requires Go 1.25+. See ARCHITECTURE.md → Build from source for cross-compile + release builds.

System requirements

• macOS 13+ on Apple Silicon (M1 or newer). Intel Macs not tested.
• Linux x86_64 or arm64 (Ubuntu 22.04+, Debian 12+, Fedora 39+, RHEL 9+).
• Linux + NVIDIA: NVIDIA driver 535+ (for vLLM); CUDA installed via the standard NVIDIA repos.
• RAM: 8 GB minimum, 16+ GB recommended; whatever model you load needs to fit.
• Disk: 50 GB for the binary + configs + small model cache; 200+ GB if you'll cache 70B-class models.
• Network: outbound HTTPS to GitHub + HuggingFace for downloading.

Troubleshooting installation

Configuration

Flock follows a strict "no config required for defaults" rule. Every flag has a sensible default. The config file is YAML at ~/.flock/config.yaml, or use env vars (FLOCK_LISTEN, FLOCK_DATA_DIR, …).

Minimal config (auto-generated on first flock up)

# ~/.flock/config.yaml
listen: ":8080"
data_dir: "~/.flock"
auth:
  require_keys: true   # set false for local-only dev mode

The initial admin key is auto-generated on first flock up and printed to stderr — copy it then. There is no auth.initial_admin_key field; the key lives in the SQLite store, not the YAML.

Full reference

Every field below is parsed by internal/config/config.go. Anything not in this list is silently ignored.

listen: ":8080"                       # HTTP listen address (used by leader and workers)
external_url: ""                      # public URL printed in UI; empty → use listen addr
data_dir: "~/.flock"                  # root for state.db, models, logs
log_level: "info"                     # debug | info | warn | error
catalog_dir: ""                       # empty → built-in catalog/ directory
max_body_bytes: 0                     # request-body cap on /v1/* in bytes;
                                      # 0 → built-in 32 MiB ceiling

storage:
  type: "sqlite"                      # only sqlite ships today
  dsn: "~/.flock/state.db"
  models_dir: "~/.flock/models"

auth:
  require_keys: true                  # set false to disable API-key auth (dev only)

engine:
  preferred: "ollama"                 # ollama | vllm | mlx | llamacpp
  ollama_endpoint:   "http://127.0.0.1:11434"
  vllm_endpoint:     "http://127.0.0.1:8000"
  mlx_endpoint:      "http://127.0.0.1:8080"
  llamacpp_endpoint: "http://127.0.0.1:8089"   # llama-server (single-node or RPC coordinator) — port chosen to avoid Flock leader :8080 and worker :8081
  whisper_endpoint: ""                # optional Whisper-compatible server for
                                      # /v1/audio/transcriptions; empty → 501
  piper_endpoint: ""                  # optional Piper-compatible server for
                                      # /v1/audio/speech; empty → 501

router:
  default_model: ""                   # empty → auto-pick on first up
  sticky_sessions: true               # legacy boolean; superseded by the TTL below
  sticky_session_ttl_seconds: 0       # >0 → pin (user, model) to its last worker
                                      # for this many seconds (KV-cache reuse)
  placement_allowed_fails: 0          # consecutive engine errors before a worker
                                      # is parked in cooldown (circuit breaker);
  placement_cooldown_seconds: 0       # both must be >0 to enable
  hedge_replicas: 0                   # >1 → allow per-request hedging across the
                                      # N least-loaded workers (`flock.hedge: true`)
  latency_fallback_p95_seconds: 0     # 0 = disabled. When >0, the router
                                       # walks the catalog `fallback:` chain
                                       # for a faster candidate FIRST whenever
                                       # the primary's recent p95 latency
                                       # exceeds this many seconds. Bet #1.
  fallback:
    enabled: false                    # true → forward unknown claude-*/gpt-* models to vendor
    anthropic_url: "https://api.anthropic.com"
    openai_url:    "https://api.openai.com"
    # Bedrock (AWS) — signed via aws-sdk-go-v2 using the standard AWS
    # credentials chain (env, shared config, instance role). Supports
    # the anthropic.* model family non-streaming; amazon.*/meta.*/mistral.*
    # return 501 (body translation not yet shipped).
    bedrock_region: ""                # e.g. us-east-1
    bedrock_url: ""                   # optional endpoint override
    # Vertex (GCP) — ADC auth probe wired; body translation for
    # generateContent not yet shipped. Set the project and a 501 with
    # ADC status returns until then.
    vertex_project:  ""               # GCP project id
    vertex_location: "us-central1"
    vertex_url: ""                    # optional endpoint override
    # OpenAI-compatible hosted gateways — URL overrides only; the keys
    # come from env (OPENROUTER_API_KEY, GROQ_API_KEY, …).
    openrouter_url: ""
    groq_url: ""
    together_url: ""
    fireworks_url: ""
    cohere_url: ""
    mistral_url: ""
    perplexity_url: ""

observability:
  otlp_endpoint: ""                   # e.g. http://localhost:4318 — empty disables tracing (no-op overhead)
  callbacks: []                       # usage/audit event sinks — list of
                                      # {kind: webhook|langfuse|s3, id, url, secret,
                                      #  events, host, public_key, secret_key,
                                      #  bucket, region, prefix, endpoint,
                                      #  access_key_id, secret_access_key,
                                      #  batch_size, flush_seconds, queue_size};
                                      #  see "Observability callbacks"
  guardrails: []                      # synchronous content checks — list of
                                      # {name, kind: webhook, mode: pre|logging_only,
                                      #  url, auth_key, headers, fail_open,
                                      #  timeout_seconds}; see "Guardrails framework"
  response_cache:
    enabled: false                    # cache embeddings responses by request hash
    driver: "memory"                  # memory | sqlite
    max_entries: 0                    # memory driver only; 0 → 1000

_…[view the full README on GitHub](https://github.com/hadihonarvar/flock)._