claudeers.
// MCP Servers

mcp-whichtool

Does the model pick the right tool? Tool-routing tests for your MCP server: run them on Claude and OpenAI models, see where each request lands, and fix the d…

// MCP Servers[ cli ][ api ][ web ][ claude ]#claude#ai-agents#anthropic#cli#developer-tools#evals#function-calling#github-actions#mcp-servers◷ Apache-2.0$open-sourceupdated 19 days ago
Actively maintained
100/100
last commit 20 days ago
last release 20 days ago
releases 1
open issues 1

Install with your AI

Paste into Claude Code, Cursor, or any agent — it reads the repo and wires the tool into your project.

Install and set up mcp-whichtool (git-clone project) into my current project.
Found on https://claudeers.com/mcp-whichtool
Repo: https://github.com/tastewise/mcp-whichtool
Homepage/docs: —
Detected install method: git-clone → git clone https://github.com/tastewise/mcp-whichtool
Category: mcp-servers. Platforms: cli, api, web.
Read the repo's README for exact setup and env vars, then install it and wire it into my project.

Claudeers Health Verdict:
active; community-verified: false. Confirm the source before running anything.
// or clone
git clone https://github.com/tastewise/mcp-whichtool

// compatibility

Platformscli, api, web
Operating systems—
AI compatibilityclaude
LicenseApache-2.0
Pricingopen-source
LanguageTypeScript

Get your FREE $2.50 API credits to access TickAtlas financial data ↗

mcp-whichtool

Does the model pick the right tool? Tool-routing tests for your MCP server: check that models pick the right tool, and fix the descriptions that make them pick the wrong one.

A model reads your tool descriptions, and nothing else, to decide which tool answers a user's request. mcp-whichtool runs your prompts against Claude and OpenAI models with your server's tools, shows where each one lands, proposes a better description for the ones that miss, and measures the edit before you ship it.

Status: early. It works end to end, the interface may still change.

whichtool on the bookshop demo: AI writes a test per tool, they run on Claude Haiku 4.5, one misroutes, Ask AI proposes a description, and the test passes with it

On the bookshop demo: AI writes a test per tool, they run three draws each on Claude Haiku 4.5, and "Can I pick up Piranesi in Lyon today?" goes to search_books instead of check_stock. Ask AI explains why and proposes a description; tried on that test, it passes 3/3. Nothing is written until you save it. (Waits sped up.)

Try it in your browser

Open in GitHub Codespaces, above, installs whichtool in a cloud environment and opens its console in a new tab: nothing to install on your machine. Try a demo opens the bookshop server below with its thirteen tests; to run them, add an Anthropic or OpenAI key in the page (it stays in your browser). A Codespace runs on your GitHub account's free monthly quota.

Scan your serverIts testsThe guides
The scan: drop your server's code, or give its URLThe test libraryThe Docs tab: naming and describing tools

Try it on the demo

The repository ships a small bookshop server whose descriptions are vague on purpose.

uv sync --all-groups
uv run python demo/server.py &  # http://127.0.0.1:8000/mcp
cd demo
uv run whichtool keys           # your Anthropic and/or OpenAI key, kept in demo/.env
uv run whichtool tools          # the seven tools a model will be offered
uv run whichtool eval           # thirteen cases, three draws each; asks before spending
uv run whichtool serve          # the same, in the browser

On your own server

cd your-project
whichtool scan                                     # the tools your source declares
whichtool scan --server http://127.0.0.1:8000/mcp  # or your running server's, any language
whichtool keys
whichtool generate                                 # test cases drafted by the model

scan lists the tools it found with their descriptions, then writes whichtool.toml and an empty routing_cases.json once you say yes. It reads Python (@mcp.tool() with a docstring, or description=), TypeScript and JavaScript (registerTool, tool, and tool objects with an inputSchema); for a server in another language, give its URL. Without a URL the models see the descriptions but not the tools' arguments, so a running server gives the truer result.

In the browser, whichtool serve opens on the same scan, from any folder: drop your server's files, a folder or a .zip, or give its URL. Try the demo there opens the bookshop with its tests ready (copied once to ~/.whichtool/demo/). API keys, at the top of the page, takes your Anthropic or OpenAI key: it stays in your browser, goes with each request, and the console never stores it. A key in your environment or the project's .env works too. Dropped files are copied to ~/.whichtool/uploads/, where their tests are kept; scanning another project switches to it.

In the browser, tests are written by AI (or by hand) and reviewed before they are kept: Try them runs the proposals as an eval would, with its progress, logs and report, and Open this run analyses a failed one. The test library then holds what was kept, and its ticks choose what to export.

Export, in the test library, downloads the tests as a kit for your server's repository: the cases, a whichtool.toml that reads the repository's own source, a GitHub Actions workflow that runs them on every pull request, and a short README. Unzip it at the root, add your API key as a repository secret, push.

A case is a prompt and the tool it should reach:

[
  {"id": "stock-pickup", "prompt": "Can I pick up Piranesi in Lyon today?", "expect": "check_stock"}
]

Write the canonical ask for each tool, then the oblique ones: the words a user types when they don't know your tool names. generate and the web console draft cases for you: read them over, since a model writing prompts from your descriptions tends to borrow their words.

Your keys and your tools

whichtool runs on your machine, and there is no whichtool server. Your Anthropic and OpenAI keys stay in your browser or your project's .env, the tools it scans stay in your files, and none of them is ever sent to us or stored anywhere else. To run a test, your tools and the test's request go to the model you chose, Anthropic's or OpenAI's, with your key, and nowhere else. The details are in Your keys and your tools.

What a run tells you

Each case is drawn N times (-n, default 3) per model. It passes when a strict majority of draws reach the expected tool, and it is weak when it passes without a clean sweep. A draw where the model answers in words instead of calling a tool counts as no pick. A run of the demo looks like this:

claude-haiku-5-5  12/13 passed  ·  3 draws  ·  $0.0020
  ✗ genres-cookbooks  expected list_genres, got search_books ×3
      'Do you sell cookbooks?'

Before it spends anything, eval prints an estimate from the price table and the token counts of your previous runs (or of a typical call, before the first one).

Fixing a description

In the console (whichtool serve), open a run and unfold a failed case. It shows the prompt, the tool expected and what each draw picked, then two ways to fix it:

  • Ask AI: a model reads the whole catalogue, the prompt and where it went, says why it failed, and proposes one description to change, or says the prompt itself is ambiguous. The proposal is editable.
  • Edit the description: write it yourself, for the tool expected or for the one picked instead.

Try it on this case replays that case alone, three draws, on the run's model, with the new description in place of the current one: "Passed 3/3" or "Still failing". Nothing is written: the description is a draft the run carries. Once it holds, Run all tests with it runs every case with the same draft, to check it moved nothing else, and Save to source writes it into your server's source, Python, TypeScript or JavaScript. Committing it is up to you. For dropped files, it is Save and copy: the copy the tests read takes it, and you paste it into your own source.

A description built at run time (a variable, a template with ${}) is read as none and left as it is; the console shows the proposal for you to copy.

In CI

whichtool eval --yes -n 3 -m claude-opus-5-5 -m gpt-6-sol

Exit code 0 means every case passed, 1 means at least one misrouted, and 2 means the eval could not run (server unreachable, no key, no cases). Keys come from the environment first, then .env.

Export, in the test library, does this setup for you: a kit to unzip in your server's repository (the tests, a project file and a GitHub Actions workflow), or the tests alone as JSON, to run in your own code. Exporting your tests compares the two and shows how to use each.

Commands

CommandWhat it does
whichtool scan [dir or URL]Finds the tools in a project's source or on its server, then writes whichtool.toml (--server URL; -y no question)
whichtool initWrites whichtool.toml and an empty case list, without looking
whichtool generateAsks the model for test cases, added to the case list (-t tool, repeatable; -y no question)
whichtool keysStores API keys in the project's .env (readable by you only)
whichtool toolsLists the tools the models are offered, and which descriptions come from your source
whichtool evalRuns the cases (-m model, repeatable; -n draws; -k filter on case ids; -y no question)
whichtool resultsShows each model's newest run again
whichtool try "prompt"Routes prompts that are in no case yet
whichtool serveOpens the console on 127.0.0.1:8790, on the scan; the tests are on /routing

Development

uv sync --all-groups
uv run pytest -q
uv run ruff check .

cd web && npm install && npm run test && npm run build  # the page, built into whichtool/static/

The tests run against real servers on sockets (Anthropic, OpenAI and MCP fakes in tests/fakes/, and the demo server itself), never against mocks of our own code.

License

Apache License 2.0. See NOTICE.

Built and maintained by Tastewise.

// faq

What is mcp-whichtool?

Does the model pick the right tool? Tool-routing tests for your MCP server: run them on Claude and OpenAI models, see where each request lands, and fix the descriptions that lose.. It is open-source on GitHub.

Is mcp-whichtool free to use?

mcp-whichtool is open-source under the Apache-2.0 license, so it is free to use.

What category does mcp-whichtool belong to?

mcp-whichtool is listed under mcp-servers in the Claudeers registry of Claude-compatible tools.

7 views
★ 13 stars
unclaimed
updated 19 days ago

// embed badge

mcp-whichtool on Claudeers
[![Claudeers](https://claudeers.com/api/badge/mcp-whichtool.svg)](https://claudeers.com/mcp-whichtool)

// retro hit counter

mcp-whichtool hit counter
[![Hits](https://claudeers.com/api/counter/mcp-whichtool.svg)](https://claudeers.com/mcp-whichtool)

// reviews

// guestbook

0/500

// related in MCP Servers

🔓

f.k.a. Awesome ChatGPT Prompts. Share, discover, and collect prompts from the community. Free and open source — self-host for your organization with complete…

// mcp-serversf/⟨HTML⟩★ 177,810◷ NOASSERTION[ claude ]
🔓

A cross-platform desktop All-in-One assistant for Claude Code, Codex, OpenCode, OpenClaw, Gemini CLI & Hermes Agent. Only official website: ccswitch.io

// mcp-serversfarion1231/⟨Rust⟩★ 155,814◷ MIT[ claude ]
🔓

🪨 why use many token when few token do trick — Claude Code skill that cuts 65% of tokens by talking like caveman

// mcp-serversJuliusBrussee/⟨JavaScript⟩★ 117,780◷ MIT[ claude ]
🔓

An open-source AI agent that brings the power of Gemini directly into your terminal.

// mcp-serversgoogle-gemini/⟨TypeScript⟩★ 108,744◷ Apache-2.0[ claude ]
→ see how mcp-whichtool connects across the ecosystem