Half the MCP Registry Does Not Start

Every statistic you have read about the Model Context Protocol ecosystem has a selection problem, and a paper submitted on 10 September 2026 states it in the first sentence:

Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, popularity lists, hand-curated frames, or pipelines that repair a server until it starts.

Reference sets are chosen because they are good examples. Popularity lists rank by adoption, and a server nobody can run does not get adopted. And a pipeline that repairs a server until it starts has, by construction, deleted the thing you were trying to measure.

So the author did the boring, correct thing. Take a census of 24,135 servers, draw 400 npm/stdio servers at random with a published seed, probe each one over the wire, and repair nothing.

What a random draw contains

Only 48.8% complete an initialize handshake.

That is the first thing an MCP client does. Not a useful task, not a tool call, just the opening exchange that establishes the connection. Under half of a random draw gets through it.

The same instrument pointed at a hand-curated frame gives 66.7%, which quantifies the bias directly. Curation is worth roughly eighteen points of apparent health, and every figure derived from a curated frame inherits that.

Then the useful detail, the one that changes what you do about it:

the dominant failure is not missing credentials (13.3%) but servers that never start at all (37.5%)

The comfortable explanation was always credentials. Of course a random server fails when probed; it wants an API key you did not give it. That explanation covers 13.3%.

37.5% never start. No key required, no auth wall reached. The process does not come up. That is not an artifact of probing without secrets; it is broken packaging, missing dependencies, a bad entry point, an install that does not survive contact with a clean machine.

The genuinely surprising part

Here is where it gets interesting, because the picture flips completely for the servers that do run.

Among the 195 that start, conformance is perfect:

hard conformance is total: zero fatal JSON Schema violations across 2,766 advertised tools

Zero. Across nearly three thousand tools. Not a low rate, an absence.

So the ecosystem is not sloppy. It is bimodal. A server either does not work at all, or it works and its schemas are clean. There is almost no middle ground of half-conformant servers advertising malformed tools.

That is good news about the protocol and its SDKs. The schema layer is doing its job; if your server runs, the shape of what it advertises is correct. The failure is entirely in getting it to run, which is a packaging and distribution problem, not a protocol problem.

Where variance does show up is in the optional parts:

the tool-level omission rate on a random draw is 58.8% against 41.5% on the curated frame, so curation flatters this figure too

Safety annotations, the optional metadata describing whether a tool is read-only or destructive, are missing on most tools either way. And note the pattern repeating: curation flatters this number too, by about seventeen points. Any ecosystem-health figure you have read is probably measuring the curator's taste.

The second half is about benchmarks, and it is worse

The paper then compares real MCP tool descriptions against two tool-use benchmark corpora, holding the method constant. This part deserves its own attention.

Real MCP tools show 2.8% near-duplication at a cosine threshold of 0.70, and all of it is within single servers. Cross-author near-duplication is 0.0% at every threshold tested. Independent authors do not accidentally write the same tool description. That is what you would hope, and it is now measured.

BFCL v4 shows 16.7%, of which 16.4 points sit between independently presented tasks. Tasks the benchmark presents as separate are, semantically, near-copies of each other.

Then the number that should stop you:

68.8% of raw BFCL rows and 85.6% of raw UltraTool rows are exact name-plus-description repeats, against 0.4% for real MCP

Not similar. Exact repeats. Over two thirds of one widely-used benchmark's raw rows and over five sixths of another's are the same name and description appearing again.

The consequence is stated plainly: any statistic computed over these releases without global deduplication measures repetition rather than tools. If you have seen a chart of model performance across tool-use benchmarks, some of what it measures is how often the same item recurs.

One point of fairness the author is careful about: UltraTool shows only 0.3% near-duplication at the cosine threshold, cleaner than real tools. So the near-duplication problem is a property of BFCL specifically, not of synthetic corpora as a class. That distinction is the mark of someone reporting what they found rather than building a case.

Why this one is credible despite being a v1 preprint

Single author, unreviewed, submitted last week. Normally that earns a large asterisk, and it still earns some. But two features do real work.

The seed is published and the scripts are released. The paper's closing claim is that all figures regenerate from released scripts and a published seed, with materials archived at Zenodo under a DOI. That is the actual remedy for trusting a single author: not credentials, but the ability to rerun it.

And the paper's whole argument is methodological. Its thesis is that other people's sampling was biased. A paper making that claim while hiding its own sampling would be self-refuting, and this one publishes the frame, the seed, the draw and the per-server outcomes.

Still: v1, unreviewed, one instrument, npm/stdio servers only. The 48.8% is a number about that population, not about every MCP server everywhere.

What to actually do with this

Assume roughly half of any MCP server list you find will not run. Budget for that when you evaluate. It is packaging, not you.

Test on a clean machine. The 37.5% that never start are mostly failing in ways that a developer's own machine, with its accumulated global installs, will hide completely. If you publish a server, this is the check that matters most and the one nobody runs.

Do not trust ecosystem-health numbers from curated lists. Eighteen points on handshake success, seventeen on annotation omission. The direction of the bias is always flattering.

Write safety annotations. Most tools do not have them, which means doing it puts you in a minority and it is the metadata that tells a client whether your tool is safe to call without asking. Given how much of the tool-description problem turns on what a model is told about a tool, this is not decoration.

And discount tool-use benchmark scores accordingly. The deduplication finding is the most portable result here, and it compounds with what benchmark scores already fail to tell you, covered in the best agent scores 25.6 percent.

The line to keep

The finding underneath all of this is not about MCP. It is that the way you draw your sample can be the entire result, and a field measuring itself with convenience samples will produce a consistently flattering picture that nobody has to lie to create.

Under half of a random draw starts. Everything that starts is schema-perfect. Both of those are true, and you only find out by refusing to repair anything.

Source. What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead, Haseeb Mohammed Afsar, submitted 10 September 2026, with materials archived at Zenodo (doi:10.5281/zenodo.21347997). Every figure and quote above is from that paper. Related here: MCP explained, MCP went stateless and left the tool descriptions alone and what breaks in the 2026-07-28 spec.