Skip to content

Benchmarking My MCP Security Scanner: 14 of 14 Findings, Zero False Positives

Published Last updated 2 min read

Key takeaways

  • The benchmark used 5 synthetic MCP tool manifests with 24 tools in total and hand-labeled ground truth for six rules, MCP-001 to MCP-006.
  • Each fixture was scanned 1000 times, single thread, on Python 3.13, with no warmup tricks.
  • The scanner detected 14 of 14 planted findings with 0 false positives, at a median of 82 microseconds per manifest.
  • The clean safe_notes fixture returning zero findings is the real test, because a scanner that cries wolf gets ignored.
  • The results measure the rule engine on synthetic manifests, not detection quality on real world servers.

A scanner without numbers is a demo. A scanner with made-up numbers is worse. So I hand-labeled a small benchmark suite and ran the scanner against it, 1000 times per fixture.

The setup is deliberately boring:

  • 5 synthetic MCP tool manifests: one clean (3 tools), one openly risky (4 tools), one realistic mixed shop (8 tools), one payments flavored (6 tools), and one leaky agent (3 tools) built to exercise the two newest rules. 24 tools total.
  • Ground truth per tool: which rule IDs should fire. MCP-001 unsafe command capability, MCP-002 excessive scope, MCP-003 secret-like value, MCP-004 untrusted input into a sensitive operation, MCP-005 credential forwarded in metadata, MCP-006 write tool without tenant scoping.
  • 1000 timed scans per fixture, single thread, Python 3.13, no warmup tricks.

The results:

fixturetoolsexpected findingsdetectedfalse positivesmedian scan
safe_notes300028 us
risky_ops466063 us
mixed_shop833094 us
payment_tools633073 us
leaky_agent322089 us

Aggregate: 14 of 14 planted findings detected, 0 false positives, median 82 microseconds per manifest.

Two honest notes. First, this measures the rule engine on synthetic manifests, not detection quality on real world servers; synthetic ground truth is easy to trust too much. Second, the clean fixture matters as much as the risky ones. Zero findings there is the actual test, because a scanner that cries wolf gets ignored, and an ignored scanner is worse than none.

The harness, fixtures, and raw results ship with the repo under benchmark/, so anyone can rerun it: clone, one command, verdict. Same zero dependency habit as the test suites.

Repo: https://github.com/glatinone/mcp-security-preflight

If you want to run checks like these on an MCP server or a .claude/ directory before you install it, I maintain mcpscan, an open-source scanner for tool poisoning, risky permissions and leaked secrets. Issues and false-positive reports are welcome.

Originally published on DEV. Discussion happens there.

Frequently asked questions

How do you benchmark an MCP security scanner?
Hand-label which rule IDs should fire for each tool in a set of fixtures, then run the scanner 1000 times per fixture and compare detections and false positives to the labels. Include a clean fixture, since zero findings there is the real test.
How fast is the MCP security scanner in this benchmark?
The median scan was 82 microseconds per manifest, ranging from 28 us for the 3-tool clean fixture to 94 us for the 8-tool mixed shop fixture.
Does a 14 of 14 benchmark prove the scanner works on real MCP servers?
No. It measures the rule engine on synthetic manifests, and the author warns that synthetic ground truth is easy to trust too much.
Share Share on X Share on LinkedIn

About the author

Security Engineer & Technical Writer

Security engineer and technical writer based in Batam, Indonesia. I review and harden MCP servers and AI agents, build security automation, and write up what holds and what fails. I also train on DNS security as a PANDI .id Academy trainer.

Related articles

All writing | RSS | Markdown