# Benchmarking My MCP Security Scanner: 14 of 14 Findings, Zero False Positives

> MCP security scanner benchmark: 14 of 14 planted findings detected, 0 false positives across 5 fixtures and 24 tools, at a median 82 microseconds per manifest.

- URL: https://www.kielltampubolon.id/writing/benchmarking-my-mcp-security-scanner-14-of-14-findings-zero-false-positives/
- Author: Kiell Tampubolon
- Published: 2026-09-12
- Last updated: 2026-09-12
- Language: en
- Tags: MCP Security, Security Engineering

## Key takeaways

- The benchmark used 5 synthetic MCP tool manifests with 24 tools in total and hand-labeled ground truth for six rules, MCP-001 to MCP-006.
- Each fixture was scanned 1000 times, single thread, on Python 3.13, with no warmup tricks.
- The scanner detected 14 of 14 planted findings with 0 false positives, at a median of 82 microseconds per manifest.
- The clean safe_notes fixture returning zero findings is the real test, because a scanner that cries wolf gets ignored.
- The results measure the rule engine on synthetic manifests, not detection quality on real world servers.

A scanner without numbers is a demo. A scanner with made-up numbers is worse. So I hand-labeled a small benchmark suite and ran the scanner against it, 1000 times per fixture.

The setup is deliberately boring:

- 5 synthetic MCP tool manifests: one clean (3 tools), one openly risky (4 tools), one realistic mixed shop (8 tools), one payments flavored (6 tools), and one leaky agent (3 tools) built to exercise the two newest rules. 24 tools total.
- Ground truth per tool: which rule IDs should fire. MCP-001 unsafe command capability, MCP-002 excessive scope, MCP-003 secret-like value, MCP-004 untrusted input into a sensitive operation, MCP-005 credential forwarded in metadata, MCP-006 write tool without tenant scoping.
- 1000 timed scans per fixture, single thread, Python 3.13, no warmup tricks.

The results:

| fixture | tools | expected findings | detected | false positives | median scan |
|---|---|---|---|---|---|
| safe_notes | 3 | 0 | 0 | 0 | 28 us |
| risky_ops | 4 | 6 | 6 | 0 | 63 us |
| mixed_shop | 8 | 3 | 3 | 0 | 94 us |
| payment_tools | 6 | 3 | 3 | 0 | 73 us |
| leaky_agent | 3 | 2 | 2 | 0 | 89 us |

Aggregate: 14 of 14 planted findings detected, 0 false positives, median 82 microseconds per manifest.

Two honest notes. First, this measures the rule engine on synthetic manifests, not detection quality on real world servers; synthetic ground truth is easy to trust too much. Second, the clean fixture matters as much as the risky ones. Zero findings there is the actual test, because a scanner that cries wolf gets ignored, and an ignored scanner is worse than none.

The harness, fixtures, and raw results ship with the repo under benchmark/, so anyone can rerun it: clone, one command, verdict. Same zero dependency habit as the test suites.

Repo: https://github.com/glatinone/mcp-security-preflight

---

*If you want to run checks like these on an MCP server or a `.claude/` directory before you install it, I maintain [mcpscan](https://github.com/glatinone/mcpscan), an open-source scanner for tool poisoning, risky permissions and leaked secrets. Issues and false-positive reports are welcome.*

*Originally published on [DEV](https://dev.to/kielltampubolon/benchmarking-my-mcp-security-scanner-14-of-14-findings-zero-false-positives-438g). Discussion happens there.*

## FAQ

### How do you benchmark an MCP security scanner?

Hand-label which rule IDs should fire for each tool in a set of fixtures, then run the scanner 1000 times per fixture and compare detections and false positives to the labels. Include a clean fixture, since zero findings there is the real test.

### How fast is the MCP security scanner in this benchmark?

The median scan was 82 microseconds per manifest, ranging from 28 us for the 3-tool clean fixture to 94 us for the 8-tool mixed shop fixture.

### Does a 14 of 14 benchmark prove the scanner works on real MCP servers?

No. It measures the rule engine on synthetic manifests, and the author warns that synthetic ground truth is easy to trust too much.