We Planted 10 Vulnerabilities to Test Free Semgrep. It Reported 3.

Before you read: we are not security experts. We are a small team learning security tooling by testing it directly and writing down what happened. This is a field note, not a best-practices guide. If something here is wrong, corrections in the comments are welcome.

---

TL;DR

We put 10 common vulnerabilities into a React and TypeScript app on purpose, then ran the free, zero-config Semgrep (--config=auto) against them. It reported 3 of the 10. The gap is not a defect in Semgrep. It reflects what free static analysis is built to find and what it is not. The details are below.

Why we ran this

TL;DR: We wanted a measured answer to one question. On its own, how much does a free SAST scan actually find?

“Add SAST to your CI” is common advice. The part that rarely gets measured is what you get from it when you pay nothing and write no custom rules. Instead of guessing, we set up a small test:

• A Security Operations Dashboard built with React 18, TypeScript, and Vite.
• A folder, src/security-test-cases/, holding 10 deliberately vulnerable files, one per weakness.
• A GitHub Actions workflow that runs semgrep scan --config=auto on every push.

Then we read the CI logs and recorded the results.

---

How Semgrep works

TL;DR: Semgrep parses your code into a tree (an AST) and matches rule patterns against that tree. It does not run your program. If no rule describes a given pattern, that pattern is not reported.

The same idea as a text diagram:

  Source code  ->  Parser  ->   AST      ->  Match rules  ->  Findings
  (.ts/.tsx)                  (a tree of       (~1074           (file + line
                               your code)       patterns)         + rule id)

A few things follow from this design:

• It is fast and cheap to run. There is no compile or execution step, so a full repo scans in seconds, which suits CI.
• It handles bad code that is present. A call like crypto.createHash('md5') has a stable shape that a rule can match.
• It struggles with correct code that is absent. The missing presence of an authorization check is hard to express as a pattern.

---

The 10 cases we planted

TL;DR: One file per weakness, each mapped to its CWE identifier. All are isolated and never imported by real application code.

#	Vulnerability	CWE	Risk in one line
1	Hardcoded credential	CWE-798	Secrets in git history are hard to rotate and are exposed to anyone with repo access
2	IDOR	CWE-639	Trusting a user-supplied ID lets one user read another user’s data
3	Sensitive info leak (logs or localStorage)	CWE-532/312	Tokens in logs or localStorage are easy to read
4	Broken crypto (MD5)	CWE-327	MD5 is unsafe for password or integrity use
5	XSS via dangerouslySetInnerHTML	CWE-79	Rendering raw user HTML lets a script run in another user’s session
6	SQL or NoSQL injection	CWE-89	String-concatenated queries let an attacker rewrite the query
7	Insecure deserialization or unsafe eval	CWE-502	Evaluating untrusted input allows code execution
8	CSRF (no token)	CWE-352	Cookie-only state changes can be triggered by another site
9	Open redirect	CWE-601	Redirecting to an unchecked URL enables convincing phishing
10	Improper error handling	CWE-209	Raw stack traces reveal internal paths and logic

---

The setup did not work on the first try

TL;DR: The pipeline broke several times before it produced results. Each failure had a specific cause. The sequence is below.

Attempt 1  Failed. We used a flag that does not exist, `--soft-fail`.
           CI stopped with "unknown option". Lesson: check the CLI reference, not memory.

Attempt 2  Failed. The SARIF upload hit "Resource not accessible by integration".
           Cause: missing `permissions: security-events: write`. Tokens are read-only by default.

Attempt 3  Partial. On a private repo, GitHub code scanning (Advanced Security) was unavailable.
           We made the upload non-blocking. The Security tab is not free on private repos.

Attempt 4  Not useful. `--sarif --output=file` suppressed the readable report.
           The log showed only a count, no paths. We split it into two steps.

Attempt 5  Worked. Findings appeared in the log with path, line, and rule id.

Most of our time went into configuration rather than security: flags, permissions, and output formats. That is worth planning for.

---

Results

TL;DR: The first round, with generic mock code, reported 1 of 10. After we rewrote the fixtures to resemble real framework code, it reported 3 of 10. The remaining 7 stayed unreported across six different rule packs.

#	Vulnerability	Round 1 (generic)	Round 2 (realistic)
4	Broken crypto (MD5)	✅	✅
7	Unsafe eval()	❌	✅
9	Open redirect	❌	✅
1	Hardcoded secret	❌	❌
2	IDOR	❌	❌
3	Sensitive info leak	❌	❌
5	XSS	❌	❌
6	SQL or NoSQL injection	❌	❌
8	CSRF	❌	❌
10	Stack-trace leak	❌	❌

The unplanned finding

TL;DR: The same scan reported 3 real issues we did not plant, which is useful evidence that the tool works on real code.

• A mutable GitHub Actions tag (@v4 rather than a pinned SHA), which is a supply-chain risk.
• An incomplete-sanitization issue in our actual metricsApi.ts.
• A ReDoS risk from a non-literal RegExp in our actual emlParser.ts.

The 7 misses are not a sign of a broken scanner. On production code, it reported genuine problems.

---

Why 7 were not reported

TL;DR: Three structural reasons, none of them a bug. Pattern matching has limits.

It matches shapes, not runtime behavior

A static matcher cannot follow a value from an HTTP request through several functions to a dangerous call unless a rule already describes that flow. Deeper cross-file taint tracking is mostly a paid feature.

Missing-check bugs are hard for generic rules

IDOR and CSRF are not cases of bad code being present. They are cases of expected code being absent, such as a missing ownership check or a missing CSRF token. A generic community rule has no way to know what your application’s auth is supposed to look like.

Secrets detection is a separate tool

Finding hardcoded keys relies on entropy analysis and provider-specific patterns, closer to TruffleHog or GitGuardian than to AST matching. Our AWS and Azure style placeholders were not reported even by the dedicated secrets pack, because that capability is a different product.

What static SAST handles well, and what it does not

Where static SAST does well	Where it does not
Known dangerous functions such as eval and md5	Business-logic flaws such as IDOR and broken access control
Risky configuration such as mutable CI tags	Cases where a required check is simply missing
Syntactic anti-patterns	Data flow across multiple files, without paid taint tracking
Fast, cheap, consistent at scale	Secrets, which need a dedicated scanner

---

Is it still worth running? Yes, as a first layer

TL;DR: A clean Semgrep run means no known pattern matched. It does not mean there are no vulnerabilities. Use it as the fast first filter, not the only control.

What we would suggest for a small team:

1. Keep Semgrep OSS in CI. It is free and reported 3 real issues for us, so there is no reason to remove it.
2. Do not read “0 findings” as “secure”. It means no generic pattern matched.
3. Write a small number of custom rules against your own auth helpers, for example flagging any /api/ handler that does not call requireOwnership(). A handful of targeted rules will find more of your IDOR and CSRF cases than any generic pack.
4. Add a dedicated secrets scanner. Do not rely on --config=auto for keys.
5. Keep code review in the loop. A reviewer asking whether an endpoint checks ownership catches what a free static tool will not.

---

What we took away from it

TL;DR: The value of a tool comes from knowing where it stops.

• A green check is a starting point, not a guarantee.
• Much of the work is configuration, not analysis. Plan for that.
• Free SAST, custom rules, a secrets scanner, and code review work together. No single tool covers all of it.
• Measuring your own coverage, such as this 3 of 10, is more useful than a vendor’s claim.

If you are early in learning security, a small test like this is worth building. Testing the tool directly taught us more in a day than reading about it would have.

---

We are still learning, so corrections and additions in the comments are welcome.

Originally published on DEV. Discussion happens there.