# The eMudhra Fallout, Part 2: I Scanned 30 Docker Images. 10 Still Trust the Root Chrome Is Dropping

> Chrome drops emSign ECC Root CA - G3 on October 27. I read the CA bundles in 30 popular Docker images: a third still ship the root, some built after the fix.

- URL: https://www.kielltampubolon.id/writing/the-emudhra-fallout-part-2-i-scanned-30-docker-images-10-still-trust-the-root-chrome-is-dropping/
- Author: Kiell Tampubolon
- Published: 2026-10-25
- Last updated: 2026-10-25
- Language: en
- Tags: Security Engineering, Deployment

## Key takeaways

- Mozilla's NSS removed emSign ECC Root CA - G3 on October 20 and Chrome removes it on October 27, but container images do not update themselves.
- Of 30 popular Docker images scanned on October 24, 2026, 10 still ship G3, 10 have a bundle with G3 removed, and 10 have no CA bundle at all.
- python:3.12-slim has a clean system bundle but pip's vendored certifi still contains G3, and Java images carry a separate cacerts store.
- amazonlinux:2023 and amazoncorretto:21 were built on October 23, three days after the NSS fix, and still carry all four emSign roots.
- The fix is to find every trust store in the image, check each one, and rebuild with docker build --pull --no-cache.

On October 27, Chrome removes emSign ECC Root CA - G3 from its root store. Mozilla's NSS already dropped it on October 20. Your browser is fine. The question I wanted answered is simpler and less comfortable: what about the containers you pulled this week?

So I took 30 of the most common images and opened every trust store inside them. Ten of them still trust the root that Mozilla has already removed and Chrome is about to drop. Two of those ten were built three days *after* the NSS fix.

## TL;DR

> - After the eMudhra key incident, Mozilla removed the emSign ECC Root CA - G3 root on October 20, and Chrome removes it on October 27. Browsers update themselves; container images don't.
> - I scanned 30 popular Docker images on October 24, 2026. **10 of 30 still ship G3** in at least one trust store. **10 have no CA bundle at all**, and **10 have one with G3 removed**.
> - A "patched" image can still be exposed: `python:3.12-slim` has a clean system bundle but pip's vendored certifi still contains G3. Java images carry a second store, `cacerts`, that the OS package never touches.
> - Freshly built isn't the same as fixed: `amazonlinux:2023` and `amazoncorretto:21` were built on October 23 and still carry all four emSign roots.
> - The fix is cheap: find every bundle in your image, check each one, rebuild with `--pull`. The commands are at the end of this post.

This is a follow-up to [Part 1](https://www.kielltampubolon.id/writing/the-emudhra-ca-keys-were-one-byte-long-does-your-container-still-trust-them/), which covered why the eight compromised intermediates were a browser problem for a few hours and a backend problem for much longer.

## What changed, and when

The short version, from primary sources:

- **Oct 17, 2026.** Chrome received the report about eight eMudhra intermediate CAs with weak keys and blocked them through CRLSet within about two hours ([Google Security Blog](https://blog.google/security/protecting-chrome-and-android-users-from-compromised-emudhra-intermediate-ca-certificates/)).
- **Oct 19.** Firefox users were protected through OneCRL.
- **Oct 20.** NSS 3.131.1 removed trust in emSign ECC Root CA - G3, emSign Root CA - C1 and emSign ECC Root CA - C3. The notice says the G3 removal is "in response to a recent security incident which impacted some intermediate certificates" ([oss-security, Oct 20](https://www.openwall.com/lists/oss-security/2026/10/20/30), CVE-2026-113470).
- **Oct 27.** Chrome removes G3 from the Chrome Root Store.

Two things matter for servers. First, CRLSet and OneCRL protect browsers only. curl, Python `requests`, Go's `net/http`, Node and the JVM don't consult them, and by default they don't check revocation at all. For those clients, the trust store *is* the control. Second, none of the eight intermediates appear in any root bundle I scanned (I checked for non-self-signed emSign certificates in all of them: zero). So the measurable question is whether the root they chain to is still trusted.

## How I scanned without Docker

I didn't want to depend on a local daemon or on `docker run` behaving the same everywhere, so the scanner talks to the registry directly:

1. Get an anonymous token and the manifest for `linux/amd64`.
2. Stream every layer, in order, as a gzip tar, without writing it to disk.
3. Keep the *final* state of each trust store path, honouring later layers and `.wh.` whiteout files.
4. Parse every certificate and check the subject for the four emSign roots.

Paths covered: the system PEM bundle (`/etc/ssl/certs/ca-certificates.crt` and the RHEL equivalents), any `certifi/cacert.pem`, and Java `lib/security/cacerts` (JKS and PKCS12, opened natively and cross-checked with a byte search). Every image was read from its registry on October 24, 2026.

The core of the layer walk, simplified (the real version also handles hardlinks and removes everything under a whited-out directory):

```python
def stream_layers(reg, layers, files):
    for layer in layers:                              # oldest layer first
        r = reg.get("blobs/" + layer["digest"], stream=True)
        tf = tarfile.open(fileobj=r.raw, mode="r|*")  # stream, never touch disk
        for m in tf:
            name = posixpath.normpath("/" + m.name.removeprefix("./").lstrip("/"))
            d, b = posixpath.dirname(name), posixpath.basename(name)
            if b == ".wh..wh..opq":                   # opaque dir: drop everything below
                for k in [k for k in files if k.startswith(d + "/")]:
                    del files[k]
            elif b.startswith(".wh."):                # whiteout: file deleted in this layer
                files.pop(posixpath.join(d, b[4:]), None)
            elif m.issym() and want(name):            # bundles are often symlinks
                files[name] = ("link", m.linkname)
            elif m.isfile() and want(name):           # later layers overwrite earlier ones
                files[name] = ("file", tf.extractfile(m).read())
```

## The results

![Grid of 30 Docker images colour-coded: 10 still ship emSign ECC Root CA - G3, 10 have a bundle with G3 removed, 10 have no CA bundle at all.](https://www.kielltampubolon.id/writing/the-emudhra-fallout-part-2-i-scanned-30-docker-images-10-still-trust-the-root-chrome-is-dropping/img-1.png)

| Group | Images | What I found |
|---|---|---|
| Still ship G3 (10) | alpine:3.20, golang:1.24-alpine, amazonlinux:2023, amazoncorretto:21, eclipse-temurin:17-jre, eclipse-temurin:21-jre, distroless base-debian12, distroless static-debian12, python:3.12-slim, python:3.12-alpine | G3 present in the system bundle, Java `cacerts`, or pip's certifi |
| Bundle present, G3 removed (10) | alpine:3.22, alpine:latest, redhat/ubi9-minimal, python:3.13, python:3.13-slim, node:22-alpine, nginx:stable, nginx:alpine, httpd:2.4, curlimages/curl | No G3 anywhere I looked |
| No CA bundle at all (10) | debian:12, debian:12-slim, debian:13-slim, ubuntu:22.04, ubuntu:24.04, node:20-slim, node:22-slim, node:24-slim, redis:7, postgres:16 | Nothing to scan; you inherit whatever you install later |

By store type: G3 was in 8 of 20 system bundles, 4 of 5 Java `cacerts` files, and 2 of 4 certifi copies.

Four results surprised me more than the headline number.

**1. One image can give two answers.** `python:3.12-slim` is built on Debian trixie, and its system bundle no longer has G3. But the image also ships pip, and pip vendors its own copy of certifi, which still has it. `python:3.12-alpine` is the same story on Alpine. The `python:3.13` images are free of G3 in both places.

![Two cards: python:3.12-slim has G3 removed from the system bundle but present in pip's certifi; eclipse-temurin:21-jre has G3 in both the system bundle and the JDK cacerts.](https://www.kielltampubolon.id/writing/the-emudhra-fallout-part-2-i-scanned-30-docker-images-10-still-trust-the-root-chrome-is-dropping/img-2.png)

**2. Java images carry a second trust store.** Both Temurin images (Ubuntu 26.04 base, built October 17) have G3 in the Ubuntu bundle *and* in the JDK's own `cacerts`. Updating `ca-certificates` in a Java image doesn't change what the JVM trusts unless your image links the two, so check `cacerts` separately.

**3. Recent build date doesn't mean the fix is in.** `amazonlinux:2023` and `amazoncorretto:21` were built on October 23, three days after NSS 3.131.1, and still carry all four emSign roots in both the PEM bundle and `cacerts`. The image is fresh; the package inside it isn't yet.

**4. Distros didn't all remove the same roots.** Patched Alpine (`ca-certificates` 20261020-r0) and UBI 9 keep only emSign Root CA - G1, which matches what NSS did. Debian trixie's update (`20250419+deb13u1`) removed G3 but still has C1 and C3, which NSS also removed. That's an observation from the scan, not something I found documented. The distroless Debian 12 images ship `20250419~deb12u1`, the version from before the fix.

A note on the ten images with no bundle: that isn't a pass. `debian:12-slim` and the `node:*-slim` images don't include `ca-certificates`, so the moment your Dockerfile runs `apt-get install ca-certificates`, you get whatever version the mirror serves that day. If your build cache is older than October 20, it's the old one.

## Check your own images

![Three steps: find every bundle in the image, grep each one for emSign ECC Root CA - G3, rebuild with --pull and re-check.](https://www.kielltampubolon.id/writing/the-emudhra-fallout-part-2-i-scanned-30-docker-images-10-still-trust-the-root-chrome-is-dropping/img-3.png)

Run this inside a container from your image, not the base image, since your own layers may add stores. Minimal images (Alpine, distroless) often have no `openssl`, `keytool` or even a shell; for those, `docker create` the image and `docker cp` the bundle files out, then check them on your machine:

```bash
# 1. Find every trust store
find / \( -name 'ca-certificates.crt' -o -name 'cacert.pem' -o -name 'ca-bundle.crt' -o -name cacerts \) 2>/dev/null

# 2. Check each PEM bundle for the root
openssl storeutl -noout -text -certs /etc/ssl/certs/ca-certificates.crt | grep 'emSign ECC Root CA - G3'

# 2b. Java keystores
keytool -list -cacerts -storepass changeit | grep -i emsign

# 3. Rebuild from fresh layers, then repeat 1 and 2
docker build --pull --no-cache -t myapp:ca-refresh .
```

Then fix what you find:

- **System bundle:** rebuild on a base that has the updated `ca-certificates`, or add `apt-get update && apt-get install -y ca-certificates` / `apk upgrade ca-certificates` in your own layer.
- **certifi:** upgrade it (`pip install -U certifi`; per Endor Labs and SafeDep, 2026.10.19 drops G3), and remember pip's vendored copy only updates when pip does. Even better, point Python at the system store with `SSL_CERT_FILE` so there's one bundle to maintain.
- **Java:** use a JDK build that includes the update, or remove the alias from `cacerts` with `keytool -delete` until your vendor ships one.

## Limitations

- One point in time (October 24, 2026), `linux/amd64` only. Tags move; rerun before you quote these numbers.
- I scanned the system bundle, certifi and Java `cacerts`. Node's built-in store, Go binaries that embed their own roots, and anything an application vendors were out of scope.
- Distro package versions for the fixes come from secondary sources (Endor Labs, SafeDep) except where I saw them in an image. Exact distro release dates are unclear.
- Root presence is a proxy. It tells you a client *would* accept a chain to that root, not that anyone is attacking you.

## The takeaway

The browser vendors handled this in hours, which is how it should work. The containers you build from are a different supply chain with a different clock, and they don't update unless you rebuild them. A check in CI that greps every trust store in the image, not just the system bundle, would have flagged every red box in that grid. A scheduled `--pull` rebuild fixes most of them; for the few where the upstream tag still ships the root, the check tells you to patch the store in your own layer.

Which trust store in your stack do you think nobody has looked at since it was built?

Versi bahasa Indonesia: [https://www.kielltampubolon.id/writing/saya-memindai-30-image-docker-populer-10-masih-percaya-root-ca-yang-dicabut-chrome/](https://www.kielltampubolon.id/writing/saya-memindai-30-image-docker-populer-10-masih-percaya-root-ca-yang-dicabut-chrome/)

## FAQ

### Which Docker images still trust emSign ECC Root CA - G3?

In the October 24, 2026 scan, 10 of 30 did, including alpine:3.20, amazonlinux:2023, amazoncorretto:21, both eclipse-temurin JRE images, the distroless Debian 12 images, python:3.12-slim and python:3.12-alpine.

### Why can a patched image still contain the root?

An image can hold several trust stores. python:3.12-slim has a clean system bundle but pip's vendored certifi still has G3, and Java images keep a separate cacerts file that the OS package never touches.

### Does a recent build date mean the fix is included?

No. amazonlinux:2023 and amazoncorretto:21 were built on October 23, after NSS 3.131.1, and still carry all four emSign roots.

### How do I check and fix my own image?

Find every bundle with find, grep each PEM bundle with openssl storeutl and each Java keystore with keytool, then rebuild with docker build --pull --no-cache and repeat the check.