The eMudhra Fallout, Part 2: I Scanned 30 Docker Images. 10 Still Trust the Root Chrome Is Dropping

On October 27, Chrome removes emSign ECC Root CA - G3 from its root store. Mozilla’s NSS already dropped it on October 20. Your browser is fine. The question I wanted answered is simpler and less comfortable: what about the containers you pulled this week?

So I took 30 of the most common images and opened every trust store inside them. Ten of them still trust the root that Mozilla has already removed and Chrome is about to drop. Two of those ten were built three days after the NSS fix.

TL;DR

• After the eMudhra key incident, Mozilla removed the emSign ECC Root CA - G3 root on October 20, and Chrome removes it on October 27. Browsers update themselves; container images don’t.
• I scanned 30 popular Docker images on October 24, 2026. 10 of 30 still ship G3 in at least one trust store. 10 have no CA bundle at all, and 10 have one with G3 removed.
• A “patched” image can still be exposed: python:3.12-slim has a clean system bundle but pip’s vendored certifi still contains G3. Java images carry a second store, cacerts, that the OS package never touches.
• Freshly built isn’t the same as fixed: amazonlinux:2023 and amazoncorretto:21 were built on October 23 and still carry all four emSign roots.
• The fix is cheap: find every bundle in your image, check each one, rebuild with --pull. The commands are at the end of this post.

This is a follow-up to Part 1, which covered why the eight compromised intermediates were a browser problem for a few hours and a backend problem for much longer.

What changed, and when

The short version, from primary sources:

• Oct 17, 2026. Chrome received the report about eight eMudhra intermediate CAs with weak keys and blocked them through CRLSet within about two hours (Google Security Blog).
• Oct 19. Firefox users were protected through OneCRL.
• Oct 20. NSS 3.131.1 removed trust in emSign ECC Root CA - G3, emSign Root CA - C1 and emSign ECC Root CA - C3. The notice says the G3 removal is “in response to a recent security incident which impacted some intermediate certificates” (oss-security, Oct 20, CVE-2026-113470).
• Oct 27. Chrome removes G3 from the Chrome Root Store.

Two things matter for servers. First, CRLSet and OneCRL protect browsers only. curl, Python requests, Go’s net/http, Node and the JVM don’t consult them, and by default they don’t check revocation at all. For those clients, the trust store is the control. Second, none of the eight intermediates appear in any root bundle I scanned (I checked for non-self-signed emSign certificates in all of them: zero). So the measurable question is whether the root they chain to is still trusted.

How I scanned without Docker

I didn’t want to depend on a local daemon or on docker run behaving the same everywhere, so the scanner talks to the registry directly:

1. Get an anonymous token and the manifest for linux/amd64.
2. Stream every layer, in order, as a gzip tar, without writing it to disk.
3. Keep the final state of each trust store path, honouring later layers and .wh. whiteout files.
4. Parse every certificate and check the subject for the four emSign roots.

Paths covered: the system PEM bundle (/etc/ssl/certs/ca-certificates.crt and the RHEL equivalents), any certifi/cacert.pem, and Java lib/security/cacerts (JKS and PKCS12, opened natively and cross-checked with a byte search). Every image was read from its registry on October 24, 2026.

The core of the layer walk, simplified (the real version also handles hardlinks and removes everything under a whited-out directory):

def stream_layers(reg, layers, files):
    for layer in layers:                              # oldest layer first
        r = reg.get("blobs/" + layer["digest"], stream=True)
        tf = tarfile.open(fileobj=r.raw, mode="r|*")  # stream, never touch disk
        for m in tf:
            name = posixpath.normpath("/" + m.name.removeprefix("./").lstrip("/"))
            d, b = posixpath.dirname(name), posixpath.basename(name)
            if b == ".wh..wh..opq":                   # opaque dir: drop everything below
                for k in [k for k in files if k.startswith(d + "/")]:
                    del files[k]
            elif b.startswith(".wh."):                # whiteout: file deleted in this layer
                files.pop(posixpath.join(d, b[4:]), None)
            elif m.issym() and want(name):            # bundles are often symlinks
                files[name] = ("link", m.linkname)
            elif m.isfile() and want(name):           # later layers overwrite earlier ones
                files[name] = ("file", tf.extractfile(m).read())

The results

Group	Images	What I found
Still ship G3 (10)	alpine:3.20, golang:1.24-alpine, amazonlinux:2023, amazoncorretto:21, eclipse-temurin:17-jre, eclipse-temurin:21-jre, distroless base-debian12, distroless static-debian12, python:3.12-slim, python:3.12-alpine	G3 present in the system bundle, Java cacerts, or pip’s certifi
Bundle present, G3 removed (10)	alpine:3.22, alpine:latest, redhat/ubi9-minimal, python:3.13, python:3.13-slim, node:22-alpine, nginx:stable, nginx:alpine, httpd:2.4, curlimages/curl	No G3 anywhere I looked
No CA bundle at all (10)	debian:12, debian:12-slim, debian:13-slim, ubuntu:22.04, ubuntu:24.04, node:20-slim, node:22-slim, node:24-slim, redis:7, postgres:16	Nothing to scan; you inherit whatever you install later

By store type: G3 was in 8 of 20 system bundles, 4 of 5 Java cacerts files, and 2 of 4 certifi copies.

Four results surprised me more than the headline number.

1. One image can give two answers. python:3.12-slim is built on Debian trixie, and its system bundle no longer has G3. But the image also ships pip, and pip vendors its own copy of certifi, which still has it. python:3.12-alpine is the same story on Alpine. The python:3.13 images are free of G3 in both places.

2. Java images carry a second trust store. Both Temurin images (Ubuntu 26.04 base, built October 17) have G3 in the Ubuntu bundle and in the JDK’s own cacerts. Updating ca-certificates in a Java image doesn’t change what the JVM trusts unless your image links the two, so check cacerts separately.

3. Recent build date doesn’t mean the fix is in. amazonlinux:2023 and amazoncorretto:21 were built on October 23, three days after NSS 3.131.1, and still carry all four emSign roots in both the PEM bundle and cacerts. The image is fresh; the package inside it isn’t yet.

4. Distros didn’t all remove the same roots. Patched Alpine (ca-certificates 20261020-r0) and UBI 9 keep only emSign Root CA - G1, which matches what NSS did. Debian trixie’s update (20250419+deb13u1) removed G3 but still has C1 and C3, which NSS also removed. That’s an observation from the scan, not something I found documented. The distroless Debian 12 images ship 20250419~deb12u1, the version from before the fix.

A note on the ten images with no bundle: that isn’t a pass. debian:12-slim and the node:*-slim images don’t include ca-certificates, so the moment your Dockerfile runs apt-get install ca-certificates, you get whatever version the mirror serves that day. If your build cache is older than October 20, it’s the old one.

Check your own images

Run this inside a container from your image, not the base image, since your own layers may add stores. Minimal images (Alpine, distroless) often have no openssl, keytool or even a shell; for those, docker create the image and docker cp the bundle files out, then check them on your machine:

# 1. Find every trust store
find / \( -name 'ca-certificates.crt' -o -name 'cacert.pem' -o -name 'ca-bundle.crt' -o -name cacerts \) 2>/dev/null

# 2. Check each PEM bundle for the root
openssl storeutl -noout -text -certs /etc/ssl/certs/ca-certificates.crt | grep 'emSign ECC Root CA - G3'

# 2b. Java keystores
keytool -list -cacerts -storepass changeit | grep -i emsign

# 3. Rebuild from fresh layers, then repeat 1 and 2
docker build --pull --no-cache -t myapp:ca-refresh .

Then fix what you find:

• System bundle: rebuild on a base that has the updated ca-certificates, or add apt-get update && apt-get install -y ca-certificates / apk upgrade ca-certificates in your own layer.
• certifi: upgrade it (pip install -U certifi; per Endor Labs and SafeDep, 2026.10.19 drops G3), and remember pip’s vendored copy only updates when pip does. Even better, point Python at the system store with SSL_CERT_FILE so there’s one bundle to maintain.
• Java: use a JDK build that includes the update, or remove the alias from cacerts with keytool -delete until your vendor ships one.

Limitations

• One point in time (October 24, 2026), linux/amd64 only. Tags move; rerun before you quote these numbers.
• I scanned the system bundle, certifi and Java cacerts. Node’s built-in store, Go binaries that embed their own roots, and anything an application vendors were out of scope.
• Distro package versions for the fixes come from secondary sources (Endor Labs, SafeDep) except where I saw them in an image. Exact distro release dates are unclear.
• Root presence is a proxy. It tells you a client would accept a chain to that root, not that anyone is attacking you.

The takeaway

The browser vendors handled this in hours, which is how it should work. The containers you build from are a different supply chain with a different clock, and they don’t update unless you rebuild them. A check in CI that greps every trust store in the image, not just the system bundle, would have flagged every red box in that grid. A scheduled --pull rebuild fixes most of them; for the few where the upstream tag still ships the root, the check tells you to patch the store in your own layer.

Which trust store in your stack do you think nobody has looked at since it was built?

Versi bahasa Indonesia: https://www.kielltampubolon.id/writing/saya-memindai-30-image-docker-populer-10-masih-percaya-root-ca-yang-dicabut-chrome/