X.509 Certificate Chains: Path Building vs. Path Validation

Certificate chaining is two processes most writeups blur: path building assembles a candidate, path validation checks it. Built and broken with OpenSSL.

A certificate is a signed claim: this public key belongs to this name, says the issuer. A certificate chain is a sequence of those claims, each one vouching for the key that signed the one before it, ending at a claim you accept without proof. That last claim is the trust anchor, and the most useful thing to know about it is that it’s an input to the process, not a participant in it.

The second most useful thing to know is that “chaining” is two separate processes that most writeups blur together. Path building assembles a candidate sequence of certificates. Path validation checks whether that sequence is acceptable. They live in different documents (RFC 4158 for building, RFC 5280 section 6 for validation), they fail differently, and the most common chain error in production, the missing intermediate, is a building failure that gets misdiagnosed as a validation failure. This post builds a three-certificate chain with OpenSSL, breaks it both ways, and ends with the 2021 incident where a chain with two valid paths lost one of them.

Figure 1: everything in this post, on one diagram. Three field-level linkages run through a chain: each certificate’s issuer DN equals its issuer’s subject DN (blue), the issuer’s public key verifies the child’s signatureValue (green), and AKI matches the issuer’s SKI (gray, dashed for a reason covered later). The trust anchor sits in the verifier’s store, outside the path. The certificates and key identifiers are real; we build them next.

Commands and output are from OpenSSL 3.0.13. The flags are stable across 3.x; -addext behaves differently on 1.1.1 and earlier, so check man openssl-req if you’re on an older build.

Three certificates

A minimal private PKI: a self-signed root, an intermediate that does the day-to-day issuing, and a leaf for a client. EC keys on P-256 to keep the output small.

# Root CA: self-signed, CA:TRUE
secdev@testbench:~/blog/cert-chaining$ openssl ecparam -name prime256v1 -genkey -noout -out root.key
secdev@testbench:~/blog/cert-chaining$ openssl req -x509 -new -key root.key -sha256 -days 3650 \
  -subj "/O=Example Lab/CN=Example Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out root.pem

The intermediate gets pathlen:0, which means it may issue end-entity certificates but no further CAs below it:

secdev@testbench:~/blog/cert-chaining$ cat > int.ext <<'EOF'
basicConstraints=critical,CA:TRUE,pathlen:0
keyUsage=critical,keyCertSign,cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

secdev@testbench:~/blog/cert-chaining$ openssl ecparam -name prime256v1 -genkey -noout -out int.key
secdev@testbench:~/blog/cert-chaining$ openssl req -new -key int.key \
  -subj "/O=Example Lab/CN=Example Lab Issuing CA" -out int.csr
secdev@testbench:~/blog/cert-chaining$ openssl x509 -req -in int.csr -CA root.pem -CAkey root.key \
  -CAcreateserial -days 1825 -sha256 -extfile int.ext -out int.pem

The leaf is an ordinary client certificate, here for an example telematics gateway.

secdev@testbench:~/blog/cert-chaining$ cat > leaf.ext <<'EOF'
basicConstraints=critical,CA:FALSE
keyUsage=critical,digitalSignature
extendedKeyUsage=clientAuth
subjectAltName=DNS:telematics-gw.example.internal
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always
EOF

secdev@testbench:~/blog/cert-chaining$ openssl ecparam -name prime256v1 -genkey -noout -out leaf.key
secdev@testbench:~/blog/cert-chaining$ openssl req -new -key leaf.key \
  -subj "/O=Example Lab/CN=telematics-gw.example.internal" -out leaf.csr
openssl x509 -req -in leaf.csr -CA int.pem -CAkey int.key \
  -CAcreateserial -days 365 -sha256 -extfile leaf.ext -out leaf.pem

The chain linkage is name-based. Each certificate’s issuer field holds the subject DN of the certificate that signed it:

secdev@testbench:~/blog/cert-chaining$ for c in leaf int root; do openssl x509 -in $c.pem -noout -subject -issuer; done

subject=O = Example Lab, CN = telematics-gw.example.internal
issuer=O = Example Lab, CN = Example Lab Issuing CA
subject=O = Example Lab, CN = Example Lab Issuing CA
issuer=O = Example Lab, CN = Example Lab Root CA
subject=O = Example Lab, CN = Example Lab Root CA
issuer=O = Example Lab, CN = Example Lab Root CA

The root certificate is self-signed. So it is its own subject and issuer as shown above.

There’s a second, key-based linkage running alongside the names. The leaf’s Authority Key Identifier matches the intermediate’s Subject Key Identifier:

$ secdev@testbench:~/blog/cert-chaining$ openssl x509 -in leaf.pem -noout -ext authorityKeyIdentifier
X509v3 Authority Key Identifier:
    59:B5:B2:0B:78:7D:3B:D1:88:2A:E6:AB:2E:D2:CC:35:53:CF:57:D4
$ secdev@testbench:~/blog/cert-chaining$ openssl x509 -in int.pem -noout -ext subjectKeyIdentifier
X509v3 Subject Key Identifier:
    59:B5:B2:0B:78:7D:3B:D1:88:2A:E6:AB:2E:D2:CC:35:53:CF:57:D4

Hold that thought. What that AKI/SKI pair is actually for surprises most people, and it’s in the gotchas section.

Two processes, two failure modes

Verify the leaf against the root, supplying the intermediate as untrusted material:

$ secdev@testbench:~/blog/cert-chaining$ openssl verify -CAfile root.pem -untrusted int.pem leaf.pem
leaf.pem: OK

The flag name is doing real work here. -untrusted is how OpenSSL describes intermediates generally: they’re candidate building material, not trusted input. Trust attaches only to what’s behind -CAfile.

Now drop the intermediate:

$ secdev@testbench:~/blog/cert-chaining$ openssl verify -CAfile root.pem leaf.pem
O = Example Lab, CN = telematics-gw.example.internal
error 20 at 0 depth lookup: unable to get local issuer certificate
error leaf.pem: verification failed

Error 20 is not a validation failure. No signature was rejected, no expiry tripped, no constraint was violated. The path builder searched for a certificate whose subject is “Example Lab Issuing CA,” found nothing, and gave up. In TLS deployments this is overwhelmingly the “server forgot to send the intermediate” bug: the operator installed the leaf, the chain works in their browser (which has other ways to find intermediates, more on that below), and an API client or embedded device fails with exactly this error.

Path building, briefly

RFC 4158 defines a certification path as an ordered list running from a certificate signed by a trust anchor down to the target certificate, and path building as the process of assembling it. The RFC also explains why building gets so little attention compared to validation: “the means used to find the path does not affect its validation.” A valid path is valid no matter how you found it. The catch is that you have to find it.

The builder starts at the target and works upward. For each certificate, it looks for candidates whose subject DN equals the current certificate’s issuer DN, using AKI/SKI to pick among candidates when several certificates share a subject (re-issued CAs, cross-signs). Candidates come from whatever the peer sent, from local stores, and on desktop platforms from fetching the URL in the Authority Information Access extension (so-called AIA chasing). Browsers do that fetch; embedded TLS stacks generally don’t, which is one reason a chain can work in Chrome and fail on a device. Because one subject and key can be certified by multiple issuers, the candidate space is a graph, not a list, and the builder is doing a graph search.

TLS itself only half-helps. RFC 8446 section 4.4.2 requires the sender’s certificate to come first in the Certificate message, but each following certificate only SHOULD directly certify the one before it. In TLS 1.2 that was a MUST. TLS 1.3 also tells implementations to tolerate out-of-order and extraneous certificates, and permits omitting the trust anchor. The practical consequence: a client that walks the wire order instead of doing real path building will choke on chains that are spec-tolerated and common in the wild, like servers that send both a current and a deprecated intermediate during a transition.

Path validation (RFC 5280 section 6)

Validation takes a finished candidate path plus three other inputs: the trust anchor information, the current time, and policy controls. The anchor is not part of the path being validated. Its self-signature gets no meaningful check; a root certificate is best understood as a packaging format for a name and a public key you’ve decided to trust. That’s also why omitting the root from a TLS chain is fine: the relying party must already have it, or the whole exercise is pointless.

The algorithm then walks the path from the anchor toward the target, carrying a working public key that starts as the anchor’s key. For each certificate it checks, roughly:

The signature verifies under the working public key.
The current time falls inside the validity window.
The certificate isn’t revoked, per whatever revocation policy is in force.
The issuer name equals the working issuer name (the previous subject).
Name constraints and policy constraints hold.
For any certificate that isn’t the target: basicConstraints asserts cA, keyUsage asserts keyCertSign if present, and pathLenConstraint hasn’t been exceeded.
Every critical extension the implementation doesn’t recognize is fatal.

Then the working public key becomes this certificate’s key, and the loop continues. Step 6 is where our intermediate’s pathlen:0 bites: if it ever signed another CA certificate, paths through that CA would fail here regardless of signatures. The cA and keyCertSign bits form a consistency pair (each one’s presence constrains the other), and implementations disagree about the inconsistent combinations; the x509-limbo test corpus has cases for all of them and is worth an afternoon.

When a chain has two paths

Cross-signing means one subject and key certified by two different issuers, which gives the builder two paths to choose from. The textbook case ran in production on September 30, 2021.

Let’s Encrypt’s root, ISRG Root X1, was cross-signed by an older root, DST Root CA X3, so that Let’s Encrypt certificates would be trusted on devices that predated ISRG Root X1’s inclusion in trust stores. A Let’s Encrypt leaf therefore had two valid paths: a short one ending at ISRG Root X1 in the local trust store, and a long one extending through the cross-sign to DST Root CA X3. On that date the DST root expired, and the long path died.

Two distinct populations failed. Devices that had never received ISRG Root X1 (no software updates, frozen trust stores) lost the only path they had. That one’s unsurprising. The interesting failures were clients that did have ISRG Root X1 in their store but whose path builder insisted on extending through the cross-sign to the expired root anyway. OpenSSL 1.0.2 behaved this way, with no fix available on the client side; OpenSSL’s own postmortem covers the server-side workarounds. (I haven’t traced exactly why 1.0.2’s builder prefers the longer path.) Trust was present. Path building chose the dead branch. The two-process distinction isn’t academic; it determined who broke that day.

Gotchas

The AKI/SKI matching from earlier is a building hint, not a validation rule. RFC 5280 says outright that applications don’t have to check key identifier matches during path validation; the binding that validation enforces is the name chain plus the signatures. Plenty of writeups present AKI/SKI as the mechanism that links a chain. It isn’t. It’s an index that helps the builder pick candidates when names collide.

Error messages tell you which process failed, if you let them. “Unable to get local issuer certificate” is the builder coming up empty. Expiry, bad signatures, and constraint violations are the validator rejecting a path the builder found. The fixes are different: building failures are fixed by supplying certificates (configure the server’s chain file, add to the local store); validation failures are fixed by replacing them.

Behavior with multiple candidate paths is implementation-defined in practice. RFC 4158 is informational and offers guidance, not requirements, so when one path is expired and another is fine, what happens depends on which library and version you’re running. Test with the stacks your clients actually use, not just your desktop browser.

Revocation got one line in the validation loop above and deserves a post of its own. CRLs, OCSP, and what browsers actually do instead are a separate swamp, deliberately skipped here.

The takeaway

When a system separates “find a candidate” from “check the candidate,” diagnose failures by first identifying which phase you’re in. Certificate chains make the distinction unusually crisp: a graph search over issuer names, then a rule check over the result. Most chain debugging time I’ve seen wasted, including my own, went to staring at validity dates and key usage bits when the actual problem was that the builder never had the intermediate in hand.