TLS 1.3 Handshake Analysis with OpenSSL and Wireshark

TLS 1.3 dramatically simplified the handshake and removed legacy cryptography, but it also changed how you debug and inspect secure traffic. If you want to defend modern networks, you need to understand what the handshake looks like on the wire, how key shares work, and how to interpret the encrypted handshake messages in Wireshark.

This walkthrough uses OpenSSL to generate traffic and Wireshark to analyze it. The goal is to make the message flow and cryptographic context concrete so you can reason about failed handshakes, downgrade attempts, and certificate issues.

Build a local TLS 1.3 server

Start with a local TLS server so you can capture traffic without legal or privacy issues. OpenSSL makes this trivial.

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 3 -nodes -subj "/CN=lab.local"
openssl s_server -cert cert.pem -key key.pem -accept 8443 -www -tls1_3

In another terminal, connect with s_client and enable message tracing.

openssl s_client -connect 127.0.0.1:8443 -tls1_3 -msg -state

The -msg output shows the handshake messages in order. Keep this output as a reference when you inspect the capture in Wireshark.

Capture and decode in Wireshark

Start a capture on your loopback or lab interface. Apply this display filter to isolate the handshake:

tls && tcp.port == 8443

In TLS 1.3, only the ClientHello and ServerHello are in clear text. Everything after that is encrypted, including the certificate and handshake finished messages. This is normal and one of the major privacy improvements.

Anatomy of the TLS 1.3 handshake

The minimal flow looks like this:

1. ClientHello with key share and supported groups
2. ServerHello with selected cipher and key share
3. EncryptedExtensions
4. Certificate
5. CertificateVerify
6. Finished
7. Application Data

In Wireshark you will see the encrypted records after ServerHello. That does not mean the handshake failed. It means the keys are already derived and the rest is protected.

Key share and cipher selection

Open the ClientHello and inspect the Key Share extension. You should see a group like x25519 with a public key. The server will reply with the selected group in its ServerHello.

The cipher suite will typically be TLS_AES_128_GCM_SHA256 or TLS_CHACHA20_POLY1305_SHA256. These are AEAD ciphers with HKDF. There are no RSA key exchanges in TLS 1.3.

Decrypting with SSLKEYLOGFILE

If you need to decrypt traffic for analysis, use the SSLKEYLOGFILE environment variable and a TLS client that supports it. OpenSSL can do this on some builds, but browsers are usually easier. For lab analysis, curl with OpenSSL 3 can export keys.

export SSLKEYLOGFILE=/tmp/sslkeys.log
curl -k https://127.0.0.1:8443/

In Wireshark, set the key log file under Preferences -> Protocols -> TLS. Once loaded, Wireshark can decrypt the handshake and application data, which makes it easier to compare with the OpenSSL output.

Resumption and 0-RTT notes

TLS 1.3 uses PSK-based resumption. The server sends a NewSessionTicket after the handshake, and the client uses it on a later connection to skip most of the handshake.

You can observe this by making two sequential connections and comparing the messages. The second connection should show a shorter handshake and may include early data if enabled. In most security-sensitive environments, 0-RTT is disabled due to replay risk.

Extension inspection and SNI

The ClientHello contains extensions that are valuable for defenders. The Server Name Indication (SNI) extension tells you which hostname the client is trying to reach. Application-Layer Protocol Negotiation (ALPN) advertises the protocols the client supports, such as h2 for HTTP/2 or http/1.1.

In Wireshark, expand the ClientHello and review the extensions. This is where you can confirm whether a client is using a normal browser stack or a custom tool. Many malware families use non-standard ALPN values or omit extensions, which can be a useful detection signal even when you cannot decrypt the traffic.

JA3 and JA4 fingerprints

JA3 fingerprints hash the ordered list of TLS ClientHello attributes, producing a stable client fingerprint. JA4 is a newer variant that is more resilient to small changes. These fingerprints are useful because they work even when the traffic is encrypted.

In a lab, capture a few connections from different clients and compute JA3 or JA4 values. If you see an unknown fingerprint on a host that typically uses a browser, that is a sign to investigate. Combine this with DNS and HTTP metadata to reduce false positives.

Certificate chain validation

TLS 1.3 still relies on the certificate chain for identity. The chain is encrypted on the wire, but the server logs and client validation errors will tell you what is wrong. Common problems include missing intermediate certificates, expired leaf certs, and mismatched Subject Alternative Names (SANs).

In a lab, deliberately use an invalid certificate and observe the errors in s_client. The output will show which part of the chain failed and why. This is a practical way to learn how chain building works without needing to intercept real traffic.

Common debugging cases

• Handshake failure: Check for incompatible cipher suites or missing key share groups. The ServerHello will indicate the selected group if it succeeds.
• Certificate errors: The certificate is encrypted, so you must use a key log file or server-side logs to diagnose.
• Downgrade attempts: TLS 1.3 embeds a downgrade sentinel in the ServerHello random if the server negotiates TLS 1.2 or lower.

Security implications

TLS 1.3 removes static RSA and weak curves. It improves forward secrecy and reduces handshake latency. For defenders, it means you rely more on metadata like SNI and JA3 fingerprints, and less on passive decryption. You should plan for that shift in your monitoring strategy.

Lab checklist

Use this short checklist to confirm your lab workflow:

• Capture a TLS 1.3 handshake and verify SNI and ALPN in the ClientHello.
• Confirm the negotiated cipher suite in both OpenSSL output and Wireshark.
• Export keys with SSLKEYLOGFILE and decrypt a small capture.
• Test session resumption and verify the NewSessionTicket behavior.

Takeaways

If you can read a TLS 1.3 handshake, you can debug most secure traffic issues in a lab. Use OpenSSL to create deterministic traffic, Wireshark to inspect the message flow, and key logs when you need decryption. This skill translates directly to incident response and network defense, where TLS misconfigurations and failed handshakes are common early indicators.