Network
Deep network troubleshooting — trace a request client to backend
Follow one failing request across DNS, routing, TCP, TLS, HTTP and app health — and read the distinct error each layer produces so you fix the right one.
Quick reference — reach for these first
dig +short app.shipreadymetrics.com AThe resolver's answer only. Empty = NXDOMAIN or no record. Compare resolvers with dig @RESOLVER.mtr -rwzbc 50 scm.ccPer-hop loss and latency (needs install + root). Loss that starts mid-path AND persists to the end is a real drop.nc -vz scm.cc 443Reachability of one port. 'succeeded' vs 'refused' (RST, port closed) vs timeout (filtered path).openssl s_client -connect scm.cc:443 -servername scm.ccFull handshake: cert chain, expiry, SNI, negotiated protocol. Append </dev/null to return.curl -sS -o /dev/null -w '%{time_namelookup} %{time_connect} %{time_appconnect} %{time_starttransfer} %{time_total}\n' https://scm.ccSplits the wall-clock into DNS / TCP / TLS / TTFB so you know which stage is slow.The request path and the error each stage produces
Text equivalent: A request crosses six stages left to right: 1 DNS resolve, 2 route/interface, 3 TCP connect, 4 TLS handshake, 5 HTTP load balancer, 6 app health. Each stage fails in its own recognizable way. DNS: NXDOMAIN or SERVFAIL. Routing: no route or unreachable. TCP: 'connection refused' is an RST meaning the port is closed, while a timeout means the path is filtered. TLS: a certificate, SNI, or protocol error. The load balancer returns 502 bad gateway when the upstream is unreachable; the app layer returns 504 or other 5xx when the backend is too slow or hung. Match the symptom to the stage before touching the app.
Common tasks
Task · Command · What to check
| Task | Command | What to check |
|---|---|---|
| Resolve a name | dig +short app.shipreadymetrics.com | Returns an A/AAAA address? Empty answer = NXDOMAIN or no such record. |
| Compare resolvers | dig @1.1.1.1 scm.cc +short; dig @8.8.8.8 scm.cc +short | Different answers = stale cache or split-horizon DNS, not the app. |
| Trace the delegation | dig +trace scm.cc | Which nameserver fails; distinguishes SERVFAIL (broken zone) from NXDOMAIN (no name). |
| Show the route to a host | ip route get 203.0.113.10 | Which interface and gateway the kernel picks; 'unreachable' = no route. |
| Per-hop loss and latency | mtr -rwzbc 50 scm.cc | Install mtr; needs root. Loss appearing mid-path and persisting to the last hop is real. |
| Test a single TCP port | nc -vz scm.cc 443 | 'succeeded' = open; 'refused' = RST (closed); hang/timeout = filtered. |
| Watch retransmits live | ss -tin 'dst scm.cc' | -i shows retrans, rtt, cwnd. Climbing retrans = packet loss on the path. |
| Probe the path MTU | ping -M do -s 1472 app.shipreadymetrics.com | 1472+28=1500. Fails with NO 'frag needed' reply = MTU black hole. macOS: ping -D -s. |
| IPv4 vs IPv6 | curl -4 https://scm.cc; curl -6 https://scm.cc | One works, the other hangs = a broken AAAA / IPv6 path. Check records and routing. |
| Capture the handshake | sudo tcpdump -ni any host scm.cc and port 443 | SYN [S] going out with no SYN-ACK [S.] back = dropped/filtered, not the app. |
| Inspect the TLS session | openssl s_client -connect scm.cc:443 -servername scm.cc </dev/null | Verify-return-code 0, cert not expired, SNI served, protocol matches. |
| Time every phase | curl -w '@curl-format.txt' -o /dev/null -sS https://scm.cc | DNS vs connect vs appconnect(TLS) vs starttransfer(TTFB) isolates the stage. |
| Check the backend directly | curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/healthz | 502 at the LB but 200 here = the LB→app hop, not the app itself. |
HTTPS works locally but times out from another network
INCIDENT A
curl -sS -m 10 https://app.shipreadymetrics.com/healthz
# curl: (28) Connection timed out after 10001 msdig +short app.shipreadymetrics.com
# 203.0.113.10 (identical to the working network → not DNS)nc -vz 203.0.113.10 443
# nc: connect to 203.0.113.10 port 443 (tcp) timed out: Operation now in progresssudo tcpdump -ni any host 203.0.113.10 and port 443
# IP ... > 203.0.113.10.443: Flags [S] ... (repeats, no [S.] reply)Small requests succeed but large transfers stall (path-MTU black hole)
INCIDENT B
curl -sS -o /dev/null -w '%{http_code} %{size_download}\n' https://app.shipreadymetrics.com/report.csv
# (hangs ~20s) curl: (28) Operation too slow ... 0 bytes receivedcurl -sS -o /dev/null -w '%{time_appconnect} %{time_starttransfer}\n' https://scm.cc/report.csv
# 0.11 (appconnect OK; then stalls before first byte of bulk data)ping -M do -s 1472 app.shipreadymetrics.com # then -s 1473
# 1472: 64 bytes ... ok · 1473: (no reply at all — not even 'frag needed')ping -M do -s 1372 app.shipreadymetrics.com
# ok → path MTU ≈ 1400Troubleshooting — symptom → first check → next check
| Symptom | First check | Establishes | Next check |
|---|---|---|---|
| NXDOMAIN / SERVFAIL | dig +short HOST; dig +trace HOST | Name doesn't exist vs resolver/zone broken | Confirm the record exists; compare @1.1.1.1 against the internal resolver |
| "Connection refused" | nc -vz HOST PORT (gets an RST) | Host reachable, but nothing listening on that port | Confirm the service is up and bound to the right address:port |
| "Connection timed out" | nc -vz HOST PORT hangs; tcpdump shows SYN, no SYN-ACK | Packet filtered/dropped on the path | Firewall / security group / routing — not the app |
| TLS handshake error | openssl s_client -connect HOST:443 -servername HOST | Expiry / chain / SNI / protocol mismatch | Fix cert or SNI; test the intended host with curl --resolve |
| 502 Bad Gateway | curl -s -o /dev/null -w '%{http_code}' to the backend directly | LB is up but the upstream is unreachable/crashed | Check the app process and the LB→app port/health check |
| 504 Gateway Timeout | curl -w '%{time_total}' against the backend | Upstream reachable but too slow / hung | Backend latency, connection-pool limits, proxy read timeouts |
| Large transfers stall, small ones fine | ping -M do -s 1472 HOST, then sweep upward | Path-MTU black hole (frag-needed ICMP filtered) | Clamp MSS or lower the interface MTU to the real path MTU |
| Every stage passes | curl -w full timing to the real endpoint | Request path is healthy end-to-end | Save the timing as a baseline to diff against next incident |
References
- dig — ISC BIND 9 manualbind9.readthedocs.io
- ss(8) — iproute2 socket statisticsman7.org
- ping(8) — iputils (DF bit, -M/-s)man7.org
- openssl-s_client — TLS client diagnosticsdocs.openssl.org
- curl --write-out variables (timing)curl.se
Linux with iproute2 (ip, ss) and GNU/iputils ping, curl ≥ 7.75, OpenSSL ≥ 3. mtr and traceroute usually need installing and raw-socket privileges (root/suid). macOS/BSD differ: ping uses -D (not -M do) to set the DF bit, and ss is replaced by netstat/lsof. app.shipreadymetrics.com and scm.cc are example hosts; replace them. Placeholders in CAPS.