[ AUDIT_ARCHIVE // #06 ]

INCIDENT POST-MORTEM & ROOT CAUSE ANALYSIS LOG

ALL_INCIDENTS: RESOLVED // ZERO_DATA_LOSS

Transparent, blameless post-mortem dossiers for major production disruptions, kernel panics, and security anomalies across our GPU computing fabric and storage clusters. Complete with minute-by-minute timelines, root cause mechanics, and preventative sysctl/architecture remediations.

> grep -i

P0-CRITICAL INC-2026-08

[STORAGE_OUTAGE]

NFSv4.2 RPC Socket Starvation During Distributed PyTorch 8x A100 Checkpoint Flush

MTTD (Tespiti Süresi): 12 MIN

MTTR (Kurtarma Süresi): 38 MIN

IMPACT: 640GB VRAM PyTorch DDP Hang

STATUS: RESOLVED // CAPA APPLIED

[ 01 // INCIDENT SUMMARY ]

During a 70B parameter distributed LLM checkpointing run across 8x NVIDIA A100 GPUs, compute nodes attempted to simultaneously burst write ~480 GB of unbuffered tensor weights to TrueNAS SCALE over NFSv4.2 with sec=krb5p. Client-side kernel TCP RPC slots were instantly saturated, triggering cascading ESTALE and RPC timeout cascades.

[ 02 // MINUTE-BY-MINUTE TIMELINE (UTC) ]

14:02:11 Slurm job #8924 initiates epoch 14 checkpoint flush (480 GB total payload).
14:03:40 TrueNAS disk write queue depth exceeds 256; NFS RPC latency spikes from 0.8ms to 14,200ms.
14:05:12 Kernel emits sunrpc: task 40182 on broken transport on compute nodes.
14:14:00 NOC on-call engineer paged via automated Slurm GPU non-responsive watchdog.
14:26:30 Emergency sysctl socket queue expansion applied live across compute cluster.
14:40:00 NFS client RPC slots recovered; checkpoint committed safely without data corruption.

[ 03 // ROOT CAUSE & REMEDIATION (CAPA) ]

Root Cause: Default Linux kernel sunrpc.tcp_slot_table_entries was pegged at 16, creating severe head-of-line blocking under parallel multi-threaded synchronous I/O.

# Corrective Kernel Parameter applied across all cluster compute nodes:
sysctl -w sunrpc.tcp_slot_table_entries=128
sysctl -w sunrpc.tcp_max_slot_table_entries=128

# TrueNAS ZFS dirty data throttle tuned to avoid RPC burst stalls:
echo 17179869184 > /sys/module/zfs/parameters/zfs_dirty_data_max_max

P1-HIGH INC-2026-05

[IDENTITY_FABRIC]

Active Directory Kerberos Clock Skew (KRB_AP_ERR_SKEW) Cluster Login Lockout

MTTD: 5 MIN

MTTR: 18 MIN

IMPACT: SSH & Slurm Submission Lockout

STATUS: RESOLVED // CHRONY PTP ENFORCED

[ 01 // INCIDENT SUMMARY ]

Following a hypervisor hardware clock desync, secondary compute node hyperthreads drifted 318 seconds ahead of the primary Active Directory Domain Controller. MIT Kerberos strictly rejected all ticket requests with KRB_AP_ERR_SKEW (Clock skew too great).

[ 02 // ROOT CAUSE & REMEDIATION (CAPA) ]

# Replaced standard systemd-timesyncd with Chrony PTP hardware synchronization:
chronyc makestep 1.0 3
chronyc sources -v

# Configured /etc/chrony/chrony.conf with local GPS/PTP stratum-1 boundary clock:
server time.lab.internal iburst minpoll 2 maxpoll 4 maxdelay 0.05

P0-CRITICAL INC-2026-02

[CONTAINER_SECURITY]

Docker Authz Plugin Silent Fail-Open Incident (CVE-2024-41110 Triage)

MTTD: 4 MIN

MTTR: 22 MIN

IMPACT: Attempted Rootless Privilege Escalation

STATUS: CONTAINED // PROXY INJECTED

[ 01 // INCIDENT SUMMARY ]

Automated eBPF audit probe intercepted an anomalous container spawn payload with Content-Length: 0 aimed at bypassing our Open Policy Agent gatekeeper. Docker daemon forwarded an empty body to OPA, allowing privileged access.

[ 02 // ROOT CAUSE & REMEDIATION (CAPA) ]

# Upgraded Docker Engine to 27.1.1+ with fail-close enforcement:
apt-get install -y docker-ce=5:27.1.1-1~ubuntu.22.04~jammy

# Verified OPA proxy interceptor blocks Content-Length 0 requests unconditionally:
iptables -A INPUT -p tcp --dport 2375 -m string --string "Content-Length: 0" --algo bm -j REJECT