[ FIELD_RUNBOOKS // #05 ]

EMERGENCY PRODUCTION RUNBOOKS & TRIAGE CHEATSHEETS

TRIAGE_ACTIVE // 6 STANDARD OPERATING PROCEDURES

High-pressure incident response checklists, recovery one-liners, and debugging triage procedures. Verified against production Linux kernels, TrueNAS OpenZFS pools, Kerberos identity brokers, and multi-tenant GPU fabrics.

> grep -E

P0-CRITICAL SOP-SEC-01

[CONTAINER_SECURITY]

Docker Authz Plugin (OPA) Fail-Open Bypass Containment

Mitigate silent authorization bypass when Docker daemon fails to forward Content-Length 0 requests to OPA authorization plugins. Enforce strict fail-close daemon configuration.

EMERGENCY MITIGATION COMMANDS:

# 1. Audit active docker authz plugins
docker info --format '{{json .Plugins.Authorization}}'

# 2. Block direct docker socket access immediately
chmod 660 /var/run/docker.sock
systemctl restart docker-authz-opa

# 3. Patch /etc/docker/daemon.json for fail-close
cat <<'EOF' > /etc/docker/daemon.json
{
  "authorization-plugins": ["opa-docker-authz:latest"],
  "userland-proxy": false,
  "live-restore": true
}
EOF
systemctl reload docker

P1-HIGH SOP-STOR-02

[TRUENAS_NFSv4]

NFSv4.2 Stale File Handle (ESTALE 116) & Kerberos Expiry Flush

Recover hung mounts caused by server-side inode replacement, ZFS snapshot rollbacks, or expired Kerberos user session TGTs without rebooting client compute nodes.

LAZY UNMOUNT & KEYTAB REFRESH:

# 1. Flush hung NFS client cache and unmount lazily
umount -l /mnt/datasets

# 2. Refresh machine host Kerberos keytab
kinit -k -t /etc/krb5.keytab host/$(hostname -f)

# 3. Invalidate kernel RPC idmap cache
nfsidmap -c

# 4. Remount with explicit Kerberos encryption
mount -t nfs4 -o proto=rdma,port=20049,sec=krb5p,timeo=600,retrans=2 truenas.lab:/mnt/tank/datasets /mnt/datasets

P1-HIGH SOP-AUTH-03

[ACTIVE_DIRECTORY]

SSSD / Active Directory Cache Invalidation & Offline Purge

Resolve stale POSIX UID/GID mappings, group membership desynchronization, and LDAP offline loop locks across cluster nodes after Active Directory schema edits.

SSSD PURGE & NSS REFRESH:

# 1. Stop SSSD daemon cleanly
systemctl stop sssd

# 2. Invalidate memory DB caches
sss_cache -E

# 3. Purge corrupted local LDB database files
rm -f /var/lib/sss/db/* /var/lib/sss/mc/*

# 4. Restart SSSD and verify deterministic POSIX resolution
systemctl start sssd
getent passwd operator_ob
id -a operator_ob

P0-CRITICAL SOP-ZFS-04

[OPENZFS_RECOVERY]

ZFS Corrupted Pool Import & Scrub Panic Recovery

Emergency procedure when a ZFS pool triggers a kernel panic or refusal to import due to metadata corruption, damaged uberblocks, or power loss in mid-transaction.

READ-ONLY RECOVERY IMPORT:

# 1. Search for existing pools without importing
zpool import

# 2. Force read-only import, rewinding damaged transactions
zpool import -f -F -o readonly=on tank

# 3. If standard rewind fails, perform extreme transaction rollback
zpool import -f -F -X -o readonly=on tank

# 4. Immediately stream critical datasets to remote replication target
zfs send -v tank/critical@snapshot | zfs receive backup_tank/critical

P2-OPERATIONAL SOP-EBPF-05

[LINUX_KERNEL]

Production eBPF Kernel Tracing & Latency Diagnostics

Zero-overhead live kernel tracing one-liners. Capture short-lived rogue processes, TCP packet drops, and NVMe block I/O tail latency bottlenecks without restarting daemons.

EBPF BPFTRACE ONE-LINERS:

# 1. Trace all process executions in real time with parent PID
bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s (pid=%d, ppid=%d): %s\n", comm, pid, curtask->parent->pid, str(args->filename)); }'

# 2. Profile NVMe / ZFS disk I/O latency distribution (Microseconds)
biolatency-bpfcc -m 10 1

# 3. Intercept TCP retransmissions with source/dest IP and ports
tcpretrans-bpfcc -l

# 4. Detect permission denied access attempts across kernel VFS
bpftrace -e 'kretprobe:vfs_open /retval == -13/ { printf("EACCES: pid=%d comm=%s\n", pid, comm); }'

P1-HIGH SOP-GPU-06

[CUDA_ACCELERATION]

NVIDIA GPU Hang, Xid Error & PCIe AER Recovery (No Node Reboot)

Recover unresponsive NVIDIA A100 / RTX 4090 GPUs after CUDA memory illegal address faults, Xid 31/79 crashes, or PCIe bus link drops without rebooting the 192-core host node.

SURGICAL GPU RESET SEQUENCE:

# 1. Identify offending process locking GPU memory
fuser -v /dev/nvidia*
kill -9 $(fuser /dev/nvidia0 2>/dev/null)

# 2. Stop NVIDIA persistence daemon & Slurm compute agent
systemctl stop slurmd
systemctl stop nvidia-persistenced

# 3. Trigger hardware PCIe secondary bus reset on GPU index 0
nvidia-smi --gpu-reset -i 0

# 4. Verify GPU responds to NVML telemetry queries
nvidia-smi -i 0 --query-gpu=temperature.gpu,utilization.gpu,memory.used --format=csv
systemctl start nvidia-persistenced
systemctl start slurmd