[ INFRA_TOPOLOGY // #04 ]
MULTI-USER GPU CLUSTER & STORAGE LAB BLUEPRINT
RACK: 42U // ENCL-01 ACTIVE
Physical hardware blueprint and architecture layout for our multi-user deep learning compute fabric, TrueNAS SCALE enterprise ZFS storage pool, Kerberos-federated identity broker, and 100GbE RoCE v2 low-latency networking. Click on any rack unit below to inspect hardware telemetry and topology.
RACK_ID: TR-IST-RACK-07
42U EIA-310-D STANDARD
41-42U
ARISTA 7060X 32x100GbE QSFP28 [RoCE v2 Fabric]
● 100G
39-40UAIRFLOW BAFFLE // CABLE MANAGEMENT
33-38U
SUPERMICRO 4U // 2x EPYC 9654 // 8x NVIDIA A100 80GB SXM4
● 640GB VRAM
31-32UAIRFLOW BAFFLE // THERMAL BUFFER
25-30U
SUPERMICRO 4U // 2x EPYC 9354 // 4x RTX 4090 + 2x RTX 6000 Ada
● 192GB VRAM
23-24UPDU DUAL-CIRCUIT FEED // 3-PHASE 16A
17-22U
TRUENAS SCALE 4U // 24x 18TB EXOS SAS [RAIDZ2] // NVMe ZIL
● 432TB RAW
15-16UAIRFLOW BAFFLE // OPTICAL PATCH PANEL
13-14U
DELL R650 1U // AD/SSSD KDC // SLURM CONTROLLER // OPA
● PRIMARY
11-12U
DELL R650 1U // HIGH AVAILABILITY SECONDARY REPLICA
● STANDBY
01-10UAPC SYMMETRA 16kVA ON-LINE UPS MODULES
[ UNIT_INSPECTOR // TOR-SWITCH-01 ]
NETWORK_FABRIC
Arista 7060X 32-Port 100GbE QSFP28 Switch
High-performance spine-leaf switch providing RoCE v2 (RDMA over Converged Ethernet) lossless transmission for GPU inter-node communication and TrueNAS NFS-over-RDMA high-speed dataset access.
THROUGHPUT: 6.4 Tbps Wire-Speed
PORT CONFIG: 32x 100G QSFP28
LATENCY: 450ns Cut-Through
PFC / ECN: Priority Flow Control Enabled
LIVE SENSOR TELEMETRY:
INTERCONNECTED TIERS:
■
IDENTITY & KERBEROS STORAGE FEDERATION ARCHITECTURE
01 // Active Directory & SSSD Broker
Centralized user lifecycle management. Users authenticate via Active Directory Kerberos V5. SSSD on compute nodes provides deterministic POSIX UID/GID resolution, avoiding colliding ranges across multi-tenant GPU workspaces.
[domain/AD.LAB]
id_provider = ad
auth_provider = krb5
krb5_realm = AD.LAB.INTERNAL
ldap_id_mapping = False02 // TrueNAS SCALE & NFSv4.2 krb5p
Shared high-capacity datasets exported exclusively with sec=krb5p (RPC integrity + packet encryption). Eliminates standard NFS spoofing risks where rootless containers masquerade as other researchers' UIDs.
mount -t nfs4 -o proto=rdma,port=20049,sec=krb5p truenas.lab:/mnt/tank/datasets /mnt/datasets03 // Slurm GPU Scheduler & OPA Gates
Workload orchestration across A100 SXM4 and RTX 4090 pools. Enforces strict cgroup v2 GPU memory limits, MIG slicing (Multi-Instance GPU), and Open Policy Agent validation preventing unprivileged host root escalation.
srun --partition=a100 --gpus=2 --cpus-per-task=16 --mem=128G --pty bash