Senior System Architect, Infrastructure Reliability job opportunity at NVIDIA.

Home Jobs In United States Of America Tag: Infrastructure Reliability Senior System Architect, Infrastructure Reliability position at NVIDIA


DatePosted 13 Days Ago bot
NVIDIA Senior System Architect, Infrastructure Reliability
Experience: 6-years
Pattern: full-time
Country:
apply Apply Now
Salary:
Status:

Infrastructure Reliability

Copy Link Report
degreeMBA
loacation US, CA, Santa Clara, United States Of America
loacation US, CA, Santa ..........United States Of America

NVIDIA is seeking a Senior System Architect: Heterogeneous EDA Systems to solve a complex challenge in accelerated computing: Failure Attribution at Scale. As EDA or equivalent experience workloads scale across thousands of heterogeneous nodes, a single failure can cause massive resource waste. We need an engineer to develop and build an automated framework. This framework will ingest telemetry from CPU and GPU clusters to identify the root cause of job failures in real-time. It will distinguish between hardware faults, infrastructure instability, and software defects. What you'll be doing: Architect Failure Attribution Frameworks: Build a scalable "flight recorder" for EDA jobs that captures high-fidelity state across the CPU, GPU, and Fabric at the moment of failure. Build automated diagnostics that correlate GPU XID errors, PCIe bus failures, and CUDA memory exceptions. Connect these errors with system-level events such as OOM kills or NUMA-related hangs. Distributed Logging & Tracing: Implement low-overhead tracing mechanisms (using tracing tools or custom agents) that provide access to job execution across multi-node Slurm or Kubernetes clusters. Root Cause Automation: Develop heuristics and models based on machine learning to classify failures as "Hardware Fault," "Software Bug," or "Environment Issue." This reduces the Mean Time to Identify (MTTI) for R&D teams. Resiliency Engineering: Work closely with hardware and infrastructure teams to define "signals of impending failure," enabling proactive job migration or check-pointing before a crash occurs. What we need to see: Distributed Systems Mastery: BS, MS, or PhD in Computer Science or Electrical Engineering (or equivalent experience) with 6+ years in systems programming. Experience building automated RCA (Root Cause Analysis) pipelines for HPC or cloud-scale environments. CPU Architecture Deep-Dive: Expert knowledge of x86/ARM node-level metrics: IPC (Instructions Per Cycle), cache contention, NUMA imbalance, and hardware interrupts. Programming Proficiency: Strong C++ and Python skills, with the ability to build high-performance daemons that monitor system health without impacting workload performance. Scale Experience: Familiarity with cluster resource managers (Slurm, LSF, or Kubernetes) and how they manage job lifecycle and signal propagation. Ways To Stand Out From The Crowd: Low-Level Diagnostics: Expert knowledge of the Linux kernel and its error-reporting interfaces (/dev/mcelog, dmesg, journald). Understand how the kernel handles hardware exceptions and memory faults. GPU Infrastructure Proficiency: Deep experience with the NVIDIA DCGM (Data Center GPU Manager) and NVIDIA Management Library (NVML) for monitoring device health and capturing state-dumps. Experience with tools doing non-intrusive monitoring of application health and syscall-level failure patterns. Experience with checkpoint/restore technologies (like CRIU) and their application in long-running EDA flows. #LI-Hybrid  Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5. You will also be eligible for equity and benefits . Applications for this job will be accepted at least until March 1, 2026. This posting is for an existing vacancy.  NVIDIA uses AI tools in its recruiting processes. NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Other Ai Matches

Senior Software Engineer, Observability Applicants are expected to have a solid experience in handling Observability related tasks
Manager, SWQA Test Development Applicants are expected to have a solid experience in handling SWQA Test Development related tasks
Research Scientist, ML Systems - PhD New College Grad 2026 Applicants are expected to have a solid experience in handling ML Systems - PhD New College Grad 2026 related tasks
Creative Director - APAC Applicants are expected to have a solid experience in handling Job related tasks
Senior Design for Debug Architect and Methodology Engineer Applicants are expected to have a solid experience in handling Job related tasks
Senior DL Algorithms Engineer - Cosmos Applicants are expected to have a solid experience in handling Job related tasks
Senior SWQA Test Developer - Embedded Applicants are expected to have a solid experience in handling Job related tasks
Senior Research Scientist, Multi-Modal Language Models Applicants are expected to have a solid experience in handling Multi-Modal Language Models related tasks
ATE Test Engineer Applicants are expected to have a solid experience in handling Job related tasks
Senior ASIC Verification Engineer, Coherent High Speed Interconnect Applicants are expected to have a solid experience in handling Coherent High Speed Interconnect related tasks
Senior Design Verification Engineer - Interconnect IP Applicants are expected to have a solid experience in handling Job related tasks
remote-jobserver Remote
Senior Software Engineer - Dynamic Storage Applicants are expected to have a solid experience in handling Job related tasks
Senior ASIC Verification Engineer – Global IP Applicants are expected to have a solid experience in handling Job related tasks
Senior Power Methodology and Modeling Engineer Applicants are expected to have a solid experience in handling Job related tasks
PR Specialist Applicants are expected to have a solid experience in handling Job related tasks
Senior Compiler Engineer, AI Inference Platforms Applicants are expected to have a solid experience in handling AI Inference Platforms related tasks
Senior Production Factory Planner Applicants are expected to have a solid experience in handling Job related tasks
Manager, GPU Accelerated Data Analytics Applicants are expected to have a solid experience in handling GPU Accelerated Data Analytics related tasks
Inventory Control Planner Applicants are expected to have a solid experience in handling Job related tasks
Senior Startups Inception Partner Manager Applicants are expected to have a solid experience in handling Job related tasks
Senior Physical Design Engineer Applicants are expected to have a solid experience in handling Job related tasks
Senior Deep Learning Software Engineer, PyTorch - TensorRT Performance Applicants are expected to have a solid experience in handling PyTorch - TensorRT Performance related tasks
Network Site Reliability Engineer Applicants are expected to have a solid experience in handling Job related tasks