СШАSenior Manager, Production & Fleet Operations
6 дн. назад
СШАSeniorHybrid
Lead day-to-day operational management of AI production infrastructure, ensuring fleet health, coordinating responses, and managing maintenance.
О компании
- Operate the production fleet safely, reliably, and consistently - providing continuous visibility into infrastructure health, rapid restoration when failures occur, and an increasingly standardized and automated operating model as the fleet scales.
Обязанности
- is building and operating some of the world's most advanced AI infrastructure. As our infrastructure footprint expands across cloud, colocation facilities, and customer environments, we need a scalable operating model that provides continuous visibility into production infrastructure and ensures issues are rapidly identified, prioritized, and resolved.
- We are seeking a Senior Manager, Production & Fleet Operations to lead the day-to-day operational management of -managed production infrastructure. Reporting to the Director of Central Operations, this leader will establish and operate the mechanisms required to understand fleet health, coordinate production response, manage maintenance and repair activities, and ensure infrastructure is safely and efficiently returned to service when failures occur.
- This role will work closely with SiteOps/DC Ops, Reliability & Incident Management, Service Ops & Enablement, Global Service Logistics & Inventory, Build & Deploy, and Engineering.
- has clear fleet-level visibility across production infrastructure.
- Production alerts have defined severity, ownership, response, and escalation expectations.
- Operational procedures are consistent across sites and clusters.
- Time to detect, acknowledge, diagnose, and restore failures improves.
- SiteOps dispatch and intervention workflows are clearly defined and measurable.
- Repair/RMA workflows have clear ownership and predictable turnaround.
- Routine production issues increasingly resolve without Engineering intervention.
- Manual operational toil decreases as automation increases.
- The number of systems/clusters supported per Operations FTE improves as the fleet grows.
Требования
- 8+ years of experience in infrastructure operations, cloud operations, fleet operations, SRE, HPC operations, data center operations, or similar technical environments.
- 3+ years leading technical operations teams.
- Experience operating complex, highly available production infrastructure.
- Strong understanding of monitoring, telemetry, incident response, maintenance, hardware troubleshooting, and service restoration.
- Experience coordinating work between centralized operations and onsite technical teams.
- Strong infrastructure troubleshooting and operational judgment.
- Demonstrated ability to establish repeatable processes in rapidly evolving environments.
- Strong cross-functional leadership and communication skills.
- Demonstrated focus on automation and reduction of operational toil.
Будет плюсом
- AI/HPC, large-scale compute, accelerator, cloud, or distributed infrastructure experience.
- Experience with hardware-intensive production environments.
- Experience building or operating NOC/Fleet Operations capabilities.
- Experience managing hardware repair/RMA processes.
- Experience operating geographically distributed infrastructure.
Условия
- People who are serious about software make their own hardware. At , we have built a breakthrough architecture that is unlocking new opportunities for the AI industry. With dozens of model releases and rapid growth, we’ve reached an inflection point in our business. Members of our team tell us there are five main reasons they joined :
- Build a breakthrough AI platform beyond the constraints of the GPU.
- Publish and open source their cutting-edge AI research.
- Work on one of the fastest AI supercomputers in the world.
- Enjoy job stability with startup vitality.
- Our simple, non-corporate work culture that respects individual beliefs.
- Find out more about what it's like to work at here !
- Apply today and become part of the forefront of groundbreaking advancements in AI!
- This website or its third-party tools process personal data. For more details, click here to review our CCPA disclosure notice.
Другое
- Systems builds the world's largest AI chip, 56 times larger than GPUs. This architecture allows to deliver industry-leading training and inference speeds; over 10 times faster than GPU-based hyperscale cloud inference services. This order of magnitude increase in speed is transforming the user experience of AI applications, unlocking real-time iteration and increasing intelligence via additional agentic computation. works with the leading model labs, global enterprises, and cutting-edge AI-native startups. OpenAI recently announced a multi-year partnership with , to deploy 750 megawatts of scale, transforming key workloads with ultra high-speed inference.
- Own day-to-day operational health of -managed production infrastructure.
- Establish consistent operational processes across clusters, sites, and operating environments.
- Maintain clear visibility into production state, degradation, outages, maintenance, and infrastructure availability.
- Establish fleet monitoring, alert response, triage, escalation, and restoration processes.
- Develop appropriate 24x7 operational coverage as fleet requirements evolve.
- Establish clear ownership of production issues from detection through restoration.
- Develop and mature the Fleet Operations/NOC capability.
- Establish standards for monitoring, telemetry, alerting, dashboards, and fleet-health reporting.
- Develop shift, handoff, escalation, and on-call operating procedures.
- Ensure operators have the tooling, runbooks, procedures, and access required to safely operate production infrastructure.
- Partner with Service Ops & Enablement to develop operator competency and training requirements.
- Drive standardization across geographically distributed infrastructure.
- Determine when physical intervention is required.
- Dispatch and prioritize SiteOps/DC Ops activities based on production impact.
- Provide clear diagnostic information and execution procedures.
- Coordinate troubleshooting between centralized Operations and onsite technicians.
- Validate successful recovery following physical intervention.
- Authorize return-to-service from an operational perspective.
- Analyze recurring physical interventions and identify opportunities for improved diagnostics, procedures, serviceability, or automation.
- Establish and manage the operational workflow for failed systems and components.
- Coordinate diagnosis, replacement, repair, and RMA activities.
- Establish clear disposition paths for failed assets.
- Partner with Global Service Logistics & Inventory to ensure replacement assets and spares are available when required.
- Maintain visibility into repair status and repair-loop performance.
- Identify recurring repair patterns and feed them into Reliability & Incident Management and Engineering.
- Reduce time from failure detection through restored production capacity.
- Establish operational governance for maintenance activities affecting production infrastructure.
- Coordinate planned maintenance across Central Ops, SiteOps, customers, Engineering, and infrastructure partners.
- Ensure production changes have appropriate validation, execution, and rollback procedures.
- Maintain clear maintenance windows and communication mechanisms.
- Measure and reduce production impact associated with planned maintenance.
- Identify high-frequency and high-toil operational activities suitable for automation.
- Partner with Operations Strategy, Automation & Performance to prioritize automation opportunities.
- Provide operational requirements and acceptance criteria to Software/Engineering teams implementing automation.
- Increase automated detection, diagnosis, remediation, and validation where appropriate.
- Reduce unnecessary Engineering involvement in repeatable operational activities.
- We are looking for someone who owns the production outcome, remains calm under pressure, thinks at fleet rather than individual-system scale, builds repeatable mechanisms instead of relying on heroics, and constantly looks for opportunities to simplify and automate operations.
- When infrastructure enters production, we need to know its state, know who owns every issue, restore failures rapidly, and make operating the next hundred systems easier than operating the first hundred.