СШАSr. Server Hardware Systems Engineer (Starlink)
вчера
СШАSeniorOnsite
telemetryautomation
Responsible for managing the lifecycle and reliability of Starlink's global server fleet through automation and coordination with various stakeholders.
Обязанности
- Own the end-to-end hardware repair lifecycle for the Starlink server fleet and its components (CPUs, memory, SSDs, NICs, optics, and power supplies): triage alerts, isolate failures to the component level, and drive safe removal from production, repair, validation, and return to service
- Manage vendor support cases and coordinate on-site work with data center remote hands and field technicians, including parts delivery, site access, and return of failed parts for warranty replacement or failure analysis
- Build the software and automation behind that path: hardware telemetry and alerting (BMC and Redfish exporters, node-level metrics, log collection), removal of manual steps from diagnostics, production drain and return, alert silencing, and post-repair validation, and component-level tracking from detection to recovery
- Define, measure, and report fleet hardware metrics, including Mean Time to Respond, Repair, and Recover (p50 and p90), backlog trends, and component failure rates, and publish regular status updates to stakeholders.
- Write runbooks, methods of procedure (MOPs), and training material so NOC and field technicians can resolve common hardware incidents
- Validate new server hardware before it enters production, including burn-in, firmware, and configuration checks, and use fleet failure trends to drive server design, selection, qualification, and corrective action with vendors and hardware engineering
Требования
- Bachelor’s degree in computer science, information systems/IT, or an engineering discipline and 5+ years of professional experience in systems administration, site reliability engineering, or DevOps; OR 7+ years of professional experience in systems administration, site reliability engineering, or DevOps in lieu of a degree
- 3+ years of professional experience troubleshooting and repairing enterprise server hardware in a production environment
- 3+ years of professional experience administering Linux systems
- Must be willing to work extended hours and weekends as needed to support hardware around the world.
- Must be willing to participate in an on-call rotation for hardware issues.
- Must be willing to travel domestically and internationally to data centers and other Starlink sites as needed, and must hold or be able to obtain a valid passport.
- Must be able to work in data center environments for extended periods, including standing, kneeling, and working in loud, variable-temperature conditions.
- Must be able to lift, push, and pull 50lbs.
- To conform to U.S. Government export regulations, applicant must be a (i) U.S. citizen or national, (ii) U.S. lawful, permanent resident (aka green card holder), (iii) Refugee under 8 U.S.C. § 1157, or (iv) Asylee under 8 U.S.C. § 1158, or be eligible to obtain the required authorizations from the U.S. Department of State. Learn more about the ITAR here .
Будет плюсом
- 3+ years of experience operating a large server fleet in production data center environments
- Hands-on experience diagnosing and replacing server components such as CPUs, DIMMs, NVMe SSDs, NICs, transceivers, power supplies, and system boards
- Experience with out-of-band management and baseboard management controllers (e.g., HPE iLO, Dell iDRAC, OpenBMC) using Redfish and IPMI
- Proficiency with Linux hardware diagnostics, such as kernel and machine check logs, EDAC, ipmitool, smartctl, nvme-cli, ethtool, etc
- Software development experience in Python or Go, including building production tools, services, and automation
- Experience building monitoring and alerting with Prometheus, Grafana, and Alertmanager, including writing custom exporters
- Experience operating Kubernetes clusters, including node lifecycle tasks such as cordoning, draining, and replacing nodes
- Experience with server networking, including NICs, optics, cabling, and link-level troubleshooting
- Experience managing vendor support cases, RMAs, and warranty processes, and coordinating remote hands across multiple data centers and time zones
- Familiarity with firmware management, TPM enrollment, secure boot, UEFI, and server provisioning at scale
- Experience designing operational workflows in ticketing systems, and building dashboards for operational metrics such as MTTR and backlog size
- Experience with server hardware qualification, burn-in testing, or new product introduction
- Excellent written and verbal communication skills, with the ability to drive work to completion across teams, vendors, and time zones
- Ability to work in a fast-paced environment and manage multiple concurrent repairs across regions
- Experience writing scripts or software in at least one language, such as Python, Go, or Bash
Другое
- was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today is actively developing the technologies to make this possible, with the ultimate goal of enabling human life on Mars.
- Starlink delivers high-speed, low-latency internet to millions of people around the world, and the network depends on a global fleet of servers running in data centers and points of presence (PoPs). The Starlink Site Reliability Engineering (SRE) team is hiring a Server Hardware Systems Engineer to own the health of this fleet as it scales rapidly. You will own the server hardware lifecycle from issue detection, to repair, through recovery. You will build the telemetry, alerting, and automation that let a small team operate a very large fleet, and coordinate with network operations, field technicians, hardware vendors, and data center partners to get failed hardware repaired and back into service quickly. You will also use fleet reliability data to shape server design, selection, and qualification for new sites across both commercial-off-the-shelf (COTS) and custom-designed server platforms. The right engineer for this job is one who is automation-first. Our goal is to scale operations for server hardware through excellent tooling, and automated process. While this role is primarily located in our offices (not in data center facilities), you are welcome and encouraged to spend time in the field with Starlink data center engineering teams