Staff/Senior Software DevOps Engineer

1 мес. назад

CanadaWorldwideLeadOnsite

cibuildtestingsimulationhardware-in-the-loop

Lead DevOps engineer responsible for build, test, and CI flows for AI hardware software stack.

Обязанности

We're searching for a Staff/Senior Software DevOps Engineer to own the build, test, and CI flows that the entire DX-1 software stack — compiler, runtime, simulator, and framework integration — depends on. Ours is a large test suite that asserts token-exact correctness against golden references, and it has to run across scarce, expensive resources that span both simulation compute and hardware-in-the-loop testing — simulator and emulator boxes alongside DX-1 and prototype-platform boards. Your mission is to keep that system fast, trustworthy, observable, and affordable as the test suite, the team, and the resource pool all grow.
This is a build-and-test- role, not product-serving SRE. You'll work where CI, the runner fleet, and the test hardware meet, partnering closely with the infrastructure, compiler, runtime, simulator, and modelling teams. At Senior/Staff level your impact is the velocity of every engineer who depends on this system — how fast they get a trustworthy signal, how rarely they wait on a machine or a flaky run, and how much they can self-serve without coming to you. That leverage — the standards, platforms, and shared-resource model others build on — is what we're hiring for, far more than any single system you ship.
Own the Build & Test Pipelines: Design, build, and own CI pipelines and test-execution — PR, merge, and nightly lanes — that gates the entire software stack, balancing fast feedback against coverage and cost.
Scale Test Execution: Split a large, slow suite into staged lanes, parallelize it with real test isolation, and cache aggressively with content-addressed keys, so feedback stays fast and cheap as the suite and the team grow rather than getting solved by throwing machines at it.
Manage the Fleet & Scarce Resources: Run CI across a heterogeneous fleet of cloud and self-hosted machines, and give the team fair, monitored, fail-fast shared access to scarce and expensive hardware — keeping it reliable, well-utilised, and never a silent bottleneck.
Build the Performance & Readiness Signal: Stand up perf-regression baselines the team trusts (pinned hardware, rolling baselines, sound metric aggregation, determinism), and turn CI and test signal into CI-health and product-readiness dashboards that drive real decisions.
Own Software Observability: Choose the metrics store that scales to many series on daily runs with long-lived history, making dashboards for observable software.
Set Standards: Define the flows that keep builds and test runs hermetic and reproducible; and make the system fail closed and contain the blast radius when something is misconfigured or a job is untrusted.

Будет плюсом

GitHub Actions or comparable CI at scale; scaling CI runner fleets on cloud infrastructure (e.g. AWS); hardware-in-the-loop or lab automation for custom silicon or FPGA bring-up; time-series and observability stacks (Prometheus/Grafana, Datadog, columnar warehouses).
Adjacent depth is welcome: HPC / cluster batch scheduling, release engineering, or developer-productivity platforms.

Условия

Competitive Salary: Commensurate with your experience, skills, and location
Equity & Ownership: Meaningful stock options. You’re not just joining the mission; you’re owning a piece of it
Proximity Bonus: We value your time. To minimise your commute and maximise your life, we offer an annual Living-Local Bonus if your residence is within 20 minutes of the office
Retirement Benefits: Employer-contributed retirement plans to help you build long-term financial security.
Due to U.S. export control regulations, candidates’ eligibility to work at depends on their most recent citizenship or permanent residency status. We are generally unable to consider applicants whose most recent citizenship or permanent residence is in certain restricted countries (currently including Iran, North Korea, Syria, Cuba, Russia, Belarus, China, Hong Kong, Macau, and Venezuela). Applicants who have subsequently obtained citizenship or permanent residency in another country not subject to these restrictions may still be eligible.

Другое

AI is growing faster than any technology in history and the explosion in demand has created a massive infrastructure gap; we can no longer build chips or power stations fast enough to keep up. The industry is still leaning on a ten-year-old hardware blueprint that has reached its limit. A new paradigm that is faster and more efficient will be the biggest economic opportunity of the next century and create the most important company of the next decade. The Decode Accelerator 1 (DX-1) is the first accelerator architected specifically for decode. Rack-scale co-design of logic, data movement, packaging, optics and interconnect enables a step change in system level performance.
Experience build/test infrastructure, CI/CD, developer-productivity, or large-scale systems / release engineering, with demonstrated end-to-end ownership of a big test or CI system.
Scaling a large test suite: staged lanes, parallelism with real isolation, content-addressed caching, and keeping feedback fast and affordable as the suite grows.
Fleet and scarce-resource management: heterogeneous CI runner fleets across cloud and on-prem (VMs, containers, and bare-metal hardware), and giving a team shared, monitored access to scarce or expensive resources — custom accelerators, FPGA/prototype rigs, lab hardware, or contended compute — including reservations, remote access, hardware-in-the-loop testing, and artifact portability across heterogeneous hosts.
Performance-analysis support: trustworthy regression baselines, determinism and noise handling, sound metric aggregation (geometric vs arithmetic mean vs median), and fast attribution and bisection.
Observability and metrics platforms: CI-health and readiness dashboards, a metrics store that scales to long-lived, high-cardinality time series, and self-serve access for engineers.
Strong scripting and systems programming (e.g. Python plus a systems language), and fluency with containers, Linux, and cloud infrastructure (AWS or similar).
A reproducible, safe-by-default mindset: hermetic builds, least privilege, fail-closed defaults, and blast-radius control.
Excellent communication and the ability to align and influence cross-functional teams (compiler, runtime, modelling) without relying on formal authority.
Bachelor's degree or higher in computer science, electrical engineering, mathematics, or a related field.