Senior/Staff Kubernetes Infrastructure Engineer

2 нед. назад

180k–250k USD / yearLeadRemote

kuberneteslinuxslurmgpuinfrastructure automationnetworking

Build and manage high-performance Kubernetes infrastructure environments supporting generative media AI products at scale.

Design, automate, validate, and deliver the complete lifecycle of customer compute environments — from provisioning through upgrades, recovery, and decommissioning
Use AI aggressively to automate and accelerate every aspect of infrastructure delivery and operations
Provision dedicated Kubernetes and Slurm clusters tailored to customer workloads
Build and maintain Linux images and automated OS-provisioning workflows
Operate the NVIDIA GPU stack: drivers, GPU Operator, NVIDIA Container Toolkit, device plugins, MIG, and GPU monitoring
Design Kubernetes and data-center networking using Cilium/Calico, MetalLB, VLAN, VXLAN, BGP, and ECMP
Configure distributed and shared storage for high-performance workloads
Build monitoring, alerting, diagnostics, and automated recovery for customer environments
Develop reusable tooling, standards, documentation, and runbooks
Collaborate with customers and internal teams to translate workload requirements into sound infrastructure designs

5+ years of experience building and operating production Linux infrastructure
Strong production experience with Kubernetes on bare metal (bootstrapping, upgrades, HA control planes, etcd, containerd, CNI, CSI, ingress, load-balancing, observability, security, troubleshooting)
Experience with Linux virtualization: KVM/QEMU, libvirt, VFIO device passthrough
Experience operating NVIDIA GPUs on Linux and Kubernetes (drivers, container runtimes, device plugins, GPU Operator, GPU telemetry)
Strong networking fundamentals: TCP/IP, L2/L3, VLANs, routing, packet-level troubleshooting (tcpdump, Wireshark)
Practical scripting experience
Experience with configuration-management tools such as Ansible
Ability to diagnose complex, cross-layer infrastructure issues
Strong communication and ability to drive technical decisions across teams
Track record of moving quickly, taking ownership, and continuously improving systems

Production Slurm experience
High-performance networking: NVLink/NVSwitch, InfiniBand, RoCEv2, GPUDirect RDMA, NCCL, IMEX
Hugepages, NUMA, CPU pinning
SR-IOV, DPDK
Distributed storage: Ceph, Lustre, Weka
KubeVirt, OpenStack
IPsec, WireGuard, Tailscale
VXLAN, BGP, ECMP
Bare-metal management: BMC, IPMI, Redfish, PXE/iPXE, Kickstart, cloud-init
Network automation: NetBox, Nautobot, Nornir
AI training, inference, or distributed GPU workload infrastructure
Python or Go proficiency

fal is the generative media ecosystem powering the next generation of AI products. We build the infrastructure, tools, and model access that teams need to move from idea to production, and do it at scale without compromise. For developers and enterprises, fal is the foundation that makes generative media not just possible, but practical: a unified platform where high-performance inference, orchestration, and observability come together to unlock new categories of AI-native products.
As generative media reshapes industries across a market projected to grow by hundreds of billions over the next decade, fal is becoming the ecosystem that ambitious teams build on.
You will build the high-performance compute environments we deliver to customers. These environments span bare-metal servers, virtual machines with GPU passthrough, Kubernetes and Slurm clusters, distributed storage, and high-speed networking.
You will work across the infrastructure stack — from Linux images and hardware provisioning to cluster networking, GPU performance, observability, and lifecycle automation. The goal is to make every customer environment performant, reliable, isolated, and repeatable.