LLM Pre-training & Distributed Engineer (AI Infrastructure)

4 мес. назад

USASenior

pytorchdeepspeedmegatron-lminfiniandrdmaslurmkubernetesc++

Engineer responsible for orchestrating large-scale LLM pre-training runs and optimizing distributed AI infrastructure.

Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
Automate checkpointing and failure recovery during month-long training runs.

We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing  distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.
Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
Experience managing SLURM or Kubernetes-based GPU clusters.
Strong systems engineering background (C++, CUDA, Python).