Компания не указана
Devops engineer (Data)
3 нед. назад
Страна неизвестнаSeniorRemoteB2
sparkairflowdbtawspythonbashansiblegitlab ci/cd
Looking for a DevOps Engineer (Data Focus) to join Data Engineering team building scalable data infrastructure for trading research and production systems.
Responsibilities
- Design, implement, and maintain DevOps tooling for data pipelines and data platforms.
- Support and automate ETL workflows (e.g. Spark, Airflow).
- Build and manage monitoring and observability for data quality (DBT), data pipelines, and storage systems.
- Implement and maintain data storage infrastructure on AWS (S3, EC2, networking).
- Integrate and support Project Nessie as a transactional catalog for Data Lakes with Git-like semantics.
- Work closely with Data Engineers to ensure smooth deployment, versioning, and scaling of data jobs and pipelines.
- Automate deployment and CI/CD processes with GitLab CI/CD.
- Ensure security best practices for sensitive data handling.
- Write infrastructure automation using Python/Bash/Ansible.
- Document systems, pipelines, and DevOps processes.
Requirements
- 5+ years in a DevOps / IT Ops / DataOps role, with hands-on experience in data-related infrastructure.
- Strong Linux knowledge (networking, kernel tuning, performance optimization).
- Hands-on experience with Airflow, ClickHouse, DBT, Spark, and Project Nessie.
- Solid knowledge of AWS services (S3, EC2, IAM, networking).
- Proficiency in scripting and automation (Python, Bash, Ansible).
- Experience with CI/CD pipelines (GitLab).
- Experience with monitoring stacks (Prometheus, Grafana, ELK).
- Familiarity with containerization and orchestration (Docker, Kubernetes).
- Understanding of data governance concepts.
- Strong communication skills and a proactive, ownership-driven mindset.
Nice to have
- Experience with real-time data streaming.
- Knowledge of infrastructure as code (Terraform).
- Familiarity with security in data engineering (IAM, encryption, access control).
- Background in high-performance systems or trading environments.