Trust Wallet
Senior Data Engineer
1 нед. назад
Страна неизвестнаSeniorRemote
databrickssparkpythondbtsqlawsterraformdocker
Senior Data Engineer role at Trust Wallet focusing on building and maintaining scalable data infrastructure and pipelines using Databricks and Spark in a fully remote global format.
Обязанности
- Architect and maintain robust, scalable and secure data infrastructure on Databricks, covering both streaming and batch workloads.
- Design, develop and maintain data pipelines, primarily in Python and Spark, to automate ingestion and transformation across internal systems, external providers and on-chain sources.
- Design and maintain dimensional data models and transformation layers in dbt, with tests and documented contracts, so metrics are consistent and reusable across the company.
- Oversee the data lake and lakehouse layers, ensuring efficient storage, effective partitioning, high data quality, and monitoring and alerting that surfaces issues early.
- Integrate Databricks with a wide range of data sources, including change data capture from operational databases, third-party APIs and blockchain data, and adapt data flows to specific business needs.
- Optimise pipelines and storage for performance, reliability and cost efficiency at scale.
- Apply best practices for governance, security and compliance in cloud and Databricks environments, including access control, encryption and monitoring.
- Work closely with platform engineers, data analysts and other stakeholders to understand data requirements, and document infrastructure, models and best practices.
Требования
- 3+ years as a Data Engineer, with hands-on ownership of production pipelines and lakehouse or data warehouse architecture.
- Strong practical experience with Databricks, Delta Lake and Spark, including both streaming and batch processing, and the judgement to choose between them.
- Solid understanding of dimensional modelling, slowly changing dimensions and data warehouse design, with the ability to define and defend the grain of the models you build.
- Experience with dbt or an equivalent framework for managing transformations, testing and lineage.
- Strong cloud fundamentals, including identity and access management, object storage, networking and infrastructure as code.
- Comfortable writing production transformations as well as services and connectors in Python and SQL.
- Experience implementing testing, monitoring and governance practices in cloud data environments.
Будет плюсом
- Experience with change data capture and replication from operational databases
- Familiarity with blockchain or on-chain data
- Infrastructure as code, ideally Terraform, and CI/CD for data workloads
- Experience with BI and semantic layers such as Holistics, Looker or similar
- Containerisation and orchestration (Docker, Kubernetes)
Условия
- Полная занятость
- Формат работы: удалённо (Remote - Global)