Data Smart Solutions
Middle Data Engineer
6 дн. назад
600–800 USDСтрана неизвестнаMiddleRemote
databrickspythonllmneo4jdelta tablesazure
Middle Data Engineer to work on clinical-research data pipelines using Databricks, Python, LLM, and Neo4j with remote work format.
Responsibilities
- Extracting and structuring data from large collections of clinical-research PDFs
- Building bronze, silver, and gold layers in Databricks with Python notebooks and Delta tables
- Integrating LLM endpoints into data pipelines and monitoring token spend
- Building a Neo4j graph alongside Delta tables for agents to query
- Preparing gold-layer data for an agentic layer used as an API tool
- Consolidating scattered and redundant Databricks job pipelines
Requirements
- Strong hands-on experience with Databricks, especially Databricks Jobs
- Advanced Python skills, writing notebooks daily from ingestion to the gold layer
- Solid understanding of medallion architecture and how to define bronze / silver / gold boundaries
- Experience working with Delta tables
- Experience processing large volumes of PDF documents (100K+ files)
- Experience calling LLM endpoints from notebooks and tracking token usage and cost
- Experience with Azure cloud
- Ability to design and consolidate job orchestration pipelines
Nice to have
- Experience with Neo4j or other graph databases
- Experience with healthcare or clinical-research data
- Experience building data layers consumed by AI agents