Бразилия[Job-31837] Master Data Developer - Data Ops, Brazil
2 дн. назад
БразилияSeniorRemote
sql serversnowflakecicdmonitoring
Master DataOps Engineer responsible for environment provisioning and platform readiness in a finance data platform modernization.
О компании
- At CI&T, our rapid growth is fueled by the innovative solutions we create for our global clients. We are seeking a Master DataOps Engineer to own the operational backbone — environments, CI/CD, and monitoring — for a large-scale Finance data platform modernization at a major US mortgage lender, so the platform being built is reliable and cost-governed from day one, not bolted on at the end.
Требования
- Solid, hands-on experience in DataOps or platform engineering roles supporting a production cloud data platform end-to-end — covering provisioning, CI/CD, monitoring, and operational handoff.
- Strong CI/CD experience for data pipelines: Azure DevOps Pipelines (YAML), Git-based workflows with branch policies and approval gates, automated testing and regression gates, and multi-environment promotion paths.
- Azure infrastructure fluency: Virtual Networks, NSGs, Private Link and Private Endpoints, Azure Key Vault (secrets, certificates, and rotation), Microsoft Entra ID (service principals, managed identities, PIM, and access reviews), Azure Monitor and Log Analytics, and Azure Cost Management.
- Infrastructure as Code: hands-on Terraform experience including remote state management on Azure Blob Storage, the Snowflake Terraform provider, and IaC security scanning integrated into CI pipelines.
- Snowflake platform experience: RBAC design, warehouse sizing and segregation, resource monitors, row-access policies, and dynamic data masking — experience across Enterprise or Business Critical editions preferred.
- CDC replication pipeline monitoring and operations: experience monitoring and alerting on a CDC-based replication pipeline in production, including lag monitoring, restart and recovery procedures, and schema-drift detection at the landing layer.
- Secrets management discipline: experience managing pipeline credentials via Azure Key Vault with no plaintext credentials in repositories or pipeline definitions, and documented rotation procedures.
- Security-aware development practices: experience integrating code scanning, dependency scanning, and IaC scanning into CI pipelines; familiarity with least-privilege access design and audit logging requirements in regulated or sensitive-data environments.
- Platform cost governance as a named responsibility: experience owning FinOps attribution, budget alerts, and right-sizing reviews — not as a shared or ambient concern.
- Fluent English (B2+): this role is the named client-facing point of contact for all platform operations topics across delivery and client teams.
Будет плюсом
- Qlik Replicate: hands-on configuration of full-load and CDC replication — source and target endpoint setup, table scoping, schema-evolution handling, throttling and change-window management, latency monitoring, and restart and recovery procedures.
- Azure SQL Server / SQL Server on Azure VM: familiarity with CDC prerequisites (SQL Server Agent, capture jobs, and service account permissions), connectivity requirements for private Azure environments, and coordination with DBA teams on change windows and backup schedules.
- Microsoft Fabric / OneLake: operational experience with capacity management, landing-zone health, and monitoring in a Fabric environment.
- Azure ExpressRoute and hybrid networking: experience designing or operating the connectivity layer between on-premises SQL Server instances and cloud-hosted data infrastructure.
- Microsoft Entra ID Privileged Identity Management (PIM): experience implementing just-in-time access with approval workflows and audit trails for vendor and contractor access in a client environment.
- Microsoft Dynamics 365 Business Central / SQL Server source systems: familiarity with Business Central table structures, company-prefixed naming conventions, and schema-drift risk from vendor-managed upgrades.
- Power BI semantic model operations: familiarity with workspace publication, scheduled refresh configuration, and gateway management in a governed enterprise deployment.
- Production runbook authorship and knowledge transfer facilitation: demonstrated experience writing runbooks that a client's operations team can execute without vendor support, and leading structured knowledge transfer with competency-evidence checkpoints.
- Mortgage, lending, or financial-services environments: experience operating under strict change-control discipline with sensitive financial data and audit-trail requirements covering every access and deployment event.
- dbt or dbt-style versioned SQL transformation frameworks: experience operating dbt in a CI/CD pipeline — running tests, managing model promotion, and debugging failed runs in a Snowflake environment.
Другое
- At CI&T, we help large enterprises transform the potential of AI into real business impact with AI Deployment, AI-native execution, and tech-integrated business solutions.
- With 30 years of experience in technological transformation, we accelerate innovation with expertise in Agentic SDLC, Application modernization, Data & AI, Martech and Business strategy.
- We are 8,000 CI&Ters across more than 25 countries, collaborating to build solutions with real impact. AI is already part of how we work, evolve, and innovate every day.
- Own end-to-end environment provisioning and platform readiness across the full data stack — from SQL Server CDC source through Snowflake landing, raw, and curated layers — ahead of and during the build, with a formal readiness checklist delivered before sensitive-data build begins.
- Provision and maintain Snowflake environments (DEV → QA → UAT → Prod) with dedicated, right-sized warehouses per workload class — replication/ingestion, curated-layer transformation, and BI serving — each with auto-suspend, resource monitors, and cost attribution from day one.
- Coordinate Azure infrastructure provisioning: Virtual Networks, NSGs, Private Endpoints, Azure Key Vault, self-hosted Azure DevOps agents, and VM infrastructure for the CDC replication tool — including firewall rules, TLS certificate requests, and DNS resolution for private Snowflake endpoints, all with documented lead times surfaced proactively.
- Manage Snowflake RBAC provisioning: dedicated project-specific role hierarchy, clearly separated from broad IT, developer, analyst, and AI roles; positive and negative access tests executed before the environment is handed over to the build team.
- Manage the full vendor and service account access lifecycle: named, individual, time-bound, least-privilege access for all delivery personnel; dedicated non-personal service accounts for replication, Snowflake, and BI integrations; credential storage and rotation via Azure Key Vault; revocation with documented evidence at engagement end.
- Build and operate the CI/CD pipeline for all data artifacts — transformation logic, metric definitions, replication configurations, Snowflake object DDL, and versioned SQL transformation models — using Azure DevOps Pipelines with YAML-based, version-controlled pipeline definitions.
- Enforce a DEV → QA → UAT → Prod promotion path with branch policies, required reviewers, build validation on PRs, and written approval gates before any production deployment — no silent or automatic production changes.
- Integrate automated regression, schema validation, row-count checks, and data quality assertions at every promotion stage — not only before go-live.
- Externalize all environment-specific configuration from code; integrate Azure DevOps Variable Groups linked to Azure Key Vault so no credentials are stored as plaintext in pipeline definitions or repository files.
- Integrate code scanning, dependency scanning, and IaC scanning (e.g., Checkov, Microsoft Defender for DevOps) into CI pipelines; remediate or formally document exceptions for all material findings before promotion to production.
- Design, document, and rehearse rollback procedures for every promotion stage, including a full mock cutover and rollback rehearsal before the production readiness gate.
- Implement and own end-to-end pipeline monitoring using Azure Monitor and Log Analytics: replication health, CDC lag alerts, replication agent availability, Snowflake warehouse freshness against agreed SLA targets, schema-change and drift detection events at the landing zone, and semantic-definition change alerts.
- Configure Azure Monitor Alert Rules routed via Action Groups to the client's approved ITSM tool (e.g., ServiceNow), with named alert owners, defined thresholds, and documented escalation paths — every alert fully operable by the client's DataOps team without vendor involvement post-handoff.
- Enable and maintain Diagnostic Settings on all Azure resources (Key Vault, VMs, NSGs, Storage Accounts) feeding a centralized Log Analytics Workspace with appropriate retention, access control, and audit trails covering authentication, access grants, configuration changes, and deployments.
- Implement Snowflake resource monitors and Azure Cost Management budget alerts for cost anomaly detection; produce periodic FinOps attribution reports by workload class.
- Own platform cost attribution and consumption monitoring across all Snowflake warehouses and Azure compute resources, with per-workload attribution established from day one using a consistent resource tagging strategy enforced via IaC.
- Deliver a Snowflake and Azure cost model at architecture gate, including monitored budgets, right-sizing recommendations, and auto-suspend configurations — not estimated retrospectively.
- Conduct periodic right-sizing reviews using Azure Advisor and Snowflake usage data; present findings in structured FinOps reviews with the client's Cloud Architecture team.
- Manage all Azure infrastructure as code using Terraform (or Bicep where appropriate), including VNets, NSGs, Key Vault, VMs, storage accounts, Entra ID service principals, and RBAC role assignments — no manual portal configurations for anything that belongs in the production baseline.
- Manage Snowflake object provisioning (databases, schemas, warehouses, roles, resource monitors) via the Snowflake Terraform provider, maintaining remote state in Azure Blob Storage with state locking.
- Enforce configuration standards via Azure Policy guardrails (e.g., soft-delete on Key Vault, no public IPs on data resources, mandatory resource tagging) so IaC deployments are validated against policy before they reach production.
- Maintain Architecture Decision Records (ADRs) and other living delivery artifacts alongside the Lead Architect — including network and connectivity design, environment topology, CI/CD promotion model, and security access design — in client-controlled repositories from the first commit.
- Write and validate operational runbooks covering: replication failure, recovery, and resynchronization; CDC lag breach response; deployment and rollback procedures; reconciliation-discrepancy triage; access request and revocation workflows; cost-anomaly response; and incident routing via client ITSM — each exercised with the client's DataOps team before final handoff acceptance.
- Drive progressive knowledge transfer from the architecture phase onward through paired working sessions, shadowed deployments, and runbook exercises — not a single end-of-project handoff.
- Act as the named point of contact for platform operations across the delivery team and the client's DataOps, Cloud Architecture, Infrastructure/Network, and Information Security teams — translating technical platform concerns into clear client-facing communications in fluent English.
- Quantify and track client infrastructure dependencies — Snowflake environments, Azure network paths, service accounts, replication tool licensing, and production change windows — with named need-by dates and schedule-impact statements, escalating proactively when readiness is at risk.