Job Title: Sr. Data Platform Engineer
Job Type: Contract
Job Location: Hybrid
Overview
The Data Analytics Platform spans the full data lifecycle on Azure. Batch and file-based sources flow through a multi-zone Delta Lake medallion lakehouse on ADLS Gen2, processed with Azure Synapse Analytics. A real-time tier on Azure Data Explorer handles streaming and sensor telemetry. Azure SQL Database holds orchestration metadata and operational state, master data management provides enterprise golden records, containerized Python jobs on Azure Container Apps handle API extraction, and Logic Apps handle lightweight intake and notification workflows. Power BI delivers reporting tiered by data velocity: high-velocity real-time dashboards from Azure Data Explorer, near-real-time views from Azure SQL, and analytical reporting from curated serverless SQL marts in the lakehouse. Access is governed by a domain-based classification model aligned to government data practices requirements, and all code deploys through Azure DevOps CI/CD.
Every ingestion pipeline runs through a metadata-driven orchestration framework: a central metadata database that drives source configuration, scheduling, watermarking, logging, and layer promotion through shared Synapse notebook toolboxes. New work is done inside this framework, not alongside it.
Responsibilities:
• Source onboarding: Build ingestion for new data sources (REST APIs, SFTP drops, network file shares, vendor exports, sensor and telemetry feeds) through the metadata-driven orchestration framework, landing raw data and promoting it through the medallion zones with PySpark notebooks and Synapse pipelines.
• Curated data products: Develop serverless SQL views and Delta Lake tables in the curated consumption layer that apply business rules and serve analytics, reporting, and downstream feeds, following the platform's schema-per-mart and access-control conventions.
• Extraction jobs: Develop and maintain containerized Python jobs on Azure Container Apps for API extraction workloads, including managed identity authentication, structured logging to the orchestration database, and YAML-based CI/CD deployment.
• Troubleshooting and defect resolution: Diagnose and resolve pipeline failures, data quality issues, and performance problems across the platform, tracing issues through orchestration metadata, Spark logs, and source systems to root cause.
• Framework improvement: Extend and refactor shared toolbox notebooks and framework components when the backlog exposes gaps.
Requirements:
• 5+ years of data engineering experience with your own code running in production, including substantial hands-on work in Azure.
• Production proficiency with Azure Synapse Analytics: Spark notebooks, pipeline development, triggers, and serverless SQL pools that you have built and operated in a live environment. Note that Databricks and Microsoft Fabric are not used in this environment; native Synapse experience matters.
• PySpark and T-SQL at the depth that comes from regular production use: performance tuning, window functions, and incremental load patterns.
• Every item requires sustained, hands-on production work. That is code the candidate personally designed, built, deployed, and supported in a live environment over time, not coursework, certifications alone, proof-of-concept exposure, or brief contact with a technology on someone else's project.
• Evaluation will include a detailed technical discussion of production work the candidate has personally delivered (design decisions, failure modes encountered, and how solutions held up under real data and real change) and a hands-on technical exercise using provided materials. Candidates will not be asked to show code or other proprietary artifacts belonging to prior employers or clients.
Preferred Qualifications:
• Experience with Azure Data Explorer (KQL, ingestion mappings, continuous export) or other real-time analytics stores.
• Experience working in a metadata-driven ingestion framework, where pipeline behavior is configured in database metadata rather than hard-coded per source.
• Experience with Azure Container Apps or Docker-based job workloads, Logic Apps, and infrastructure-as-code (Bicep).
• Familiarity with Power BI as a consumer of the platform, including DirectQuery and Government Community Cloud constraints.
• Microsoft certifications such as Azure Data Engineer Associate or Azure Solutions Architect.
• Prior work in government, transportation, aviation, utilities, or other regulated environments handling non-public data.
The projected hourly range for this position is $65 to $72.
On-Demand Group (ODG) provides employee benefits which includes healthcare, dental, and vision insurance. ODG is an equal opportunity employer that does not discriminate on the basis of race, color, religion, gender, sexual orientation, age, national origin, disability, or any other characteristic protected by law.