Azure Databricks Integration Expert

I need an exceptional engineer who can take full ownership of designing and building production-grade ETL/ELT pipelines on Azure Databricks. The bulk of the work centres on Apache Spark jobs that ingest, transform and load high-volume data into analytics-ready structures.

Your pipelines must expose and consume REST APIs that interact with SQL databases, NoSQL stores and various third-party APIs so that downstream systems can query clean, well-modelled data in real time. Enterprise-scale integration experience is critical; if you have prior Salesforce connectivity know-how, that will be a welcomed advantage.

You will be expected to:
• Architect and code Spark-based transformations in Databricks notebooks and/or repos
• Orchestrate jobs, handle schema evolution and enforce data quality rules
• Build secure, well-documented REST endpoints for cross-system data exchange
• Optimise SQL queries and cluster configurations for performance and cost efficiency
• Set up CI/CD, automated testing and monitoring to keep everything reliable in production

Deliverables include fully functioning Databricks notebooks (or repos), API source code with documentation, deployment scripts and a brief runbook so my team can maintain what you build. If you thrive on hands-on problem solving and enjoy seeing your data pipelines drive real business value, let’s talk.