Key skills
Azure DatabricksAzure Data Lake StorageApache SparkScalaSQLHealthcare domain experienceSpark Structured StreamingPythonPySparkMongoDBAzureAIData LakeDatabricksSparkCI/CD
About this role
Optum is a global organization that delivers care, aided by technology, to help millions of people live healthier lives. The Senior Data Engineer will design and develop Azure cloud-native enterprise data solutions, focusing on data integration and transformation while leading a team of data engineers to enhance health outcomes through innovative data solutions.
Responsibilities:
- Design and develop Azure cloud-native enterprise data solutions with emphasis on data integration, transformation, and governance
- Build solution designs and technical designs for data pipelines based on business and technical requirements
- Develop and standardize reusable plug-and-play components that can be orchestrated into:
- Data ingestion patterns (batch/streaming)
- Data flows across multiple zones (raw/curated/consumption as applicable)
- Data quality frameworks and validation rules
- Implement and maintain pipelines leveraging Databricks + ADLS, ensuring reliability, scalability, and maintainability
- Identify opportunities to enhance/streamline existing codebase for:
- Automation
- Performance improvements
- Scalability and reliability
- Operational efficiency and reduced run-cost
- Support backlog execution by helping prioritize pipeline development, estimate effort, and create implementation roadmaps
- Lead a team of data engineers by providing technical direction, code reviews, and design guidance
- Provide QA, UAT, and implementation support, including deployment readiness, operational handoffs, and production troubleshooting
- Ensure adherence to engineering standards (documentation, logging/monitoring expectations, and CI/CD practices where applicable)
- Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI
Requirements:
- Bachelor's degree
- 8+ years of experience in each of the following: Azure Databricks
- 8+ years of experience in each of the following: Azure Data Lake Storage (ADLS)
- 8+ years of experience in each of the following: Apache Spark (Core concepts and performance tuning)
- 8+ years of experience in each of the following: Scala
- 8+ years of experience in each of the following: SQL
- Healthcare domain experience (claims, clinical, eligibility, provider, member, HIPAA-aware handling preferred)
- Data Streaming (e.g., Spark Structured Streaming and streaming design patterns)