Senior Data Engineer

Careforth is a company focused on digital and technology solutions, and they are seeking a Senior Data Engineer to join their data platform team. The role involves architecting and operating a cloud-native infrastructure that supports various AI and analytics products, ensuring reliable and scalable data systems.

Responsibilities:

• Architect and build robust, scalable, and secure data pipelines leveraging Databricks (Delta Lake, Spark, Unity Catalog, Workflows) and AWS (S3, Redshift, Athena, Glue, Lambda, Step Functions, Kinesis), supporting both batch and real-time processing modes
• Design and evolve the enterprise lakehouse architecture, including data modeling, Delta table optimization, data versioning, and a semantic data layer and metrics store serving BI tools and downstream ML feature engineering
• Build and maintain ETL/ELT orchestration (Airflow, dbt) and stream processing pipelines (Kafka/Kinesis) ingesting claims, EHR/EMR, pharmacy, IoT, survey, CRM, and digital engagement event data
• Partner with Data Scientists to design and maintain feature store architecture, real-time member context assembly, and temporal feature computation services supporting risk and predictive models
• Develop data infrastructure for LLM and NLP use cases: vector databases, knowledge base ingestion, transcript storage and indexing, secure audio storage, and LLM response logging and audit trails
• Build APIs and integrations for downstream consumers: care team dashboards, lead scoring, pathway recommendation, queue prioritization, CRM connectors (Salesforce, HubSpot), and payer-facing data exchange
• Implement HIPAA-compliant data handling across all pipelines including PHI detection, redaction, audit trails, row-level security, SSO, and secrets management (AWS Secrets Manager, Vault)
• Build reporting infrastructure including a report template engine, multi-format export (PDF, Excel, CSV), scheduled report generation, and delivery APIs for internal and payer-facing reporting
• Conduct pull request reviews, enforce engineering best practices, mentor junior engineers, and research and apply emerging data engineering tools and patterns
• Participate in design discussions with technical leads across product lines; perform other duties and special projects as assigned

Requirements:

• Bachelor's or Master's Degree in Computer Science, Data Engineering, or a related field
• 7–10 years of professional data engineering experience, including 3+ years with modern cloud-based data lake or lakehouse architectures
• Deep expertise in Apache Spark (PySpark, Scala, or Java) for large-scale distributed data processing and strong hands-on Databricks experience across Delta Lake, Unity Catalog, and Workflows
• Hands-on experience with AWS data services: S3, Redshift, Athena, Glue, Lambda, Step Functions, Kinesis, DynamoDB, EMR, Bedrock, Transcribe, and Comprehend Medical
• Experience in healthcare or regulated data environments; familiarity with HIPAA technical safeguards required
• Demonstrated experience building data infrastructure in support of ML feature stores and model serving pipelines
• Proficient SQL across relational, NoSQL, and lakehouse environments
• ETL/ELT orchestration with Airflow, dbt, and AWS Step Functions; stream processing with Kafka/Kinesis
• Strong communicator able to translate data architecture decisions to clinical, operational, and executive stakeholders
• Collaborative by nature; comfortable driving alignment across data science, product, engineering, and clinical teams
• Detail-oriented and committed to data quality, HIPAA compliance, and operational excellence; self-directed with a continuous learning mindset
• Strong Python (preferred); Java or Scala also valued
• Experience with vector databases, embedding stores, document ingestion pipelines, and RAG pipeline data layers for LLM use cases
• Container orchestration with Docker and Kubernetes/EKS; CI/CD for data pipelines; familiarity with data quality and data contract frameworks
• BI tool integration, ad-hoc query execution engine design, query result caching, and multi-format report delivery pipelines