Data Engineer

Clevanoo LLC is seeking a Databricks Engineer to consult with customers and optimize Databricks environments. The role involves designing scalable ETL/ELT pipelines and integrating Databricks with various cloud platforms while ensuring data governance and compliance best practices are followed.

Responsibilities:

• Consult directly with customers to design, optimize, and fine-tune Databricks environments
• Design, build, and maintain scalable ETL/ELT pipelines
• Optimize Databricks clusters, workflows, and architecture for cost, scalability, and performance
• Integrate Databricks with AWS, Azure, and GCP environments
• Implement data governance, security, compliance, and data management best practices
• Utilize Spark, Delta Lake, MLflow, and Databricks Machine Learning technologies
• Collaborate with engineers, analysts, data scientists, and business stakeholders
• Troubleshoot platform and infrastructure issues while driving continuous improvements
• Support advanced analytics, machine learning, and AI-focused customer initiatives

Requirements:

• Active Secret Clearance
• U.S. Citizenship
• Proven experience with Databricks in production environments
• Strong Python and/or SQL skills
• Apache Spark experience
• Experience building and maintaining ETL/ELT pipelines
• Knowledge of Data Lakehouse architecture
• Experience with AWS S3, Azure Data Lake, or GCP Storage
• Data modeling, data quality, and performance tuning experience
• CI/CD and Git experience
• Bachelor's degree preferred (experience considered in lieu of degree)
• Ability to travel as needed
• Master's Degree or PhD
• MLflow experience
• Delta Lake experience
• Unity Catalog experience
• Kafka, Kinesis, or Event Hubs
• Tableau, Power BI, or Looker
• Terraform, ARM, or CloudFormation
• FedRAMP, HIPAA, SOC2, or similar compliance experience
• DevOps/MLOps experience
• Prior consulting or customer-facing experience
• Agile project delivery experience and familiarity with professional services environments