Senior Databricks Data Engineer

job Title: Senior Databricks Data Engineer

Location: Austin, TX (Hybrid)

Interview Mode: In-Person

Job Summary

We are seeking an experienced Senior Databricks Data Engineer with 8+ years of IT experience in designing, developing, and delivering enterprise data solutions. The ideal candidate will have extensive expertise in Databricks, Apache Spark, Python/Scala, SQL, and modern data lakehouse architectures. This role requires a highly motivated professional who can independently analyze business requirements, design scalable data solutions, optimize data pipelines, and deliver analytics-ready datasets that support business decision-making.

The candidate should be capable of planning and executing complex technical tasks with minimal supervision while collaborating effectively with business and technical stakeholders.

Key Responsibilities

• Analyze business requirements and translate them into scalable data engineering solutions.
• Design, develop, and optimize ETL/ELT pipelines using Databricks and Apache Spark (PySpark or Scala).
• Build and maintain Lakehouse architectures using Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Develop and manage Lakeflow Declarative Pipelines (formerly Delta Live Tables) and orchestrate workflows using Lakeflow Jobs or similar scheduling tools.
• Design and implement dimensional data models, including Star and Snowflake schemas, to support enterprise reporting and analytics.
• Create Databricks SQL dashboards, Databricks Apps, and other analytical solutions that provide actionable business insights.
• Implement data governance, security, validation, and data quality frameworks to ensure reliable and trusted data.
• Optimize workload performance, scalability, monitoring, and operational efficiency while managing infrastructure costs.
• Perform system analysis, evaluate alternative technical solutions, and conduct cost-benefit analyses to recommend optimal approaches.
• Collaborate with cross-functional teams to gather requirements, document technical specifications, and deliver high-quality data solutions.
• Support CI/CD implementation for data engineering projects using Git-based workflows and DevOps best practices.
• Communicate technical concepts and project updates effectively to both technical and non-technical stakeholders.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent experience).
• 8+ years of experience designing, developing, deploying, or supporting enterprise technology solutions.
• 8+ years of hands-on experience with Databricks and Apache Spark for building and optimizing ETL/ELT pipelines.
• Strong expertise in SQL and Python (or Scala) for large-scale distributed data processing.
• Experience with Delta Lake, Lakehouse architecture, and Medallion Architecture (Bronze, Silver, Gold).
• Experience implementing Lakeflow Declarative Pipelines (DLT) and scheduling jobs using Lakeflow Jobs or similar orchestration platforms.
• Strong experience in data warehousing concepts and dimensional data modeling (Star/Snowflake schemas).
• Experience developing dashboards and applications within Databricks using Databricks SQL and Databricks Apps.
• Strong understanding of data governance, data quality, data security, and enterprise data management best practices.
• Experience with CI/CD pipelines, Git, and DevOps practices for data engineering.
• Excellent analytical, problem-solving, verbal, and written communication skills.
• Ability to work independently while managing multiple complex initiatives.

Preferred Qualifications

• Experience working in public sector or government environments.
• Databricks Certified Data Engineer Associate or Professional certification.
• Experience with workflow orchestration tools such as Apache Airflow or similar platforms.
• Familiarity with cloud-based data platforms and modern data engineering best practices.