Tech Lead/Architect - Databricks Data Engineering & AI/ML

EazyML specializes in Responsible AI solutions that support proactive compliance and sustainable automation for enterprises adopting AI at scale. The company is seeking a Tech Lead/Architect to design and deliver scalable Databricks data engineering and AI/ML solutions while leading engineers, guiding architecture, and remaining hands-on with coding and technical problem-solving.

Responsibilities:

• Lead the technical design and architecture of large-scale data engineering and AI/ML solutions on Databricks (Lakehouse Platform)
• Provide hands-on technical leadership across the full data lifecycle: ingestion, transformation, storage, and consumption using Spark, Delta Lake, and Databricks workflows
• Design and implement scalable ETL/ELT pipelines, ensuring performance, reliability, and cost optimization
• Guide the team in building and deploying AI/ML models and GenAI/LLM-based solutions, integrating them into production data pipelines
• Collaborate with data scientists, ML engineers, and business stakeholders to translate requirements into robust technical solutions
• Establish and enforce best practices for data governance, security, CI/CD, and DataOps/MLOps within Databricks environments
• Mentor and coach a team of data engineers, conducting code reviews and driving engineering excellence
• Own technical decision-making, including platform architecture, tool selection, and performance tuning
• Act as the primary technical point of contact for client/stakeholder discussions, ensuring alignment between business goals and technical execution
• Stay current with evolving Databricks features, Delta Lake updates, and AI/GenAI trends, driving adoption of relevant innovations

Requirements:

• 8+ years of overall experience in data engineering, with 3+ years in a technical leadership or lead role
• Strong hands-on experience with Databricks, including:
• Spark (PySpark/Scala), Delta Lake, Unity Catalog
• Databricks Workflows/Jobs, Cluster management & optimization
• Databricks SQL and performance tuning
• Solid experience in Data Engineering, including:
• Building and optimizing ETL/ELT pipelines at scale
• Data modeling (batch and streaming)
• Working with cloud data platforms (Azure/AWS/GCP – specify as needed)
• Experience with orchestration tools (e.g., Airflow, ADF, Databricks Workflows)
• Practical experience in AI/ML, including:
• Building, training, and deploying ML models
• MLOps practices for model lifecycle management
• Strong programming skills in Python and SQL; familiarity with Scala is a plus
• Experience with CI/CD pipelines and version control (Git-based workflows)
• Solid understanding of data security, governance, and compliance best practices
• Excellent communication and stakeholder management skills, with the ability to work directly with onshore clients/business teams
• Proven experience leading and mentoring technical teams
• Strong ownership mindset with the ability to balance hands-on delivery and team leadership
• Excellent problem-solving and analytical thinking
• Strong communication skills to bridge technical and business conversations
• Ability to work independently in a fast-paced, onshore client-facing environment
• Familiarity with GenAI/LLM frameworks (e.g., LangChain, RAG pipelines, vector databases) is a strong plus
• Experience in the Life Sciences domain (pharma, biotech, clinical, or healthcare data) is preferred
• Databricks certification(s) (e.g., Databricks Certified Data Engineer Professional, Databricks Certified Machine Learning Professional)
• Cloud certification (Azure/AWS/GCP)
• Experience in Agile/Scrum delivery environments
• Prior experience working in client-facing consulting or professional services roles