Lead Data Software Engineer with Databricks

EPAM Systems is seeking a Lead Data Software Engineer with Databricks expertise to drive the technical design and evolution of end-to-end traceability solutions. This role provides technical leadership for traceability platforms and data products, collaborating with various teams to ensure best practices and long-term architectural vision are maintained.

Responsibilities:

• Provide technical leadership and ownership for traceability data platforms, pipelines, and data products
• Define, document, and enforce architectural principles, technical standards, and best practices for traceability solutions
• Oversee the design of robust pipelines, quality controls, and proactive monitoring frameworks to ensure data availability, quality, integrity, and timeliness
• Guide and support data engineers and technical contributors on implementation choices, troubleshooting, performance optimization, and best practices
• Contribute strategically to Run execution and continuous improvement, including data quality, lineage, auditability, and operational monitoring
• Translate traceability and regulatory requirements into technical solutions, data models, and platform capabilities
• Collaborate closely with solution architects, data platform teams, and external partners to ensure alignment and scalability
• Support adoption of traceability solutions by business and sustainability teams through technical guidance and enablement

Requirements:

• 5+ years of experience in technical roles within data, analytics, or platform environments, with exposure to leadership or solution ownership responsibilities
• Expertise in cloud-based data platforms, preferably Azure
• Background in data architecture, data modeling, and data platform design, including complex, large-scale datasets
• Hands-on experience with Databricks, Spark/PySpark, and SQL for data transformation pipelines and analytics platforms
• Proficiency in SQL and data storage concepts, including relational databases, NoSQL, and data lakes, with performance optimization for large and complex datasets
• Knowledge of Python and data manipulation ecosystems such as Pandas and NumPy, with ability to review and guide code even if not hands-on full time
• Familiarity with data visualization tools including Power BI, Tableau, and Looker
• Experience designing or governing data quality frameworks, lineage, monitoring, and auditability
• Familiarity with orchestration tools and data integration patterns such as ADF, Airflow, and AWS Glue, or equivalent
• Experience with software engineering best practices including version control with git, branching strategies, code reviews, and CI/CD
• Understanding of data governance, security, and access control for sensitive business and regulatory data
• Proficiency in English at a B2+ level