Senior Data Engineer

Live Nation Entertainment is a global live entertainment and eCommerce company operating across ticketing, concerts, artist management, and entertainment marketing. The Senior Data Engineer will build scalable data pipelines and large-scale data systems using Databricks and SQL Server, modernize the data ecosystem, and leverage AI tools to improve productivity and pipeline reliability. The role also involves cross-functional collaboration, mentoring, documentation, and continuous improvement.

Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL
• Optimize data workflows and SQL/Spark queries for performance, cost efficiency, and reliability
• Contribute to the design of our enterprise Data Lake and curated data assets
• Build robust ingestion frameworks, transformation logic, and reusable data engineering components
• Support data migrations and modernization efforts from legacy systems to cloud-based platforms
• Use AI coding copilots (Databricks Assistant, GitHub Copilot, SQL copilots) to accelerate development, code review, test creation, and documentation
• Automate repetitive tasks (schema evolution, config generation, logging templates, test data creation) with generative AI tools
• Leverage AI to auto-detect pipeline anomalies, data drift, quality issues, and performance bottlenecks
• Incorporate AI-assisted query optimization tools to improve Spark & SQL performance
• Ability to learn and utilize LLM-powered tools to generate, refactor, and explain complex SQL, PySpark logic, and pipeline configurations
• Work cross-functionally with Product, data consumers to deliver high-quality data solutions
• Participate in Agile ceremonies & contribute to sprint planning & technical design discussions
• Mentor junior data engineers, promote best practices in AI-assisted data engineering
• Ensure strong documentation, standards adoption, & continuous improvement across the team

Requirements:

• strong hands-on experience building scalable data pipelines, optimizing performance, and enabling AI use cases
• Ability to learn and utilize LLM-powered tools to generate, refactor, and explain complex SQL, PySpark logic, and pipeline configurations
• 5–7+ years of hands-on data engineering experience
• Advanced proficiency in Python, SQL, PySpark, and ETL/ELT frameworks
• Strong experience with Databricks and distributed data processing
• Demonstrated ability to optimize SQL/Spark workloads for performance and cost
• Experience using AI development tools such as: GitHub Copilot, Databricks Assistant, SQL/IDE AI copilots, AI-based data quality or monitoring tools
• Solid understanding of data modeling, governance, and CI/CD for data pipelines
• Excellent communication skills and ability to work in a cross-functional environment
• Experience working in Agile environments with tools like Jira and Confluence