Senior Data Engineer

Cars Commerce builds technology solutions that simplify automotive commerce for dealerships, manufacturers, and consumers. The Senior Data Engineer will own data-platform domains end to end by designing scalable pipelines, data models, quality systems, and warehouse assets while advancing AI-assisted development and knowledge-graph capabilities.

Responsibilities:

• Design, build, and maintain robust production pipelines and reusable data assets for analytics, Data Science, and BI, using PySpark and SQL-based transformation patterns
• Own data quality and reliability for your domain through data contracts with upstream producers, automated data quality checks, and anomaly detection
• Tune PySpark jobs and pipeline performance for runtime, memory, partitioning, and compute cost at production scale
• Design and maintain dimensional models and medallion-style (bronze/silver/gold) data layering, producing reusable, well-documented assets for Data Science and BI teams
• Optimize Redshift performance; query tuning, Spectrum external tables, Glue Data Catalog, and materialized views for the warehouse assets your domain owns
• Own warehouse pipeline SLAs, freshness monitoring, and incident response for your domain; participate in on-call
• Break down ambiguous business problems and resolve them, mapping day-to-day work to broader increment goals
• Drive adoption of Al-assisted development practices across the team - using tools like Claude Code, Cursor, or Amazon Q to accelerate pipeline development, testing, and documentation, not just code completion
• Partner directly with stakeholders on requirements and prioritization; establish and document engineering standards for your area
• Review code and ensure the quality of what the team ships
• Mentor and coach more junior engineers, leveling up their technical judgment and ownership over time

Requirements:

• Remote and must be located in the Chicago area. May require onsite interviews
• 5+ years building and maintaining production data pipelines, with demonstrated ownership of a domain, a significant migration, or an architecture decision
• Proficient in PySpark and Python, including performance tuning for runtime, memory, and cost
• Advanced SQL for complex data manipulation, joins, aggregation, and window functions
• Solid cloud experience; AWS strongly preferred
• Strong data modeling skills; dimensional design and grain, and medallion-style layering
• Delta Lake/Iceberg fluency — table format internals, compaction, vacuuming, and schema evolution
• Experience with Redshift — query optimization, Spectrum external tables, Glue Data Catalog, and materialized views
• Experience establishing data contracts and schema validation with upstream producers, and working with observability tooling (e.g., Great Expectations, Metaplane, Monte Carlo, or similar)
• Proficient with Airflow (or similar orchestration)
• API development for consuming and serving data or model outputs
• Comfort with infrastructure-as-code and mature CI/CD practices (e.g., GitHub Actions, Jenkins)
• Hands-on experience with AI-assisted development tools (e.g., Claude Code, Cursor, or Amazon Q)
• Interest in or experience with knowledge graphs and graph-based data modeling — you'll help design and build out this capability as a core part of our semantic layer and catalog work
• Demonstrates ability to mentor without being the go-to answer key — you teach people how to reason through a problem rather than just handing them the solution
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience
• For US-based Positions: Applicants must be authorized to work in the United States. Please note that we are unable to sponsor employment visas at this time
• Bonus:
• Experience with streaming/CDC patterns (Kafka, MSK, or Flink)
• Exposure to data cataloging and lineage tools (Atlan or similar)
• Experience in a high-availability, distributed environment
• AWS certifications