Key skills
SQLSparkCloud experience in GCPCloud experience in AWSDistributed data systemsData pipelinesPythonGCP BigQueryGCP DataflowGCP AirflowGCP Pub/SubGCP AI PlatformAWS GlueAWS KinesisAWS LambdaAWS SageMakerAWS AthenaCI/CD pipelinesApache AirflowCloud ComposerAWS Step FunctionsApache BeamApache KafkaApache FlinkSpark Structured StreamingBigQueryDynamoDBAuroraRDSRedisMemcachedDistributed systems principlesAIMachine LearningMLETLELTData EngineeringAnalyticsKafkaAirflowAWSGCPLambdaGlueAthenaSageMakerKinesisPub/SubcomposerCachingPerformance OptimizationCI/CD
About this role
Best Buy is seeking a Senior Data Engineer to join their Ad Serving team to design, build, and operate a machine learning-powered Ad Serving platform. The role involves integrating large-scale data pipelines and ensuring seamless data flow for optimal ad experiences.
Responsibilities:
- Implement scalable ETL/ELT pipelines to process large volumes of clickstream, behavioral, and ad performance data in real time and batch modes
- Build and maintain integrations between the Ad Server, ML models, and supporting systems such as UIs, analytics platforms, and reporting databases
- Work closely with data scientists to operationalize ML models for real-time ad decisioning and bidding
- Leverage GCP Big Query, Dataflow, Airflow, Pub/Sub, AI Platform and AWS Glue, Kinesis, Lambda, SageMaker, Athena for data ingestion, transformation, and model deployment
- Architect cloud-native data workflows ensuring high availability, low latency, and fault tolerance
- Work with the operations team to develop CI/CD pipelines for data pipelines and ML model deployments, enabling fully automated testing and releases
- Participate in on-call rotation and provide operational support for data pipelines and ML-driven ad serving
- Anticipate and solve data architecture and integration challenges before they impact production
Requirements:
- Bachelor's in Computer Science, Data Engineering; or equivalent combination of relevant professional experience and education
- 3 years of modern development with SQL and distributed data processing frameworks such as Spark
- 3 years of experience diagnosing and resolving complex, production-grade data and/or model serving issues
- 3 years of hands-on cloud experience in GCP or AWS
- 3 years of experience in designing and managing distributed data systems and pipelines
- 2 years of Python experience
- Demonstrated experience collaborating with data scientists, product managers, and operations engineers
- Experience with high-throughput, real-time Ad Serving systems processing thousands of requests per second
- Strong understanding of ad delivery, targeting, and performance optimization concepts
- Proficiency in data orchestration tools (Apache Airflow, Cloud Composer, Step Functions)
- Deep familiarity with streaming data frameworks (Apache Beam, Kafka, Flink, Spark Structured Streaming)
- Experience with cloud-native data stores (Big Query, DynamoDB, Aurora, RDS) and caching systems (Redis, Memcached)
- Knowledge of distributed systems principles—state management, concurrency, messaging, and inter-service data consistency