Big Data Platform Engineer

Job Summary We are seeking an experienced Big Data Platform Engineer to design, develop, and optimize large-scale data processing platforms and cloud-native data solutions. The ideal candidate will have deep expertise in Apache Spark, Kubernetes, AWS, and distributed data processing technologies. This role focuses on building scalable data pipelines, architecting Kubernetes infrastructure, optimizing big data workloads, and supporting enterprise data platforms through modern software engineering and DevOps practices. Key Responsibilities Design, develop, and maintain large-scale data processing pipelines using Apache Spark, Hadoop, Python, and Scala. Architect, deploy, and optimize containerized big data workloads using Amazon EMR on EKS. Design and build Kubernetes infrastructure to support scalable Spark applications and enterprise data platforms. Develop scalable solutions for data ingestion, storage, transformation, and analytics. Optimize existing data pipelines for performance, scalability, reliability, and cost efficiency. Monitor, troubleshoot, and resolve production issues affecting data pipelines and big data platforms. Develop automated testing frameworks and implement continuous testing to ensure data quality and platform reliability. Create and maintain unit, integration, and end-to-end tests for data processing applications. Manage Kubernetes clusters, pods, deployments, services, namespaces, ConfigMaps, Secrets, and storage resources. Collaborate with architects, data scientists, analysts, and engineering teams to deliver enterprise data solutions. Research and evaluate emerging Big Data, cloud, and AI technologies to improve platform capabilities. Participate in architecture discussions and contribute to technical design decisions. Required Qualifications Bachelor's degree in Computer Science, Information Systems, or a related field, or equivalent professional experience. Minimum of 5 years of experience designing and developing enterprise Big Data solutions. Strong experience building Kubernetes infrastructure for enterprise applications. Hands-on experience with Kubernetes architecture, including pods, services, deployments, namespaces, ConfigMaps, Secrets, networking, storage, and security. Experience running Apache Spark workloads on Kubernetes, including Amazon EMR on EKS. Strong experience with Apache Spark, including Spark architecture, performance tuning, partitioning, caching, broadcast joins, DAG execution, executors, and resource optimization. Experience designing and supporting large-scale data pipelines processing massive datasets. Experience with Hadoop, Spark, Hive, and Trino. Experience troubleshooting data skew, resource constraints, scalability issues, and Spark job failures. Strong experience with AWS services, including Amazon S3, EMR, EMR on EKS, Glue, Lambda, Athena, Amazon EKS, CloudWatch, and CloudTrail. Experience with AWS IAM Roles for Service Accounts (IRSA), VPC networking, subnets, and security groups. Experience with Kubernetes resource management, scheduling, auto-scaling, Helm charts, kubectl, and YAML manifests. Experience integrating Spark with Kubernetes operators and dynamic resource allocation. Strong programming experience using Python or Scala. Experience writing clean, modular, scalable, and high-performance code. Strong understanding of functional programming concepts, concurrency, memory management, and collections. Strong SQL skills, including window functions, complex joins, aggregations, and query optimization. Experience using AI development tools such as GitHub Copilot, Amazon Q Developer, ChatGPT, or Claude. Experience applying AI tools to improve software development workflows and engineering productivity. Strong analytical, troubleshooting, communication, and problem-solving skills. Experience working in Agile software development environments. Preferred Qualifications Experience managing enterprise ETL and production data pipeline platforms. Experience with CI/CD tools such as Jenkins, GitLab CI, GitHub Actions, or ArgoCD. Experience with Infrastructure as Code (Terraform or AWS CloudFormation). Experience with Docker and container image optimization. Experience with service mesh technologies such as Istio or Linkerd. Experience with monitoring and observability tools including Prometheus, Grafana, or the ELK Stack. AWS certifications such as AWS Certified AI Practitioner, AWS Certified Solutions Architect, or AWS Certified Data Analytics/Specialty. Kubernetes certifications such as Certified Kubernetes Administrator (CKA) or Certified Kubernetes Application Developer (CKAD). Experience with GitOps deployment practices. Master's degree in Computer Science, Information Systems, or a related field. Experience working within the Financial Services industry. Education: Bachelors Degree