Key skills
AirflowKafkaKubernetesPythonSparkShellAI
About this role
Role Overview
- Design and execute test plans validating the end-to-end cluster creation flow on a kubernetes platform.
- Managing complex data modeling and schema drift, as well as embedding automated data quality checks and statistical anomaly detection directly into pipelines.
- Working with governance layers to ensure policies like tag-driven Attribute-Based Access Control (ABAC), column-level masking, row-level filters, and zero-code lineage ingestion are accurately enforced at the data layer.
- Architect the automation framework and tools that validate a highly distributed, multi-cluster control plane.
Requirements
- 4+ years of software engineering experience with a focus on test automation, infrastructure, or backend development
- Ability to learn and develop AI enabled test automation frameworks
- Hands-on understanding of modern compute and streaming engine internals like Spark, Kafka, Trino, Airflow
- Understanding of Kubernetes internals (CRDs, Controllers, Operators, Namespaces)
- Expert-level proficiency in Python/Shell for scripting and automation.
- Bachelor’s or Master’s degree in Computer Science or equivalent experience.
Tech Stack
- Airflow
- Kafka
- Kubernetes
- Python
- Spark
Benefits
- Generous PTO Policy
- Support work life balance with Unplugged Days
- Flexible WFH Policy
- Mental & Physical Wellness programs
- Phone and Internet Reimbursement program
- Access to Continued Career Development
- Comprehensive Benefits and Competitive Packages
- Paid Volunteer Time
- Employee Resource Groups