About this role

The ideal candidate will be responsible for designing, deploying, maintaining, and scaling Kafka clusters in mission-critical environments, while also supporting the Linux-based infrastructure that forms the foundation of our real-time data platform.

Responsibilities

Design, implement, and maintain highly available Kafka clusters in production environments.

Perform Kafka upgrades and patching ensuring minimal downtime and seamless transitions.
Administer security protocols and access controls to protect the platform.
Monitor cluster health and performance using our established monitoring stack (Prometheus, Grafana)
Troubleshoot and resolve cluster-related incidents such as broker outages, replication lag.
Develop and enhance automation scripts to streamline operational tasks and reduce manual intervention.
Assist with planning and managing cluster resources to support future growth.
Develop Kafka topics, partitions, ACLs, schema registry, and optimize throughput and reliability.
Manage a tune of Kafka Connect, Kafka Streams, and Mirror Maker as needed for data pipelines.
Ensure platform reliability and performance on enterprise-grade Linux systems (RHEL, Ubuntu).
Automate Kafka provisioning, configuration, and monitoring using Infrastructure as Code (IaC) and configuration management tools (e.g., Ansible, Terraform).
Deep knowledge of Kafka ecosystem components like ZooKeeper, Kafka Connect, Schema Registry, MirrorMaker, KRaft (Kafka without ZooKeeper).
Build custom Kafka tooling and dashboards for producers and consumers, monitor lag and health metrics.
Work with engineering teams to onboard new Kafka use cases and ensure security and compliance standards.
Perform root cause analysis and resolution of platform-level incidents and outages.
Develop and maintain Linux shell scripts for automation of deployments, monitoring, and maintenance tasks.

Qualifications we seek in you!

Qualifications

BE/B Tech/MCA
Excellent written and verbal communication skills

Qualifications/ Skills

Working experience with OpenShift Kubernetes Cloud services to deploy Kafka clusters using vendor provided helm charts & docker images
Kafka Resiliency dc/dr setup & observability for Kafka nodes
Data ingestion & streaming pipeline setup for other data platforms to Kafka & Kafka to data sources like Solr
Five or more years of Kafka administration experience, ideally with Confluent Kafka
Proficiency in Linux, Python, or Bash scripting
Hands-on knowledge of Kafka monitoring tools and experience with security protocols.

Kafka Engineer / Admin

Key skills

About this role