Kafka Developer/Data Engineer

Quantum World Technologies Inc. is seeking a Kafka Developer/Data Engineer to lead the architecture and development of enterprise-scale streaming data pipelines. This role involves hands-on building and ownership of Kafka-based solutions, requiring collaboration with onsite US clients and a focus on design and operational excellence.

Responsibilities:

• Architect, build, and operate real‑time and near‑real‑time Kafka‑based pipelines
• Lead development using Confluent Kafka (core focus)
• Design and implement stream processing using:
• Apache Flink (preferred) or
• Apache Spark Structured Streaming
• Define Kafka best practices for:
• Topic design, partitioning, and delivery semantics
• Error handling and retries
• Backpressure management and performance tuning
• Design and maintain batch and hybrid ETL pipelines using SQL Server
• ETL tools preferred: SSIS
• Own data quality, reliability, and observability across streaming pipelines
• Work directly with onsite clients to:
• Translate business requirements into technical solutions
• Defend architectural decisions
• Troubleshoot complex Kafka and streaming production issues
• Mentor engineers and raise overall engineering standards
• Collaborate with DevOps and platform teams in containerized, distributed environments

Requirements:

• 8+ years of hands-on data engineering experience
• Deep expertise in Confluent Kafka: Architecture and cluster design, Topic and partition strategy, Delivery semantics (exactly-once, at-least-once, etc.)
• Strong hands-on experience with Apache Flink or Apache Spark Structured Streaming
• Excellent SQL skills (complex analytics, performance tuning)
• Strong experience in Python, Java, or .NET (C#) (At least one OOP language is mandatory)
• Solid understanding of: Streaming and event-driven architectures, ETL / ELT patterns, Data modeling and orchestration
• Experience with SQL Server
• Apache Flink
• SSIS
• Docker / Kubernetes
• CI/CD for data pipelines
• Cloud or hybrid data platforms