Capture & Ingestion: Own the full capture path from wire to lake
Build batch + stream pipelines (Airflow, Spark, dbt) for tick and reference data
Own L2/L3 order-book reconstruction with gap handling
Storage & Modeling: Own the Iceberg-over-S3 lakehouse
Drive storage cost optimisation via compaction, tiering, and snapshot expiry
Tooling & Libraries: Build libraries for schema management, data contracts, validation, and lineage
Reliability & Observability: Embed monitoring, alerting, SLAs/SLOs, and CI/CD across capture and pipeline layers on Kubernetes
Collaboration: Partner with Quant Research, Data Science, Backend, and DevOps to translate requirements into platform capabilities

5+ years building production-grade data systems
Proven expertise architecting and launching data lakes / lakehouses from scratch
Hands-on experience with Apache Iceberg (or comparable table formats — Delta / Hudi)
Experience with market data and/or network packet capture
Experience normalizing market data from multiple vendors
Expert-level Python (incl. Polars and/or PySpark); Rust a strong plus
Modern orchestration (Airflow) and distributed processing (Apache Spark)
Advanced SQL: complex aggregations, window functions, query optimization, partition pruning
Solid fundamentals in Linux, containerization (Docker, Kubernetes / EKS), and cloud object storage (AWS S3)
DevOps & observability: CI/CD, infrastructure-as-code (Terraform), GitOps (ArgoCD)
Strong grasp of structured + unstructured / binary data, and storage optimization
English fluency for documentation and collaboration in an international team

Market Data Engineer – Domain Trading Expertise Required

Key skills