Software Engineer II (Data Pipelines & Event Processing)

Torch.AI is a company focused on building a government-owned reasoning infrastructure that enables machine reasoning at scale for mission-critical environments. The Software Engineer II role involves designing, building, and maintaining scalable data pipelines and backend services to process diverse data sources, supporting mission-critical decision making with reliability, adaptability, and operational security. The position requires collaboration with AI/ML and security engineers to integrate advanced data fusion and ensure compliance and security in data workflows.

Responsibilities:

• Design, build, and maintain backend services and pipelines that acquire and process data from diverse data sources, including: Extensible connector and adapter patterns so new data sources, workflows, and processing methods can be added without rebuilding the core platform
• Systems that monitor and respond to real-world events by collecting, correlating, and updating relevant data streams
• Data transformation, enrichment, and normalization processes to support downstream analytics and mission applications
• Build and implement workflows that: Allow users to configure, run, review, and manage data processing tasks through controlled application interfaces
• Support human-in-the-loop review and promotion workflows so raw or enriched data can be validated before becoming available to downstream users or systems
• Incorporate automated data access workflows using APIs, web interfaces, and distributed systems while adhering to controlled access and usage constraints
• Optimize performance, scalability, and resilience of data acquisition and processing systems
• Partner with AI/ML engineers to: Integrate data fusion capabilities that combine multiple sources to improve data completeness, context, and usability
• Integrate AI-enabled capabilities into data workflows, including structured extraction, classification, summarization, validation, and routing through approved internal services
• Partner with security engineers to: Implement policy enforcement, fail-closed validation, audit evidence, and infrastructure safeguards for sensitive data workflows
• Implement safeguards that reduce attribution risk and ensure responsible interaction with external data sources
• Apply operational security practices to data acquisition and processing workflows, including controlled access patterns, traffic shaping, and protection of system and infrastructure
• Ensure systems meet compliance, auditability, and mission requirements
• Contribute to CI/CD pipelines, containerized deployments, and automated testing workflows
• Troubleshoot production issues and perform root-cause analysis across distributed systems
• Document system behavior, data flows, and integration patterns to support maintainability and operational use
• Participate in code reviews and contribute to engineering standards and best practices

Requirements:

• B.S. or M.S. in Computer Science, Engineering, or related field
• 3–6 years of professional software engineering experience
• Strong proficiency in Python (preferred) or similar backend programming languages
• Strong understanding of data pipeline patterns, including ETL/ELT workflows
• Experience building production systems for data ingestion, integration, or processing
• Experience working with APIs, automated data access techniques, and distributed systems
• Experience working with structured and semi-structured data formats (JSON, CSV, Parquet)
• Experience designing systems that operate reliably under variable data availability and latency conditions
• Experience with CI/CD pipelines and modern software development practices
• Experience implementing controlled or privacy-aware data acquisition techniques, such as controlled access, managed attribution, fail-closed validation, etc
• Familiarity with SQL and NoSQL databases and query optimization
• Familiarity with cloud environments (AWS preferred) and containerization (Docker)
• Understanding of secure software development practices and controlled data access patterns
• Strong problem-solving skills and ability to operate in fast-paced, mission-driven environments
• Experience with workflow orchestration tools (Airflow, NiFi) or similar systems
• Experience with streaming or event-driven architectures (Kafka or equivalent)
• Experience working with geospatial, temporal, or multi-source datasets
• Experience supporting defense, intelligence, or other regulated environments
• Familiarity with large-scale data ingestion, indexing, or search systems
• Familiarity with infrastructure patterns that support identity protection, request management, or traffic routing in distributed systems
• Familiarity with distributed systems and microservices architectures
• Exposure to observability tooling (logging, monitoring, metrics)
• Exposure to data enrichment, entity resolution, or relationship mapping techniques