Senior Data Engineer

VivSoft develops secure software factories and technology solutions for federal, healthcare, and civilian agencies. The Senior Data Engineer will lead the design, optimization, and operation of a scalable AWS-based enterprise lakehouse for a United States Air Force program, supporting analytics, AI/ML, and operational reporting. The role also provides technical leadership, mentorship, and expertise in data platforms, governance, distributed processing, and cloud infrastructure.

Responsibilities:

• Architect, build, and manage a cloud-based lakehouse environment using Apache Iceberg, AWS S3, and AWS Glue Catalog
• Develop and optimize Apache Spark data pipelines on EMR and Kubernetes environments
• Design and implement event-driven data ingestion solutions using S3 events and Amazon SQS
• Optimize query performance across Athena, Trino, and Spark SQL environments
• Orchestrate data workflows and ETL pipelines using Apache Airflow
• Manage infrastructure deployment and automation using Terraform
• Develop, publish, monitor, and maintain data products and analytical dashboards
• Implement data quality, governance, lineage, access controls, and cost management best practices
• Create technical documentation, architectural designs, and operational runbooks
• Provide technical leadership and mentorship to junior engineering team members

Requirements:

• Must possess an active Secret Clearance
• 8+ years of professional experience in Data Engineering or large-scale data platform development
• Expertise in Apache Spark, including performance tuning, partitioning, memory optimization, and handling data skew
• Strong proficiency in Python or Scala and advanced SQL development
• Experience with open table formats such as Apache Iceberg (preferred) or Delta Lake
• Strong understanding of distributed query engines, including Athena, Trino, and Spark SQL
• Hands-on experience with AWS services, including S3, Glue, EMR, Athena, EC2, SQS, and event-driven architectures
• Experience with Apache Airflow for workflow orchestration
• Proficiency with Terraform and Infrastructure as Code (IaC)
• Experience working with Kubernetes environments
• Experience developing dashboards and data products using Grafana or similar visualization platforms
• Strong understanding of data quality, data governance, lineage, and access control frameworks
• Excellent technical leadership, mentoring, and stakeholder communication skills
• Ability to translate business, operational, and analytics requirements into scalable data platform solutions
• Strong collaboration skills with Data Scientists, AI Engineers, Cloud Engineers, and business stakeholders
• Proven ability to lead technical discussions and mentor junior engineers
• Excellent written and verbal communication skills
• Strong attention to detail regarding data quality, governance, lineage, security, and operational reliability
• Experience operating and maintaining Apache Iceberg tables at enterprise scale
• Experience with Trino administration and performance optimization
• Knowledge of dbt or similar data transformation frameworks
• Experience with Helm and GitOps deployment methodologies
• Experience evaluating and implementing modern data visualization platforms beyond Grafana
• Familiarity with data catalog, metadata management, lineage, and governance tools
• AWS Data Analytics Certification and/or Certified Kubernetes Administrator (CKA)
• Prior DoD, USAF, or Federal Government data platform experience
• Experience supporting AI/ML, data science, or advanced analytics workloads in cloud environments