Senior Data Engineer - Remote (USA)

ICF is a mission-driven company focused on improving lives and making the world a better place. They are seeking a Senior Data Engineer to design and maintain scalable data pipelines, develop data processing workflows, and ensure data quality and security.

Responsibilities:

• Design, develop, and maintain scalable data pipelines using Spark, Hive, and Airflow
• Develop and deploy data processing workflows on the Databricks platform
• Develop API services to facilitate data access and integration
• Create interactive data visualizations and reports using AWS QuickSight
• Build required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS and SQL technologies
• Monitor and optimize the performance of data infrastructure and processes
• Develop data quality and validation jobs
• Assemble large, complex sets of data that meet non-functional and functional business requirements
• Write unit and integration tests for all data processing code
• Work with DevOps engineers on CI, CD, and IaC
• Read specs and translate them into code and design documents
• Perform code reviews and develop processes for improving code quality
• Improve data availability and timeliness by implementing more frequent refreshes, tiered data storage, and optimizations of existing datasets
• Maintain security and privacy for data at rest and while in transit
• Other duties as assigned

Requirements:

• Bachelor's degree
• 7+ years of hands-on software development experience
• 4+ years of experience building data pipelines using Python, Java, and cloud technologies, with hands-on experience leveraging Spark and Hive for large-scale data processing
• Candidate must be able to obtain and maintain a Public Trust clearance
• Candidate must reside in the US, be authorized to work in the US, and work must be performed in the US
• Must have lived in the US 3 full years out of the last 5 years
• Experience building job workflows with the Databricks platform
• Strong understanding of AWS products including S3, Redshift, RDS, EMR, AWS Glue, AWS Glue DataBrew, Jupyter Notebooks, Athena, QuickSight, EMR, and Amazon SNS
• Familiar with work to build processes that support data transformation, workload management, data structures, dependency and metadata
• Experienced in data governance process to ingest (batch, stream), curate, and share data with upstream and downstream data users
• Experienced in data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up
• Demonstrated understanding using software and tools including relational NoSQL and SQL databases including Cassandra and Postgres; workflow management and pipeline tools such as Airflow, Luigi and Azkaban; stream-processing systems like Spark-Streaming and Storm; and object function/object-oriented scripting languages including Scala, C++, Java and Python
• Familiar with DevOps methodologies, including CI/CD pipelines (Github Actions) and IaC (Terraform)
• Ability to obtain and maintain a Public Trust; residing in the United States
• Experience with Agile methodology, using test-driven development