Sr. AWS Data Engineer (10+years) - Full Time Role- Remote

Visionary Innovative Technology Solutions LLC is seeking a Senior AWS Data Engineer with over 10 years of experience. The role involves developing and managing data pipelines, optimizing SQL queries, and ensuring cloud security best practices are followed.

Responsibilities:

• Must have below:
• 10+ Years Experience
• Great Communicator/Client Facing/ Attention to detail
• Individual Contributor and ability to work as a team
• 100% Hands on in the mentioned skills
• Advanced Proficiency in Python concepts like Code Structures, Modules, Packages, Class, SubClass, Inheritance, Multi-Threading and Functional Programming
• Experience in developing reusable Python packages for internal or public usage
• Ability to write automating ETL processes and scheduling jobs like Airflow DAG
• Ability to track job pipeline runs to reprocess error records
• Ability to orchestration different pipelines to run in sequence or parallel
• Troubleshoot data pipeline errors and fix issues
• Export or Import data to/from various formats like CSV, JSON, XML etc preferably from S3 or other cloud storage
• Experience in using AI IDE tool
• Advanced SQL skills, including complex joins, CTE's and subqueries
• Experience in optimizing SQL queries for performance and optimization in data warehouse technologies preferably Snowflake
• Proficiency in Python unit, integration, and system test
• Proficiency in implementing DBT tests for data validation and quality checks
• Experience in generating code using configurations using python and jinja templates
• Experience in GitHub, including implementing CI/CD process from scratch
• In depth understanding of AWS S3 for data storage, ECS, IAM including best practices for organization and security
• Knowledge of AWS security best practices, including IAM roles, encryption standard, secure coding guidelines, DBT profiles access configurations and more
• Experience with AWS lambda for serverless data processing tasks
• Proficiency in using Apache Airflow on AWS to design ,schedule and monitor complex data flows
• Ability to integrate Airflow with AWS services and DBT models such as triggering a DBT model or EMR or reading from s3 writing to redshift or snowflake
• Experience in API integrations to Salesforce, SFTP, SharePoint, One Drive, and others
• Experience in creating complex DBT models including full refresh, incremental models, snapshots, and documentation
• Ability to write and maintain DBT macros for reusable code
• Experience in creating custom DBT macros using jinja and Python allowing for reusable components within DBT models
• Familiarity with AWS cloud watch, Data Dog for monitoring the pipelines and setting up alerts for workflow failures

Requirements:

• 10+ Years Experience
• Great Communicator/Client Facing/ Attention to detail
• Individual Contributor and ability to work as a team
• 100% Hands on in the mentioned skills
• Advanced Proficiency in Python concepts like Code Structures, Modules, Packages, Class, SubClass, Inheritance, Multi-Threading and Functional Programming
• Experience in developing reusable Python packages for internal or public usage
• Ability to write automating ETL processes and scheduling jobs like Airflow DAG
• Ability to track job pipeline runs to reprocess error records
• Ability to orchestration different pipelines to run in sequence or parallel
• Troubleshoot data pipeline errors and fix issues
• Export or Import data to/from various formats like CSV, JSON, XML etc preferably from S3 or other cloud storage
• Experience in using AI IDE tool
• Advanced SQL skills, including complex joins, CTE's and subqueries
• Experience in optimizing SQL queries for performance and optimization in data warehouse technologies preferably Snowflake
• Proficiency in Python unit, integration, and system test
• Proficiency in implementing DBT tests for data validation and quality checks
• Experience in generating code using configurations using python and jinja templates
• Experience in GitHub, including implementing CI/CD process from scratch
• In depth understanding of AWS S3 for data storage, ECS, IAM including best practices for organization and security
• Knowledge of AWS security best practices, including IAM roles, encryption standard, secure coding guidelines, DBT profiles access configurations and more
• Experience with AWS lambda for serverless data processing tasks
• Proficiency in using Apache Airflow on AWS to design, schedule and monitor complex data flows
• Ability to integrate Airflow with AWS services and DBT models such as triggering a DBT model or EMR or reading from s3 writing to redshift or snowflake
• Experience in API integrations to Salesforce, SFTP, SharePoint, One Drive, and others
• Experience in creating complex DBT models including full refresh, incremental models, snapshots, and documentation
• Ability to write and maintain DBT macros for reusable code
• Experience in creating custom DBT macros using jinja and Python allowing for reusable components within DBT models
• Familiarity with AWS cloud watch, Data Dog for monitoring the pipelines and setting up alerts for workflow failures