Applied Machine Learning & Data Engineering Senior Consultant II

National General is a part of The Allstate Corporation, which means we have the same innovative drive that keeps us a step ahead of our customers’ evolving needs. The role supports high-impact AI and data automation initiatives focused on improving data ingestion, transformation, and mapping processes across the organization, responsible for advancing AI-assisted data mapping capabilities to enhance accuracy and deliver scalable solutions.

Responsibilities:

• Design and build AI-powered data pipelines to process and transform structured and unstructured data (PDF, Excel, CSV, etc.)
• Develop and optimize LLM-based solutions (prompt engineering, mapping, extraction, and transformation)
• Translate complex data and technical outputs into actionable insights for business stakeholders
• Partner with cross-functional teams (product, engineering, business SMEs) to define and deliver solutions
• Validate and improve solution accuracy through testing, iteration, and stakeholder feedback
• Support integration of ML/AI solutions into enterprise systems and workflows (APIs, databases, cloud platforms)
• Document solutions to ensure scalability, maintainability, and long-term adoption
• Analyze and process carrier and CMS datasets
• Build AI/ML mapping logic using LLMs and prompts
• Develop data pipelines (Python/C# data processing logic, SQL, APIs, Azure)
• Validate outputs against expected results with business stakeholders
• Troubleshoot accuracy, performance, and edge cases
• Document pipelines and ensure production-readiness
• Support integration into enterprise systems (C#, APIs, Azure)

Requirements:

• 3+ years of related experience
• Strong proficiency in Python and SQL for data analysis and engineering
• Experience working with Generative AI / LLMs (prompt engineering, APIs, or similar)
• Hands-on experience with data transformation, file parsing, and pipeline development
• Knowledge of DevOps practices (CI/CD pipelines, version control, cloud deployment)
• Familiarity with cloud platforms (preferably Azure)
• Ability to communicate effectively with both technical and non-technical stakeholders
• Proven ability to work in ambiguous, fast-evolving environments
• Experience with C# / .NET integration or enterprise application development
• Familiarity with LLM frameworks (LangChain, etc.) or document processing tools
• Knowledge of API design, JSON processing, and system integration patterns
• Background in insurance, healthcare, or other regulated industries