Lead Data Engineer / Architect

IT Associates is seeking a Lead Data Engineer / Architect to lead their data architecture and engineering efforts. This role involves designing a Data Lakehouse, overseeing a high-caliber team, and ensuring the architecture supports analytics and reporting at scale.

Responsibilities:

• Design and own the end-to-end data architecture. Example: Databricks(or Snowflake, or Hybrid)-— Unity Catalog, Delta Lake, medallion layer structure (Bronze / Silver / Gold)
• Define data modeling standards: Example: star schema, dimensional modeling, and semantic layer design using dbt
• Establish data governance frameworks including lineage, cataloging, and access controls
• Own the technology selection process in partnership with the COO, consultants, and other stakeholders
• Ensure the architecture can support investment analytics, portfolio reporting, and operational workflows at scale
• Build and own the ingestion layer: Example: Fivetran/Airbyte connectors for custodians, Bloomberg, and internal systems
• Develop and maintain dbt transformation models, tests, and documentation
• Configure and manage pipeline orchestration using Airflow or Prefect
• Write clean, production-grade Python for data pipelines, custom connectors, and analytical tooling
• Build internal dashboards and data applications using for example Plotly Dash or Streamlit
• Integrate Claude Enterprise (Anthropic) into data workflows for AI-assisted analysis and automation
• Recruit, interview, onboard, and mentor direct reports — you define who joins this team
• Define scope, manage delivery, and sign off on all work from external consultants (pipeline specialists, viz specialists)
• Ensure all consultant engagements include knowledge transfer and documentation before close
• Build a team culture grounded in engineering excellence, documentation discipline, and continuous improvement
• Report directly to the COO with regular updates on build progress, risks, and resource decisions
• Translate complex technical decisions into clear business terms for investment and operations leadership
• Collaborate with portfolio managers, operations, and compliance teams to understand and prioritize data needs
• Own the data roadmap and communicate trade-offs transparently

Requirements:

• 7+ years in data engineering, data architecture, or a combined role
• Demonstrable experience designing and building a modern Data Lakehouse or cloud data warehouse from the ground up — not just maintaining one
• 3+ years working with cloud data platforms (Databricks or snowflake strongly preferred)
• Strong hands-on dbt experience: model design, testing, documentation, and semantic layer development
• Experience managing or mentoring engineers — you have grown people, not just pipelines
• Track record of working directly with senior business stakeholders and translating requirements into technical solutions
• Databricks, Snowflake: Delta Lake, Unity Catalog, Databricks SQL, cluster configuration
• dbt Cloud: transformations, testing, documentation, multi-environment deployment
• Data ingestion: Fivetran, Airbyte, custom connector development
• Orchestration: Apache Airflow or Prefect (managed or self-hosted)
• Python: production-grade scripting, pipeline development, Plotly/Dash or Streamlit
• Data modeling: star schema, dimensional modeling, slowly changing dimensions
• SQL: expert-level; complex transformations, window functions, query optimization
• Git, CI/CD, and engineering best practices
• Financial services or asset management experience strongly preferred
• Familiarity with custodian data feeds, Bloomberg API, or investment data sources is a significant advantage
• Working knowledge of data governance, lineage, and quality frameworks
• Direct experience with Databricks Unity Catalog rollout
• Hands-on experience with data catalog tooling (Atlan, Monte Carlo, or similar)
• Familiarity with Claude Enterprise or Anthropic AI tools in a production context
• Experience with SOC 2 data environments and security-conscious engineering practices
• Exposure to portfolio management systems, order management systems (OMS), or similar investment platforms
• Experience standing up a data team from zero — first hire, first pipeline, first architecture decision