Databricks Data Engineer / Business Analyst – Data Contracts & OCDS

Scicom Infrastructure Services is seeking a Data Engineer / Business Analyst to support a large-scale Databricks implementation. The role bridges business, contracting, data-governance, and engineering teams to define requirements, construct data contracts, map procurement data to OCDS, and build reliable data pipelines and products.

Responsibilities:

• Lead requirements-gathering sessions with procurement, contracting, program, analytics, governance, and technical stakeholders
• Define, construct, document, and maintain data contracts between data producers and consumers
• Establish data-contract requirements covering:
• Dataset purpose and ownership
• Source and target systems
• Schemas, fields, and data types
• Required and optional attributes
• Business definitions and transformation rules
• Data-quality expectations
• Validation and reconciliation rules
• Refresh frequency and delivery schedules
• Versioning and schema-evolution requirements
• Security classifications and access controls
• Service-level expectations
• Issue ownership and change-management procedures
• Translate business requirements into user stories, acceptance criteria, process flows, data mappings, interface specifications, and technical requirements
• Conduct source-system analysis, data profiling, gap assessments, and source-to-target mapping
• Identify differences between existing procurement data and required OCDS structures
• Facilitate agreement among data owners, producers, consumers, architects, and governance teams
• Maintain traceability from business requirements through data models, engineering implementation, testing, and acceptance
• Evaluate requested changes for downstream effects on data products, reports, integrations, and analytical use cases
• Support backlog refinement, sprint planning, demonstrations, testing, and stakeholder acceptance
• Apply the Open Contracting Data Standard to procurement and public-contracting datasets
• Map source-system data to appropriate OCDS fields and structures
• Work with data across the contracting lifecycle, including:
• Planning
• Tender
• Award
• Contract
• Implementation
• Develop and maintain mappings for OCDS releases, records, release packages, record packages, identifiers, organizations, parties, items, milestones, documents, transactions, amendments, and related contracting elements
• Interpret and apply OCDS schemas, codelists, validation rules, and implementation guidance
• Determine whether standard OCDS fields meet project requirements or whether documented extensions are necessary
• Support the construction of complete contracting records from multiple transactional releases
• Establish rules for handling amendments, updates, cancellations, corrections, and historical changes
• Validate transformed data against applicable OCDS JSON schemas
• Identify missing, invalid, inconsistent, or nonconforming procurement data and work with stakeholders to resolve deficiencies
• Document assumptions, business rules, mappings, extensions, and exceptions
• Support the publication, exchange, analysis, or internal use of standardized contracting data
• Design, develop, test, and maintain data pipelines using Databricks, Apache Spark, PySpark, Python, and SQL
• Build ingestion and transformation pipelines for structured and semi-structured procurement data
• Process JSON, CSV, XML, Parquet, relational database, API, and file-based data sources
• Implement Bronze, Silver, and Gold data layers using medallion architecture
• Build normalized, dimensional, analytical, and OCDS-aligned data models
• Use Delta Lake capabilities for schema enforcement, schema evolution, versioning, auditability, and reliable processing
• Develop reusable frameworks for mapping source procurement data into OCDS-compatible outputs
• Implement batch and incremental ingestion patterns
• Use Databricks Workflows, notebooks, jobs, Auto Loader, Delta Live Tables or Lakeflow capabilities, as appropriate
• Develop REST API integrations for source ingestion and downstream data delivery
• Implement automated data-quality, reconciliation, completeness, and conformity checks
• Support Unity Catalog implementation for metadata, ownership, lineage, access control, and data discovery
• Optimize Spark jobs, SQL queries, clusters, partitioning, file sizes, and data layouts
• Participate in code reviews, automated testing, CI/CD, deployment, monitoring, and production support
• Investigate pipeline failures, data discrepancies, and performance issues
• Define measurable quality rules for accuracy, completeness, validity, timeliness, consistency, and uniqueness
• Develop validation controls for required OCDS fields, identifiers, dates, amounts, currencies, organizations, classifications, and contracting relationships
• Create dashboards or reports that show data-contract compliance and data-quality results
• Establish processes for detecting and managing schema drift
• Document data lineage from original procurement systems through Databricks transformations and downstream products
• Work with governance teams to assign data owners, stewards, classifications, retention requirements, and access policies
• Ensure sensitive procurement and supplier information is handled according to security and privacy requirements
• Support auditability through documented rules, version-controlled mappings, validation results, and change histories

Requirements:

• Bachelor's degree in computer science, information systems, data analytics, business analysis, engineering, public administration, supply-chain management, or a related field
• At least five years of combined data-engineering, data-analysis, business-analysis, or data-integration experience
• Hands-on experience implementing or working with the Open Contracting Data Standard
• Demonstrated experience constructing, documenting, negotiating, or maintaining data contracts
• Experience mapping procurement or contracting data to OCDS schemas
• Strong knowledge of OCDS releases, records, schemas, codelists, identifiers, contracting stages, and validation practices
• At least three years of hands-on experience with Databricks
• Strong experience with:
  • Apache Spark
  • PySpark
  • Python
  • SQL
  • Delta Lake
  • ETL and ELT pipelines
  • Data modeling
  • JSON and JSON Schema
  • REST APIs
  • Data-quality validation
  • Source-to-target mapping
• Experience gathering requirements and translating them into implementable engineering specifications
• Ability to communicate effectively with technical and nontechnical stakeholders
• Experience writing user stories, acceptance criteria, business rules, data dictionaries, interface specifications, and process documentation
• Strong analytical, troubleshooting, facilitation, and documentation skills
• Experience working within Agile delivery teams
• Ability to operate equally well in technical engineering and business-analysis discussions
• Strong understanding of how data contracts create accountability between data producers and consumers
• Ability to convert complex procurement processes into clear data structures and transformation rules
• Attention to detail when interpreting schemas, codelists, business definitions, and validation requirements
• Strong stakeholder-facilitation and conflict-resolution skills
• Ability to identify gaps and ambiguities before they become engineering defects
• Commitment to documentation, traceability, governance, and data quality
• Ability to work effectively within a large, multidisciplinary Databricks team
• Experience with government procurement, public-sector contracting, grants, acquisition, supplier, or financial data
• Experience implementing OCDS extensions or tailoring OCDS for specific organizational requirements
• Familiarity with procurement classifications, organizational identifiers, tender processes, awards, amendments, milestones, transactions, and contract implementation
• Databricks Certified Data Engineer Associate or Professional certification
• Experience with Unity Catalog, Delta Live Tables, Lakeflow, Databricks Workflows, or Structured Streaming
• Experience with Microsoft Azure, AWS, or Google Cloud
• Experience with Azure Data Factory, ADLS Gen2, AWS Glue, Amazon S3, Snowflake, dbt, Kafka, Airflow, or Power BI
• Experience using JSON Schema validation tools and automated data-quality frameworks
• Knowledge of metadata management, master-data management, data lineage, and reference-data governance
• Experience with CI/CD, Git, Azure DevOps, GitHub Actions, Jenkins, or Terraform
• Experience supporting large federal, state, local-government, or regulated-enterprise data programs
• Familiarity with federal acquisition, procurement, reporting, transparency, or open-data requirements