Key skills
PythonScalaSQLAIMLMLflowETLELTData EngineeringAnalyticsDatabricksSparkApache SparkKafkaApache KafkaAirflowApache AirflowAzureGoogle CloudPerformance OptimizationCollaboration
About this role
Job Title: Databricks Architect
Location: United States (Remote)
Job Type: Contract
Roles and Responsibilities:
Architecture & Design
- Design and implement modern data platforms using Databricks Lakehouse architecture.
- Define data architecture, data models, and governance frameworks for enterprise analytics platforms.
- Architect scalable batch and real-time data pipelines.
Data Engineering
- Develop and optimize ETL/ELT pipelines using Apache Spark, Python, and SQL.
- Implement data ingestion frameworks using tools like Azure Data Factory, Apache Kafka, or Apache Airflow.
- Work with structured, semi-structured, and unstructured data.
Platform & Performance Optimization
- Optimize Spark workloads, cluster configurations, and query performance.
- Implement data quality, monitoring, and cost optimization strategies.
- Ensure data platform scalability, reliability, and security.
Advanced Analytics & ML Enablement
- Enable ML pipelines and collaboration with data science teams using MLflow.
- Build feature engineering pipelines and support AI/ML use cases.
Required Technical and Professional Expertise:
- Strong experience with Databricks platform
- Expertise in Apache Spark, Spark SQL, and distributed computing
- Programming experience in Python, Scala, or SQL
- Experience with data lakes and lakehouse architectures
- Strong experience with cloud data platforms such as:
- Microsoft Azure
- Google Cloud Platform