Azure Databricks Engineer (Dallas, TX)

Company: Cedent Consulting Inc
Location: Dallas
Posted on: February 1, 2025

Job Description:

Key Responsibilities

• Azure Databricks Architecture & Design
• Design, build, and optimize scalable, high-performance data pipelines and data lakes using Azure Databricks.
• Architect and implement end-to-end analytics solutions leveraging Databricks and other Azure services (e.g., Azure Data Lake, Azure SQL, Azure Blob Storage).
• Lead the design of cloud-based architectures using Azure Databricks for data processing, transformation, and reporting.
• Data Engineering & Integration
• Design and implement ETL/ELT processes to ingest, process, and transform data across various sources (structured and unstructured).
• Collaborate with data scientists, analysts, and other stakeholders to understand business requirements and develop data solutions.
• Manage data integration workflows between Databricks and other platforms like Power BI, Azure SQL, Synapse, etc.
• Optimization & Performance Tuning
• Identify performance bottlenecks in Databricks environments and optimize clusters, queries, and code.
• Continuously improve and scale data pipelines to accommodate growing data volumes and business needs.
• Collaboration & Leadership
• Mentor and guide junior engineers and team members in best practices related to Azure Databricks and data engineering.
• Work closely with cross-functional teams (data science, analytics, business intelligence) to deliver integrated solutions.
• Provide technical leadership in cloud data architecture and data engineering.
• Monitoring & Security
• Implement monitoring and alerting systems to ensure data pipelines and workflows are running smoothly.
• Ensure compliance with data security and governance standards across the data ecosystem.
• Continuous Improvement & Innovation
• Stay up to date with the latest advancements in Azure Databricks, cloud technologies, and data engineering trends.
• Continuously evaluate and introduce new technologies and methodologies to enhance the data platform.Required Skills and Qualifications
• Experience
• Minimum 10 years of hands-on experience in data engineering and cloud-based data platforms.
• At least 5 years of experience working with Azure Databricks, building data pipelines, and performing data engineering tasks.
• Strong experience with Azure services such as Azure Data Lake, Azure Synapse Analytics, Azure Blob Storage, Azure SQL Database, etc.
• Technical Skills
• Proficiency in Spark, PySpark, Scala, or SQL for large-scale data processing.
• Expertise in Databricks notebooks, clusters, job scheduling, and libraries.
• Deep understanding of cloud-native data engineering practices and architecture on Azure.
• Familiarity with data modeling, data lakes, and data warehouse concepts.
• Expertise in Azure services: Data Factory, Data Lake, Synapse Analytics, Event Hubs, Cosmos DB.
• Data Pipeline and ETL/ELT Development
• Experience building and optimizing ETL/ELT workflows on Databricks.
• Knowledge of data orchestration tools such as Azure Data Factory, Apache Airflow, or similar.
• Big Data Technologies
• Proficiency in handling large-scale distributed data processing with tools like Apache Spark.
• Experience with technologies such as Kafka, Delta Lake, and Databricks Runtime.
• Strong understanding of Delta Lake and Lakehouse architecture.
• Programming Languages
• Strong programming skills in Python, Scala, or Java.
• Experience with SQL-based querying and optimizations.
• Cloud & DevOps
• Strong understanding of Azure cloud services and architecture.
• Familiarity with DevOps principles, CI/CD pipelines, and automation tools. Terraform required.
• Knowledge of Git, Azure DevOps, or similar version control and deployment systems.
• Collaboration & Leadership
• Excellent communication skills to work with both technical and non-technical stakeholders.
• Proven track record of mentoring and leading teams of data engineers.
• Experience managing complex projects and working with cross-functional teams.Preferred Qualifications
• Certifications
• Microsoft Certified: Azure Data Engineer Associate or equivalent certification.
• Databricks Certified Associate Developer for Apache Spark.
• Education
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
  #J-18808-Ljbffr

Keywords: Cedent Consulting Inc, North Richland Hills , Azure Databricks Engineer (Dallas, TX), Engineering , Dallas, Texas