Senior Data Engineer

Overview

Summary

The candidate must have a successful track record in handling complex data pipelines for high transaction volume application and reporting/analytics data with minimal supervision. The candidate will be expected to support a variety of structured and semi-structured data in streaming and batch frameworks for both migration from and synchronization with legacy systems. The Senior Data Engineer provides support to the Application database design to aid in eliminating data duplication and enabling selective event-based and schedule-based data transfer to endpoints within the cloud and legacy environment as required. The Senior Data Engineer drives towards programmatic pipeline generation and orchestration to enhance repeatability and rapid deployment, using Databricks, PySpark, and Airflow, out-of-the-box thinking, AWS native capabilities, and CI/CD tools while utilizing established design patterns, specifically the Medallion architecture.

Responsibilities

What You'll Be Doing

• Designs, implements, and operates data management systems using Databricks and PySpark.
• Defines and adheres to best practices for implementing the Medallion architecture diligently across all data layers.
• Orchestrates complex data pipelines and workflows using Apache Airflow.
• Works with product and application owners, data stewards, and business/data analysts to determine, create, and populate required data in an optimal, reliable, and measurable manner.
• Assists the Data Architects and technical leads in designing how data will be stored, accessed, used, integrated, and managed by different data regimes and digital systems.
• Plans, designs, and optimizes data throughput and query performance.
• Participates in the employment of backend database technologies, their configuration and utilization, and the optimization of the full data pipeline infrastructure to support the actual content, volume, ETL, and periodicity of data to match expected responsiveness.
• Utilizes identified COTS, Python, PySpark, and AWS tools and services as needed.
• Creates and executes medium to complex SQL scripts to validate and test data prior to and during the lifecycle of reports and dataset development.

Qualifications and Education

Required Qualifications

BA or BS degree in CS, Computer Engineering, Information Technology, or a related field.

Databricks Certified Data Engineer (Associate or Professional) or equivalent experience.

5+ years of hands-on experience with PySpark, Python, and various Python toolkits and libraries for data processing and pipelines.

Proven experience and deep understanding of the Medallion architecture, with the ability to define and adhere to best practices diligently.

Hands-on experience orchestrating complex data workflows using Apache Airflow.

5+ years of experience and ability to create complex SQL queries and functions.

3+ years of work experience on AWS-based datasources such as RDS, Athena, Redshift.

2+ years of hands-on experience in creating ETL and data pipeline design patterns for modular and reliable ETL with robust error management and alerting.

1+ year of experience in Java and Linux scripting is essential.

1+ year of experience working with CI/CD tools including Git for ETL and scripts repository.

1+ year of experience using complex data sets and formats such as JSON, AVRO, parquet.

1+ year of experience in data profiling and quality management.

Compensation

The proposed salary range for this role is $****** to $******* USD. The salary range provided is a good faith estimate representative of all experience levels. Karsun considers several factors when extending an offer, including but not limited to, the role, function and associated responsibilities, a candidate’s work experience, location, education/training, and key skills.