Job Details

The Data Engineer will be responsible for collecting, parsing, managing, analyzing, and visualizing large datasets to transform information into actionable insights. The role involves building scalable, repeatable, and secure data pipelines across various platforms. The engineer will ensure that data workflows support multiple users while maintaining high performance, security, and reliability.

Key Responsibilities

• Design, develop, and maintain robust and efficient data pipelines to ingest, transform, catalog, and deliver curated, trusted, and quality data into the Common Data Platform.
• Actively participate in Agile ceremonies and follow Scaled Agile processes defined by the CDP Program team.
• Deliver high-quality data products and services following SAFe Agile practices.
• Identify and resolve issues related to data pipelines and analytical data stores.
• Implement monitoring and alerting for pipelines and data stores, enabling auto-remediation to ensure uptime and reliability.
• Apply a security-first approach with strong testing and automation practices.
• Collaborate with product managers, data scientists, analysts, and business stakeholders to understand data needs and provide tools and infrastructure.
• Stay updated with modern data engineering trends and evaluate new tools, frameworks, and technologies to improve efficiency.

Required Skills

• Bachelors degree in computer science, Information Systems, or related field, or equivalent experience.
• 2+ years experience with Databricks, Collibra, and Starburst.
• 3+ years experience with Python and PySpark.
• Experience using Jupyter notebooks for coding and unit testing.
• Strong recent experience with relational and NoSQL data stores, STAR and Dimensional Modeling methods.
• 2+ years experience with a modern data stack (S3, Spark, Airflow, Lakehouse architecture, real?time databases).
• Experience with cloud data warehouses such as RedShift or Snowflake.
• Overall data engineering experience across traditional ETL and Big Data (on?prem or cloud).
• Experience building end?to?end pipelines for unstructured and semi?structured data using Spark.
• Ability to work with confidential supervisory information (Must meet Protected Individual requirements).

Education

Bachelors degree in computer science, Information Systems, or related field (or equivalent experience).