Job Details

Who We Are

Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to customers.

Job Description

Job Title: Lead Data Quality & Automation Engineer

Job Type: W2/C2C

Experience: 8-14 years

Location: San Francisco, California

Responsibilities

• 5 years of professional experience in software development, with at least 2 years in a lead or senior engineering role.
• Strong proficiency in Python programming and experience building data processing applications.
• Hands-on expertise with Apache Spark and Hadoop for big data processing.
• Solid experience with workflow orchestration tools like Apache Airflow.
• Experience designing and managing data ingestion and integration pipelines with Apache NiFi.
• Understanding on Data Quality automation, CI/CD, Jenkins, Oracle, Power BI, Splunk
• Deep understanding of data quality concepts, data validation techniques, and distributed data systems.
• Experience with cloud platforms (AWS, GCP, or Azure) is a plus.
• Lead the design and implementation of data quality frameworks and automated validation pipelines using Python, Apache Spark, and Hadoop ecosystem tools.
• Develop, deploy, and maintain scalable ETL/ELT workflows using Apache Airflow and Apache NiFi to ensure seamless data ingestion, transformation, and quality checks.
• Collaborate with cross-functional teams to understand data quality requirements and translate them into technical solutions.
• Define and enforce data quality standards, rules, and monitoring processes.
• Perform root cause analysis on data quality issues and implement effective fixes and enhancements.
• Mentor and guide junior engineers, conducting code reviews and fostering best practices.
• Continuously evaluate and integrate new tools and technologies to enhance data quality capabilities.
• Ensure high code quality, performance, and reliability in all data processing pipelines.
• Create comprehensive documentation and reports on data quality metrics and system architecture.

Qualifications

• Bachelor's degree or equivalent combination of education and experience.