Job Details

Position Title: Sr. Software Engineer Data PlatformLocation: San Francisco, CAType: On-siteDepartment: EngineeringAbout Multiscale AI: Multiscale AI provides advanced AI solutions for Semiconductor Manufacturing Process Optimization (SMPO). With offices in Atlanta, San Francisco, and Hyderabad, we help semiconductor manufacturers optimize fabs and achieve measurable results.The Role: As a Sr. Software Engineer Data Platform at Multiscale, you will architect and build foundational data infrastructure components that power our AI and analytics platform. Your mandate is to take our current platform from working to enterprise-grade. Youll design and implement high-performance data connectors, governance controls, storage and lakehouse components, and backend data services that integrate seamlessly with both cloud and on-prem ecosystems. This includes building MinIO-like object storage capabilities, developing Delta Lakecompatible services, and enabling interoperability with Snowflake, Databricks, BigQuery, and other modern data stacks.This role also requires hands-on experience with data engineering frameworks (Spark, Pandas, Arrow), allowing you to dive deep into compute-layer challenges and shape platform-level support for data and ML workloads. Youll work closely with the platform, AI, and data engineering teams to ensure Multiscale provides a reliable, scalable, and secure data foundation for semiconductor manufacturing AI.Key Responsibilities:Design and implement data connectors for cloud and on-prem data sources (object stores, cloud warehouses, RDBMS, APIs, event/streaming systems) to enable secure, scalable ingestion and export.Build and maintain data governance frameworks, including access control, audit logging, lineage tracking, data quality checks, policy enforcement, and metadata/catalog services.Architect and develop core backend data platform components, such as MinIO-like S3-compatible object storage layers, table/metadata services, and efficient serialization/transformation systems.Engineer Delta Lakecompatible services (ACID transactions, schema evolution, time travel, compaction) that enable seamless interoperability with Databricks, Snowflake, BigQuery, and broader lakehouse ecosystems.Develop platform abstractions and APIs that support Spark, Pandas, and distributed compute frameworks, enabling data engineers to run workloads reliably and efficiently on the platform.Optimize the data pipeline performance, throughput, and reliability across multi-cloud and hybrid environments, ensuring consistent SLOs.Collaborate with data engineers, data scientists, platform engineers, and product teams to translate platform requirements into scalable data solutions.Improve platform observability (metrics, logs, traces) for all data services and implement HA/DR strategies for mission-critical data components.Qualifications:Bachelors or Masters degree in Computer Science, Engineering, or a related field.6+ years of professional experience in data platform engineering, data infrastructure, or distributed systems.Hands-on experience building and operating large-scale data pipelines, ETL frameworks, or data warehouses (Databricks, Snowflake, BigQuery, Airflow, etc.).Strong proficiency in Python, Go, Java, or Scala for backend systems development.Deep understanding of data modeling, lakehouse architectures, Delta Lake, Apache Iceberg, or similar open-source table formats.Experience building data connectors, APIs, and integrations with external data systems.Strong knowledge of distributed systems, databases, and cloud platforms (AWS, Azure, GCP).Excellent problem-solving skills and strong communication/collaboration abilities. Preferred Skills:Experience with data governance platforms, metadata management, or lineage systems.Hands-on experience with object storage systems (MinIO, S3), Parquet/ORC, Delta/Iceberg/Hudi.Experience developing scalable ingestion frameworks (CDC, streaming, micro-batch).Familiarity with Spark on Kubernetes, Arrow, vectorized computation, and performance tuning.Benefits:Competitive salary and equity optionsComprehensive health, dental, and vision coverageFlexible paid time offOpportunity to work with a cutting-edge team in AI and semiconductor technologyMultiscale AI is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.
recblid h394or2y7ou3uh8fuho2mem45bgmp0