Member of Technical Staff - Engineering Lead, Data Ingestion

Reflection AI

Tech Lead • Principal • On-site • Full-time • United States

San Francisco, USA

📄 View original listing (EN)

Reflection builds open frontier models. As Data Ingestion Lead you'll lead the team that turns the open web, licensed corpora and other large-scale sources into well-structured, versioned, auditable pre-training datasets — spanning web crawl, ingestion pipelines and data lakes — while staying hands-on as an individual contributor.

Stack

Airflow LLM Spark

Responsibilities

▹Build, mentor and grow a team of data ingestion engineers
▹Provide front-line leadership across web crawling, extraction/normalization pipelines and data lakes
▹Stay hands-on with the technical stack to make targeted individual contributions
▹Manage day-to-day execution and run data acquisition/ingestion campaigns
▹Guide architectural decisions on crawler architecture, distributed processing (Ray/Beam/Spark), orchestration (Airflow/Prefect), storage formats (Parquet, JSONL, WARC), deduplication and dataset versioning
▹Partner with pre-training and data quality teams to tie ingestion decisions to measurable downstream model impact
▹Work with data partnerships, digitization operations and external vendors to onboard new sources within legal and licensing constraints

Requirements

▹Experience building, mentoring and growing data or infrastructure engineering teams while staying technically hands-on
▹Deep experience building web-scale data acquisition/ingestion systems with real ownership of production pipelines at multi-TB to PB scale
▹Strong coding ability and technical credibility
▹Deep expertise in at least one of web crawling & acquisition, large-scale extraction & ingestion pipelines, or data lakes/corpus storage, with working knowledge of the others
▹Fluency with distributed compute (Ray, Beam, Spark), orchestration (Airflow, Prefect), formats (Parquet, JSONL, WARC) and object-store/data-lake architectures
▹Familiarity with how LLMs are trained and evaluated, and intuition for what makes data useful for training
▹Ability to guide strategy and drive execution across concurrent data campaigns

Soft skills

Comfortable growing into leading a larger team quicklyRaises the bar for technical judgment, prioritization, communication and execution in a fast-moving environment

What we offer

▹Top-tier compensation with salary and equity
▹Stock options for all employees
▹Comprehensive medical, dental, vision and life insurance with annual wellness allowance
▹Lunch and dinner provided daily in the office
▹22 weeks paid parental leave for all new parents, including adoptive and surrogate journeys
▹Unlimited PTO in the US, 30 days in the UK
▹Visa sponsorship support
▹Regular off-sites, happy hours and team celebrations

About the company

Reflection is a research lab making intelligence open and accessible for everyone to use, customize and build on.

Languages: Angol: Felsőfok

📅 Running for a while

• Posted 45 days ago
📋 This employer: 100% freshly verified (≤3 days) (37 active listings)

These signals come from our own daily check: when the listing appeared, and when we last saw it in the source. Not information from the employer, and not a guarantee that the role is still open.

Salary range

Not specified

Listing imported from an external source.

Work typeOn-site

Location

San Francisco, US

TypeJob

Apply →

This listing appeared on the employer's own site — open on the employer's site

🗺️ View on the map

More jobs at Reflection AI

All at this company (36) →

This listing comes from the employer's public careers page; the advertiser is responsible for its content. The summary is a machine-generated extract — check the original posting before applying.

Is this your company's listing? Request removal