Freelance Agent Evaluation Engineer

Stack

Responsibilities

• ▹Build realistic developer environments: a virtual company with codebase, infrastructure and context (tickets, docs, conversations)
• ▹Design tasks from intermediate states of these environments: craft the prompt and define what counts as solved
• ▹Write tests that accept all valid solutions and reject incorrect ones
• ▹Iterate on tasks and tests based on QA feedback, analyzing failures

Requirements

• ▹5+ years in software development
• ▹Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
• ▹Experience writing tests (functional, integration)
• ▹English proficiency B2+

Soft skills

What we offer

• ▹Compensation up to $50/hr equivalent, depending on level and pace
• ▹Flexible, self-directed schedule

About the company

Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating and improving AI systems.