Software Engineer - Benchmarking

Stack

Responsibilities

• ▹Prepare and maintain benchmark datasets
• ▹Build and maintain evaluation pipelines across model APIs and terminal agents
• ▹Build containerized evaluation environments and viewers
• ▹Support fine-tuning experiments for small open-source LLMs
• ▹Build the scoreboard/leaderboard of results
• ▹Build analysis tools to identify failure modes
• ▹Build HTML viewers and internal tools for data annotation

Requirements

• ▹4+ years of professional software engineering experience
• ▹Strong Python skills
• ▹Writes robust, maintainable code

Soft skills

About the company

Office Hours is an on-demand expert network connecting organizations with trusted experts for advisory work, projects, and AI model training; clients include fast-growing digital health and tech companies.