PolymarketWe’re hiring an **AI Engineer (Model Training & Evaluation)** for Kortix Analysis Lab (BlackCube Labs × Kortix Suna). We’re building infrastructure to **measure, benchmark, and improve AI agent performance at scale**. **What you’ll do:** * Design agent benchmarks (GAIA + custom eval suites) * Build Python-based evaluation pipelines (scorers, datasets, regression tests) * Integrate Braintrust + LiteLLM for experiment tracking and tracing * Analyze agent behavior (tool use, latency, failures, task success) * Help build automated benchmarking + CI-style eval gates **What we need:** * Strong Python (async, FastAPI, testing systems) * Experience with LLM evaluation / benchmarking (not just prompting) * Familiar with tools like Braintrust / W&B / LangSmith * Understanding of agent systems (tools, multi-step reasoning, sandboxing) * Ability to define meaningful metrics + analyze results quantitatively * Self-directed ownership of full eval pipelines **Bonus:** GAIA, LiteLLM, Docker/sandboxes, open-source agent frameworks **Stack:** Python · FastAPI · Braintrust · LiteLLM · Docker · Next.js · pnpm monorepo If you’re interested in building real evaluation infrastructure for AI agents (not demos), reach out: **jd@blackcubelabs.com**
We’re hiring an **AI Engineer (Model Training & Evaluation)** for Kortix Analysis Lab (BlackCube Labs × Kortix Suna). We’re building infrastructure to **measure, benchmark, and improve AI agent performance at scale**. **What you’ll do:** * Design agent benchmarks (GAIA + custom eval suites) * Build Python-based evaluation pipelines (scorers, datasets, regression tests) * Integrate Braintrust + LiteLLM for experiment tracking and tracing * Analyze agent behavior (tool use, latency, failures, task success) * Help build automated benchmarking + CI-style eval gates **What we need:** * Strong Python (async, FastAPI, testing systems) * Experience with LLM evaluation / benchmarking (not just prompting) * Familiar with tools like Braintrust / W&B / LangSmith * Understanding of agent systems (tools, multi-step reasoning, sandboxing) * Ability to define meaningful metrics + analyze results quantitatively * Self-directed ownership of full eval pipelines **Bonus:** GAIA, LiteLLM, Docker/sandboxes, open-source agent frameworks **Stack:** Python · FastAPI · Braintrust · LiteLLM · Docker · Next.js · pnpm monorepo If you’re interested in building real evaluation infrastructure for AI agents (not demos), reach out: **jd@blackcubelabs.com**