AI Engineering Labs

Code for the articles on this site. Each card says what the project measured and what you need to run it.

Labs for the series

• Market Analyst Agent

  Series: Engineering the Agentic Stack

  A LangGraph research agent that fetches market data and writes reports. It is the series’ reference implementation.

  Result: No benchmark. Tests cover tool calls on synthetic data and the guardian’s approval routing.

  Run it: uv run pytest tests/test_tools.py needs no model. A full analysis needs Python 3.13+, Docker, and API keys, and makes paid model calls.

  Repository · Part 1

• Agent Harness Lab

  Series: Recursive Harness Engineering

  A LangChain support assistant for refunds, deliveries, and account changes. Spec files switch harness components on and off.

  Result: With gpt-6-luna, both specs passed 12 of 12 custom state checks; adding Schema-Guided Reasoning and the Jev write guard cost about 2.2 times as much and took about twice as long. On eight τ³-bench retail tasks the guarded spec passed 6 and the plain one 7, but different tasks failed on each run, so the gap favors neither.

  Agent or workflow (v0.2.1-a2): Of five designs, a workflow with no agent passed all 16 state checks, and trace review found its replies correct. The approval agent passed them too but replied wrongly in both approved-refund cases. Each task ran once and the workflow was written with these tasks in view, so this is not a general ranking.

  Run it: make test runs offline. make a1-custom, make a1-tau3, and make a2-matrix rerun the experiments through OpenRouter and need an API key.

  Repository · Harness design (v0.1.2-a1) · Agent or workflow

• Recursive Harness Engineering Lab

  Series: Recursive Harness Engineering

  The series’ small improvement loop: a fixed LLM proposer changes a memory tool’s settings, deterministic tests on 20 synthetic stories check each change, and the feedback shapes the next proposal. Nothing carries over between campaigns.

  Result: Three campaigns went from 13 to 20 of 20 with the same three patches, using 12 model calls for about USD 0.04. A sweep of all 384 configurations found 16 that pass in under a second, so this small search did not need an LLM; the sweep knew the fixtures and had a different budget.

  Run it: lab verify and lab serve check the recorded study without an API key. A new campaign needs OPENAI_API_KEY and --live.

  Repository · Article

More labs

• RAG Evals Demo

  Notebooks and modules that compute retrieval, reranking, and generation metrics on the SciFact corpus, plus a sweep over chunking, embedding, and LLM settings. Qdrant runs embedded, so you do not need Docker.

  Result: No SciFact benchmark is reported. Unit tests pin the article’s worked examples, such as Recall@5 = 0.750, MRR = 0.625, and nDCG@5 = 0.627, so you can check each metric against a hand calculation.

  Run it: make eval runs the suite. make benchmark calls hosted GPT, Claude, and Gemini models and needs their API keys.

  Repository · Article

• TypeScript Agent Service

  A pnpm monorepo for Python ML engineers who need to ship TypeScript: two agent-loop implementations, a streaming API, Postgres run history, a worker, and an MCP server.

  Result: 40 tests, of which 36 run without Postgres or an API key. The agent-loop tests finish in under a second with no network.

  Run it: Node 24 and pnpm. pnpm demo needs no API key, database, or Docker. The real API needs OPENAI_API_KEY and Postgres in Docker.

  Repository · Article

• Search Ranking Stack

  BM25, dense retrieval, hybrid fusion, cross-encoder reranking, and LLM reranking on a sample of Amazon ESCI product-search data, measured stage by stage.

  Result: NDCG@10 is 0.585 for BM25, 0.611 for dense retrieval, and 0.628 for hybrid fusion, which beats either method alone. Recall@100 is fixed at retrieval (0.842 for hybrid), and later stages cannot raise it.

  Run it: uv sync, then uv run download-data (2.5 GB download, 5 MB sample). The LLM stage runs through Ollama, the Anthropic API, or a local Qwen model.

  Repository · Article

• Model Compression Demo

  A CLI that plans LLM quantization for a given model and target hardware, then prints recipes, dry runs, serving commands, and benchmark plans for vLLM, LLM Compressor, and TensorRT-LLM.

  Result: No measured result. The benchmark numbers in the article come from other teams; the demo gives you the commands to measure your own.

  Run it: uv run python demo.py plan and the other planning commands need no GPU. Quantization, serving, and quality runs need a GPU and vLLM.

  Repository · Article