# Start here: this address

*These values were filled in by the server you are talking to, just now.*

| | |
|---|---|
| **Base URL** | `https://pipeline-admin.technov.dev` |
| **Pipeline id** | the one the admin gave you (it goes in the header `X-Pipeline-Id`) |
| **API key** | given to you by the admin; send it as the header `X-Pipeline-Key` (never in the URL, never in a file you create or an answer you write) |
| **Every request** | the headers `X-Pipeline-Id: <your pipeline id>` and `X-Pipeline-Key: <key>` (or `Authorization: Bearer <key>`) |
| **These instructions** | `https://pipeline-admin.technov.dev/agent-guide.md` (also `GET https://pipeline-admin.technov.dev/api/agent/guide`) |
| **Helper script** | `https://pipeline-admin.technov.dev/pipeline_client.py` — `curl -O https://pipeline-admin.technov.dev/pipeline_client.py`, then `python pipeline_client.py --pipeline-id <your pipeline id> --pipeline-api-key <key> submit ./my-task --follow` (base URL already set) |

If a request answers `401`, the key or the id is wrong or expired; `403` means the pipeline is private or closed (ask the admin to make it public); `429` means wait for `Retry-After` seconds.

**The whole flow in six commands:**

```bash
BASE=https://pipeline-admin.technov.dev
PL_ID=<your pipeline id>
PL_KEY="<the API key the admin gave you>"
H=(-H "X-Pipeline-Id: $PL_ID" -H "X-Pipeline-Key: $PL_KEY")

curl -s "${H[@]}" $BASE/api/agent                        # who am I, which checks run, my limits
zip -r task.zip my-task/                                   # the task folder: task.toml, instruction.md, environment/, tests/, solution/
curl -s -X POST $BASE/api/agent/upload "${H[@]}" -H "X-Filename: task.zip" --data-binary @task.zip      # -> {"upload_id": "u_..."}
curl -s -X POST $BASE/api/agent/submit "${H[@]}" -H "Content-Type: application/json" -d '{"upload_id": "u_..."}'   # -> {"run_id": "..."}
curl -s "${H[@]}" "$BASE/api/agent/run/<run_id>/poll?wait=25&cursor="     # repeat with the cursor of each answer until "done": true
curl -s "${H[@]}" "$BASE/api/agent/run/<run_id>/report"
```

---

# Local pipeline — guide for AI agents

You can submit a task, watch it run and fix it through a local HTTP API. The pipeline evaluates the task with the same stages the CI
uses (static checks, integrity, oracle/nop and an agent trial on Daytona, LLM reviews through Codex, similarity, QC gate,
difficulty). A **project** is a task (its slug). A **pull request** holds one or more **versions** of the task; every version has
a **diff** against its base. A **run** evaluates one version and has one **job** per stage; every job has **steps** with their own logs.

Base URL: the address in the box at the top of this page (the server generated this page for the address you opened it at). The same server may answer at other addresses too (its own computer, the local
network, a Cloudflare address): they all behave the same, and **you say which pipeline you mean with your pipeline id and key**. A pipeline that is *private* answers only on the server's own computer (`403`
from anywhere else): ask the admin to make it public. If the server is not running, the admin starts it: `python app.py --no-open`.

The quickest way in is the helper script, which does everything below for you (upload, submit, live log, report, exit code):

```bash
BASE=http://127.0.0.1:<port>; PL_ID=osc-tb; PL_KEY=lpk_...     # the address in the box at the top of this page
python scripts/pipeline_client.py --base-url "$BASE" --pipeline-id "$PL_ID" --pipeline-api-key "$PL_KEY" submit ./my-task --follow
```

## Authentication: a pipeline id and its API key

The app runs several **pipelines** (for example *OSC Terminal Bench*, *Insight*, *Rustlab TB3*). Each is a separate setup with its own settings, projects and runs, and each has its
own **API key**. You are given two things by the admin (the pipeline's **Agent access** tab shows them and has copy buttons):

```
pipeline-id:       osc-tb
pipeline-api-key:  lpk_...          (shown to the admin once, when it was made; it may have a lifetime)
```

Send both on every request to `/api/agent/…`:

```
X-Pipeline-Id:  <pipeline-id>
X-Pipeline-Key: <pipeline-api-key>      # or  Authorization: Bearer <pipeline-api-key>
```

(`X-API-Key: <key>` is accepted too. The key alone is enough to say which pipeline you mean; if you send an id as well, it must be the key's own.)
**A key in the URL (`?token=…`) is not accepted**: addresses end up in proxy logs, browser history and error reports.

- **The key opens exactly one pipeline.** You see and submit to that pipeline's projects and runs only. Another pipeline's run, project or pull request answers `404`, as if it did not exist, and
  a `pipeline` field in a request body or query cannot widen that.
- A wrong key, or a right key with another pipeline's id, is `401 {"error":"unauthorized: …"}` (the answer does not say which pipelines exist); many wrong keys in a row from one address are
  throttled (`429`, with `Retry-After`). An **expired** key is `401 … has expired`: ask the admin for a new one. A pipeline the admin switched off, or closed to API access, answers `403` to its own key.
- The admin can rotate or revoke the key at any moment; the old one stops working at once. Keep it out of files that are committed or zipped.
- `GET /api/agent` says which pipeline your key opens, which endpoints exist and **your limits** (see below).

Every agent response is **scrubbed**:

- no private keys (`_…`), no machine paths, no config, no secrets — the pipeline's scripts and sources are never returned;
- logs run through a sanitiser: ANSI removed, tracebacks cut to their last line, runner paths replaced with `<path>/`;
- only statuses, timings, notes, annotations, diffs, the report and sanitised logs are exposed;
- an unexpected server error is `500 {"error":"internal error (reference abcd1234)"}`: nothing about the inside; give the reference to the admin.

You therefore see *results*, never the evaluator's code: study the report and the task files, not the pipeline.

The key lets you submit and read. It cannot change settings, switch pipeline steps, rebuild the image, make or revoke keys or delete anything —
those belong to the admin (see the next section). **You also cannot name a file or folder on the server's computer**: you send the task itself (see §1).

- Machine-readable index of every endpoint: `GET /api/agent` (also lists the stage ids and your limits)
- This guide as JSON: `GET /api/agent/guide` → `{"markdown": "...", "url": "/agent-guide.md"}`
- Human UI (for the person at the server's computer): `/#/p/<pipeline-id>/<slug>/pr/<n>` (pull request), `/#/run/<id>` (Actions-style run)

---

## Which checks run — decided by the admin, not by you

The admin chooses which steps run in each pipeline (the pipeline's **Steps** tab). **Your submissions always follow your pipeline's setting.**

- `GET /api/agent/stages` → every stage with `enabled` and, for a switched-off one, `disabled_reason`.
- A switched-off stage is **not run**. It is still **listed in the run**, so you can see it was skipped and why:
  its job has `raw: "DISABLED"` (status `skipped`), a `Disabled by admin` step carrying the reason, and `disabled: true`.
  The run detail has `disabled_stages: [...]`.
- Stages that cannot work without a switched-off one are switched off with it (determinism needs oracle+nop; the trial
  analysis needs the trial; the difficulty check needs the analysis).
- The options `skip_codex`, `force_codex`, `stages` and `disabled_stages` in a submission **have no effect** for an agent. The
  submit response lists what was ignored in `ignored_options`, and what really applies in `disabled_stages` and `steps_from`.
- A `PASS` with checks switched off is **not a full evaluation**: the verdict note and the report say which checks did not run,
  and a missing agent trial additionally says "difficulty is NOT proven". Do not present such a result as upstream-ready.
  If you need a check the admin has switched off, ask the admin.

---

## 1. Submit a task: upload it, then start the run

The task travels as an **archive** (`.zip`, `.tar.gz`, `.tgz`, `.tar.xz`, `.tar.bz2`, `.rar`, `.7z`). Two steps:

```bash
BASE=http://127.0.0.1:<port>; PL_ID=osc-tb; PL_KEY=lpk_...    # the address of this guide; the id and key are in Agent access
zip -r task.zip my-task/                 # task.toml, instruction.md, environment/, tests/, solution/ ... (at the top of the zip, or in one folder)

# 1. send it: the raw bytes are the body; the answer is an upload_id
curl -s -X POST $BASE/api/agent/upload -H "X-Pipeline-Id: $PL_ID" -H "X-Pipeline-Key: $PL_KEY" \
     -H "X-Filename: task.zip" --data-binary @task.zip
#  -> {"upload_id": "u_0c1f...", "name": "task.zip", "bytes": 48213, "sha256": "...", "expires_in": 86400}
#     (optionally send X-Sha256: <hex> and the server checks the upload against it)

# 2. start the run from it
curl -s -X POST $BASE/api/agent/submit -H "X-Pipeline-Id: $PL_ID" -H "X-Pipeline-Key: $PL_KEY" \
     -H "Content-Type: application/json" -d '{"upload_id": "u_0c1f...", "note": "first look"}'
```

An upload belongs to your pipeline, is used by **one** submission and is deleted after it (or after 24 hours). For a small task you may skip step 1 and put the archive in the
submission: `{"archive_base64": "<base64>", "archive_name": "task.zip"}` (limited by the same size limit).

What is **checked before anything is unpacked** — a task that breaks a rule is refused with `400` and a message, and nothing is written:
no `..` or absolute names or drive letters in the archive, **no symbolic links or special files**, no encrypted archive, at most 20 000 files, at most 1 GB unpacked
(512 MB per file), paths not absurdly long or deep.

Optional fields in the submission body:

| field | meaning |
|---|---|
| `project` | pin the slug (default: `task.toml`'s `name`, else the folder name) |
| `pr` | submit a new version to this **open** pull request of `project` (a fix push) |
| `new_pr` | `true` → always open a new pull request instead of continuing the open one |
| `note` | a short note shown on the version ("fixed the duplicate-path check"), at most 4000 characters |
| `title` | the pull request's title, at most 200 characters |
| `options.fail_fast` | `false` → report every stage instead of stopping at the first failure. By default a step moves the run on only when it passed: the first step that fails (or gives no result) stops the run and the later steps are not started (`stopped_at` in the run says which) |

`path`, `archive_path` and `file` (a location on the server's computer) are **refused for an agent** (`403`) unless the admin listed that folder as one you may name
(Agent access → *Folders on this computer an agent may name*; empty by default). Upload instead.

Response (`201`):

```json
{
  "pipeline": "osc-tb", "project": "oci-layer-inventory", "pr": 1, "version": 2, "run_id": "20261014-160413-6766af",
  "run_number": 3, "base": "v1", "stats": {"files": 4, "additions": 96, "deletions": 4},
  "disabled_stages": ["verifier"], "steps_from": "settings", "ignored_options": [],
  "urls": {"pr": "/#/p/osc-tb/oci-layer-inventory/pr/1", "run": "/#/run/20261014-160413-6766af",
           "api_run": "/api/agent/run/20261014-160413-6766af", "poll": "/api/agent/run/20261014-160413-6766af/poll",
           "events": "/api/agent/run/20261014-160413-6766af/events", "guide": "/agent-guide.md"}
}
```

**Behaviour**: submitting the same task again (same slug) continues its **open** pull request as a new version — exactly
like pushing a fix to a GitHub PR — and the new version's `stats`/diff show what you changed since the previous version.
Use `new_pr: true` to start a fresh PR (its diff is against the project's merged baseline), or `pr: <n>` together with
`project` to target a specific open PR.

### Limits (set by the admin, shown in `GET /api/agent` as `limits`)

| limit | default | what happens when you hit it |
|---|---|---|
| runs at once (queued or running) for your pipeline | 3 | `429` "already has N run(s)…"; wait for one to finish |
| submissions per hour (a re-run counts) | 30 | `429` with `Retry-After` (seconds) |
| uploads per hour | twice the submissions | `429` with `Retry-After` |
| biggest archive | 256 MB | `413` before the body is read |
| live connections (event streams, long polls, waits) | 8 | `429`, `Retry-After: 5` |
| uploads waiting for a submission | 20 | `429`; submit them or wait for them to expire |

Respect `Retry-After`; do not retry in a tight loop (the helper script backs off by itself).

## 2. Watch it live

Every line the pipeline writes reaches you **as it is written**. There are two ways, with the same events and the same cursor.

### a) Long poll — works everywhere, one request per batch of news

```bash
cursor=""
while true; do
  r=$(curl -s "$BASE/api/agent/run/<run_id>/poll?wait=25&cursor=$cursor" -H "X-Pipeline-Id: $PL_ID" -H "X-Pipeline-Key: $PL_KEY")
  # print what is new:   .events[] | select(.type=="log") | .text
  cursor=$(echo "$r" | jq -r .cursor)          # send the cursor of the last answer next time
  [ "$(echo "$r" | jq -r .done)" = "true" ] && break
done
```

`GET /api/agent/run/<id>/poll?cursor=<cursor>&wait=<seconds, at most 30>` answers **as soon as something happened** (or after `wait` seconds with no events and the same
cursor, so asking again is free). Answer: `{"cursor": "...", "events": [...], "done": false, "more": false}`. The first request has no cursor (you get the log from its
start; `cursor=now` starts at the present). `more: true` means more log is waiting: ask again at once. A cursor is opaque: use the one you were given. The **same cursor never repeats a line and
never skips one**, whatever your connection did in between, so after a network error just ask again with the last cursor.

Events:

| `type` | fields | when |
|---|---|---|
| `status` | `status` (`queued`/`in_progress`/`completed`), `conclusion`, `verdict`, `current_stage`, `jobs[]` (`id`, `n`, `name`, `status`, `raw`, `disabled`, `sec`, `note`) | the run's or a job's status changed |
| `log` | `stage` (which stage printed it; `null` when a batch spans a stage change), `text` (whole lines, sanitised) | new output |
| `done` | `status`, `conclusion`, `verdict`, `report_url` | once, after the last line of a finished run |

### b) Server-sent events — one connection, streaming

```bash
curl -N "$BASE/api/agent/run/<run_id>/events" -H "X-Pipeline-Id: $PL_ID" -H "X-Pipeline-Key: $PL_KEY"
```

The same `status` / `log` / `done` events as `event:` lines. The last event of each batch carries an `id:` (the cursor): reconnect with `Last-Event-ID: <id>` (or `?cursor=<id>`) and you
continue exactly where you stopped. A `: keep-alive` comment arrives every 15 s on a quiet stream; the stream closes after two hours (reconnect). Use the poll behind a proxy that buffers.

### Other ways to look

- `POST /api/agent/run/<id>/wait {"timeout": 3600}` — blocks until the run finishes (or the timeout, `202`), then returns the full detail **with the report**.
- `GET /api/agent/run/<id>?report=1` — the job list, the per-stage Codex account (`codex`), and the final report.
- `GET /api/agent/run/<id>/job?stage=<stage>` — one job with **annotations** and its **steps** (the pipeline's summary
  sections): each step has a name, a status and its log lines, so you can read exactly where it failed.
- `GET /api/agent/run/<id>/log?stage=<stage>&tail=200000` — the tail of one stage's log.
- `GET /api/agent/run/<id>/accounts` — which Codex account worked on which stage, its 5-hour headroom and the account
  pool table (names only, never e-mails or tokens).

Statuses you will see: run `status` ∈ `queued | in_progress | completed`; `conclusion` ∈ `success | failure | cancelled |
neutral`; per-job `status` ∈ `queued | in_progress | success | failure | skipped`; per-step `status` ∈ `success | failure |
skipped | in_progress | queued`. The raw pipeline status (`PASS`, `FAIL`, `INFRA`, `OVERRIDDEN`, `PIPELINE_BUG`,
`SKIPPED`, `NOT_RUN`, and `DISABLED` for a check the admin switched off) is each job's `raw` field in `run.jobs[]`
(`id` = the stage id, `status` the GitHub-style status).

The run detail also carries the verdict: `PASS` (all required stages clean), `FAIL` (a task defect), or `PIPELINE_ERROR`
(an evaluator-side problem — not your task; retry). A `PASS` with checks switched off, or whose agent trial did not run,
is not a full evaluation: the run's `verdict_note` and the report say so.

## 3. Fix and resubmit

1. Read the failing job: `GET /api/agent/run/<id>/job?stage=<stage>` → `annotations` (errors/warnings, like GitHub job
   annotations) and `blocks` (the log, split by `::group::`).
2. Change the task files, zip them again.
3. Upload the new archive and submit it to the same project: `POST /api/agent/submit {"upload_id": "...", "note": "fixed X"}`. It continues
   the open PR as the next version; you get a new `run_id` to watch. `GET /api/agent/pr/<slug>/<n>/files` shows the diff
   (added/modified/removed, +/- counts, unified patch per file via `/patch?path=<file>`).

Common failures and what they mean:

| stage / note | fix |
|---|---|
| `01 static`: canary missing, headings in instruction.md, unpinned `FROM`/pip, CRLF | fix the file the check names |
| `02 integrity`: missing files, agent-visible leaks, hardcoded fixtures, reward never 1 | align the task with the folder layout; remove leaks |
| `04 oracle+nop`: oracle ≠ 1.0 or nop ≥ 1 | make the reference solution pass the tests, and a do-nothing agent fail |
| `05 determinism` | seed randomness, fix ordering, no clock/network dependence |
| `06/07 similarity` | the task is too close to an existing one: make it distinct |
| `08 rubric` / `09 deep review` | implementation quality, bloat; often `OVERRIDDEN` automatically by triage |
| `10 verifier audit` | impossible/undisclosed tests, wrong solutions passing, reward path |
| `11 hardcoding audit` | the oracle embeds expected values, or difficulty comes from volume instead of reasoning |
| `13 trial` / `15 difficulty` | the agent solved the task (too easy) or failed for a reason unrelated to its difficulty |

### The MLE pipeline (`rwe-mle`, RealWorldEngineer ML-engineering tasks)

If your key opens the MLE pipeline, the stages are the GitHub checks of the RWE repo, in this order: `overview, static, rubric, validate, kit, checks_passed, trial, analysis, bar, cheat,
cheat_analysis` (`GET /api/agent/stages` is the truth). The steps run **in strict order**: a step moves the run on only when it passed, so the first step that fails (or gives no result) stops the run and the steps after it are not
started (`stopped_at` in the run names it). In particular the paid agent stages (`trial`, `cheat` and what reads them) never start while `static`, `rubric` or `validate` is red. Submit one task folder (`task.toml`, `instruction.md`, `environment/`, `tests/`, `README.md`);
submitting to the same project again is a task fix and needs a new `## Change Log` bullet in the README.

| stage / note | fix |
|---|---|
| `static`: a named check of `static-checks` | fix the file the check names; the checks are deterministic |
| `rubric`: a failed criterion (the explanation cites the file and a quote) | fix it at its source. A criterion marked **at risk** passed, but one review pass (or a failure rated minor, or one a second look overturned) flagged it: the real reviewer may raise it, so read what was found and fix it if it is right. "Worth a second look" is a strict reader's list: it never fails the stage |
| `validate`: invalid anchors / oracle below its anchor / nop above the floor / regrade spread | fix `[metadata.scoring]` and `evidence/scores.json`, the solution, or the starting state or verifier that hands out credit; a **regrade** spread means the verifier gives the same submission different rewards: remove the noise (seeds, timing, sampling), do not widen `regrade_tolerance` |
| `INFRA` anywhere (auth error, rate limit, a container that did not start, a task bigger than this PC's Docker, a GPU task) | **not your task**: change nothing and rerun, or tell the admin |
| `analysis`: `task_specification` or `reward_hacking` failed on a real trial | the instruction does not say what is scored, or the score can be earned by a shortcut |
| `bar`: saturated | the agent scored at or above `min(0.666-bar, 0.5)`: make the task harder in content, never move the anchors |
| `cheat_analysis`: flagged / not established | close the weakness the analysis names and add a regression case; a refusal or an early stop is not evidence that the verifier holds |

Do not copy the maintainers' slash commands into anything you write: the reports of this pipeline already put them into words.

## 4. Manage pull requests and runs

- `GET /api/agent/projects` — every project with its last run.
- `GET /api/agent/projects/<slug>` — project, PRs, runs, merged baseline.
- `GET /api/agent/pr/<slug>/<n>` — PR with versions, runs, head, latest run and per-stage `checks`.
- `GET /api/agent/pr/<slug>/<n>/files` — the head version's diff (added/modified/removed, +/− counts).
- `GET /api/agent/pr/<slug>/<n>/patch?path=<file>` — the unified patch of one file.
- `POST /api/agent/pr/<slug>/<n>/status {"status":"merged"}` — mark the PR merged (its head version becomes the project's
  baseline, so later PRs diff against it). Also accepts `closed` / `open`.
- `GET /api/agent/actions` — every run, newest first.
- `POST /api/agent/run/<id>/rerun` — run the same frozen version again (new run number). It follows the admin's current step
  settings and counts against your limits, like any agent submission.
- `POST /api/agent/run/<id>/cancel` — stop a run (a queued run leaves the queue; a running one is stopped).

## 5. Status codes

| code | meaning |
|---|---|
| `400` | the request or the task is not acceptable; the message says why (a rule of the archive, a bad cursor, an unknown field value) |
| `401` | no key, a wrong key, or an expired key |
| `403` | the pipeline is switched off, closed to the API, or **private** (it answers on the server's own computer only: ask the admin to make it public); or you named a path on the server |
| `404` | no such run, project, pull request or upload **in your pipeline** (another pipeline's are 404 too) |
| `411` | a request body without a `Content-Length` (chunked bodies are not accepted) |
| `413` | the archive is bigger than your pipeline's limit |
| `429` | too many wrong keys, too fast, too much at once: wait `Retry-After` seconds |
| `500` | an error inside the server; the reference in the message identifies it for the admin |
| `503` | the server is busy; try again in a couple of seconds |

## 6. Notes

- A full run takes ~30–60 minutes (the trial and the oracle runs use real Daytona sandboxes and Codex quota). A local run executes **one**
  agent trial, and the admin can cap paid trials per task. The admin may have switched the agent trial off: check
  `GET /api/agent/stages` before you expect a difficulty result.
- Stage ids and names: `static, integrity, artificiality, oracle_nop, determinism, similarity, semantic_similarity, rubric, deep,
  verifier, hardcode, qc_gate, codex, trial_analysis, difficulty` (`GET /api/agent/stages`).
- One machine, several agents: submissions are queued if another run is active (`max_parallel_runs` in App settings); a
  queued run starts by itself when the slot frees.
- The task you upload is **untrusted input that the pipeline runs**: do not put secrets in it. Whatever a task's scripts print appears in the run's log, which the admin
  and anyone with access to the pipeline can read.