Terra

Reinforcement learning environment for training agentic LLMs on real-world multi-step tasks with verifiable rewards.

Terra extends the GAIA evaluation methodology into a full RL training loop — producing clean binary reward signals suitable for GRPO-based RLVR training. Built on the goku multimodal agentic-evaluation engine, OpenHands, and the Agent SDK.

This is the harness submodule of the Terra project. Companion repos: dataset (the 10K corpus) · trajectories (recorded runs) · delivery (client deliverable) · samples (public evaluation sample). Terra's task path lives at benchmarks/gaia/.

• 10K-task corpus — GAIA-derived tasks across 3 difficulty levels
• Sandboxed tool surface — web search, file I/O, code execution
• Binary reward oracle — exact-match scoring for GRPO/RLVR
• Trace collection — full action/observation histories for offline training
• End-to-end pipeline — inference → scoring → trace export in one harness

┌─────────────┐     ┌──────────────────┐     ┌────────────────┐     ┌────────────┐
│  GAIA Tasks │────▶│  Agent Execution │────▶│  Reward Oracle │────▶│   Traces   │
│  (10K)      │     │  (sandboxed)     │     │  (exact-match) │     │  (RLVR)    │
└─────────────┘     └──────────────────┘     └────────────────┘     └────────────┘

1. Tasks — Multi-step questions requiring web search, file processing, and reasoning
2. Agent execution — LLM agents solve tasks in sandboxed environments (browser, terminal, code interpreter)
3. Reward signal — Binary exact-match against gold answers → clean signal for GRPO
4. Traces — Full conversation histories stored for offline RL training

• Python 3.12+
• uv ≥ 0.8.13

git clone git@github.com:EtharaOrion/terra-harness.git
cd terra-harness
make build

# Create config
cat > .llm_config/your-model.json << 'EOF'
{
  "model": "your-model-name",
  "base_url": "https://your-endpoint",
  "api_key": "YOUR_API_KEY"
}
EOF

# Validate
uv run validate-cfg .llm_config/your-model.json

TAVILY_API_KEY=xxx uv run gaia-infer .llm_config/your-model.json \
    --level 2023_level1 \
    --split validation \
    --num-workers 4

uv run python -m benchmarks.gaia.get_score --file outputs/gaia/output.jsonl

run_terra.sh drives every stage from the Harbor task bundle to the final delivery bundle. It lives in harness/ but operates on the Terra root (the same TERRA_ROOT := .. convention the Makefile uses), so it works from any directory.

harness/run_terra.sh --infer --oracle-run --uuid <uuid> --probe-models

That single invocation runs, in order: the pre-spend gate (LLM proxy reachable, docker daemon up, and the mandatory oracle/NOP task test) → Harbor bundle converted to JSONL in a temp file → oracle reference run → model run → generated pytest suite and rubric frozen → judge council and pytest executed → verifier output written under trajectories/<uuid>/verification/ → final bundle assembled under delivery/<uuid>/ → delivery layout conformance check.

Runs are stored as trajectories/<uuid>/<model>/run_N/, and each --infer pass allocates the next free N rather than overwriting. A trajectory is already scored by the time a second one exists, so there is no flag to overwrite it — re-running a model adds run_2 beside run_1.

<model> is the canonical slug: the model id with its routing vendor and the claude- product prefix stripped, the rest kept verbatim — anthropic/claude-opus-4-8 → opus-4-8, openai/gpt-5.6-sol → gpt-5.6-sol. canonical_model_slug in verification/terra/harbor.py is the only implementation; this script and the delivery emitter both call it, and --model is normalized through it as well, so one model cannot end up owning two directories.

Each run carries its own verifiers/ (report.json, rubric_results.json, pytest_results.json), the same shape the delivered bundle uses, so a run directory is self-describing. The instrument those verdicts came from is not per-run: the rubric and generated suite are frozen once per task under verification/verifiers/, because a rubric stored per run is a rubric that can be edited per run.

harness/run_terra.sh --help          # every flag and its default
harness/run_terra.sh --dry-run …     # print the exact commands, execute nothing

# score and deliver what is already on disk, no model calls at all
harness/run_terra.sh --rubric-deterministic --skip-judge --no-bridge

Three properties are enforced rather than merely defaulted. The oracle/NOP gate has no skip flag anywhere in the script: the task's own solution must score 1.0 and a no-op must score 0.0, because a corpus built on a task that does not discriminate is worse than no corpus — the numbers look fine. The JSONL is never written into dataset/: the bundle is content-addressed (uuid5 over its own file hashes), so an extra file would re-identify the task and break recompute.py verify; conversion goes to a temp path and the converter refuses any output path inside the dataset root. And a delivered run must be judged: --skip-judge implies --skip-deliver, because an unjudged run has fewer decided checks and so reports a misleadingly higher graded_score than a judged one.

Exit codes: 0 every run accepted · 1 at least one quarantined or the delivered layout does not conform · 2 blocked by the pre-spend gate.

Every flag accepted by run_terra.sh. Defaults in brackets.

Task authoring — off by default; the bundle usually already exists.

Trajectory generation — off by default; enable with --infer.

Scope

Pre-spend checks — run before any model call.

Stages — all run by default.

Judge — a council of distinct models; the verdict is the vote, not one model.

General

Environment — every judge knob is also an env var; the flags above just export them.

The stages can also be run individually — see COMMANDS.md in the knowledge repo for the per-stage invocations and the make targets. (harness/ is its own repository, so that file is one level up in the parent checkout rather than in this tree.)

Validation subset (20 tasks; Level 1 = 4, Level 2 = 7, Level 3 = 9), end-to-end with full execution traces:

Frontier cohort — PENDING PILOT. The next evaluation cohort is GPT-5.6-Sol and Claude Opus 4.8. Those runs are not yet complete, so no frontier numbers are reported here; this table is populated only from real scored trajectories. See the knowledge repo and the dashboard.

ethara/terra on HuggingFace — 10,000 GAIA-derived tasks.

Also includes full agent execution traces for Kimi K2.5 and Qwen3 VL.

terra-harness/
├── benchmarks/
│   └── gaia/
│       ├── run_infer.py        # Inference pipeline
│       ├── get_score.py        # Exact-match scoring
│       └── README.md           # Benchmark-specific docs
├── vendor/
│   └── software-agent-sdk/     # OpenHands Agent SDK (submodule)
├── .llm_config/                # LLM configuration files
├── outputs/                    # Inference results & traces
├── Makefile                    # Build automation
└── pyproject.toml              # Dependencies & CLI entrypoints

make build       # Install deps + pre-commit hooks
make format      # Format with ruff
make lint        # Lint with ruff
make pre-commit  # Run all pre-commit checks
make clean       # Remove cache files

See CONTRIBUTING.md for guidelines.

@misc{terra2026,
  title={Terra: A Verifiable-Reward RL Environment for General AI Assistants},
  author={Ethara AI},
  year={2026},
  url={https://github.com/EtharaOrion/terra}
}

MIT — see LICENSE for details.