Reference vs. candidate · sealed holdout

Prove the cheaper workflow keeps its quality.

Kourum compares one reference workflow with one candidate through a vendor-diverse panel, deterministic checks, and a Final Exam that revision logic cannot read.

Define the task

Plain text or JSON. Uploaded documents are never parsed on the server.

Task definitionDescribe the inputs, required output, constraints, and acceptance boundary.

Add context or reference filesUp to 5 files, 1 MB each. PDF/DOCX extraction remains disabled until its library TBD is resolved.

Choose the pair

Choose from Kourum's verified, priced model catalog. Same-vendor pairs are allowed.

Reference workflow

VendorOne of Kourum's four configured providers

Funded modelVerified pricing and provider support are required.

Workflow instructionsThe candidate instructions may be revised; the reference remains fixed.

Reasoning effortThe only orchestration-level consistency lever; sampling fields are omitted.

Candidate workflow

VendorOne of Kourum's four configured providers

Funded modelVerified pricing and provider support are required.

Workflow instructionsThe candidate instructions may be revised; the reference remains fixed.

Reasoning effortThe only orchestration-level consistency lever; sampling fields are omitted.

Add evaluation cases

Cases are split once into Study, Practice, and the sealed Final Exam at creation.

Case 1

PromptRequired

Reference answerOptional

Task familyDefaults to General

Human-approved canonical example

Set the quality gate

Deterministic checks run before rubric aggregation; any failure blocks a pass.

Deterministic checks

Sandbox check definitions

[
  {
    "name": "candidate-output-present",
    "type": "contract",
    "config": {
      "runtime": "node24",
      "entrypoint": "check.mjs",
      "timeoutMs": 5000,
      "files": [
        {
          "path": "check.mjs",
          "content": "import fs from 'node:fs';\nconst rows = JSON.parse(fs.readFileSync('inputs/manifest.json', 'utf8'));\nif (!rows.length) process.exit(1);\nfor (const row of rows) {\n  if (row.role === 'candidate' && !fs.readFileSync(`inputs/${row.path}`, 'utf8').trim()) process.exit(1);\n}\n"
        }
      ]
    }
  }
]

Node 24 or Python 3.13 only. Each check runs in a fresh, network-denied sandbox.

Optional human anchor calibration

Upload anchor JSON

Human anchors JSONOptional array of human-approved dimension scores keyed to dataset case IDs.

Funded runs need NEXT_PUBLIC_TURNSTILE_SITE_KEY before launch.

Start with a validated Study roundThe Final Exam remains unread until you explicitly proceed.