Creating an Eval

How to create an eval on the Datalab.

Portex makes it easy to monetize your expert domain knowledge through custom evaluations or "evals" that you can offer to model builders on the Datalab. Follow this guide to get started with monetizing evals on the Datalab.

Creating an Eval Dataset Bundle

To get started, create a new dataset and choose Eval Dataset - from here you will be prompted to upload your files. Eval datasets on Portex are uploaded as a bundle of 4 files (2 optional). This forms a Core Dataset which can be offered to sale to model builders in addition to per-eval runs. We walk through file in the Core Dataset each below.

Formatting your Eval Dataset Bundle

There are four files in an Eval Dataset Bundle. A Task List, Reference Files (Optional), Answer Key, and Knowledge Reference (Optional). Below is a description of each dataset and how to structure it.

---

🧩 Task List

A JSON file containing your tasks. Each record must include:

• task_id: unique identifier

• task_prompt: the question for the model

• If you’re referencing files (e.g. a 10-K PDF), include task_reference_file

Optional fields (you may include other metadata fields, just make sure you cover the three above):

• skills_tested: metadata about the specific skills tested by the task

• difficulty: an assessed difficulty of the question

• question_type: e.g. multiple choice, exact answer, summary

Example:

[
  {
    "task_id": "apple_net_margin_2024",
    "task_prompt": "Using Apple’s FY 2024 10-K, compute its net profit margin (net income ÷ revenue).",
    "task_reference_file": "apple_2024_10K.html"
  }
]

This file will be downloadable by eval buyers so they can generate responses.

---

📂 Reference Files (Optional)

If your tasks refer to documents, images, or other supporting files, bundle them into a .gz archive. This is only needed if you have reference files that accompany your tasks list.

Example archive structure:

reference_files/
  apple_2024_10K.html
  revenue_breakdown_chart.png

Buyers will extract this archive to access the referenced files.

---

🧠 Answer Keys

A JSON file mapping task_id to the correct output. Must include:

• task_id

• answer: the expected result (numerical, textual, set of objective criteria etc.)

Optional fields:

• rationale: explanation of how the answer was derived

• answer_type: e.g. "percentage", "multiple_choice", "criteria"

• knowledge_reference_file: filename from the Knowledge Reference archive that supports the answer e.g. expert notes or chain-of-thought steps

Example:

[
  {
    "task_id": "apple_net_margin_2024",
    "answer": 0.2397,
    "rationale": "Net income $93,736M ÷ Revenue $391,035M = 23.97%",
    "answer_type": "percentage",
    "knowledge_reference_file": "expert_notes.csv"
  }
]

The answer key will be available for purchase in the core dataset price.

---

📘 Knowledge Reference (Optional)

A .gz archive containing additional files that support your reasoning or calculations (analyst notes, data tables, or expert commentary).

Example archive structure:

knowledge_reference/
  expert_notes.csv
  margin_calculation_factors.csv

Buyers can use these to validate or audit the answer logic, or even refine their models using reinforcement learning methods. This will be offered for sale within the core dataset pricing.

Creating an Eval Listing

Once you've uploaded your eval dataset, you can create a listing for it.

The first step will be setting two prices:

1. A per-eval price: a set price model builders pay each time they submit model responses and receive a performance report.

2. Core Dataset price and minimum bid: a fixed "buy now" price and minimum bid for model builders to access the Core Dataset (tasks and answers, as well as reference files and knowledge reference if applicable).

The next step is configuring your license and all other relevant details on the listing editor. Once you publish your listing, your eval will be ready for model builders to access.