Creating an Eval

How to create an eval on the Datalab.

Portex makes it easy to monetize your expert domain knowledge through custom evaluations or "evals" that you can offer to model builders on the Datalab. Follow this guide to get started with monetizing evals on the Datalab.

Creating an Eval Dataset Bundle

To get started, create a new dataset and choose Eval Dataset - from here you will be prompted to upload your files. Eval datasets on Portex are uploaded as a bundle of 4 files (2 optional). This forms a Core Dataset which can be offered to sale to model builders in addition to per-eval runs. We walk through each file in the Core Dataset below.

Formatting your Eval Dataset Bundle

Eval datasets on Portex use 4 parts. Two are required.

Upload Checklist

1. Required: tasks.json

2. Required: answers.json

3. Optional: reference_files.zip

4. Optional (but recommended for premium offering): core_dataset.zip

Rule: every task_id in answer_keys.json must also exist in task_list.json

🧩 Task List

A JSON file containing your tasks. Each record must include:

• task_id: unique identifier

• task_prompt: the question for the model

• reference_file: if your task requires a reference file, include the file name here.

Example:

[
  {
    "task_id": "apple_net_margin_2024",
    "task_prompt": "Using Apple’s FY 2024 10-K, compute its net profit margin (net income ÷ revenue).",
    "reference_file": "apple_2024_10K.txt"
  }
]

This file will be downloadable by eval buyers so they can generate model responses.

---

🧠 Answer Keys

A JSON file mapping task_id to the correct output. Must include:

• task_id: unique identifier used above

• answer: the expected result (numerical, textual, objective criteria etc.)

Optional fields you might include to supplement:

• rationale: explanation of how the answer was derived, reasoning traces

• answer_type: e.g. "percentage", "multiple_choice", "exact_match"

Example:

[
  {
    "task_id": "apple_net_margin_2024",
    "answer": 0.2397
  }
]

By default, your answer key is blinded from eval buyers. You may optionally add it to the Core Dataset below for purchase.

---

📂 Reference Files (Optional)

If your tasks refer to text documents or images or other supporting files, bundle them into a .zip or .gz archive.

Example archive/zip structure:

reference_files/
  apple_2024_10K.txt
  revenue_breakdown_chart.png

Buyers will extract this archive to access the referenced files and generate model responses.

Accepted file types include: .json, .csv, .md, .html, .txt, .webp, .jpg, .jpeg, .png, .gif, .pdf

---

📘 Core Dataset (Optional)

A .zip or .gz archive that may optionally contain the answer key and/or additional files that support your reasoning or calculations (analyst notes, data tables, or expert commentary).

Example archive structure:

core_dataset/
  answers.json
  expert_notes.json
  margin_calculation_factors.json

Buyers can use this core dataset to validate or audit the answer logic, or even refine their models using reinforcement learning methods. The contents of this dataset are up to you as the expert to decide but including the answer key can generally support a higher price point.

---

Creating an Eval Listing

Once you've uploaded your eval dataset, you can create a listing for it.

The first step will be setting two prices:

1. A per-eval price: a set price model builders pay each time they submit model responses and receive a performance report.

2. Core Dataset price and minimum bid: a fixed "buy now" price and minimum bid for model builders to access the Core Dataset (tasks and answers, as well as reference files and knowledge reference if applicable).

The next step is configuring your license and all other relevant details on the listing editor. Once you publish your listing, your eval will be ready for model builders to access.