BioTradingArena
Sign In

Benchmark

Benchmark Results

How each prediction strategy performs on the 413-case biotech catalyst benchmark

Scoring Methodology

Close-to-Close Impact Score

Every case uses the same reproducible return window: the previous trading-session adjusted close to the first trading-session adjusted close on or after the reviewed event date.

impact_score = clamp(percent_change / 5, -10, 10)

Historical market capitalization is not used because it could not be verified consistently across the full dataset. The legacy adjusted_score field is retained only as an alias of impact_score.

Example Calculations:

% Change:
-20%
Calculation:
-20 / 5 = -4.0
Final Score:
-4.0
Category:
Very negative
% Change:
+2%
Calculation:
2 / 5 = 0.4
Final Score:
0.4
Category:
Neutral
% Change:
+20%
Calculation:
20 / 5 = 4.0
Final Score:
4.0
Category:
Very positive

Categorical Evaluation Metrics

Exact Match:
Prediction matches the actual category exactly (e.g., predicted "positive" and actual was "positive")
Close Match:
Prediction is within 1 category (e.g., predicted "positive" and actual was "very_positive" or "slightly_positive")
Direction Correct:
Prediction has the correct sign (positive/neutral/negative)

Numeric Evaluation Metrics

MAE:
Mean Absolute Error between predicted and actual scores (lower is better)
Pearson Correlation:
Linear correlation between predictions and actual outcomes (higher is better, range -1 to +1)

Evaluation Framework

Dataset Overview

413
Audited Stock-Event Cases
De-identified events with known outcomes
233
Oncology Cases
Cancer drug trials and FDA decisions
180
Non-Oncology Cases
Other therapeutic areas

How Predictions Are Compared

  1. 1
    Strategy makes prediction: Each strategy analyzes the press release and clinical trial data to predict stock impact
  2. 2
    Actual outcome measured: Stock price change is calculated from the previous trading-session adjusted close to the first session adjusted close on or after publication
  3. 3
    Impact score calculated: The return is divided by five and clamped to the -10 to +10 range
  4. 4
    Metrics calculated: Exact match, close match, direction accuracy, MAE, and correlation metrics are computed

7-Category Impact Scale

The benchmark uses a standardized 7-category scale derived from the adjusted-close impact score.

VERY NEGATIVE
Impact score: < -3
NEGATIVE
Impact score: ≥ -3 and < -1
SLIGHTLY NEGATIVE
Impact score: ≥ -1 and < -0.4
NEUTRAL
Impact score: ≥ -0.4 and ≤ +0.4
SLIGHTLY POSITIVE
Impact score: > +0.4 and ≤ +1
POSITIVE
Impact score: > +1 and ≤ +3
VERY POSITIVE
Impact score: > +3

Note: Numeric predictions are evaluated against impact_score, not raw percent change. Score boundaries are applied exactly as shown above.

De-identification & Data Cleaning

Before AI models score cases, each press release is de-identified and direct answer-bearing fields are removed from the model-input endpoint.

Three-Stage De-identification Pipeline

01

Regex pre-processing

Strips wire service attributions, location datelines, URLs, email addresses, stock exchange references, NCT trial IDs. Pre-redacts known trial names, institutions, cities, and conferences with placeholders.

02

LLM redaction (GPT-5)

Identifies and replaces identifying information with standardized placeholders. Generalizes unique identifiers (first-in-class mechanisms, unique trial names) to broader categories.

03

Regex post-processing

Catches any remaining identifiers missed by the LLM: trial names, city names, institution names, companion diagnostics, and stock exchange references.

Placeholder tokens

[COMPANY][DRUG][TICKER][DATE][EXECUTIVE][TRIAL_NAME][LOCATION][INSTITUTION][TRIAL_ID][CONFERENCE][FINANCIAL_DETAIL]

What Gets Preserved vs. Removed

What Gets Preserved

  • Efficacy data (response rates, survival, p-values)
  • Safety signals and adverse event profiles
  • Trial design (endpoints, enrollment, arms)
  • Mechanism of action (generalized)
  • Regulatory context and decision rationale

What Gets Removed

  • Company and drug names
  • Ticker symbols and financial details
  • Named investigators and executives
  • Specific trial names (e.g., KEYNOTE-XXX)
  • Institution and conference names
  • Dates, locations, and URLs

Current Input Contract

Benchmark inputs use opaque case IDs and exclude exact event dates, ticker-bearing identifiers, OHLCV price action, and ground-truth labels.

Evaluation boundary: The cases endpoint supplies inputs only. Ground truth is applied server-side by the verify and submit endpoints after a prediction is provided.

Ready to Run Benchmarks?

Execute strategies against the 413-case benchmark to see which approach performs best at predicting biotech catalyst outcomes.

View All Strategies