Scoring & Metrics

Price Window

Every ground-truth return uses the same rule:

price_before = previous trading-session adjusted close
price_after  = first trading-session adjusted close on or after the reviewed event date
percent_change = (price_after - price_before) / price_before × 100

These values use split/dividend-adjusted closes where available. This is a date-based session return; the dataset does not infer an unrecorded release timestamp.

Impact Score

impact_score = clamp(percent_change / 5, -10, 10)

The legacy adjusted_score field is retained as an exact compatibility alias. Historical market capitalization is not used because it could not be verified consistently across all published cases.

7-Category Impact Scale

Impact categories are derived from the impact score:

These categories form an ordinal scale from 0 (very_negative) to 6 (very_positive), used for categorical evaluation metrics.

Metrics

Every submission is evaluated on three accuracy metrics:

Exact Match Accuracy

The percentage of predictions that exactly match the ground truth category.

exact_match_accuracy = (exact_matches / total_predictions) * 100

Directional Accuracy

The percentage of predictions that get the direction correct:

• Positive: slightly_positive, positive, very_positive
• Negative: slightly_negative, negative, very_negative
• Neutral: neutral

A prediction is directionally correct if both the predicted and actual impact fall in the same direction group.

Close Match Accuracy

The percentage of predictions that are within one category of the ground truth on the ordinal scale:

very_negative (0) → negative (1) → slightly_negative (2) → neutral (3) → slightly_positive (4) → positive (5) → very_positive (6)

A prediction is a "close match" if |predicted_order - actual_order| <= 1.

Mean Absolute Error (MAE)

If you submit predicted_score alongside your categorical prediction, MAE measures the average absolute difference between your predicted score and the canonical impact score.

MAE = mean(|predicted_score - actual_impact_score|)

Confusion Matrix

The verify endpoint returns a direction confusion matrix showing how your predictions distribute across actual directions:

{
  "direction_confusion_matrix": {
    "positive": { "positive": 45, "neutral": 5, "negative": 2 },
    "neutral":  { "positive": 8,  "neutral": 12, "negative": 6 },
    "negative": { "positive": 3,  "neutral": 7,  "negative": 35 }
  }
}

Rows = actual direction, columns = predicted direction.

De-identification Pipeline

Before AI models score cases, each press release passes a versioned de-identification and leakage-validation process intended to reduce direct lookup cues.

Pipeline Stages

1. Curated redaction — Candidate-specific company, drug, ticker, executive, date, location, and trial-name terms are replaced with placeholders.

2. Deterministic cleanup — The publication build removes residual exact dates, years, and known identity terms and records a redaction version.

3. Runtime isolation — The benchmark endpoint recursively redacts the press release and selected trial fields again, omits NCT IDs and sponsors, and never returns price action or labels with model inputs.

Placeholder Tokens

What Is Preserved

The pipeline preserves scientific content needed for reasoning: efficacy data (response rates, survival, p-values), safety signals, trial design details, mechanism of action (generalized), and regulatory context.

Input Isolation

The model-input endpoint uses opaque case IDs and excludes exact dates, ticker-bearing identifiers, NCT IDs, sponsors, OHLCV price action, and ground truth. The verify and submit endpoints apply labels server-side after predictions are supplied.

Leaderboard Ranking

Published leaderboard runs are ranked by exact match accuracy. The submit endpoint sends a scored run for review; it does not publish a row automatically.

Verify Before Submitting

Use the verify endpoint to test your predictions without saving them:

POST /api/benchmark/verify

{
  "predictions": [
    {
      "case_id": "case_0123456789abcdef",
      "predicted_impact": "positive",
      "confidence": 0.85
    }
  ]
}

The response includes per-case results so you can debug individual predictions:

{
  "metrics": {
    "cases_evaluated": 233,
    "exact_match_accuracy": 28.5,
    "directional_accuracy": 62.3,
    "close_accuracy": 55.0,
    "avg_confidence": 0.72
  },
  "results": [
    {
      "case_id": "case_0123456789abcdef",
      "predicted_impact": "positive",
      "actual_impact": "positive",
      "percent_change": 12.5,
      "impact_score": 2.5,
      "exact_match": true,
      "close_match": true,
      "direction_correct": true
    }
  ]
}