PertEMA model card

A post-hoc, model-agnostic reliability estimator for single-cell perturbation predictors. Version 0.1.0. Provenance flag CLEAN

Model

Type: xgboost gradient-boosted tree (GBT)
Hyperparameters: n_estimators=300, max_depth=6, learning_rate=0.05
Post-processing: isotonic calibration, then a split-conformal interval
Version: 0.1.0

Training data

Source: Gladstone CD4 activation-transfer errors of the per-condition mean predictor, all folds
Training perturbations: n = 64,024
Data hash (sha256[:16]): 95dd4facf378a78b
Seeds: 42, 43, 44

No-leakage rule: a test perturbation's true error never reaches the estimator during training. The estimator is trained on out-of-fold errors on gene-disjoint splits.

Feature spec

Eleven leakage-safe, prediction-time feature groups. None uses any quantity derived from a perturbation's true effect, so every group can be computed at inference before the ground truth is known.

Feature group	Plain description	Leakage
`pred_magnitude`	Size of the predicted perturbation effect (the predictor's own output magnitude).	no true effect
`baseline_src`	Control-state mean expression in the source context.	no true effect
`dropout_src`	Fraction of zero counts (dropout rate) in the source context.	no true effect
`donor_var_src`	Cross-donor variability of the gene in the source context.	no true effect
`baseline_dst`	Control-state mean expression in the destination context.	no true effect
`dropout_dst`	Fraction of zero counts (dropout rate) in the destination context.	no true effect
`donor_var_dst`	Cross-donor variability of the gene in the destination context.	no true effect
`src_onehot(Rest,Stim8hr,Stim48hr)`	One-hot indicator of the source activation context.	no true effect
`dst_onehot(Rest,Stim8hr,Stim48hr)`	One-hot indicator of the destination activation context.	no true effect
`coexpr_embedding(50d)`	50-dimensional co-expression neighborhood embedding of the perturbed gene.	no true effect
`training_set_similarity`	How similar this perturbation is to the estimator's training set.	no true effect

Target

Per-perturbation transfer error, defined as 1 - Pearson on the source-training high-variance gene set. Lower is better. The estimator predicts this error, then reports it as a reliability score and a calibrated error with a conformal interval.

Calibration (held-out)

Expected calibration error: raw 0.0076, improved to isotonic 0.0028
Conformal coverage: 0.900 at the 0.90 target

These are held-out numbers from the frozen evaluation, not in-sample. The empirical coverage is reported on every result surface (invariant N2).

Accuracy ceiling

Accuracy ceiling: Pearson r = 0.73 on hit genes, 0.11 on all genes (the best any predictor could reach given cross-donor replicate noise). Higher is better. The corresponding 1 - Pearson error ceiling is about 0.27 on hit genes and 0.89 on all genes. This ceiling is low, which is why gains from any reliability layer on the noisy primary data are modest, and it is why the honest out-of-fold reliability is reported next to every result rather than a headline win.

Provenance note: the frozen artifact stores this under the key measured_accuracy_ceiling_1minus_pearson, but the stored value is Pearson r, not 1 - Pearson. The key name is a known label bug held for a future model version. The value is read as-is and labeled correctly as Pearson r here.

Intended use and out-of-scope

Intended use. Rank which predictions from a single perturbation predictor to trust, so a user can abstain from the least-reliable predictions (selective abstention, a within-predictor ranking).

Out of scope.

NOT predictor routing. Picking the most-reliable predictor per perturbation does not beat the best single predictor, on either the noisy primary data or the cleaner external Replogle axis.
NOT biological importance. PertEMA scores which predictions to trust, not which genes matter, and it does not recover known CD4 and IL-2 regulators better than a trivial magnitude heuristic.
On single-context Norman, the PRESCRIBE method's intrinsic uncertainty beats PertEMA. Gains are small and are reported against the measured accuracy ceiling above.

Honest scope (verbatim from the artifact): "Reliability estimator. It scores which predictions to trust, not which genes are important. Gains are modest on noisy data."

SHAP attribution explains the estimator, not biological importance.

Provenance, license, and regeneration

Provenance flag: CLEAN (no known pretraining overlap with benchmark data)
Version: 0.1.0
Regenerate: rebuild the frozen artifact with freeze_model.py
Citation: No published paper yet — cite the software: Bishal Shrestha, PertEMA: Perturbation Error Meta-Assessment (v0.1.0), 2026, github.com/OfficialBishal/PertEMA. The repository's CITATION.cff generates BibTeX and APA.

Numbers on this card come from the frozen artifact provenance. The page attempts a same-origin fetch of /model-card.json to stay in sync with the artifact and falls back to these baked values if it is unavailable.