PertEMA: methods and honest scope

PertEMA is a post-hoc, model-agnostic reliability estimator. It wraps any single-cell perturbation predictor and predicts the per-perturbation error of its outputs from prediction-time features only, so it can say which predictions to trust. It is a reliability layer, not a predictor, and it does not identify biologically important genes.

Why a reliability layer

Two 2025 benchmark papers showed that deep perturbation predictors often do not beat a trivial mean baseline. We reproduced that on our own roster of twelve predictors: six trained-from-scratch (constant mean, linear ridge, a nonlinear MLP, local kNN, no-change) and six frozen-adapted foundation-model predictors (ridge and kNN heads over the pretrained gene embeddings of Geneformer, scGPT, and gene2vec). None beats the per-condition mean baseline, and the best foundation predictor comes closest at 1.001 times the mean error but still loses. The useful next question is not a better predictor but a calibrated way to say which predictions are worth acting on.

How it works

A gradient-boosted tree is trained on a predictor's out-of-fold errors on gene-disjoint splits, using only prediction-time features (predicted effect magnitude, control-state baseline expression and dropout, a co-expression embedding, source and destination context indicators, and training-set similarity). No quantity derived from a test perturbation's true effect is ever a feature. The predicted error is recalibrated with isotonic regression and wrapped in a split-conformal interval.

What we show (measured)

It ranks realized error above simple heuristics, a deep-ensemble intrinsic uncertainty, and no selection across three transfer settings, with clean negative controls.
The clearest result is external and cross-cell-line: on the Replogle K562-to-RPE1 screens it closes 58 percent of the gap between no selection and a perfect oracle (Spearman 0.567).
Calibration: isotonic expected calibration error 0.008 to 0.003, split-conformal empirical coverage 0.900 at a 0.90 target.
Ranking a validation shortlist by reliability yields a more reproducible shortlist than the default effect-magnitude ranking.

What we do NOT claim (reported plainly)

PertEMA does NOT recover known CD4 and IL-2 regulators better than a trivial magnitude heuristic. Reliability and biological importance are different axes.
A routing procedure that picks the most-reliable predictor per perturbation does NOT beat the best single predictor, on either the noisy primary data or the cleaner external Replogle axis. Reliability enables selective abstention (a within-predictor ranking, which works) but not predictor routing (a cross-predictor comparison, which fails on both axes).
Gains on the noisy primary data are small and are reported against the measured accuracy ceiling (cross-donor Pearson about 0.73 on hit genes, 0.11 on all genes). On single-context Norman, PRESCRIBE's intrinsic uncertainty beats PertEMA.
The frozen-adapted foundation-model predictors (Geneformer, scGPT, gene2vec) carry an UNKNOWN-OVERLAP pretraining-provenance flag, so possible memorization of a benchmark dataset is never mistaken for clean generalization. Even with that caveat in their favor, none beat the mean baseline.

The contribution

A standardized open reliability benchmark for perturbation prediction, scoring uncertainty-quantification methods across datasets and context-transfer settings with provenance flags, plus this self-hostable tool. See the benchmark.

How to cite

No preprint DOI exists yet. Manuscript in preparation. Until a DOI is available, cite PertEMA by tool name and version 0.1.0 using the entry below.

@software{pertema_v0_1_0,
  title   = {PertEMA: a post-hoc reliability estimator for single-cell perturbation predictors},
  author  = {The PertEMA developers},
  year    = {2026},
  version = {0.1.0},
  note    = {Manuscript in preparation}
}

Citations

Ahlmann-Eltze C, Huber W, Anders S. Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. Nature Methods. 2025. doi:10.1038/s41592-025-02772-6.
Wong D, Hill A, Moccia R. Perturb-seq prediction benchmark. Bioinformatics. 2025, 41(6):btaf317. doi:10.1093/bioinformatics/btaf317.
Cheng J, et al. PRESCRIBE: Predicting Single-Cell Responses with Bayesian Estimation. NeurIPS 2025. arXiv:2510.07964.
Xing H, Yau C. GPerturb: Gaussian process modelling of single-cell perturbation data. Nature Communications. 2025. doi:10.1038/s41467-025-61165-7.
Angelopoulos AN, Bates S. A Gentle Introduction to Conformal Prediction. Foundations and Trends in Machine Learning. 2021. arXiv:2107.07511.

Full reference list and traceable numbers: see the manuscript and results/ in the repository.