Skip to main content

m35 · Machine learning in EP mapping: annotation, driver detection, validation pitfalls

Warning

Education only. Not for diagnosis, treatment or any clinical decision.

Note

Draft, stages 2 to 4 of 6. The derivations and three interactive demos are written. Each equation is checked by tests in tests/test_m35_evaluation.py, against scikit-learn where it has the same function, and the demos’ math in tests/test_part6_demo_math.py. The code lab is in place too. The questions and the remaining sections follow in later stages (plan: docs/outlines/m35-outline.md). Nothing here has had expert review yet.

In 60 seconds

Machine learning can annotate electrograms and flag candidate arrhythmia drivers faster and more consistently than people, but only as well as its labels and its validation allow. Labels made by experts disagree, which caps how good any model can look. Records from the same patient resemble each other, so splitting records instead of patients between training and testing makes a model look better than it is. A good score also needs the right measure: predictive values depend on how common the condition is, and predicted probabilities must mean what they say. This module is about those pitfalls and the arithmetic that exposes them.

From the field

Planned in the outline (docs/outlines/m35-outline.md); written in a later stage.

Derivations

Symbols used throughout. \(y \in \{0, 1\}\) true class. \(\hat p\) predicted probability that \(y = 1\). \(\pi\) prevalence, the share of cases with \(y = 1\). \(\mathrm{Se}\) sensitivity and \(\mathrm{Sp}\) specificity. \(m\) records per patient, \(r\) correlation between records of the same patient, \(n\) number of records. \(\epsilon\) rate at which labels are wrong.

Mapping tasks as learning problems

Most mapping tasks fit one of a few shapes. Classification assigns a class: is this electrogram local or far-field, does this region show rotational activation? Regression predicts a number, such as an activation time. Detection finds events in a signal, such as each local activation in a long recording. Every one needs labels, and the labels come from somewhere:

  • Experts. In one study, experts labeled 175,000 Hilbert-transform image grids of atrial fibrillation (AF, a rapid, irregular rhythm of the upper chambers) from 35 patients for rotational activation, and those labels trained a convolutional neural network (Alhusseini et al. 2020).
  • An independent reference. Another study validated driver detection by multi-electrode mapping against near-infrared optical mapping of explanted human atria, a reference that does not depend on reading the electrograms (Zolotarev et al. 2020).

Key takeaways

  • Name the task (classify, regress, detect) and the unit being predicted (record, site, patient).
  • Ask where the labels came from before asking how good the model is.

How much do the labels agree?

Two labelers who agree 80% of the time may be agreeing largely by chance if one class is common. Cohen’s kappa corrects for that (Cohen 1960):

\[ \kappa = \frac{p_o - p_e}{1 - p_e} \tag{1}\]

where \(p_o\) is the observed agreement and \(p_e\) the agreement expected by chance from each labeler’s class frequencies. \(\kappa = 1\) is perfect agreement and \(\kappa = 0\) is chance. The experts in the AF study above were consistent but not identical, with \(\kappa\) = 0.79 (Alhusseini et al. 2020).

Disagreement has a cost that no model can remove. Suppose the labels are wrong at rate \(\epsilon\), independently of the model’s own errors. A model whose true accuracy is \(a\) then scores

\[ a_{\text{measured}} = a\,(1 - \epsilon) + (1 - a)\,\epsilon \tag{2}\]

against those labels. A perfect model (\(a = 1\)) scores only \(1 - \epsilon\). Since the expression is \(\epsilon + a(1 - 2\epsilon)\), no model can score above \(1 - \epsilon\) while its errors are independent of the labelers’. A higher score is a warning sign: the model’s errors line up with the labelers’, for example because it has learned their habits.

Key takeaways

  • Report agreement between labelers with kappa, not raw percentages.
  • Label noise caps measured accuracy at \(1 - \epsilon\); an independent reference avoids the cap.

Splitting the data

Cross-validation estimates how a model will do on data it has not seen. That estimate is meaningful only if the split between training and validation data mimics the split between the training data and the data the model will meet in use (Saeb et al. 2017). A model meant for new patients must be tested on patients it never saw in training.

Records from one patient are correlated: same anatomy, same catheter, same noise. With \(m\) records per patient and correlation \(r\) between them, the \(n\) records carry the information of only

\[ n_{\text{eff}} = \frac{n}{1 + (m - 1)\, r} \tag{3}\]

independent ones; \(1 + (m - 1) r\) is the design effect (Killip et al. 2004). With many records per patient, even a modest correlation makes the effective sample close to the number of patients, not the number of records. Splitting records at random puts records of the same patient on both sides of the split, and the model is then partly tested on patients it has already seen.

Simulation

Assumptions

  • Model: 30 synthetic patients, half labeled 1. Each record has six features: a patient signature (a share of the variance set by the slider, the same for every record of that patient), independent noise, and the class signal along one feature.
  • Classifier: one nearest neighbour. Record-wise scoring leaves out one record at a time; patient-wise scoring leaves out all of a patient’s records. Only the patient-wise number matches use on new patients (Saeb et al. 2017).
  • All values are synthetic and dimensionless, with a fixed random seed; no clinical data.
  • What would change the result: models that memorize less than nearest neighbour (smaller gap), more patients, or features that carry no patient signature.

Key takeaways

  • Split by patient (or by site, or by hospital) to match how the model will be used.
  • Count patients, not records, when you judge how much evidence you have.

Measuring performance

Sensitivity is the share of true positives the model finds, and specificity the share of true negatives it clears. What a user needs is the other direction: given a positive result, how likely is it to be right? By Bayes’ rule (Altman and Bland 1994),

\[ \mathrm{PPV} = \frac{\mathrm{Se}\, \pi}{\mathrm{Se}\, \pi + (1 - \mathrm{Sp})(1 - \pi)}, \qquad \mathrm{NPV} = \frac{\mathrm{Sp}\, (1 - \pi)}{\mathrm{Sp}\, (1 - \pi) + (1 - \mathrm{Se})\, \pi} \tag{4}\]

The same model has a different PPV wherever the prevalence \(\pi\) differs, for example between a curated research dataset and routine use.

A model that outputs a score is judged at every threshold at once by the receiver operating characteristic (ROC) curve. The area under it has a direct meaning: it is the probability that a randomly chosen positive case gets a higher score than a randomly chosen negative one, the same quantity estimated by the Wilcoxon statistic (Hanley and McNeil 1982):

\[ \mathrm{AUC} = P\!\left( \hat p_{+} > \hat p_{-} \right) + \tfrac12 P\!\left( \hat p_{+} = \hat p_{-} \right) \tag{5}\]

(ties count one half). AUC measures ranking only. It does not depend on prevalence, and it says nothing about whether the scores are good probabilities.

Exact

Assumptions

  • Scores of negative cases are N(0, 1) and of positive cases N(\(d\), 1). Then AUC = \(\Phi(d/\sqrt2)\), because the difference of a positive and a negative score is N(\(d\), 2) (checked in tests/test_m35_evaluation.py). Predictive values from Equation 4.
  • Defaults (separation 1.5, threshold 0.75, prevalence 10%) are illustrative inputs.
  • What would change the result: real scores are rarely normal with equal variances; the ROC curve then becomes asymmetric.

Key takeaways

  • Quote PPV and NPV at the prevalence where the model will be used.
  • AUC is a ranking probability; it hides the threshold and the calibration.

Probabilities that mean what they say

When a model says “70%”, the event should happen about 70% of the time. The Brier score measures the overall error of predicted probabilities (Brier 1950):

\[ \mathrm{BS} = \frac1n \sum_{i=1}^{n} \left( \hat p_i - y_i \right)^2 \tag{6}\]

This is the binary form. Brier’s original definition sums over both classes and so gives exactly twice this value.

Calibration asks how predicted and observed risks relate. Fit

\[ \operatorname{logit} P(y = 1) = \alpha + \beta \operatorname{logit} \hat p \tag{7}\]

to the outcomes. A well-calibrated model has \(\alpha = 0\) and \(\beta = 1\). A slope \(\beta < 1\) means the predictions are too extreme, too close to 0 and 1; \(\beta > 1\) means they are too timid. Poorly calibrated models can mislead even when they rank cases well (Van Calster et al. 2019).

Simulation

Assumptions

  • Model: 6,000 synthetic cases with true logits drawn from N(0, 1.5²) and outcomes drawn from those probabilities; the model’s predictions are distorted exactly as in Equation 7. Fixed random seed.
  • The diagram shows bins of width 0.1 that hold at least 20 cases. The readout compares the Brier score (Equation 6) with that of the true probabilities.
  • All values are synthetic and dimensionless. With fewer cases, the binned points scatter more around the line.

Key takeaways

  • Check calibration as well as discrimination; a high AUC can come with misleading probabilities.
  • A model fitted too closely to too little data tends to give a calibration slope below 1 on new data (shown by simulation in the tests).

Reporting what you did

The TRIPOD+AI statement gives a checklist for reporting studies that develop or evaluate prediction models, whether they use regression or machine learning (Collins et al. 2024). Module m36 covers how regulators handle AI-enabled device software, including changes after clearance.

Key takeaways

  • Report the data source, the split, the labels and their agreement, discrimination and calibration.

Clinical correlate

NotePending clinical review

Planned in the outline. Requires sign-off by an EP physician.

Visual summary

Planned in the outline.

How experts think

Planned in the outline.

Cheat sheet

Planned in the outline.

Interview questions

Planned in the outline.

Code lab

The lab rebuilds this module’s results in code, so you can see where each one comes from and break it on purpose. It runs in a few seconds.

Part What you do Data
A Generate labeled electrograms (local versus far-field) from the m03 model, with patient-specific gain and noise synthetic
B Fit logistic regression with scikit-learn and evaluate it patient by patient: accuracy, AUC (Equation 5), Brier score and calibration (Equation 7) synthetic
C Leakage: compare record-wise and patient-wise cross-validation for a nearest-neighbour model and for logistic regression (Equation 3) synthetic
D A confidence interval for AUC that resamples patients, against one that resamples records synthetic
E Flip a share of the labels and compare the measured accuracy with Equation 2 synthetic

Each part ends with checks: lines that pass with the code as given. If you change something and a check fails, that is information, not an error to hide.

Self-check

Planned in the outline.

Literature figures

Planned in the outline.

Further reading

Planned in the outline. - AI competency areas. Machine learning; Bayesian networks (Bayes’ rule and predictive values); search (why searching many designs leaks the validation set). See the AI competency map.

References

Alhusseini, Mahmood I., Firas Abuzaid, Albert J. Rogers, et al. 2020. “Machine Learning to Classify Intracardiac Electrical Patterns During Atrial Fibrillation: Machine Learning of Atrial Fibrillation.” Circulation: Arrhythmia and Electrophysiology 13: e008160. https://doi.org/10.1161/CIRCEP.119.008160.
Altman, D. G., and J. M. Bland. 1994. “Diagnostic Tests 2: Predictive Values.” BMJ 309 (6947): 102. https://doi.org/10.1136/bmj.309.6947.102.
Brier, Glenn W. 1950. “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review 78 (1): 1–3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.
Cohen, Jacob. 1960. “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement 20 (1): 37–46. https://doi.org/10.1177/001316446002000104.
Collins, Gary S., Karel G. M. Moons, Paula Dhiman, et al. 2024. “TRIPOD+AI Statement: Updated Guidance for Reporting Clinical Prediction Models That Use Regression or Machine Learning Methods.” BMJ 385: e078378. https://doi.org/10.1136/bmj-2023-078378.
Hanley, J. A., and B. J. McNeil. 1982. “The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve.” Radiology 143 (1): 29–36. https://doi.org/10.1148/radiology.143.1.7063747.
Killip, Shersten, Ziyad Mahfoud, and Kevin Pearce. 2004. “What Is an Intracluster Correlation Coefficient? Crucial Concepts for Primary Care Researchers.” Annals of Family Medicine 2 (3): 204–8. https://doi.org/10.1370/afm.141.
Saeb, Sohrab, Luca Lonini, Arun Jayaraman, et al. 2017. “The Need to Approximate the Use-Case in Clinical Machine Learning.” GigaScience 6: 1–9. https://doi.org/10.1093/gigascience/gix019.
Van Calster, Ben, David J. McLernon, Maarten van Smeden, et al. 2019. “Calibration: The Achilles Heel of Predictive Analytics.” BMC Medicine 17: 230. https://doi.org/10.1186/s12916-019-1466-7.
Zolotarev, Alexander M., Brian J. Hansen, Ekaterina A. Ivanova, et al. 2020. “Optical Mapping-Validated Machine Learning Improves Atrial Fibrillation Driver Detection by Multi-Electrode Mapping.” Circulation: Arrhythmia and Electrophysiology 13: e008249. https://doi.org/10.1161/CIRCEP.119.008249.