m43 · Foundation models and large language models: self-supervised ECG encoders, LLM evaluation, retrieval
Education only. Not for diagnosis, treatment or any clinical decision.
In 60 seconds
A foundation model is pretrained once on many unlabeled examples and then adapted to many tasks with few labels. For the ECG, the usual pretraining task is contrastive: two views of the same patient should get similar embeddings, and views of different patients should not (Sangha et al. 2024; Khattak et al. 2026; Ko et al. 2026). Such models reached a given accuracy with far fewer labels than models trained from scratch (Ko et al. 2026; Sangha et al. 2024), but what they learn depends on the cohort they were pretrained on (Khattak et al. 2026). Large language models (LLMs) are transformers trained on text; general-purpose LLMs were not reliable readers of ECGs or pacemaker electrograms in recent studies (Chai et al. 2026; Zeljkovic et al. 2025; Bozyel et al. 2025). This module derives the contrastive loss and attention, builds a retrieval step over this course’s own text, and shows how to evaluate a language model’s answers: accuracy with its uncertainty, consistency, and whether its citations exist.
From the field
- Fewer labels for the same accuracy. An ECG foundation model pretrained on more than 1.7 million ECGs with their text reports matched the best comparator’s AUC in disease-detection tasks with, on average, 90.8% less labelled training data, in external data from another health system (Ko et al. 2026). A model pretrained to match two ECG images of the same person beat ImageNet and generic contrastive pretraining when only 0.1% of the labels were used (Sangha et al. 2024).
- The pretraining cohort shapes the model. Pretraining on a multi-centre, diverse cohort improved in-distribution accuracy but reduced out-of-distribution generalization, because the model also learned cohort-specific artifacts; building each batch from one cohort helped (Khattak et al. 2026). Another foundation model kept its accuracy for left ventricular systolic dysfunction when its supervised training data were skewed by sex or race (Hsieh et al. 2026). Neither result transfers automatically to a new model or task: test it.
- General LLMs and EP signals. Two current general-purpose LLMs had low macro F1 for rhythm and axis on ECGs from a public database (Chai et al. 2026). GPT-4 interpreted 19% of 150 ECGs correctly without clinical context and 45% with it (Zeljkovic et al. 2025). On 20 pacemaker electrogram cases, ChatGPT-4o’s accuracy rose from 57% to 66% with context but fell to 30% and 25% in the complex scenarios, where its answers two months apart agreed only fairly (Bozyel et al. 2025).
- Keep data on site. A locally run open-source LLM answered 90 of 100 questions about patients’ echocardiogram report histories correctly, retrieving relevant snippets first; its errors included misread numbers and hallucinations (Vaid et al. 2024).
- Retrieval with citations (Stanford). Almanac, a retrieval-augmented LLM from Stanford, was preferred by clinicians over general LLMs on 314 clinical questions and gave correct citations for 91% of them. When an adversarial instruction was appended to a question, the similarity between question and retrieved passages fell below Almanac’s threshold and it abstained (Zakka et al. 2024).
Derivations
Pretraining without labels
Take a batch of \(N\) anchors with unit embeddings \(\mathbf z_i\) and \(N\) candidates \(\mathbf z'_k\), where candidate \(i\) is anchor \(i\)’s positive (for example a second recording of the same patient) and every other candidate is a negative. With similarities \(S_{ik} = \mathbf z_i^\top \mathbf z'_k\) and a temperature \(\tau > 0\), the InfoNCE loss (Oord et al. 2018) is a cross-entropy over candidates:
\[ L = -\frac{1}{N}\sum_{i=1}^{N} \ln \frac{\exp(S_{ii}/\tau)}{\sum_{k=1}^{N} \exp(S_{ik}/\tau)} \tag{1}\]
If every similarity is equal, \(L = \ln N\): that is chance. The loss falls as positives become more similar than negatives, and \(\tau\) sets how hard it pushes on a given margin. Writing \(P_{ik} = \exp(S_{ik}/\tau)/\sum_j \exp(S_{ij}/\tau)\), the gradient is
\[ \frac{\partial L}{\partial S_{ik}} = \frac{P_{ik} - \delta_{ik}}{N\tau} \tag{2}\]
so each anchor pulls its positive closer with weight \(1 - P_{ii}\) and pushes each negative away with weight \(P_{ik}\): the negatives it confuses most get the largest push. In practice the loss is averaged over both directions (anchors from view 1, then from view 2), the embeddings come from a small projection head on top of the features, and the features before the head are kept for downstream use (Chen et al. 2020). What counts as a positive decides what the model learns to ignore: two augmentations of one recording teach invariance to the augmentations; two recordings of one patient, taken at different times, also teach invariance to day-to-day changes (Sangha et al. 2024; Khattak et al. 2026).
Simulation
Assumptions
- Model: an encoder with one hidden layer of 64 tanh units (the features) and a 16-dimensional projection, trained with Equation 1 averaged over both directions, \(\tau = 0.1\), batches of 128 patients, Adam with step size 0.002, for 300 steps. Precomputed by
tools/make_m43_d1.pywithepml.foundationand shipped asdemos/m43-d1-data.json. - Data: synthetic beats, the Gaussian-wave beat of m37 (McSharry et al. 2003) with patient-level wave sizes and QRS width, 250 samples per second; recordings differ by small beat changes, a shift of up to 12 ms, baseline wander and noise. All ranges are illustrative, not physiological reference values.
- The scatter shows the embeddings’ first two principal components at each step, computed afresh at each step. The probe AUC is the mean over five sets of 10 labelled patients, tested on 1,000 others; “wide QRS” means a width factor above 1.4 (illustrative).
- What would change the result: the temperature, the batch size (more negatives), the augmentations, and the pretraining cohort (lab part C).
Key takeaways
- Contrastive pretraining needs pairs, not labels; the loss is a cross-entropy whose chance level is \(\ln N\).
- The choice of positive pairs decides what the embedding keeps and what it ignores.
- The useful output is the feature layer, judged by what a simple model can do with it.
Linear probes and label efficiency
A linear probe is a logistic regression trained on frozen features; it measures how much a task is already encoded. In lab part B, after pretraining on 2,000 unlabeled synthetic patients, a probe trained on 10 labelled patients detected a wide QRS with a mean AUC of about 0.84, against about 0.63 for the same regression on the raw samples and 0.64 on the features of an untrained encoder. With 300 labels the three came within about 0.03 of each other: pretraining matters most when labels are scarce, as in the published label-efficiency curves (Ko et al. 2026; Sangha et al. 2024; Yang et al. 2026). In part C, an encoder pretrained on a cohort with no wide complexes gave a weaker probe at every label budget, and a weaker one than the raw samples once 30 or more labels were available. That is the synthetic version of the cohort effect reported by Khattak et al. (2026).
Key takeaways
- Report a pretrained model’s benefit as a curve over label budgets, not one number.
- Compare against an untrained encoder of the same size, so the gain is credited to pretraining.
- A cohort that lacks a feature yields an encoder that does not represent it.
Attention
A transformer (Vaswani et al. 2017) mixes information between positions (ECG samples, image patches or text tokens) with attention. Each position has a query \(\mathbf q\), a key \(\mathbf k\) and a value \(\mathbf v\), linear maps of its input. With queries, keys and values stacked as rows of \(Q\), \(K\) and \(V\), and key length \(d\),
\[ \operatorname{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt d}\right) V \tag{3}\]
with the softmax taken over keys, so each output is a weighted average of the values. Why \(\sqrt d\)? If the entries of \(\mathbf q\) and \(\mathbf k\) are independent with mean 0 and variance 1, each product \(q_j k_j\) has variance 1, and the \(d\) terms add:
\[ \operatorname{Var}\!\left(\mathbf q^\top \mathbf k\right) = d, \qquad \operatorname{Var}\!\left(\mathbf q^\top \mathbf k / \sqrt d\right) = 1 \tag{4}\]
Without the division, scores grow with \(d\) and the softmax approaches a one-hot choice, which starves the other positions of gradient (Vaswani et al. 2017). In lab part D, with 32 keys and \(d = 256\), the largest unscaled weight averaged 0.92; scaled, 0.17.
Key takeaways
- Attention is a softmax-weighted average of values; the weights come from query–key similarity.
- Dividing by \(\sqrt d\) keeps the scores’ variance at 1 whatever the key length.
- The same block serves ECG foundation models and LLMs (Hsieh et al. 2026; Ye et al. 2026).
Retrieval
Retrieval-augmented generation puts passages from a chosen collection in front of a language model before it answers, so that the answer can rest on them and cite them (Lewis et al. 2020). The retrieval step needs no language model. The classic method weights each word \(t\) in document \(d\) by how often it occurs there and how rare it is across the \(N\) documents of the collection (Spärck Jones 1972):
\[ w_{t,d} = \operatorname{tf}_{t,d}\left[\ln\frac{1 + N}{1 + \operatorname{df}_t} + 1\right] \tag{5}\]
where \(\operatorname{tf}_{t,d}\) counts \(t\) in \(d\) and \(\operatorname{df}_t\) counts the documents containing \(t\) (this “smoothed” form never divides by zero). Documents and the question become vectors of these weights, and documents are ranked by cosine similarity:
\[ \cos(\mathbf a, \mathbf b) = \frac{\mathbf a^\top \mathbf b}{\lVert\mathbf a\rVert\,\lVert\mathbf b\rVert} \tag{6}\]
Dense retrievers replace word counts with learned embeddings, trained with the loss of Equation 1 on matched pairs of texts; NV-Embed from NVIDIA is one such text-embedding model (Lee et al. 2024), and Almanac used a commercial embedding model with a cosine threshold below which it returned no passage (Zakka et al. 2024). Whatever the retriever, measure its recall on questions written by someone else.
Exact
Assumptions
- Method: Equation 5 with raw counts and the smoothed inverse document frequency, rows scaled to unit length, ranked by Equation 6. Text is lower-cased and split into words of letters and digits; a short list of stop words is removed, and a crude rule drops plural endings. The scores are exact for this method; the JavaScript is tested against
epml.retrieval. - Corpus: the course glossary and formula sheet at build time (
demos/m43-d2-corpus.json, written bytools/make_m43_d2.py). Test questions: 24 plain-language questions written for the course; the same author wrote many glossary entries, which makes them easier than real users’ questions. - What would change the result: a different corpus or question set, stemming, or a dense retriever, which can match synonyms that share no words.
Key takeaways
- Retrieval can be tested on its own: recall at \(k\) and mean reciprocal rank on known question–answer pairs.
- Word-matching retrieval misses synonyms; dense retrieval learns them from matched pairs.
- A similarity threshold can flag questions with no good source; in Almanac it also made the system abstain when an adversarial instruction was appended (Zakka et al. 2024).
Evaluating a language model’s answers
An evaluation needs a fixed question set with answers decided in advance, a metric, and an estimate of uncertainty. For \(k\) correct answers out of \(n\) independent questions, the 95% Wilson score interval (Wilson 1927) solves \((\hat p - p)^2 = z^2 p(1-p)/n\) for \(p\), with \(\hat p = k/n\) and \(z = 1.96\):
\[ \frac{\hat p + \dfrac{z^2}{2n} \pm z\sqrt{\dfrac{\hat p(1-\hat p)}{n} + \dfrac{z^2}{4n^2}}}{1 + \dfrac{z^2}{n}} \tag{7}\]
Unlike the textbook \(\hat p \pm z\sqrt{\hat p(1-\hat p)/n}\), it stays inside 0 to 1 and still gives an upper bound when \(k = 0\). To plan a benchmark, the number of independent questions for an interval of half-width about \(h\) around an accuracy near \(p\) is
\[ n \approx \frac{z^2\,p(1-p)}{h^2} \tag{8}\]
which is 385 questions for \(\pm 5\) percentage points at \(p = 0.5\). With 150 ECGs, an accuracy near 19% has a 95% Wilson interval of about 13% to 26% (our calculation, taking 28 of 150 for the 19% reported by Zeljkovic et al. (2025) and treating the ECGs as independent).
Accuracy is not enough for a model that samples its words:
- Consistency. Ask each question several times, or months apart, and report agreement with Cohen’s kappa (m35); Bozyel et al. (2025) found only fair agreement for complex electrogram cases.
- Citations. Check that every cited source exists and says what is claimed, by resolving its DOI or PMID; general LLMs have cited non-existent pages (Zakka et al. 2024).
- Breadth. Clinical work is more than exam questions. MedHELM, from Stanford, evaluated nine LLMs on 35 benchmarks covering 121 clinical tasks, and found the lowest scores in administration and workflow tasks; its panel of three LLM judges agreed with clinicians (intraclass correlation 0.47) about as well as clinicians agreed with each other (0.43) on two benchmarks (Bedi et al. 2025).
Key takeaways
- Report accuracy with an interval; 20 cases cannot separate 60% from 80% accuracy.
- Measure consistency across repeats and check every citation.
- A model judge must itself be validated against clinicians, task by task.
Clinical correlate
This section describes published findings only. It is not guidance for patient care, and it has not yet been reviewed by an EP physician.
Where these methods could enter EP practice. ECG foundation models are studied for screening and risk prediction from the 12-lead ECG (Ko et al. 2026; Hsieh et al. 2026; Yang et al. 2026), building on the AI-ECG work of m37. LLMs are studied as readers of ECGs and device electrograms (Chai et al. 2026; Zeljkovic et al. 2025; Bozyel et al. 2025), and as tools for querying a patient’s report history on site (Vaid et al. 2024). A review describes the evidence for cardiovascular foundation models as promising but unevenly validated, and lists interpretability, under-representation of populations, regulatory uncertainty and workflow integration as open problems (Ye et al. 2026).
What the evidence is. The foundation-model studies here are retrospective, some with external validation (Ko et al. 2026; Sangha et al. 2024; Hsieh et al. 2026); one is a preprint (Yang et al. 2026). The LLM studies are benchmarks on curated cases. We found no randomized trial of an ECG foundation model or an LLM in EP among them. Results are reported in the papers’ own units (AUC, accuracy, kappa) and none is evidence of patient benefit (m45).
In the terms of this module:
- Check the cohort. Which patients, devices and centres were in pretraining, and does the target population look like them (Khattak et al. 2026)?
- Check the labels. How many labelled examples did the reported accuracy need, and is the gain over an untrained baseline shown as a curve?
- Check the reader. For an LLM, was accuracy reported with uncertainty, with and without context, across repeats, and with verified citations?
Visual summary
Schematic
Assumptions
- Schematic: no quantitative claim. Stages follow Equation 1, Equation 5, Equation 6 and the interval of Equation 7.
How experts think
Before trusting a foundation-model or LLM result, ask:
- What was the model pretrained on? Size, centres, devices and patient mix of the pretraining cohort; a diverse cohort can still teach cohort artifacts (Khattak et al. 2026).
- What were the positives? Augmentations of one recording, recordings of one patient, or an ECG and its report (Sangha et al. 2024; Ko et al. 2026).
- How many labels did the result need? Ask for the label-efficiency curve and the untrained baseline (lab part B).
- How was the LLM tested? Fixed cases, decided answers, uncertainty, repeats, with and without context (Zeljkovic et al. 2025; Bozyel et al. 2025).
- Where do the numbers come from? In an answer, every number and citation should trace to a retrieved source or a tool, not to the model’s memory (Zakka et al. 2024; Vaid et al. 2024).
One sentence to keep: pretraining buys labels, not truth; a fluent answer is not a measured one.
Cheat sheet
| Quantity | Result |
|---|---|
| InfoNCE (Equation 1) | \(L = -\frac{1}{N}\sum_i \ln \frac{e^{S_{ii}/\tau}}{\sum_k e^{S_{ik}/\tau}}\); chance \(= \ln N\) |
| Its gradient (Equation 2) | \(\partial L/\partial S_{ik} = (P_{ik} - \delta_{ik})/(N\tau)\) |
| Attention (Equation 3) | \(\operatorname{softmax}(QK^\top/\sqrt d)\,V\) |
| Score variance (Equation 4) | \(\operatorname{Var}(\mathbf q^\top\mathbf k) = d\) |
| TF-IDF (Equation 5) | \(w = \operatorname{tf}\,[\ln((1+N)/(1+\operatorname{df})) + 1]\) |
| Cosine (Equation 6) | \(\mathbf a^\top\mathbf b / (\lVert\mathbf a\rVert\lVert\mathbf b\rVert)\) |
| Wilson interval (Equation 7) | Roots in \(p\) of \((\hat p - p)^2 = z^2 p(1-p)/n\) |
| Questions needed (Equation 8) | \(n \approx z^2 p(1-p)/h^2\); 385 for \(\pm 0.05\) at \(p = 0.5\) |
Studies in this module
| Study | What was done | Evidence |
|---|---|---|
| Ko et al. (2026) | ECG and report contrastive pretraining, many downstream tasks | Retrospective, external data set |
| Sangha et al. (2024) | Same-patient ECG-image pairs as positives | Retrospective, external cohorts |
| Khattak et al. (2026) | Effect of pretraining cohort on transfer | Retrospective, cohorts from four continents |
| Hsieh et al. (2026) | Foundation model under skewed training demographics | Retrospective, external test |
| Yang et al. (2026) | Pediatric-dominant pretraining, transfer to adults | Preprint |
| Chai et al. (2026), Zeljkovic et al. (2025) | General LLMs reading ECG images | Benchmarks |
| Bozyel et al. (2025) | ChatGPT-4o reading pacemaker electrograms | Benchmark, 20 cases |
| Vaid et al. (2024) | Local LLM over echo report histories | Feasibility, 100 questions |
| Zakka et al. (2024) | Retrieval-augmented clinical answers with citations | Clinician-graded benchmark |
| Bedi et al. (2025) | Holistic LLM evaluation on clinical tasks | Benchmark suite |
Rules of thumb
- Pretraining helps most when labels are few; show the curve.
- Retrieval quality caps answer quality: test it on its own.
- 95% intervals on small benchmarks are wide; plan the number of questions first.
Interview questions
Try each question before you open its model answer. In a real interview, say which assumptions you are making.
Q1. Write the InfoNCE loss and explain what the temperature and the negatives do.
Model answer
\(L = -\frac1N\sum_i \ln\big(e^{S_{ii}/\tau}/\sum_k e^{S_{ik}/\tau}\big)\) (Equation 1): a cross-entropy that asks each anchor to pick its positive among \(N\) candidates. The other \(N-1\) candidates are the negatives; more negatives make the task harder and the chance level \(\ln N\) higher. A small \(\tau\) sharpens the softmax, so the loss concentrates on the negatives most similar to the anchor (Equation 2) and rewards larger margins; too small can make training unstable.
Q2. Why does contrastive pretraining help most when labels are scarce?
Model answer
The pretrained features already separate patients by the factors that vary between them, so a simple probe needs few labels to find the direction that matters. With many labels, a model trained from scratch can learn those factors itself, and the gap closes. Lab part B shows this: about 0.84 against 0.63 AUC with 10 labels, within about 0.03 with 300. Published curves show the same shape (Sangha et al. 2024; Ko et al. 2026).
Q3. For ECGs, what can count as a positive pair, and why does the choice matter?
Model answer
Two augmentations of one recording (noise, gain, baseline shifts), two recordings of the same patient at different times (Sangha et al. 2024; Khattak et al. 2026), or an ECG and its text report (Ko et al. 2026). The model learns to ignore whatever differs within a pair. Augmentations teach invariance only to what you simulate; same-patient pairs also teach invariance to day-to-day variation but preserve stable patient traits. Demo m43-d1 shows same-patient pairs pulling a patient’s recordings closer than augmentation pairs do.
Q4. How can a more diverse pretraining cohort make a model generalize worse?
Model answer
If cohorts differ in recording devices or processing, the easiest way to tell patients apart within a mixed batch can be the cohort itself, so the model encodes cohort artifacts. Khattak et al. (2026) found that a multi-centre, diverse pretraining cohort improved in-distribution accuracy but reduced out-of-distribution generalization, and that building each batch from a single cohort helped.
Q5. Why does attention divide by the square root of the key length?
Model answer
For query and key entries independent with mean 0 and variance 1, \(\mathbf q^\top\mathbf k\) has variance \(d\) (Equation 4). Without scaling, scores grow with \(d\), the softmax becomes nearly one-hot, and gradients through the other positions vanish. Dividing by \(\sqrt d\) keeps the variance at 1 (Vaswani et al. 2017).
Q6. What is a linear probe, and why use one to evaluate a foundation model?
Model answer
A linear classifier trained on frozen features. It measures what the features already encode rather than what a large fine-tuned network can learn on top of them, so it isolates the value of pretraining. Compare it with the same probe on an untrained encoder of the same size, and repeat it over several label budgets.
Q7. Design an evaluation of an LLM that reads pacemaker electrograms.
Model answer
Fix a set of cases with answers agreed by experts in advance, kept out of public view to limit contamination. Score each predefined feature and the overall interpretation; report accuracy with a Wilson interval (Equation 7) and plan the number of cases (Equation 8). Test with and without clinical context, repeat over time and report kappa, as Bozyel et al. (2025) did. Report “no answer” separately from wrong answers, and break results down by case difficulty.
Q8. A vendor says its LLM “reads ECGs at cardiologist level”. What do you ask?
Model answer
On which cases, chosen how, labelled by whom; what metric (macro F1 across rhythm classes, not only overall accuracy, since rare rhythms are where general LLMs struggled (Chai et al. 2026)); with what interval; with or without clinical context, which changed GPT-4’s accuracy from 19% to 45% (Zeljkovic et al. 2025); how consistent across repeats; and on what external data.
Q9. Explain retrieval-augmented generation and two ways it can fail.
Model answer
Retrieve passages from a chosen collection, then ask the model to answer from them and cite them (Lewis et al. 2020). It fails when retrieval misses the right passage (word mismatch, as in lab part E, or a passage split badly), and when the model ignores or misreads the passage. A similarity threshold on retrieved passages can flag that nothing relevant was found; in Almanac, an appended adversarial instruction pushed the scores below the threshold and the system abstained (Zakka et al. 2024).
Q10. How would you check the citations in a model’s answer automatically?
Model answer
Require identifiers (DOI or PMID), resolve each one against the registry, compare title and authors, and flag anything that does not resolve. Then check, by retrieval or by a person, that the cited passage supports the sentence. Report the share of valid citations, as Zakka et al. (2024) did (91% for Almanac). Lab part F plants an invented key and checks that the harness finds it.
Q11. Why might a hospital run a language model locally, and what still needs checking?
Model answer
So that patient data do not leave the site: Vaid et al. (2024) ran an open-source LLM locally over echocardiogram reports. Still to check: accuracy on the hospital’s own questions, misread numbers and hallucinations (the main error sources there), consistency, and access controls and logging.
Q12. Can an LLM judge replace clinicians for grading free-text answers?
Model answer
Only where it has been validated. In MedHELM a panel of three LLM judges agreed with clinicians about as well as clinicians agreed with each other (intraclass correlation 0.47 against 0.43), on two benchmarks (Bedi et al. 2025). That supports using it for those tasks with periodic human audits, not for any task without checking.
Q13. How many questions does a benchmark need to estimate an accuracy within ±5 percentage points?
Model answer
About \(z^2 p(1-p)/h^2\) independent questions (Equation 8): 385 at \(p = 0.5\), 246 at \(p = 0.8\). Questions grouped by patient or case are not independent, so more are needed (the design effect of m35).
Code lab
The lab pretrains a small ECG-beat encoder without labels, measures what it buys with few labels, shows how the pretraining cohort matters, checks why attention divides by \(\sqrt d\), retrieves from this course’s own text, and builds an offline harness for evaluating language-model answers.
| Part | What you do | Data |
|---|---|---|
| A | The contrastive loss and its gradient by hand (Equation 1, Equation 2) | numbers |
| B | Pretrain without labels, then a linear probe with 10 to 300 labels | synthetic beats |
| C | An encoder whose pretraining cohort had no wide QRS | synthetic beats |
| D | Attention and the square root of \(d\) (Equation 3, Equation 4) | random numbers |
| E | Retrieval over the glossary and formula sheet (Equation 5, Equation 6) | course text |
| F | An evaluation harness: interval, consistency, citations (Equation 7, Equation 8) | scripted stand-in |
| G | Pretraining on unlabeled PTB-XL beats, then a 10-record probe | PTB-XL (download) |
| H | Optional: the same pretraining with a 1-D CNN in PyTorch | synthetic beats |
- Python notebook: labs/python/m43_lab.ipynb. Run it from the course repository after
pip install -r requirements.txt, or open it in Google Colab. - Data: synthetic beats and course text, plus lead I of PTB-XL 1.0.3 (Wagner et al. 2020) in part G (Creative Commons Attribution 4.0; downloaded and checked against the published SHA-256 list, skipped without internet access).
- No external AI service. The language model in part F is a scripted stand-in with faults we planted, so the numbers test the harness, not any model. An optional cell lets you plug in your own model; it is skipped in CI and must never receive patient data.
- Part H runs only if PyTorch is installed.
Each part ends with checks: lines that pass with the code as given. If you change something and a check fails, that is information, not an error to hide.
Self-check
Short checks you can do in your head. Open each answer after you have tried it.
S1. A batch has 64 anchors and every similarity is the same. What is the InfoNCE loss?
Answer
\(\ln 64 \approx 4.16\) (Equation 1): chance level.
S2. Keys have length 64 with independent, unit-variance entries, as do queries. What is the variance of a raw score, and of a scaled one?
Answer
64 and 1 (Equation 4).
S3. A word appears in every one of 100 documents. What is its inverse document frequency in Equation 5?
Answer
\(\ln(101/101) + 1 = 1\), the smallest possible: the word cannot tell documents apart, so it only contributes through its count.
S4. A model answers 18 of 20 questions correctly. Roughly what is the 95% Wilson interval?
Answer
About 0.70 to 0.97 (Equation 7): 20 questions say little.
S5. What is the cosine similarity between a vector and three times itself?
Answer
1 (Equation 6): cosine ignores length.
S6. True or false: a model that gives the same answer three times in a row is probably right.
Answer
False. Consistency and accuracy are different measurements; a deterministic retriever in lab part F is perfectly consistent and still wrong on a third of the questions.
Literature figures
These figures are worth looking at alongside this module. We link to them instead of copying them: some licenses do not allow reuse here, and others we could not confirm. Each entry says what to look for and where this module explains it.
- Same-patient positives. Khattak and colleagues, Figure 7: two ECGs from each patient, augmented and encoded; the loss pulls a patient’s embeddings together and pushes other patients’ apart [Khattak et al. (2026); free full text]. Compare with Equation 1 and demo m43-d1.
- Embeddings that cluster by cohort. The same paper, Figure 4: a model pretrained with random mixed-cohort batches forms clusters by cohort, and Figure 5 shows the clusters fade with single-cohort batches [Khattak et al. (2026); free full text]. This module: lab part C and interview question Q4.
- The experimental design. The same paper, Figure 1: five pretrained models, each probed on five labelled cohorts, 25 evaluations in all. A good template for testing transfer.
Further reading
- Contrastive and self-supervised learning. Oord et al. (2018), Chen et al. (2020); for the ECG, Sangha et al. (2024), Khattak et al. (2026), Ko et al. (2026), and the review by Ye et al. (2026).
- Transformers and retrieval. Vaswani et al. (2017), Lewis et al. (2020), Spärck Jones (1972); NVIDIA’s text-embedding model Lee et al. (2024).
- Evaluating LLMs in medicine. Stanford’s MedHELM (Bedi et al. 2025) and Almanac (Zakka et al. 2024); for EP signals, Chai et al. (2026), Zeljkovic et al. (2025), Bozyel et al. (2025).
- Next module. m44 turns retrieval and tools into agents, with guardrails and human approval.
- AI competency areas. Machine learning (self-supervised and language models); search (retrieval). See the AI competency map.