Skip to main content

m21 · Image integration: CT/MRI segmentation, landmark and ICP registration

Warning

Education only. Not for diagnosis, treatment or any clinical decision.

In 60 seconds

Image integration places a CT or MRI anatomy in the mapping system’s coordinates. First the chamber is segmented: in a contrast-enhanced image, blood is brighter than tissue, and Otsu’s threshold picks the split that maximizes the between-class variance \(\omega_0\omega_1(\mu_0 - \mu_1)^2\). Segmentations are scored by overlap, the Dice coefficient. Then the anatomy is registered. With paired landmarks, the least-squares rotation comes from an SVD, with a sign fix against reflections, and a scale can be added for a distorted map. Without pairs, iterative closest point (ICP) alternates nearest-point matching and that least-squares fit. ICP never increases the mean squared distance, but it stops at the nearest local minimum. A surface also leaves some motions unconstrained: a sphere can turn about its center with zero surface distance. So the error the system reports, a distance between surfaces or landmarks, is not the error at the target.

From the field

  • Landmarks alone were coarse; surfaces fit closer. In 30 patients, registering CT to the map with pulmonary vein landmarks gave an error of 6.4 ± 2.8 mm, while a best fit of the CT surface to the map geometry gave 2.3 ± 0.4 mm (Kistler et al. 2006).
  • Target error was measured directly in animals. In 9 dogs with CT markers on each chamber, ablations guided only by the registered CT landed 1.8–2.7 mm from the markers, depending on the chamber; landmark plus surface registration gave under 3 mm in all four chambers (Dong et al. 2006).
  • Reported error is not target error. The error at a target grows with distance from the landmarks and depends on their layout (Fitzpatrick et al. 1998); m55 works through the formula.

Derivations

Symbols used throughout. \(t\) intensity threshold; \(\omega_0\), \(\omega_1\) class fractions; \(\mu_0\), \(\mu_1\) class means; \(\sigma_B^2\), \(\sigma_W^2\), \(\sigma_T^2\) between-class, within-class and total variance; \(A\), \(B\) segmentations; \(D\) Dice; \(J\) Jaccard; \(\mathbf p_i\), \(\mathbf q_i\) paired points; \(R\) rotation; \(\mathbf t\) translation; \(s\) scale; \(H\) cross-covariance with SVD \(UDV^\top\); \(S\) a sign matrix; \(\mathbf n_i\) surface normals; \(\boldsymbol\omega\) small rotation; \(J_s\) point-to-plane Jacobian.

Threshold segmentation

Split intensities at \(t\) into a dark class (fraction \(\omega_0\), mean \(\mu_0\)) and a bright class. The total variance splits into within- and between-class parts, and Otsu’s method maximizes the between-class part (Otsu 1979):

\[ \sigma_T^2 = \sigma_W^2(t) + \sigma_B^2(t), \qquad \sigma_B^2(t) = \omega_0\omega_1(\mu_0 - \mu_1)^2, \qquad t^* = \arg\max_t \sigma_B^2(t) \tag{1}\]

It needs no labeled data. In the lab, a synthetic contrast image thresholded at Otsu’s level gave a Dice of 0.80; keeping the largest connected component raised it to 0.993, and smoothing first to 0.997 (lab part A). Real segmentation uses more than a global threshold (m40, m51); this is the baseline.

Overlap

\[ D(A, B) = \frac{2\lvert A\cap B\rvert}{\lvert A\rvert + \lvert B\rvert}, \qquad J(A, B) = \frac{\lvert A\cap B\rvert}{\lvert A\cup B\rvert}, \qquad D = \frac{2J}{1 + J} \tag{2}\]

Dice was introduced to measure association between species (Dice 1945). For two Gaussian intensity classes the expected Dice of a threshold has a closed form. When the blood pool is a small fraction of the image, Otsu’s threshold sits closer to the minority class than the Dice-optimal one: with 9.5% blood, Otsu’s 211 gave an expected Dice of 0.86 against 0.95 at 256 (lab part B).

Exact

Assumptions

  • Model: two Gaussian intensity classes with a common SD; Otsu’s threshold from the exact mixture histogram (Equation 1); expected Dice in closed form (Equation 2). Tested against epml.registration.
  • Parameters: blood intensity, noise and blood fraction are illustrative and adjustable; tissue intensity is 100.
  • What it ignores: spatial structure (connectivity, smoothing), partial-volume voxels, beam hardening and contrast timing (m51).

Key takeaways

  • Otsu maximizes between-class variance; it is not the same as maximizing overlap.
  • Report Dice together with a surface distance (m20).

Landmark registration

For paired points, minimize \(\sum_i\lVert\mathbf q_i - (sR\mathbf p_i + \mathbf t)\rVert^2\). With centered coordinates and the cross-covariance \(H = \frac1N\sum_i(\mathbf q_i - \bar{\mathbf q})(\mathbf p_i - \bar{\mathbf p})^\top = UDV^\top\):

\[ R = U S V^\top, \quad S = \operatorname{diag}\bigl(1, \ldots, 1, \operatorname{sign}\det(UV^\top)\bigr), \quad s = \frac{\operatorname{tr}(DS)}{\frac1N\sum_i\lVert\mathbf p_i - \bar{\mathbf p}\rVert^2}, \quad \mathbf t = \bar{\mathbf q} - sR\bar{\mathbf p} \tag{3}\]

The sign matrix rules out reflections, which unconstrained solutions can return when data are corrupted (Umeyama 1991). With \(s = 1\) this is the rigid fit. In the lab, four landmarks located with 3 mm RMS error gave a mean fiducial registration error of 2.0 mm and an RMS error of 2.1 mm at a target; a map with a 10% scale error left a 2.7-mm residual after a rigid fit and none after a similarity fit (lab part C). The expected target error in closed form is in m55 (eq-m55-tre) (Fitzpatrick et al. 1998).

Iterative closest point

Without known pairs, alternate matching and fitting (Besl and McKay 1992):

\[ j(i) = \arg\min_j \lVert R_k\mathbf p_i + \mathbf t_k - \mathbf q_j\rVert, \qquad (R_{k+1}, \mathbf t_{k+1}) = \arg\min_{R,\mathbf t}\sum_i\lVert R\mathbf p_i + \mathbf t - \mathbf q_{j(i)}\rVert^2 \tag{4}\]

Each step cannot increase the mean squared distance, so ICP converges, but to the nearest local minimum; trying several starting poses is the remedy Besl and McKay proposed (Besl and McKay 1992). On an ellipsoid with exact correspondences, starts of 10° to 60° converged to the truth; starts of 120° and 170° settled 43 mm (RMS) from it with a surface distance under 1 mm, because an ellipsoid looks the same turned half around. The best of eight starts recovered the truth (lab part D).

Simulation

Assumptions

  • Model: 2D point-to-point ICP, Equation 4, with brute-force nearest neighbors and the 2D least-squares rotation of Equation 3; an illustrative ellipse with one bump; mapped points are exact samples of the same wall, rotated and shifted. Tested against epml.registration.
  • Parameters: starting rotation and number of iterations are adjustable.
  • What it ignores: noise, partial coverage, deformation between image and map, and the third dimension.

What a surface cannot pin down

For a small rotation \(\boldsymbol\omega\) and translation \(\mathbf t\), the point-to-plane residuals are linear:

\[ r_i = \mathbf n_i\cdot(\boldsymbol\omega\times\mathbf p_i + \mathbf t) = (\mathbf p_i\times\mathbf n_i)\cdot\boldsymbol\omega + \mathbf n_i\cdot\mathbf t, \qquad J_s^\top J_s \text{ has a zero eigenvalue for every motion that slides the surface along itself} \tag{5}\]

On a sphere centered at the origin \(\mathbf p_i\times\mathbf n_i = 0\), so all three rotations are free: a sphere turned by 10° has zero surface distance while its points moved 3.6 mm (lab part E). Real chambers are not spheres, but smooth, nearly symmetric regions constrain registration weakly. That is why a small surface error (Kistler et al. 2006) does not by itself guarantee a small error at a target (Dong et al. 2006).

Key takeaways

  • Landmarks fix pose but carry localization error; surfaces fit closer but can slide.
  • ICP finds the nearest minimum; test several starts.

Clinical correlate

NotePending clinical review

This section describes published findings only. It is not guidance for patient care, and it has not yet been reviewed by an EP physician.

Where this shows up in EP. Image integration lets a mapping system show CT or MRI anatomy with the catheter in it. In one clinical series, landmark registration alone was much less accurate than surface registration (Kistler et al. 2006). In dogs, targets marked on CT were reached within about 2–3 mm, and combining landmarks with the surface kept errors below 3 mm (Dong et al. 2006). Segmentation quality, the time between imaging and procedure, rhythm and volume status all change the anatomy being registered (m51, m52).

What the evidence is. A clinical validation series (Kistler et al. 2006), an animal study with physical markers (Dong et al. 2006) and registration theory (Fitzpatrick et al. 1998; Besl and McKay 1992; Umeyama 1991).

What a reader should ask:

Visual summary

Schematic

segment the chamber landmarks: least-squares fit surfaces can slide
Figure 1: Segmentation, landmark registration, and the sliding a surface fit cannot see.

Assumptions

How experts think

Before trusting an integrated image, ask:

  1. How good is the segmentation, and how was that measured? (Equation 1, Equation 2)
  2. Which registration: landmarks, surface, or both? (Equation 3, Equation 4)
  3. What error is reported, and where is the target relative to the landmarks? (m55)
  4. Is the shape distinctive enough to stop the fit sliding? (Equation 5)
  5. Has the anatomy changed since the scan? (m51, m52)

One sentence to keep: a small fit error says the shapes agree; it does not say the target is where it appears.

Cheat sheet

Quantity Result
Otsu (Equation 1) maximize \(\omega_0\omega_1(\mu_0 - \mu_1)^2\); \(\sigma_T^2 = \sigma_W^2 + \sigma_B^2\)
Overlap (Equation 2) \(D = 2\lvert A\cap B\rvert/(\lvert A\rvert + \lvert B\rvert) = 2J/(1 + J)\)
Landmarks (Equation 3) \(R = USV^\top\) with a sign fix; \(s\), \(\mathbf t\) from centroids
ICP (Equation 4) match, fit, repeat; monotone; local minimum
Sliding (Equation 5) \(r_i = (\mathbf p_i\times\mathbf n_i)\cdot\boldsymbol\omega + \mathbf n_i\cdot\mathbf t\); null modes slide

Values used in this module

Value Source
Landmark 6.4 ± 2.8 mm, surface 2.3 ± 0.4 mm; 30 patients Kistler et al. (2006)
Target errors 1.8–2.7 mm by chamber; 9 dogs; under 3 mm combined Dong et al. (2006)
Image intensities, noise, landmark error, rotations, shapes illustrative
Thresholds, Dice values, FRE and TRE, ICP outcomes our own code

Interview questions

Try each question before you open its model answer. In a real interview, say which assumptions you are making.

Q1. How does Otsu’s method choose a threshold?

Model answer

It maximizes the between-class variance \(\omega_0\omega_1(\mu_0 - \mu_1)^2\), equivalently minimizes the within-class variance (Equation 1) (Otsu 1979).

Q2. Why might Otsu’s threshold not maximize Dice?

Model answer

It weighs both classes by their size; with a small blood pool it moves toward that class and over-segments (Equation 1, Equation 2).

Q3. Relate Dice and Jaccard.

Model answer

\(D = 2J/(1 + J)\); Dice is always at least Jaccard (Equation 2).

Q4. Derive the least-squares rotation between paired point sets.

Model answer

Center both sets, form the cross-covariance, take its SVD \(UDV^\top\), and set \(R = USV^\top\) with \(S\) fixing the determinant’s sign (Equation 3) (Umeyama 1991).

Q5. Why the sign matrix?

Model answer

Without it the solution can be a reflection when data are noisy or nearly planar (Equation 3) (Umeyama 1991).

Q6. When would you fit a scale as well?

Model answer

When one geometry is uniformly scaled, as an impedance map with a scaling error can be (m17); a rigid fit leaves a residual that a similarity fit removes (Equation 3).

Q7. Explain ICP and its main weakness.

Model answer

Alternate closest-point matching and a least-squares fit; it decreases the error monotonically but stops at the nearest local minimum (Equation 4) (Besl and McKay 1992).

Q8. How do you make ICP more robust?

Model answer

Start from landmarks, try several initial poses, reject outlier pairs, and use point-to-plane distances (Equation 4) (Besl and McKay 1992).

Q9. Why can surface registration look excellent yet miss a target?

Model answer

Motions that slide the surface along itself do not change the surface distance (Equation 5); the target still moves (Kistler et al. 2006; Dong et al. 2006).

Q10. FRE was 1 mm. What can you conclude about the target error?

Model answer

Little: target error depends on landmark error, number and layout, and the target’s position, not on the observed FRE (Fitzpatrick et al. 1998).

Q11. What did one clinical series find for landmark against surface registration?

Model answer

Landmarks 6.4 ± 2.8 mm; surface fit 2.3 ± 0.4 mm (Kistler et al. 2006).

Q12. How was target accuracy measured in the animal study?

Model answer

Markers sewn on each chamber were imaged by CT, ablations were aimed at them using only the integrated image, and the distance was measured at autopsy (Dong et al. 2006).

Code lab

The lab segments and registers synthetic anatomy.

Part What you do Data
A Threshold segmentation of a synthetic CT (Equation 1) synthetic
B Overlap: Dice and Jaccard (Equation 2) synthetic
C Landmark registration (Equation 3) synthetic
D Surface registration by ICP (Equation 4) synthetic
E What a surface cannot pin down (Equation 5) synthetic

Each part ends with checks: lines that pass with the code as given. If you change something and a check fails, that is information, not an error to hide.

Self-check

Short checks you can do in your head. Open each answer after you have tried it.

S1. Jaccard 0.8. Dice?

Answer

\(1.6/1.8 \approx\) 0.89 (Equation 2).

S2. Two equal-size masks overlap in half of each. Dice?

Answer

0.5 (Equation 2).

S3. Classes of fraction 0.5 each with means 100 and 300. Between-class variance at the split between them?

Answer

\(0.25 \times 200^2 = 10\,000\) (Equation 1).

S4. Point sets that differ by a uniform 10% enlargement. Similarity scale?

Answer

1.1 (Equation 3).

S5. ICP’s error went up between two iterations. Possible?

Answer

Not for exact closest points and least-squares fits; check the implementation (Equation 4).

S6. How many rotations leave a centered sphere’s surface unchanged?

Answer

All of them: three free directions (Equation 5).

Literature figures

These figures are worth looking at alongside this module. We link to them instead of copying them: some licenses do not allow reuse here, and others we could not confirm. Each entry says what to look for and where this module explains it.

  • Landmark against surface registration. Kistler and colleagues’ registration errors in patients (Kistler et al. 2006). This module: what a surface cannot pin down.
  • Target accuracy with markers. Dong and colleagues’ errors at CT-visible markers by chamber (Dong et al. 2006). This module: landmark registration and ICP.
  • ICP on shapes. Besl and McKay’s registrations of point sets, curves and surfaces (Besl and McKay 1992). This module: iterative closest point.

Further reading

References

Besl, Paul J., and Neil D. McKay. 1992. “A Method for Registration of 3-D Shapes.” IEEE Transactions on Pattern Analysis and Machine Intelligence 14 (2): 239–56. https://doi.org/10.1109/34.121791.
Dice, Lee R. 1945. “Measures of the Amount of Ecologic Association Between Species.” Ecology 26 (3): 297–302. https://doi.org/10.2307/1932409.
Dong, Jun, Hugh Calkins, Stephen B. Solomon, et al. 2006. “Integrated Electroanatomic Mapping with Three-Dimensional Computed Tomographic Images for Real-Time Guided Ablations.” Circulation 113 (2): 186–94. https://doi.org/10.1161/CIRCULATIONAHA.105.565200.
Fitzpatrick, J. M., J. B. West, and C. R. Maurer. 1998. “Predicting Error in Rigid-Body Point-Based Registration.” IEEE Transactions on Medical Imaging 17 (5): 694–702. https://doi.org/10.1109/42.736021.
Kistler, Peter M., Mark J. Earley, Stuart Harris, et al. 2006. “Validation of Three-Dimensional Cardiac Image Integration: Use of Integrated CT Image into Electroanatomic Mapping System to Perform Catheter Ablation of Atrial Fibrillation.” Journal of Cardiovascular Electrophysiology 17 (4): 341–48. https://doi.org/10.1111/j.1540-8167.2006.00371.x.
Otsu, Nobuyuki. 1979. “A Threshold Selection Method from Gray-Level Histograms.” IEEE Transactions on Systems, Man, and Cybernetics 9 (1): 62–66. https://doi.org/10.1109/TSMC.1979.4310076.
Umeyama, Shinji. 1991. “Least-Squares Estimation of Transformation Parameters Between Two Point Patterns.” IEEE Transactions on Pattern Analysis and Machine Intelligence 13 (4): 376–80. https://doi.org/10.1109/34.88573.