Solving a specific registration problem requires careful selection of the individual components, parameters, and preprocessing procedures that together form a registration method. Given two images to be registered, one of the first such choices is concerned with which image should represent the fixed image. In the ideal case, the registration method should be inverse consistent, i.e., provide the same results independent on the selection of the fixed image. However, in practice most of the registration methods available will not provide identical solutions [9, 10], mainly because most similarity measures are not designed to be symmetric with respect to the images. The images will typically have different resolution and, in some applications, different dimensionality. When the differences in image resolution are significant, it can be beneficial to choose the moving image such that it has the higher spatial resolution. Finally, the choice of the fixed image is often dictated by the needs of the clinical application. As an example, resolution of PET images is typically inferior to that of MRI. However, structural images of the anatomy provided by the MR are used as a reference for radiological review of the various additional types of imaging.

Preprocessing of the input images can be applied to simplify the registration problem. Blurring and denoising of the images can discard the high-frequency components from the data, leading to reduced interpolation errors during image resampling [11], and prevent the optimization procedure to be trapped in local minima of the similarity measure. Various approaches can be used for equalizing intensities between the registered images (in particular, while registering MR images) [12] to improve the performance of the registration [13], especially when the simple sum of squared differences (SSD) similarity measure is used which assumes that homologous structures in the two images are depicted by the same image intensity. Inconsistent spatial inhomogeneities of the intensities over the image domain, such as those introduced due to the spatially varying sensitivity of the radiofrequency (RF) coils in MRI, may necessitate inhomogeneity bias correction [14, 15]. Preprocessing can also be used to emphasize certain image features [16]. Finally, limiting the calculation of the similarity metric to a certain region of interest in the image can improve the robustness of the method. As an example, segmentation of the brain volume, or skull-stripping [17], is often required for reliable co-registration in neuroimaging applications.

The degrees of freedom of the deformable registration, and, in turn, the computational burden associated with the optimization procedure, are determined by the transformation model used to represent the mapping between the registered images. A successful deformable registration is typically contingent on the successful rigid or affine alignment of the registered objects that must be completed prior to the modeling of the deformation. Transformation models that utilize various forms of spline functions have been utilized in a variety of image-based registration methods. B-splines are commonly used since they use local support and thus are less sensitive to the variation of the parameters over the image domain, enabling more flexible representation of localized deformations [18–20]. The flexibility of the B-spline model can be controlled by varying density of the control point grid, which facilitates multi-resolution registration schemes. Thin-plate spline transformations do not require regular sampling of the control points but are more sensitive to the local deformations since the interpolated displacement at any given point is affected by each of the control points [21].

A disadvantage of the spline-based methods is that they do not include any patient-specific modeling to produce realistic deformations of biological tissue. On the contrary, physically based methods use biomechanical models discretized by finite elements (FE) for including patient-specific modeling, e.g., mechanical properties of different biological tissue. The most popular approach is to model tissue deformations with an elastic model, and assuming infinitesimal strains leads to the computationally efficient linear elastic model. Dense transformation models, i.e., where a displacement vector is computed at every voxel such as in the Demons method [22], have a high number of degrees of freedom and can consequently model complex deformations but are also susceptible to unrealistic deformations and local foldings.

Nonrigid registration is a highly underdetermined problem, e.g., at each voxel in a 3D fixed image, a 3D displacement vector needs to be estimated to find the corresponding position in the moving image, and consequently we have three times as many unknowns as data points. The transformation model generally does not prevent folding or tearing of the deformation field, which can be characterized as nonphysical deformations that do not represent the deformation of typical human tissue. Typically, a regularizer is added to the registration objective function to properly constrain the problem to finding solutions that are physically plausible. Commonly, the regularizer aims at constraining the deformation to adhere to some physical properties. Depending on the underlying tissue to be registered, such properties can be smoothness, volume preservation, elastic [23–25], fluid [9], or hyper-viscoelastic [26]. Another popular means of constraining the deformation, especially in atlas-based registration, is to force the deformation and its inverse to be smooth functions, i.e., to be a diffeomorphism. Many methods are available, of which the most popular are the diffeomorphic demons [27] and the large deformation diffeomorphic metric mapping (LDDMM) [28, 29].

The similarity measures used in rigid registration are typically applicable in deformable registration applications. The choice of the metric is based on the specifics of the registered modalities and the computational constraints imposed by the IGT application. Registration of the modalities that maintain consistent mapping of the image values, such as CT, can be feasible using simple metrics such as sum of squared differences. Information theoretic metrics, such as mutual information, have proven practical and robust in a variety of applications and modalities [30, 31]. Specific to deformable registration, template matching schemes [32] were proposed to improve the sensitivity of the metric to local deformations. Sampling of the full image domain for the purposes of evaluating the similarity metric is frequently not practical for time-critical IGT applications; various sampling strategies have been investigated to reduce the computational burden while maintaining or improving smoothness of the cost function [33, 34]. Some of the popular optimization approaches used for registration purposes were surveyed by Maes et al. [35].

Validation of the results produced by the deformable registration methods is significantly more challenging than for the problems that involve linear transformations. In the context of IGT, the question of primary interest is the registration error in the area of clinical interest. Such estimations typically require identification of homologous points that correspond to the salient image structures that can be consistently identified in both images. Such points can correspond to the intrinsic anatomical features that can be identified in a population of patients [36, 37], subject-specific image features, such as calcifications in the prostate imaging [16], or implanted markers [38]. Localization of such homologous points is typically done by a domain expert, and the registration errors should be considered in comparison with the feature localization errors. Reliable and consistent identification of the corresponding features is not always feasible due to the limited resolution, lack of localized image features, and the fact that the registered images often provide complementary information about the imaged anatomy, and the features present in one modality may not be reliably identifiable in the other image. Therefore, digital [39] and physical phantoms [40, 41] that can model realistic deformations of the studied organ are often employed. Measures based on the overlap of the corresponding segmented structures (such as Dice and Jaccard metrics) [42] and the Hausdorff distance [43] have also found applications in assessing the results of NRR methods.

Precise quantification of the absolute error in an arbitrary region of the image domain is often not possible because of the lack of ground truth and difficulties in identifying corresponding image features. In such situations characterization of the registration algorithm performance may still be possible by quantifying the uncertainty in the registration result. In point-based rigid body registration, registration uncertainty has been studied extensively to quantify the distribution of the target registration error (TRE) [1, 2, 44], which can be important for the surgeon, for instance during image-guided biopsies. Conversely, in nonrigid intraoperative registration, registration uncertainty is mostly neglected, and research has been focused on finding the most likely point estimate of the registration problem. In the case of neurosurgery, because of intensity differences due to contrast enhancement, degraded image quality and missing data due to resection, the registration uncertainty will increase in and around the resection site [45]. Therefore, reporting both registration uncertainty and the most likely estimate can potentially provide the surgeon with important information that affect the understanding of the registration results and consequently the decisions made during surgery, especially when the tumor is adjacent to important functional areas.

For intensity-based nonrigid registration, few methods are available for estimating the registration uncertainty. Kybic [46] models images as random processes and proposed a general bootstrapping method to compute statistics on registration parameters. Hub et al. [47] developed a heuristic method for quantifying uncertainty in B-spline image registration by perturbing the B-spline control points and analyzing the effect on the regional similarity criterion. In Simpson et al. [48], the posterior distribution on B-spline deformation parameters were modeled with local Gaussian distributions and approximated by a variational Bayes method. It was shown that utilizing the registration uncertainty in classification of Alzheimer patients improved the classification results. Gee et al. [49] described a Bayesian approach to uncertainty estimation, where the posterior distribution, composed of local Gaussian approximations of the likelihood term and a linear elastic prior, was explored with a Gibbs sampler. Another Bayesian approach was taken by Risholm et al. [50] which used Markov chain Monte Carlo (MCMC) to characterize the posterior distribution on registration parameters which does not require any (Gaussian) approximations to the posterior distribution. Because registration likelihood terms must depend on the complicated spatial structure of images, they are generally highly non-convex with respect to the deformation parameters, thus the assumption that the posterior distribution is unimodal and Gaussian is suspect. Recent results [51] demonstrate that the distribution on registration results may be multimodal and have heavier tails than Gaussian distributions, reflecting the possibility of results that are essentially outliers. In Risholm et al. [45] methods were introduced for summarizing and visualizing the registration uncertainty in the case of neurosurgery for tumor resection, and it was demonstrated that uncertainty information is important to convey to a surgeon when eloquent areas are located close to the resection site. An example of the uncertainty associated with registration results when registering pre- with intraoperative MRI of the brain is included in Fig. 14.2.