Medical image segmentation is a process to separate structures of interest in an image from its background or other neighboring structures. It is a necessary prerequisite step for many medical imaging applications in radiology and radiation therapy. These applications may include automatic organ delineation, quantitative tissue classification, surface extraction, visualization and image registration, etc. [64, 65]. For instance, Pham and coworkers divided segmentation algorithms into eight different categories: thresholding, region growing, classifiers, clustering, Markov random field models, artificial neural networks, deformable models, and atlas-guided approaches. In our work on PET-guided treatment planning in radiotherapy, we presented a comparative survey of the current methods applied for tumor segmentation [66, 67]; an example in head and neck cancer using different segmentation algorithms is presented in Fig. 12.4.

There are several commercial and academic software tools that support different segmentation algorithms. In general, commercial software packages have better implementations with a user-friendly interface for manual and semiautomatic segmentation methods, but often lag behind the latest development in the field. In contrast, academic software packages, such as ITK [68], BioImage Suite [69], MIPAV [70], ImageJ [71], and 3D slicer [72], may tend to be oriented toward single-modality applications and less friendly in handling multimodality images as proposed here.

Most automatic algorithms attempt to utilize image intensity variations or image gradient information. However, for low-contrast images, many of these algorithms tend to provide suboptimal solutions that are not clinically acceptable. For such cases, it has been demonstrated that if multiple images are available for the same object (the same image modality or different image modalities), all the available complementary information can be fed into the segmentation algorithms to define the so-called biophysical target [73]. Thus, the segmentation algorithms would benefit from the complementary information from different images, and consequently the accuracy of the final segmentation results could be improved. Similar approaches have been applied for detecting blood-wall interface of heart ventricles from CT, MRI, and ultrasound images using a snake deformable model [74], for classifying coronary artery plaque composition from multiple contrast MR images using K-means clustering algorithm [75], and for defining tumor target volumes using PET/CT/MR images for radiotherapy treatment planning by using a multivalued deformable level set approach as in our previous work. Mathematically, such an approach is a framework that could be thought of as a mapping from the imaging space to the “perception” space as identified by the radiologists [73]:
where is the mapping function from the different imaging modalities to the target space parameterized by λ, which represents users’ defined set of parameters representing the prior knowledge. This framework is highlighted in Fig. 12.5.

The robust image segmentations methods are based on deformable models, which are geometric representations of curves or surfaces that are defined explicitly or implicitly in the imaging domain. These models move under the influence of internal forces (contour curvature) and external forces (image boundary constraints) [76, 77]. The level set is a state-of-the-art variational method for shape recovery [76, 78–80]. They were originally developed in curve evolution theory to overcome the limitations encountered in parametric deformable models (e.g., snakes [81]) such as initialization requirement, generalization to 3D, and topological adaptation such as splitting or merging of model parts. Our generalization to multimodality imaging is based on redefining the concept of a boundary as a logical or “best” combination of multiple images by learning and incorporating expert’s knowledge on subregional or even voxel levels. An example showing combination of PET/CT in lung cancer is shown in Fig. 12.6 using a multivalued level set algorithm [73].

In another example the PET/CT images were taken from patients with cervix cancer. The PET image was sharpened using a deconvolution approach [82]. The 40 % maximum SUV (standard uptake value) thresholding is adopted in many institutes to estimate gross tumor volume for cervix cancer patients due to the high target to background ratio of these tumors in PET and the difficulty to distinguish their boundary in CT. In Fig. 12.7, the active contour algorithm is initialized with a circle (in white) of 15.9 mm diameter around the PET lesion. The evolved contour took ten iterations (in blue) and the final estimated contours (in thick black) are shown. The algorithm converged in just 30 iterations. This fast convergence could be attributed in part to the almost spherical shape of the tumor and the sharpness of the gradient. It is noticed that the results of the algorithm match the PET ground truth (99 %) as delineated by an experienced nuclear medicine radiologist. Hence, the delineation results were explained mainly by PET in this case, although information from CT could still be used to steer the algorithm, if desired.

The extraction of quantitative information from imaging modalities and relating information to biological and clinical endpoints is a new emerging field referred to as “radiomics” [32, 33]. Traditionally, quantitative analysis of FDG-PET or other PET tracer uptake is conducted based on observed changes in the standard uptake value (SUV). For instance, decrease in SUV postirradiation has been associated with better outcomes in lung cancer [83, 84]. However, SUV measurements themselves are potentially pruned to errors due to the initial FDG uptake kinetics and radiotracer distribution, which are dependent on the initial dose and the elapsing time between injection and image acquisition. In addition, some commonly reported SUV measurements might be sensitive to changes in tumor volume definition (e.g., mean SUV). These factors and others might make such approach subject to significant intra- and inter-observer variability [25, 26, 34].

Radiomics consist of two main steps, extraction of static and dynamic features as discussed in Sect. 17.​3 and outcome modeling as presented in the following. Outcomes in oncology and particularly in radiation oncology are characterized by tumor control probability (TCP) and the surrounding normal tissue complication probability (NTCP) [85, 86]. A detailed review of outcome modeling in radiotherapy is presented in our previous work [87]. DREES is a dedicated software tool for modeling of radiotherapy response [88]. In the context of image-based treatment outcome modeling, the observed outcome (e.g., TCP or NTCP) is considered to be adequately captured by extracted image features [34, 89]. We will highlight this approach using classical logistic regression.

Logistic modeling is a common tool for multi-metric modeling. In our previous work [90, 91], a logit transformation was used: