Intervertebral discs are unique structures that absorb shocks between adjacent vertebrae. They act as the ligaments that connect the vertebrae together and the pivot point which allows the spine mobility by bending and rotating. They make about one fourth of the spinal column length [88].

An intervertebral disc is composed of two parts: an outer strong ring called Annulus Fibrosus and a soft gel-like inner called Nucleus Pulposus. The nucleus pulposus consists of 80–85 % water in normal cases. By aging the disc dehydrates limiting its ability to absorb shocks. The outer rings gets weaker as well and start having tears that causes various abnormalities. The bottom up view is the anterior while top down is the posterior direction. Figures 6 and 7 show T1- and T2-weighted MRI for the same lumbar disc from our dataset, respectively.

A vertebra (plural: vertebrae) is a bone with specific structure that supports and protects the spinal cord. A typical vertebra consists of two segments: Anterior (front) and posterior (back). The anterior part of the vertebra is the body while the posterior, which is known also as the vertebral (neural) arch, includes: the vertebral foramen, a pair of pedicles and a pair of laminae, and supports seven processes [four articular, two transverse, and one spinous (aka neural spine)].

In addition to the main structures within the vertebral column, there are few other structures including.

Nerves: The spinal cord hangs inside a bony ring through the vertebral column that is made up of millions of nerve fibers. The spinal cord extends down to the L2 vertebra. Below L2, a bundle of nerves named as Cauda Equina hangs down in what is known as the Thecal Sac. Two large nerves branch off the spinal cord, one from each side passing through the neural foramina of each vertebra. These spinal nerves group together to form the main nerves that go to the organs and limbs. The nerves of the lumbar spine (Cauda Equina) go to the pelvic organs and lower limbs [88].

Connective tissues: They are the fibrous connections that hold the cells of the body together. The ligaments are strong connective tissues that attach bones together. There are many long ligaments that connect on the front and back sections of the vertebrae. The anterior longitudinal ligament runs lengthwise down the front of the vertebral bodies. Two other ligaments run full-length within the spinal canal. The posterior longitudinal ligament attaches on the back of the vertebral bodies. The Ligamentum Flavum is a long elastic band that connects to the front surface of the lamina bones (just behind the spinal cord). Thick ligaments also connect the bones of the lumbar spine to the sacrum (the bone below L5) and pelvis.

Muscles in the lower back are arranged in layers. The superficial layer, the closest to the skin, is covered by a thick tissue called Fascia. The middle layer, called the Erector Spinae, has strap-shaped muscles that run up and down over the lower ribs, chest, and low back. They join in the lumbar spine to form a thick tendon that binds the bones of the low back, pelvis, and sacrum. The deepest layer of muscles attaches along the back surface of the spine bones, connecting the low back, pelvis, and sacrum. These deepest muscles coordinate their actions with the muscles of the abdomen to help hold the spine steady during activity [95].

Spinal segments is a notion that includes two vertebrae separated by an intervertebral disc, the nerves that leave the spinal column at each vertebra, and the small facet joints that link each level of the spinal column. The intervertebral disc separates the two vertebral bodies of the spinal segment. The disc normally works like a shock absorber. It protects the spine against the daily pull of gravity. It also protects the spine during heavy activities that put strong force on the spine, such as jumping, running, and lifting. The spinal segment is connected by two facet joints described earlier. When the facet joints of the lumbar spine move together, they bend and turn the low back [95].

More than two decades ago, Hedlund and Gallagher [39] performed vertebral morphometry on lateral thoracic and lumbar X-ray radiographs of 153 women with a preliminary diagnosis of Spinal Osteoporosis. Measurements included anterior and posterior vertebral height, width, area, wedge angle, percent reduction of Anterior to Posterior Height (PRH) and Percent Difference in Anterior Height between adjoining vertebrae (PDAH). They showed that among individuals with mild Osteoporosis (0–2 fractures) PDAH identified 86 % of the fractures and 95 % of the individuals with fractures.

Manual selection of anatomical points for vertebral abnormality diagnosis is time consuming, imprecise and subjective. To obtain a more objective and accurate description of the vertebral body shape, semi-automatic methods were proposed that were based on statistical models of vertebral bodies in the sagittal view. Very early on, in 1993, a computerized quantifying technique for vertebral morphometry on lateral radiographs of the spine was proposed by Nicholson et al. [73]. Although fracture detection was improved by expanding the description of the vertebral body shape from six points to a contour, the amount of traumatic spinal injury or latent vertebral fracture was often underestimated. The main reason for the wrong diagnosis originated from the limited measurement possibility caused by the lack of depth perception in X-ray radiographs. Later in 1997, Smyth et al. [87] described how Active Shape Models (ASM) could be used to locate both normal and fractured vertebrae from Dual energy X-ray Absoptiometry (DXA) images of the spine. However, three initialization points have to be manually selected. To overcome the lack of depth perception in X-ray images, Benameur et al. [8] performed projection of a three-dimensional (3D) statistical shape model of a vertebra to a pair of orthogonal 2D X-ray radiographs. They validated this method on 57 scoliotic vertebrae images. However, the proposed segmentation was highly dependent on model initialization.

Various efforts that target the diagnosis of certain vertebra conditions involved localization and segmentation. In 2000, Long and Thoma [60] investigated the segmentation of C2 and C3 vertebrae from the cervical area using an ASM as a first step for building an image based retrieval system for a dataset consisting of 7,000 lumbar X-ray radiographs and 10,000 cervical spine X-ray radiographs. They built the Web-based Medical Information Retrieval System (WebMIRS) based on the National Health and Nutrition Examination Surveys (NHANES). Later, Cherukuri et al. [16] proposed image processing techniques for computing size-invariant, convex hull-based features to highlight anterior Osteophytes. Feature evaluation of 714 lumbar spine vertebrae using a multi-layer perceptron yielded normal and abnormal average correct discrimination of 90.5 and 86.6 %, respectively. Despite that the main purpose of this work is diagnosis, a great portion of this effort was to identify the vertebra.

Another work was presented by de Bruijne and Nielsen [11] who used Shape Particle Filter [23] and k-nearest neighbor pixel-level classification for a semi-automatic segmentation of the lumbar vertebrae. Around the same time, Kaminsky et al. [47] presented a standardized protocol that combined newly developed interactive tools (rotation transformation, warped dissection plane) with standard segmentation tools to provide both a fast and accurate 3D spine segmentation procedure. Howe et al. [42] proposed a multi-level segmentation technique for vertebrae from cervical and lumbar X-ray radiographs using an ASM and a generalized Hough transform. Their validation is based on a leave-one-out test with an error of 2 mm for 57 % of the cervical cases and less than 4 mm for 68 % of the lumbar cases. However, their method needs manual intervention for initialization.

Later, Crimi et al. [22] presented a Bayesian approach and used prior information to estimate the covariance matrix from a small number of samples in a high dimensional shape to segment the vertebrae from X-ray radiographs. Moreover, Zewail et al. [103] segmented the vertebrae from X-ray radiographs using a contourlet-based salient point matching and a localized multi-scale shape prior. They tested their work on 100 X-ray radiographs and obtained an average segmentation error of 1.2 mm. Later, Crimi et al. [22] presented a Bayesian approach and used prior information to estimate the covariance matrix from a small number of samples in a high dimensional shape to segment the vertebrae from X-ray radiographs. Moreover, Zewail et al. [103] segmented the vertebrae from X-ray radiographs using a Contourlet-based salient point matching and a localized multi-scale shape prior. They tested their work on 100 X-ray radiographs and obtained an average segmentation error of 1.2 mm.

More recently, Lecron et al. [59] presented a fully automated vertebrae detection. They used an edge polygonal approximation to detect vertebral edges and a SIFT descriptor to train an SVM-model. They achieved a corner detection rate of 90.4 % and a vertebra detection rate from 81.6 to 86.5 % on 250 cervical cases.

Despite that X-ray radiographs are mainly used for bone visualization, few efforts have been performed in the analysis of the intervertebral discs and the spinal canal. Chamarthy et al. [14] introduced image analysis techniques, including scale-invariant, distance transform-based features to characterize the disc space narrowing (with four grades zero to three) in cervical vertebrae. For a data set of 294 vertebrae X-ray images, experimental results yielded average correct grade assignment of greater than 82.10 %. However, the testing images had to be manually labeled with the boundary points.

Later on, Koompairojn et al. [56] described a fully automatic Spinal Stenosis diagnosis system via vertebral morphometry [73], using an Active Appearance Model (AAM) for segmentation and a Bayesian framework for classification. Experimental results on 86 lumbar spine X-ray images from the NHANES II database showed accuracy ranging from 75 to 80 %. Moreover, Stanley et al. [89] investigated new size-invariant features (claw and traction) for the detection of anterior Osteophytes for efficient Content Based Image Retrieval (CBIR). Using a K-means clustering and nearest neighbor classification approach, average correct classification rates of 85.80, 86.04 and 84.44 % were obtained for claw, traction and anterior Osteophytes, respectively, on 390 cervical vertebrae.

A number of automatic and semi-automatic methods for segmentation of vertebrae and vertebral structures in CT have been proposed [27, 50, 52, 64, 80, 84, 91, 96, 98, 102] over the last decade. On one hand some researchers proposed techniques that segment each vertebra separately [50, 61, 64, 84, 98], which might lead to mis-segmentation due to absence of a clear boundary between vertebrae. To overcome this issue, some authors proposed techniques for simultaneous segmentation of all vertebrae [52, 80].

Early last decade, Hahn [36], proposed a fully automated approach to evaluate rotation of the cervical vertebrae in 3D using a multidimensional Powell minimization algorithm for spiral CT scans. Later, in 2004, the same research group [37] presented a method for determination of the planes separating the individual vertebrae of the spine from CT volumes using a Balloon based model. This model requires careful initialization similar to the 2D active contours (snakes) besides its high dependency on the edge detector.

Meanwhile, Ghebreab and Smeulders [27] presented a combination of Strings [26] and Necklaces [28] to model the spine in the lumbar area using both a priori knowledge about natural variation and anatomical saliency in the visual appearance of the spine. The Strings model focuses on learning the most relevant biological variation in the visual appearance of the spine as a whole, and Necklaces aims at exploiting inhomogeneities in multiple continuous shape and gray-level features of vertebrae. Thus they were able to use both a priori knowledge about natural variation and anatomical saliency in the visual appearance of the spine. However, they tested their method on only six CT cases and with minimal spinal and vertebral deformations. Furthermore, manual intervention is used for initialization of their model. On the other hand, Vrtovec et al. [96] detected the spine curve from CT using a polynomial model to provide a Curved Planar Reformation (CPR) of the 3D spinal column. They fit the spinal curve to a set of points extracted from a distance map that emphasized the vertebral bodies and tested the method on five cases, including one Scoliotic case achieving mean positional errors between 2 and 6 mm.

Furthermore, Yao et al. [102] presented a systematic algorithm for segmenting the spinal column from chest and/or abdominal CT scans, without labeling of vertebrae. Their method is based on thresholding, Watershed, and directed graph search besides modeling the vertebral bony tissue as a four-part model. They showed correct segmentation of 69 cases out of the 71 total cases. Meanwhile, Mastmeyer et al. [64] segmented the lumbar vertebral bodies in CT images by combining viscous deformable models with the geometrical shape of the vertebral body, starting from a point in the center of each vertebral body. Tan et al. [91] presented a level set-based segmentation algorithm for the vertebrae and validated their work on synthetic 3D vertebrae volumes. After parameter selection, they tested the algorithm on 50 vertebrae (from ten subjects), obtaining 90 % success rate. Later, Shen et al. [84] presented a segmentation technique for vertebrae from 3D CT scans using prior knowledge with a set of high level features to form a surface model. However, they did not perform labeling. They tested their model on 150 vertebrae with a comparative segmentation to two experts’ segmentation. In the same year, Klinder et al. [51] presented a two-scale framework for modeling and segmenting the spine from thoracic CT scans, achieving a segmentation accuracy of 1.0 mm in average for ten thoracic CT volumes. By applying statistical models of shape, gradient and appearance of spinal structures in 3D, the same research group [52] detected, identified and segmented the vertebrae in CT volumes. However, their identification algorithm is based on vertebrae Active Appearance Model for spatial registration and matching which is very computationally expensive (20–30 min per case). Their framework was tested on 64 CT images including pathologies like Scoliosis, Kyphosis and collapsed vertebrae. Later, Kim and Kim [50] automatically segmented the vertebrae by a region growing algorithm inside a volume limited by a 3D fence that was obtained from a deformable model. They obtain 80 % success on a 50 patient dataset. More recently, Ma and Lu [61] proposed a method for segmentation and identification of thoracic vertebrae in CT images by training an edge detector to bone structures via steerable gradient features and using a deformable surface model in a two-stage coarse-to-fine scheme. They achieve point-to-surface error 0.95 ± 0.91 mm on 40 volumes.