Upstream Machine Learning in Radiology

Key points

•

This article reviews multiple applications of artificial intelligence (AI) in which it has been used to vastly improve upstream components of the medical imaging pipeline.
•

All medical imaging examinations can be accelerated by using AI to help manage examination schedules and imaging protocols.
•

In ultrasound, AI has been used to form high-quality images with enhanced robustness to speckle noise.
•

In computed tomography, AI-based image reconstruction approaches have been used to reconstruct high-quality images from low-dose acquisitions.
•

In MR imaging, AI has been used to design optimal data sampling schemes, and reconstruction approaches to accelerate scan time.

Introduction

Machine learning (ML) approaches have many applications in radiology. Over the last several years, much attention has been garnered by applications focused on assisted interpretation of medical images. However, a significant component of radiology encompasses the generation of medical images before interpretation, the upstream component. These elements include modality selection for an examination, hardware design, protocol selection for an examination, data acquisition, image reconstruction, and image processing. Here, the authors give the reader insight into these aspects of upstream artificial intelligence (AI), conveying the breadth of the emerging field, some of the techniques, and the potential impact of the applications.

Discussion

The standard radiology workflow can undergo 7 steps: (1) conversion of a patient’s clinical question into a radiology examination order, usually performed by a nonradiologist physician; (2) conversion of an examination order into an examination protocol, usually performed by a radiologist or radiology technologist based on institution-specific guidelines; (3) scheduling of an examination onto a specific scanner, usually performed by a radiology scheduler or technologist; (4) adaption of a protocol to a specific device, usually performed by a radiology technologist; (5) acquisition and reconstruction of acquired imaging data into images, usually performed automatically with software algorithms directly on the imaging hardware; (6) image processing, to retrospectively improve image quality and produce multiplanar reformations and quantitative parameter maps, typically performed directly on the imaging hardware; and (7) evaluation of images to render a diagnosis related to the patient’s clinical question, usually performed by a radiologist ( Fig. 1 ). Applications and research areas for AI algorithms in radiology have typically focused on the final 2 steps in the radiology workflow. Several AI techniques exist to perform automated classification, detection, or segmentation of medical images, many of which are even approved for clinical use by the Food and Drug Administration. Furthermore, the acquisition of raw data for generating diagnostic quality images is also being used for reducing scan time or radiation dosage of medical images, or retrospectively improving resolution or signal-to-noise ratio (SNR) of images.

Despite a dearth of research studies, there exist substantial challenges for all steps in the “preimage acquisition” workflow (steps 1–4) that may be alleviated using data-driven benefits that AI techniques can offer. For example, a patient’s initial examination order may not always be correctly defined by a radiology specialist, which can lead to delays in insurance authorizations and the schedule of the correct examination. Next, conversion of an order to a protocol is commonly performed by radiology fellows at many academic institutions, whereas many private practice organizations may not have such a luxury. Furthermore, with “protocol creep” of new protocols being aggregated in an unorganized manner, choosing the appropriate protocol for the patient becomes a more challenging task. The scheduling of patients to different scanners also requires considerations for complex cases that may require a radiologist present. Following successful protocoling and scheduling, tuning the scanner parameters to cater to the specific examination order needs and patient requirements (such as image resolution, coils, and breath holds for MR imaging, dose optimization and contrast needs for computed tomography [CT], and so on) can directly affect image quality, and consequently, downstream utility for eventual image interpretation.

Errors can occur during any part of the radiology preimage-acquisition workflow and can lead to delayed diagnoses, or, in the worst case, missed diagnoses. Each of these 4 steps is repetitive, but because of the innate variability between patients, these tasks have not lent themselves to automated techniques. Using manual techniques to complete such tasks is prone to user error, leads to delays in the workflow, and is an inefficient use of human capital particularly when these tasks require radiologist oversight. The application of premage-acquisition AI can lead to potential gains in efficiency and patient care. There are several factors that make these tasks ripe for automation with AI techniques. First, all the tasks are already performed on computer systems, entailing that all inputs to and output of the tasks are already digitized and recorded and can be retrospectively extracted. Second, as large volumes (hundreds to thousands) of most examination types are performed every year, even at a single academic center, large institution-specific data sets needed to train AI algorithms are already available. Finally, a stepped pathway for incorporating these algorithms into the radiology workflow can be achievable. Initial versions of the ML algorithms can suggest or preselect choices for use as a clinical decision support system with continued human oversight. Following this, continual-learning strategies can be used to improve such algorithms to have higher reliability than humans for eventual transition to full automation.

Recent advances in natural language processing (NLP) have been used to perform automated assignment for CT and MR imaging protocols using procedure names, modality, section, indications, and patient history as inputs to guide a clinical decision support model. Similar NLP systems have also been previously used to use unstructured data for performing automated protocoling for musculoskeletal MR imaging examinations ^, as well as brain MR imaging examinations. Newer AI techniques for NLP that use the Bidirectional Encoder Representations from Transformers model have also been used in the context of automated protocoling. Advances in multimodality data integration present an exciting opportunity to improve the upstream workflow by further combining data from a patient’s medical record, along with AI models that rely on conventional computer vision and NLP techniques.

Although the techniques and applications of upstream AI are broad, the authors later discuss some representative examples across various commonly used medical imaging modalities.

Improved Image Formation in Ultrasound

In ultrasound image reconstruction, raw echo data are received by an array of transducer sensors. These “channel data” are processed to produce an image of the underlying anatomy and function. Traditional ultrasound image reconstruction can include various noise artifacts that degrade signal and image quality. Rather than attempting to correct these using already-reconstructed images, upstream ultrasound research uses the raw channel data to address the artifacts at their source. AI has found a wide range of early uses in upstream ultrasound, a few of which are highlight here.

A major source of ultrasound image degradation is reverberation clutter. Clutter arises from shallow reverberations that are incorrectly superimposed on echoes from deeper tissue, resulting in decorrelated channel data. This decorrelation manifests in ordinary B-mode images as a dense haze and also degrades the ability to perform Doppler and other advanced imaging techniques. Several non-AI techniques have been proposed to remove clutter from channel data directly, improving channel data correlations and B-mode image contrast, albeit at great computational cost. Recently, AI has been used to achieve the same goal using a data-driven approach. ^, In the authors’ work, they used matched simulations of ultrasound sensor data with reverberation clutter (input) and without reverberation clutter (output) to train a convolutional neural network (CNN). Fig. 2 demonstrates how the CNN removes high-frequency clutter from simulated channel data. In Fig. 3 , the CNN removes clutter from in vivo channel data, resulting in B-mode images with improved structure visualization and contrast. The CNN restores correlations in the raw channel data and thus enables subsequent advanced imaging techniques, such as sound speed estimation and phase aberration correction.

Another major noise source in ultrasound images is a pervasive grainy texture called speckle. Speckle is an unavoidable artifact arising from the finite resolution of ultrasound imaging systems. Although speckle can be useful for certain applications, it is largely treated as an undesirable noise that degrades B-mode images of the underlying tissue echogenicity. Rather than filtering speckle from already-reconstructed B-mode images, the authors trained a CNN to estimate the tissue echogenicity directly from raw channel data using simulations of ultrasound speckle and known ground-truth echogenicity. In simulations, the trained CNN accurately and precisely estimated the true echogenicity. In phantom and in vivo experiments, where the ground-truth echogenicity is unknown, the CNN outperformed traditional signal processing and image processing speckle reduction techniques according to standard image quality metrics, such as contrast, contrast-to-noise ratio, and SNR. Fig. 4 shows an in vivo example of a liver lesion acquired using a clinical scanner. This CNN has been further demonstrated in real time on a prototype scanner.

Another emerging application in ultrasound is contrast-enhanced imaging, including with targeted contrast agents, to detect the presence of disease. Contrast-enhanced images are presented alongside ordinary B-mode images to highlight any disease-bound contrast agents in the field of view. A key challenge in contrast-enhanced imaging, particularly molecular imaging, is to detect echoes from contrast agents while suppressing echoes from background tissue. Current state-of-the-art imaging identifies molecular signals retrospectively after a strong destructive ultrasound pulse eliminates them from the field of view and thus cannot be used for real-time imaging. The authors recently used CNNs to nondestructively emulate the performance of destructive imaging, enabling real-time molecular imaging for early cancer detection. Rather than treating B-mode and contrast-enhanced modes as separate images, the CNN combines raw data from both to improve the final molecular image. As shown in Fig. 5 , the trained CNN (panel D) was able to mimic the performance of the state-of-the-art destructive approach (panel C), enabling high-quality real-time ultrasound molecular imaging.

In addition to these, AI has found numerous other upstream ultrasound applications, including adaptive beamforming, ultrasound localization microscopy, surgical guidance, vector flow Doppler imaging, ^, as well as many others. The range and variety of applications highlight the extraordinary adaptability of AI techniques to ultrasound signal and image processing.

Computed Tomography Dose Optimization

More than 75 million CT scans are performed annually in the United States, with the number expected to continue increasing. Despite clear clinical value, ionizing radiation dose has been a major concern for CT. In addition, there are significant variations in radiation dose and image quality across institutions, protocols, scanners, and patients. Although CT has evolved from the manual adjustment of dose control parameters, such as tube current, to automatic tube current modulation, CT dose optimization still has major challenges in achieving the most efficient use of radiation dose for each patient and clinical task. ^, An overview of existing technologies for controlling dose and image quality is shown in Fig. 6 .

Reconstructed CT images contain quantum noise when the ALARA (as low as reasonably achievable) principle is followed to reduce radiation exposure. As a result, important details in the image could be hidden by the noise, leading to degradation in the diagnostic value of CT. It is therefore crucial to assess the quality of reconstructed CT images at reduced dose. Dose and image noise have a complex dependency on patient size (ranging from pediatrics to obese adults), patient anatomy, CT acquisition, and reconstruction. As can be seen in Fig. 7 , CT images at lower doses relative to the routine dose induce higher noise. Unfortunately, there is no consistent image quality assessment measure that has been established to objectively quantify images from different imaging protocols, vendors, sites, and so forth. Having such capabilities would help balance image quality and radiation dose. At every step of the CT scan, from system design to acquisition to reconstruction, efforts could then be made to lower radiation dose. As a result, the quality of the acquired CT images could be varied and assessed to guide dose reduction. In addition, effective dose to the patient needs to account for the different radiosensitivity of different organs (for example, using the International Commission on Radiological Protection tissue weights). Therefore, an accurate, reliable, and region-focused assessment of dose and image quality is extremely important.

In an effort to combat increased noise in low-dose settings, CT image reconstruction has been going through a paradigm shift, from conventional filtered backprojection (FBP) to model-based iterative reconstruction (MBIR), and now to deep learning (DL) image reconstruction. FBP, although fast and computationally efficient, incurs higher noise and streak artifacts at low dose. Toward improving image quality at low dose, iterative reconstruction has emerged that preserves diagnostic information, although low-contrast detail remains a challenge and can impact detectability of subtle pathologic conditions. With the advent of machine learning and DL, in particular, high diagnostic quality is created with reduced noise at low dose, with the additional benefit of faster reconstruction than iterative reconstruction, for example, (1) generating standard-dose images from low-dose images in DL-based CT denoising ; (2) through incorporating deep CNN in the complex iterative reconstruction process, less noisy images can be obtained at low dose ; (3) improved image quality of dual-energy CT, particularly for large patients, low-energy virtual monoenergetic images, and material decomposition ; (4) reduction of metal artifacts through DL techniques. ^, DL-based image reconstruction enables significantly lower image noise and higher image quality (IQ score and better CNR [contrast to noise ratio]) over FBP and MBIR, which can be traded off for dose reduction. However, dose reduction needs to be balanced against an acceptable level of diagnostic accuracy. Therefore, not only phantom data but also patient data are needed to understand the impacts of dose reduction on image quality (using objective metrics) as well as the task-specific diagnostic performance (such as detecting liver lesions).

Conventionally, medical images are assessed subjectively requiring manual labor, long times, and observer variations, which is impractical for use in a clinical workflow. It is therefore desirable to perform fast, automatic, and objective image quality assessment. Objective assessment can be of 2 types: reference-based and non-reference-based. Several reference-based image assessment metrics have been widely used in medical imaging. For example, PSNR, MSE, CNR, SSIM, and RMSE are often used. However, these metrics rely on the availability of high-quality reference images. Because the goal is to acquire CT scans at reduced dose, one cannot assume to have access to high-quality (and high-dose) images of the same patient. Therefore, nonreference assessments are preferred so that any CT image can be quantified without requiring the corresponding reference images at high dose. In addition to the established image quality assessment metrics, a few nonreference assessment methods are also available, such as a blind/referenceless image spatial quality evaluator based on natural scene statistics that operates in the spatial domain. A growing body of research has recently emerged focusing on no-reference and automated image quality assessment using DL. For example, Patwari and colleagues used a regression CNN to learn mapping noisy CT images to GSSIM scores, as an assessment of the reconstructed image quality. Kim and colleagues proposed a no-reference image quality assessment framework via 2-stage training: regression into objective error map and subjective score. Another DL-based nonreference assessment has recently been proposed leveraging self-supervised prediction of noise levels.

In summary, reduction of CT dose directly impacts image quality. Preserving image quality at low dose has been an active area of research. DL-based methods have achieved some promising results in CT image reconstruction, denoising, artifact reduction, and image quality assessment. However, to bring dose optimization to CT acquisition, prediction of the expected dose and image quality must be real time, patient specific, organ specific, and task specific. This problem is a complex and challenging one for which DL algorithms are likely to excel.

Optimized Data Acquisition in MR Imaging via Deep Learning

In MR imaging, data are acquired in the Fourier domain, or k-space, which represents spatial and temporal frequency information in the object being imaged. Sampling of k-space is necessarily finite. Therefore, optimizing sampling patterns and designing k-space readout trajectories that enable collecting the most informative k-space samples for image reconstruction can improve the resulting diagnostic information in images. Besides improved image quality, the design of optimized sampling trajectories can lead to shortened data acquisition for improved efficiency. Recently, several DL methods have been proposed for learning k-space sampling patterns jointly with image reconstruction algorithmic parameters. These methods have shown improved reconstructed image quality compared with reconstructions obtained with predetermined trajectories, such as variable density sampling. DL methods for data sampling can be separated into 2 categories: active and nonactive (fixed) strategies. These 2 strategies differ primarily in whether the learned sampling schemes are fixed or not at inference time.

Given a target k-space undersampling or acceleration factor, the nonactive (fixed) strategies produce undersampling masks or sampling trajectories using a set of fully sampled images. After the training procedure is completed, the learned sampling pattern is fixed, and new scans can be acquired using the learned trajectory ( Fig. 8 A, B ). Image reconstruction can then be performed using the reconstruction network learned as a part of the training process. Several studies ^, focus on the Cartesian sampling case and model binary sampling masks probabilistically. These methods can be applied to either 2-dimensional (2D) or 3-dimensional (3D) Cartesian sampling to determine the optimal set of phase encodes. As an extension to more general parameterized, or non-Cartesian, k-space trajectories, one can directly optimize for k-space coordinates by restricting the optimization to a set of separable variables, such as horizontal and vertical directions in the 2D plane. The framework presented in Weiss and colleagues additionally incorporates gradient system constraints into the learning process. This enables finding the optimal gradient waveforms in a data-driven manner.