1.1 Prominence of Medical Image Perception in Medicine

Medical images form a core portion of all the information a clinician utilizes to render diagnostic, treatment, and management decisions while a patient is under her/his care. The goal of this chapter is to provide a broad picture of the importance of medical image perception from a general healthcare enterprise perspective. Here we treat perception not only in terms of visual perception, though that is currently by far the most prominent method to interpret medical images, but also computational perception, where images are “read” and “understood” by computational algorithms.

Medical imaging has been primarily ascribed to the subspecialty of radiology, with about two billion radiological imaging exams performed worldwide every year. The images include a variety of exam types such as single-projection X-ray projections used in musculoskeletal, chest, and mammography applications; dynamic X-ray exams such as fluoroscopy, three-dimensional computed tomography (CT) and magnetic resonance imaging (MRI) exams; nuclear medicine emission images, and ultrasound. With the advent of digital imaging, multidetector CT, and protocol diversification in MRI, the number of radiology examinations has been increasing. The range of image types is also expanding rapidly with newer modalities of tomosynthesis (Dobbins et al., 2017; Gilbert et al., 2016), hyperspectral (Guolan and Fei, 2014) and molecular imaging (Fei and Schuster, 2017; Liang et al., 2017) all being used for numerous applications from identifying lesion margins during surgical removal to identifying cancer cells in the blood.

While imaging is the central technology behind the subspecialty of radiology, imaging today is playing an expanding and changing role beyond radiology and embraces other subspecialties including cardiology, radiation oncology, pathology, and ophthalmology, to name a few. Pathology used to be limited to the glass slide specimen “images” rendered by the microscope for the pathologist to view. With the advent of digital slide scanners in recent years and the preponderance of evidence supporting its feasibility, acceptance, equivalence to light microscopy, and cost efficiency (Bashshur et al., 2017), virtual slides are becoming more prevalent not only in telepathology applications but in everyday reading (Kaplan and Rao, 2016; Weinstein et al., 2001). Clinical use is likely to accelerate with the recent approval by the US Food and Drug Administration (FDA) for marketing of whole slide imaging for review and interpretation of digital surgical pathology slides prepared from biopsied tissue (FDA, 2017). In many medical school and pathology residency programs (Christensen et al., 2017; Wilbur, 2016), students are no longer required to purchase a microscope and box of glass specimen slides. They are simply purchasing a CD with directories of virtual slides to view as soft-copy images and learn from.

Ophthalmology has relied on images for years (mainly as 35-mm film prints or slides) for evaluating conditions such as diabetic retinopathy. With the advent of digital images and high-performance color displays, screening raters are increasingly using soft-copy images (Tan et al., 2017). Although most of the original applications were in diabetic retinopathy detection, teleophthalmology has expanded to include glaucoma, emergency eye care, and numerous other retinal diseases (Sim et al., 2016). Telemedicine in general has opened up an entirely new area in which medical images are being acquired, transferred, and stored to diagnose and treat patients (Krupinski et al., 2002). Specialties such as teledermatology, teleophthalmology, telewound/burn care, and telepodiatry are all using images on a regular basis for store-and-forward telemedicine applications. Real-time applications such as telepsychiatry, teleneurology, and telerheumatology similarly rely on video images for diagnostic and treatment decisions. In every case, issues that digital radiology has addressed for years are being addressed in these newer image-based clinical scenarios. For example, the development of standards for image acquisition and presentation (American Telemedicine Association Ocular Telehealth Special Interest Group, 2004; Badano et al., 2015; McKoy et al., 2016; Pantanowitz et al., 2014; Theurer et al., 2017) and the impact of image quality on diagnostic decisions are key research and clinical implementation topics.

There are a number of ways to examine the pervasiveness of medical imaging. One approach used a few years ago is to examine the amount of money spent each year on healthcare and then portion out the amount devoted to medical imaging (Beam et al., 2006). Relying on 2004 data from the Centers for Medicare and Medicaid Services, approximately 16% of the gross domestic product (GDP) or $1.6T is allotted to national healthcare expenditures (www.cms.hhs.gov/home/rsds.asp). Medicare expenditures represent 17% of national healthcare expenditures, of which Part B (43%) accounts for the nonfacility or physician-related expenditures. Approximately 8% of Part B (or nearly $10B) constitutes physician-based imaging procedures. Imaging also accounts for over 40% of all hospital procedures reported in the discharge report, according to the Agency for Healthcare Research and Quality (www.ahrq.gov/data/hcup/). If, based on Medicaid Part B spending, one conservatively assumes that imaging procedures comprise only 8% of non-Medicaid Part B health spending, medical imaging in the USA is estimated to amount to $56B ($10B/17%/43%) or 0.5% of GDP. More recent studies (America’s Health Insurance Plans (AHIP), 2008; Glabman, 2005; Medicare Payment Advisory Commission (MEDPAC), 2014) place the cost of imaging in the USA at over $100B annually, despite recent trends toward stabilization of utilization (Dodoo et al., 2013; Lee et al., 2013).

Imaging technologies are extremely varied. Medical images can be grayscale or color, high-resolution and low-resolution, two-dimensional or multidimensional, hard-copy or soft-copy, uncompressed or compressed (lossy or lossless), acquired with everything from sophisticated dedicated imaging devices to off-the-shelf digital cameras. With the pervasiveness of imaging in modern medicine, there has been significant attention and interest in the technological aspects of imaging operations ranging from hardware features to software functionalities. What is less appreciated is the perceptual act underlying the interpretation of these images (Krupinski, 2016; Manning et al., 2005; Wolfe, 2016). In order to impact patient care, an image must be perceived and interpreted (i.e., understood in the context of patient care) (Figure 1.1). If one assumes each of the one billion imaging examinations performed worldwide annually involves an average of four individual images per exam, one could compute, that on the average, 120 medical image perception events take place every second! This astounding frequency speaks further of the pervasiveness of medical image perception in healthcare enterprise.

Figure 1.1 As a fundamentally visual discipline, medical imaging requires psychophysical interpretation of the images to draw “meaning” from the viewing information and understand their clinical relevance.

The need for interpretation of medical images comes from the fact that medical images are not self-explanatory. In popular culture, a picture is “worth a thousand words,” reflecting the power and utility of images. Ironically, however, the interpretation of a medical image involves summarizing a multidimensional image into a few words, which is not necessarily an easy task (Bracamonte et al., 2017; Ware et al., 2017). That is necessary because medical images, like other complex and sometimes ambiguous images (Figure 1.2), by themselves do not deliver the certainty that they promise. This lack of certainty, which necessitates interpretation, stems from the nature of medical imaging. Visual interpretation is impacted by psychophysical processes involved, while computational interpretation is likewise impacted by image-processing methods. Medical images involve variety, where anatomical structures can camouflage a feature of clinical interest. That feature can have very low prevalence (in the case of screening), which impacts the psychology and processes of interpretation (Fanshawe et al., 2016; Littlefair et al., 2016). Added to those complexities, there are notable variations from case to case and a multiplicity of compounding abnormalities and related factors that the interpreter or the computational operator needs to accommodate for.

Figure 1.2 Detecting a subtle abnormality is somewhat similar in difficulty to identifying the dog in a popular visual illusion.

There are clearly a significant number of images interpreted in a variety of clinical specialties. As such, diagnostic accuracy cannot be defined independently of the interpretation, and any limitations or suboptimality in terms of how images are used can have a measurable impact on the diagnostic and therapeutic clinical decisions that they enable. Given a one-to-one link between an image and its interpretation, imaging technology alone can offer little in terms of patient care if the image is misinterpreted. The complexities of image interpretation can lead to interpretation errors. Clinicians do make mistakes in the interpretation of image data (Berlin, 2005, 2007; Waite et al., 2017a, 2017b). Estimates in radiology alone suggest that in some areas there may be up to a 30% miss rate (omission errors) and an equally high false-positive rate. Errors can also occur in the recognition of an abnormality (e.g., whether a lesion is benign or malignant). Such errors can have significant impact on patient care due to delays or misdiagnoses. Other sources of error include satisfaction of search, cognitive bias, prevalence effects, presence of and information in a clinical history, fatigue, workload, level of training or experience, distractions and interruptions, and even ergonomic considerations (Waite et al., 2017a, 2017b). What is less well appreciated is the prominent contribution of the inherent limitations of human perception to these errors. Image perception is the most prominent yet least appreciated source of error in diagnostic imaging. The prominence of imaging reading errors in malpractice litigation is an example of this ignorance.

The likelihood of error in the interpretation of those images emphasizes the need to understand how the clinician interacts with the information in an image during the interpretation process. Such an understanding enables us to determine how we can further improve decision making. That brings us to the science of medical image perception. Error is one reason to study medical image perception.

1.2 The Science of Medical Image Visual Perception

First and foremost, it is important to understand the nature and causes of interpretation error. For that objective, one needs to distinguish between errors that are visual in nature (estimated to amount to about 55% of the errors) because the clinician does an incomplete search of the image data (Giger et al., 1988; Waite et al., 2017a, 2017b) and those of a cognitive nature (45%), where an abnormality is recognized but the clinician makes a decision-making error in calling the case negative (Kundel et al., 1978). Visual errors are further subdivided into error where the clinician fails to look at the territory of the lesion (30%) (Kundel, 1975; Kundel et al., 1978) and those when he/she does not fixate on the territory for an adequate amount of time to extract the relevant lesion features (25%) (Carmody et al., 1980).

Contributing to interpretation errors are a host of psychophysical processes. Camouflaging of the abnormality by normal body features (so-called anatomical noise) is one of the main contributors to interpretation error. Masking of subtle lesions by normal anatomical structure is estimated to affect lesion detection threshold by an order of magnitude (Samei et al., 1997). Visual search is another important process, necessitated by the limited angular extent of the high-fidelity foveal vision of the human eye (Van der Gijp et al., 2017; Wolfe et al., 2016). Preceded by a global impression or gist, visual search involves moving the eye around the image scene to closely examine the image details (Nodine and Kundel, 1987). Studies on visual search have highlighted the prominent role of peripheral vision during the interpretation where there is an interplay between foveal and peripheral vision as the observer scans the scene (Kundel, 1975). As a result there are characteristic dwell times associated with correct and incorrect decisions that are influenced by the task and idiosyncratic observer search patterns (Kundel et al., 1989). Satisfaction of search is yet another contributing factor to errors where, once an abnormal pattern is recognized, it takes additional diligence on the part of the clinician to look for other possible abnormalities within an image (Berbaum et al., 1989; Smith, 1967; Tuddenham, 1962, 1963). Studies have explored the impact of expertise and prior knowledge in that behavior, as well as the use of tools such as systematic search patterns and checklists to alleviate (Berbaum et al., 2016; Kok, 2016).

Image quality is yet another topic of interest. While intuitively recognized, image quality has been more elusive to characterize in such a way that it would directly relate to diagnostic accuracy (or its converse, diagnostic error). In that regard, it is important to understand how best to assess image quality and its impact on perception in order to optimize it and minimize error (Krupinski and Jiang, 2008). Studies have focused on the impact of image acquisition, imaging hardware, image processing, image display, and reading environment on image quality and diagnostic accuracy.

Ergonomic aspects of interpreting medical images also play a role in the perception process. There is a need to understand the impact of ergonomic and presentation factors to minimize error (Krupinski and Kallergi, 2007; Krupinski et al., 2017; Ratwani et al., 2016). Topics include determining the causes of fatigue and how that can be minimized, the contribution of fatigue to error (Krupinski et al., 2017; Rohatgi et al., 2015; Waite et al., 2017a, 2017b), the environmental distractions (Balint et al., 2014; Williams and Drew, 2017), the impact of viewing interface, especially with soft-copy images, and the impact of the color tint of the image.

Though we hope and aim for consistent and correct clinical decisions on every case, that aim is hard to achieve. The likelihood of two clinicians rendering two different interpretation of an image is unsettlingly high. The expertise of the clinician plays an important role in that respect (Van der Gijp et al., 2017). Medical expertise is the ability to efficiently use contextual medical knowledge toward accurate and consistent diagnosis. Medical imaging expertise further involves perceptual and cognitive analysis of image features and manifests itself in a rich structured knowledge of normalcy and “perturbations” from the normal, an efficient hypothesis-driven search strategy, and an ability to generalize visual findings to idealized patterns. Achieving such expertise requires talent further honed by motivated effortful study, preferably supervised, and dedicated work, where accuracy is roughly proportional to the logarithm of number of cases read annually (Nodine and Mello-Thoms, 2000). Topics of interest in that line of investigation include the impact of clinician’s experience, age, and visual acuity on accuracy, toward better training and utilization of medical imaging clinicians.

Considering the impact of image perception on diagnostic accuracy, it is often necessary to test various imaging technologies and methods in terms of the associated impact on image perception. Such studies require the use of experienced clinicians, which is an expensive undertaking. Thus, there is a great need for accurate computational models/programs that could model visual perception and predict human performance. A host of such perception models have been developed over years, including the ideal human observer model, nonprewhitening models, channelized models, and visual discrimination models (Abbey and Bochud, 2000). These models naturally require a reasonably accurate understanding of the image interpretation process. As our knowledge in that regard is limited, so is the accuracy of these models. As such, their use often requires certain assumptions, verifications of their accuracy and relevance in pilot experimentations, and certain calibrations (e.g., adding internal noise to make the model predictions fit the human results). Nonetheless, these models have demonstrated valuable, though limited, utility in many applications, and their advancement continues to shed light on the image interpretation process.

Surprisingly, mathematical models are not the only ones being used to try to understand how humans visually process medical images. Key insights into this human behavioral tasks were reported using pigeons (Columba livia), which share many visual system properties with humans. The birds had a remarkable ability to distinguish benign from malignant human breast histopathology and, even more importantly, were able to generalize what they had learned when confronted with novel image sets. Their accuracy, like that of humans, was affected by the presence or absence of color as well as by degrees of image compression, but could be ameliorated with further training. In radiology, the birds were quite capable of detecting cancer-relevant microcalcifications on mammogram images. However, when given the more difficult task with mammographic masses the pigeons proved to be capable only of image memorization and were unable to successfully generalize when shown novel examples. The birds’ successes and difficulties suggested that pigeons are well suited to help understand human medical image perception (Levenson et al., 2015).

By and large, image interpretation is currently a human task. However, increasingly, artificial intelligence tools are being used to aid the human in the interpretation process or all together replace the human (Brink et al., 2017; Jha and Topol, 2016). The most common technology currently used is computer-aided diagnosis (CAD), computer algorithms that examine the image content for certain abnormal features of clinical interest and then prompt the clinician for a closer examination of those features (Al Mohammad et al., 2017; Doi, 2007; Pande et al., 2016). CAD is becoming an important tool for interpreting medical images considering the exponential growth of imaging and shortage of specialized expertise. There is currently a need to understand the impact of CAD on accuracy. Issues in that regard include how best to integrate the human and the machine in such a way that the strength of both can be fully utilized toward improved diagnosis. An experienced clinician might ignore the CAD prompts or be distracted by them if the system indicates too many false positives. However, an inexperienced clinician might overly depend on CAD, initiating unnecessary follow-up procedures or dismissing an abnormality that might not have been picked up by the CAD algorithm. Such patterns might also change over time as a clinician gets used to a system, and such “getting used to” might not necessarily lead to improved diagnosis or efficiency. Thus, there is a need to understand the impact of CAD on the clinician’s psychology, expertise, efficiency, and specialization paradigms.

Fundamental to most topics noted above is a need to measure diagnostic accuracy (Metz, 2006; Obuchowski, 2005; Wagner et al., 2007). There are a number of simple measures of performance such as fraction correct, sensitivity, or specificity. However, such simple measures do not adequately reflect accuracy as they can be dependent on disease prevalence or the criterion level applied by the clinician (e.g., a clinician who calls all cases abnormal will have a perfect sensitivity but poor specificity, and vice versa). Seeking an overall performance measure independent of disease prevalence and criterion, receiver operating characteristic (ROC) analysis has served as the current gold standard for measuring diagnostic accuracy. However, ROC analysis has a number of limitations, including being primarily limited to single tasks, nonbinary confidence ratings, and location-independent decisions. In recent years, a number of variants and advancements of the ROC methodology have been developed, a welcome expansion which has shown continued advancement.

1.3 Why a Clinician Should Care About Medical Image Perception

Medical image perception is a mature science which continues to be advanced by expert scientists. In this age of overspecialization in which specialized “territories” are left to the experts, one may ask why a clinician involved with medical images needs to care about medical image perception. It is needless to say that no one expects a clinician to also be a medical perception scientist. However, knowledge of perception issues and concerns can provide vital advantages in the work of a clinician involved with medical images. Those advantages can be grouped into five categories.

1. 
   

   1. Patient care: Understanding perceptual issues could help a clinician to improve his/her performance. Knowledge of key perceptual factors such as satisfaction of search, the relevance of prolonged dwell time, search strategies, and psychological impacts of decision aids (such as CAD) can directly impact the way he/she interprets medical images. It further enforces a greater care about the way the images are created, a greater appreciation for image quality and its relevance in terms of accuracy and efficiency, an appreciation for proper ergonomic design of working environment and fatigue factors, and higher confidence in the use of new display technologies.

   
   
2. 
   

   2. Science: Being better informed about key perceptual factors enables proper design of projects involving medical images, ability to better answer perceptual questions that inevitably arise in the review of imaging-related papers and grant applications, and proficiency to review such papers and grants.

   
   
3. 
   

   3. Teaching and learning: Knowledge of perceptual factors can help a clinician be a better teacher in communicating his/her expertise to trainees. Similarly he/she would be able to better hone in his/her perceptual skills.

   
   
4. 
   

   4. Consumer: Understanding the importance of perceptual factors enables a clinician to be a better shopper of medical image-related products and services. For example, he/she will be more mindful of image quality performance aspects of acquisition and display devices, and the importance of graphical user interface of the picture archiving and communications system (PACS) workstations.

   
   
5. 
   

   5. Profession: Awareness of image perception issues enables a clinician to better educate patients, other medical professionals, and the public about the statistical nature of medical image interpretation, and to play a more effective role in related malpractice litigations.

1.4 About This Book

As outlined above, medical image perception is a frequent clinical task and a notable component of modern medicine. With perceptual error as one of the major sources of medical decision errors, our knowledge of perceptual issues gives us resources to better control and minimize these errors and to educate future medical imaging clinicians and scientists. This book aims to provide a comprehensive reflection of medical perception issues and concepts within one single volume. Chapters in this text deal with a variety of perceptual issues in great detail.

In this second edition, we have retained the core chapters that summarize the history of medical image perception, as well as those that cover foundational methodologies for image perception research. Most chapters have been updated by the original authors to reflect advances in their specific topic areas. Some of the more outdated chapters have been replaced by new ones that better reflect either state-of-the-art technologies being used today in clinical settings, and/or newer assessment methods, tools, and techniques. A number of new chapters have also been added that address new topics in medical imaging that have either developed or matured since the first edition, thus warranting inclusion.

The first part of the book retains chapters by four prominent scientists (Harold Kundel, MD, Calvin F. Nodine, PhD, Arthur Burgess, PhD, Robert Wagner, PhD), reflecting on historical developments of the field and its theoretical foundations. Each of these authors is considered today a “father” of medical image perception, from different but related perspectives. Their pioneering research has been paramount in shaping the field of medical image perception as we know it today. A new chapter has been added discussing the overall context of medical image perception.

The second part of the book includes chapters discussing the science of medical image perception. Main topics include visual and cognitive factors, satisfaction of search, and the role of expertise. A new chapter on the role of subsecond and peripheral vision/perception in image interpretation has been added.

Part III focuses on perception metrology with chapters focused on the logistical aspects of designing perception experiments, ROC methodology, and its variants. A new chapter has been added on the impact of memory effects for images in the context of running observer studies and another on three- and four-dimensional models. The part includes discussion of perceptual observer models and their implementation as well as an assessment of the overall value and limitations of such models.

A new part (IV) has been added describing perception in the context of multisource imaging and data and two international programs designed to assess clinical performance of mammographers over time and in comparison to their peers for overall quality assessment.

Part V focuses on computational perception and CAD issues with topics ranging from the design of CAD studies and perceptual impact of CAD to perceptual factors associated with the use of CAD in interpreting chest, breast, and volumetric images. A new chapter has been added on the evaluation of CAD, and another one on the overall process of images as a source of quantitative information.

The final part (VI) on applied perception offers chapters dealing with specific optimization and use considerations from a perceptual standpoint. New and revised chapters offer topics on display optimization, reading environment and ergonomic design of workplaces for radiologists, image perception in pathology, and perceptual basis for developing human search-based training and computer-based training methods. The book ends with a chapter summarizing image perception from the perspective of a practicing radiologist and a final chapter outlining future possible directions for medical image perception science.

We hope readers benefit from this new edition, to learn from its content and to find inspiration from its diverse topics that still need more comprehension, innovation, and application to advance the value and utility of medical images in medicine. We eagerly anticipate that the insights and methods described in these pages can lead to a positive impact on patient care and human health, which shall remain the main objective of health science and healthcare.