The next least complex methodology, which was used in early rheumatoid arthritis studies in particular [7], is to have two single independent readers read 100 % of all the images, without knowing the results of the other reader (Fig. 5.2). Where there is a discrepancy or discordance, the average of the two reads would be used as the outcome. This averaging smoothes out the variation and provides a single result, but does not address the issue of why there may have been a variation in the read. Therefore, there are two issues from a GCP viewpoint: the first includes that while two pairs of eyes have reviewed the images, where there is discordance, it is unknown whether this is a case of incorrect data entry, misinterpretation, or true disagreement. Secondly, averaging these results does not solve the first issue and from an audit trail perspective does not provide a complete data trail back to one image; there is no such thing as an average “read” by a radiologist.

The two-reader paradigm with a third reader to adjudicate the discordances, is arguably the most common scenario (Fig. 5.3). When conducted in its simplest form, there are just two primary readers who both read 100 % of all the images and a third reader as an adjudicator. Within this paradigm there are two variations: the so-called forced adjudication and the open adjudication. In the forced adjudication scenario, the two agree with a set of results in its entirety, from one of the two primary readers. In this manner, all images are reviewed by two pairs of eyes with the same results, and it precludes the potential for a third set of discordant results from occurring.

There are some subtleties to the adjudication depending on therapeutic area; in oncology the adjudicating reader is normally asked to choose one reader over the other at the patient level. In osteoporosis, for example, the adjudicator should be able to choose at the vertebral level between readers.

The alternative methodology to the previous paradigm is to allow the third reader not to truly adjudicate but to independently evaluate the images and provide a third set of results (Fig. 5.3). This now provides a further set of data from the interpretation without truly fulfilling the GCP requirements for two pairs of eyes to review the same data. Three people have reviewed the data, but with three different sets of results, there is no decision regarding which one should be considered “truth.” While the third reader’s results are often used, there is no logical reason why the first or second reader’s interpretations should be used. The other variation is to average the results from the closest two readers. Oncology studies have generally favored the use of forced adjudication. Studies in rheumatoid arthritis have tended to use the open style of adjudication.

In the field of rheumatoid arthritis, there is yet another variation on this theme that is being evaluated: the images that are identified as needing to be adjudicated are returned to the readers in a blinded manner, and they reread as if they were reading them for the first time. Only after this second double evaluation are they sent to an adjudicator if they still pass the adjudication threshold (Fig. 5.4).

Some organizations that provide a central review process have developed a methodology where there are several readers who are available for reading clinical trials. Each day they read the images from any of the trials that require reading. In this situation, the “pair of readers” can be any two radiologists who are available to read the data that day. In this situation there is no consistency to the pairing (Fig. 5.5). The adjudication that is used can be either of the two methods described but with any available radiologist, and again there is no consistency with the adjudicator. If a forced adjudication process is used, this methodology does provide a process that meets GCP requirements. However, the downside is that there is a higher degree of variability of the readers, more closely meeting the variability of the so-called local readers. One of the many reasons that central reads were developed was to reduce the variability of the reads, and this only increases the variability. A second factor that plays into this paradigm is that the readers all tend to be from the same institution. This provides more consistency but also bias, since they all tend to read the same way. Originally, the read designs were developed to have readers from different institutions, so that the inter-institution variability would be picked up, making the design more robust with the third forced adjudicator.

As pharmaceutical studies get larger, then it is impossible for two readers and one adjudicator to evaluate all the images in a timely manner. A number of different scenarios have been developed to accommodate this issue, from pairs of pairs to a more elaborate process recently described, with four readers. If we take a scenario of four readers being used for a study, there are six potential parings. If the images are sent in a randomized manner to two readers (pair), any adjudication that is required can be carried out by one of the other two radiologists not from that pair (Fig. 5.6). As a study progresses and a statistically significant number of images are read, the rate of concordance and the rate requiring adjudication between the readers can be evaluated. It is therefore possible to determine if one of the readers is evaluating the images differently by the higher adjudication rate observed with his/her pairings.

One can also assess the readers by determining the percentage of reads proceeding to adjudication for which the final determination is consistent with one of the original readers. If one assumes equal performance among all readers, the variation in adjudication will be 50 %, i.e., equal favoring to all readers in the pair. If this variation is significantly different from this value, it is suggestive of a reader that is calibrated differently to the other readers. If this process is performed correctly, high rates of adjudication or high rates of discordance with the final adjudicated result could be used to retrain, “recalibrate,” or replace readers who are reading the criteria differently to the general consensus at predetermined thresholds.

To be more specific, if we take the example of readers A, B, C, and D, there are 6 pairings: AB, AC, AD, BC, BD, and CD. If we have say 540 sets of patient images that have been read, each pair will have handled 90 sets each. (This is utilizing the equal shared methodology). Let’s assume that the adjudication rate is around the 10 % rate. We can then anticipate that each pair will have nine subjects’ images that will be read by one of the other readers. If the generally accepted maximum of 20 % is used, 18 pts per pair could be anticipated to be adjudicated. However, if pairing AB, BC, and BD have a higher adjudication rate, then it might be anticipated that reader B is not reading to the same standards as the other readers. A review of the adjudication rate would confirm this: if more of the adjudication results went to readers A, C, and D in these pairings, then it would easily highlight that reader B requires retraining or recalibration of their read methodology.