Development of a tool to aid the radiologic technologist using augmented reality and computer vision
This technical innovation describes the development of a novel device to aid technologists in reducing exposure variation and repeat imaging in computed and digital radiography. The device consists of a color video and depth camera in combination with proprietary software and user interface. A monitor in the x-ray control room displays the position of the patient in real time with respect to automatic exposure control chambers and image receptor area. The thickness of the body part of interest is automatically displayed along with a motion indicator for the examined body part. The aim is to provide an automatic measurement of patient thickness to set the x-ray technique and to assist the technologist in detecting errors in positioning and motion before the patient is exposed. The device has the potential to reduce the incidence of repeat imaging by addressing problems technologists encounter daily during the acquisition of radiographs.
Augmented reality Children Computed radiography Computer vision Digital radiography Radiation dose Technical innovation
Digital radiography (DR) has led to shorter image acquisition times compared to computed radiography (CR) and screen-film radiography as images are transferred in a matter of seconds from the image receptor to a display monitor. Despite these advances, it has been reported that repeat rates have increased from 5% for CR  to as high as 25% for DR . Technologists have stated that the reasons for this increase are the ability to quickly review and repeat images with DR and that less time is spent positioning the patient compared to CR . Repeat imaging leads to anxiety for both patient and parents and has additional negative consequences summarized in the American Association of Physicists in Medicine Report 151 : “Repeated and rejected images represent both unnecessary radiation exposure to patients and inefficiency in the imaging operation owing to wasted time and resources.”
The most common causes of repeat imaging include errors in positioning, patient motion and inappropriate exposure parameters , each of which is described in detail below. An antiquated technologist workflow is at least partly to blame and has gone largely unchanged for more than 50 years. The current workflow is as follows: A technologist accompanies the patient to the x-ray room; positions the patient against a detector, table or wall bucky surface containing an image receptor; sets the x-ray field using a collimator light; walks to a shielded control area, sets the x-ray technique (e.g., kV, mAs) and exposes the patient. The image is reviewed by the technologist and/or radiologist and repeated if it is unacceptable for diagnosis. This process is fraught with potential for errors.
First, with respect to positioning, even if the technologist initially positions the patient correctly, there is ample time for a child or sick patient to change position while the technologist returns to the control area. Once in the control area, the collimator light has usually gone out and the technologist may have a restricted view of the patient from a distance and/or an angle due to room design. While he/she may be able to turn on the collimator light remotely for short periods of time (~10 s), it is difficult to view the light on clothing through a window at a distance and angle.
Second, in the current workflow it is difficult to assess motion of a specific body part from the control workstation. The technologist may also be distracted by extraneous motion of the patient even when the body part of interest is still.
Lastly, while CR/DR is more tolerant to underexposure and overexposure than screen-film radiography due to a wide dynamic range, incorrect technique can lead to repeated images. If manual techniques are used, the body part and its thickness should be used to set exposure parameters , as advocated by the Image Gently® campaign. Despite this recommendation, many hospitals continue to use age-based technique charts due to practical limitations of physical calipers. Calipers are intimidating to children, only measure a single point at any time and require physical contact with subsequent cleaning. In the experience of the authors, calipers have been largely discarded in the transition from screen-film radiography to CR/DR. Unfortunately, the alternative, patient age, is a terrible surrogate for patient size since a large four year old can have the same thickness as a thin eighteen year old . An automated, easy, accurate and patient-friendly method of measuring thickness is needed.
A common alternative to manual technique charts is the use of automatic exposure control to terminate the exposure when a predetermined amount of radiation has reached the image receptor. When using automatic exposure control, it is critical that the body part to be imaged covers the active chamber(s). Manufacturers provide plastic sheets that can be inserted at the collimator box to identify the automatic exposure control chambers as a shadow on the collimated light field. The appropriate sheet needs to be selected based on the source-to-image receptor distance and then manually inserted. The size and position will be incorrect if the source-to-image receptor distance is changed. The time it takes to select and insert the appropriate plastic sheet interferes with the workflow and limits its use.
The best opportunities to minimize unnecessary radiation and to avoid repeat imaging occurs immediately before the patient is exposed. The necessary steps include 1) ensuring proper positioning and collimation to the area of interest; 2) setting a technique based on automatic exposure control or a manual technique based on body part and thickness, and 3) detecting patient motion. The motivation for this work was to develop a practical tool for the technologist to automatically measure body part thickness and to detect errors in positioning and motion before an x-ray is acquired.
Description of device
The device consists of four generic components: 1) a 3-D depth-sensing camera with high-resolution color video and infrared depth sensor, 2) a processing unit where all real-time computations are performed, 3) an application programming interface to communicate with the processing unit and 4) the user interface displayed on a control room monitor. The prototype device uses a Microsoft Kinect™ V2 (Microsoft Corporation, Redmond, WA) consisting of a 1,080p color video and time-of-flight infrared depth sensor. An Intel NUC (Intel, Santa Clara, CA) personal computer with Core i5 processor and 32GB random access memory (RAM) was used for the processing unit. The user interface was displayed on an Asus 19.5″ HD touchscreen monitor (Model VT207N; Asus, Taipei, Taiwan). A schematic of the system architecture and components is provided in Fig. 1. The 3-D depth camera was mounted to the tube assembly housing of a Philips Digital Diagnost (Philips Healthcare, Andover, MA) with a custom 3-D printed mount.
A proprietary processing software was developed in C++ using the Qt, OpenCV and FFMPEG libraries to analyze, in real time, the color and depth video streams from the 3-D depth-sensing camera. At start-up, the device requires a calibration procedure, which runs in less than 2 s and is fully automatic. During this calibration, a computer vision algorithm precisely localizes the imaging equipment via pattern matching of visual markers of the automatic exposure control and image receptor markers on the bucky surface. The software constructs an internal 3-D representation of the room so that the source-to-image receptor distance can be assessed at any point and time while correcting for camera tilt. Once calibrated, the software calculates patient thickness and tracks in real time (15 fps) the patient positioning and patient motion. Motion estimation is achieved by assessing the position of the tracked body part from a skeleton-tracking algorithm or by using an optical flow algorithm that allows motion estimation from the color camera only.
A screenshot of the body part selection menu is shown in Fig. 2. Prior to the exam, the body part of interest (e.g., chest) is manually selected from an anatomical model, selected by a button in the top left. In future versions, this information will be automatically extracted from the digital imaging and communications in medicine (DICOM) modality worklist. During the exam, the user interface displays real-time video of the patient along the central ray axis with overlays for position of automatic exposure control chambers, image receptor, body part thickness, patient skeleton tracking and a motion tracer (Fig. 3). The overlays consist of computer-generated images from the calibration process, superimposed on a real-time video of the patient. This composite image is typically referred to as augmented reality. An inset shows the depth map of the patient created by calculating the distance between the patient surface and bucky surface with perspective correction (i.e. correcting for camera angle). The user interface automatically displays the thickness measurement based on the preset location allowing the technologist to set the thickness-based x-ray technique. The software can be configured to display thickness at any point. For example, the body part position determined by the skeleton-tracking algorithm (i.e. chest location in Fig. 3) or an alternative such as the central ray, thickness over a specific automatic exposure control chamber, the thickest point, or at the location chosen by the technologist by touch screen or by using the mouse cursor (as shown on Fig. 3 inset).
Patient motion is displayed as a real-time trace of the motion amplitude of the examined body part. Due to very short exposure times, intra-exposure motion is difficult to control. However, a real-time (15 fps) motion trace helps to confirm the patient is not moving prior to exposure. The motion trace can also be reviewed retrospectively to confirm the patient did not move during the exposure. In Fig. 4, the motion tracer shows the peak of patient motion as the patient changes position.
The software is designed to alert the technologist when a potential problem arises, such as when the body part is not centered appropriately on the detector or the patient has moved off the automatic exposure control chambers (Fig. 4). The alert threshold is customizable by the user. For example, an alert can be generated if a percentage (e.g., 20%) of the automated exposure control cell is uncovered. Alerts can be visual, auditory or both.
As part of our proof-of-concept work, we compared the accuracy of the depth measured by the depth-sensing camera and a laser measurement tool with ± 1 mm precision (Fluke 419D; Fluke Inc., Everett, WA). With the Kinect™ V2 mounted on the x-ray tube housing assembly, a flat barrier was placed at distances ranging from 10 cm to 130 cm from the detector. The correlation factor between both methods was high (R2=0.99989, Fig. 5). It is expected that a depth precision of 1 cm is adequate for patient thickness measurements in a clinical setting and as such, the performance of the Kinect™ V2 is sufficient.
CR/DR software is typically preloaded with anatomically programmed techniques based on patient age or arbitrary size categories (e.g., child, small adult, medium adult, etc.). It is difficult to optimize techniques over the wide range of patient sizes encountered in pediatric hospitals within the constraints of these default size categories. A radiologist, medical physicist and technologist should develop the technique guide to serve as routine protocols. The technologist is expected to follow the technique guide but should be given latitude to diverge from the protocol based on experience and training as required by the clinical situation. Newer software on integrated systems (i.e. communication between detector and generator) have introduced inhibit controls aimed to prevent x-ray exposure when a deviation from the standard protocol is detected. For example, if the preprogrammed technique for an abdomen radiograph requires a grid, the technologist will be prevented from exposing the patient until a grid is inserted or they override the program. While designed to prevent inappropriate exposure conditions, these systems can potentially discount the expertise of the technologist and lead to frustration when the workflow is interrupted unexpectedly.
By contrast, we aim to empower the technologist by providing critical, real-time information. The basis of this approach is that the technologist’s interaction with the patient, in particular with children, is key to a successful exam. The information displayed is highly customizable so overlays can be added or removed and the location of measuring body-part thickness is flexible (e.g., automatically recognized body part, center of detector, thickest point or free cursor). The user interface we have developed is intended to enhance the workflow by alerting the technologist to issues that may result in repeat imaging before the patient is exposed. In our prototype, the user interface is displayed on a touchscreen in the control area. In a commercial implementation, the user interface can be integrated into the existing workstation monitor since a large area is typically empty prior to exposure.
There are secondary applications for the device. For example, the programming provides data (e.g., thickness) to enhance radiation dose reporting in CR/DR by improving the accuracy of radiation dose information included in the Radiation Dose Structured Report and DICOM metadata. Entrance dose (mGy) can be provided for individual body parts and at the patient skin surface as opposed to a reference point dose as provided today, i.e. a fixed distance from either source or detector. Another application is to provide a short clip of the patient during x-ray exposure, with equipment overlay information to the quality assurance team. Such a video could be used to troubleshoot image quality issues. Face recognition could also be used to confirm patient identity.
There are limitations to the current prototype. The current software has been developed only for the wall bucky. Future development will need to include the table bucky, tabletop and portable units. The prototype uses the Kinect™ V2, but there may be other hardware that will be better suited for commercialization in the future.
This device has only started to be tested in a clinical environment. It will be important to show that the measurements are at least as accurate as the now-discarded calipers. With adoption of thickness-based technique charts, it will be important to measure changes in exposure variation using this device, quantified by entrance exposure and exposure index, as well as reduction in repeat rates.
A novel device that has the potential to reduce repeat radiographs and exposure variation has been described. It uses augmented reality and computer vision to measure thickness automatically while confirming patient positioning and tracks for motion in real time. This device empowers technologists with the information they need to take the best radiographs.
This work was supported by the Society for Pediatric Radiology Research and Education Foundation Pilot Award and the Washington University Bear Cub Grant. Intellectual property is covered by claims of U.S. pending patent application identified as serial number 15/100,022.
Compliance with ethical standards
Conflicts of interest