Clinical research

Deep Learning for Lung Cancer Detection in Screening CT Scans: Results of a Large-Scale Public Competition and an Observer Study with 11 Radiologists

Authors: Colin Jacobs, Arnaud A. A. Setio, Ernst T. Scholten, Paul K. Gerke, Haimasree Bhattacharya, Firdaus A. M. Hoesein, Monique Brink, Erik Ranschaert, et al

Purpose

To determine whether deep learning algorithms developed in a public competition could identify lung cancer on low-dose CT scans with a performance similar to radiologists.

Materials and Methods

In this retrospective study, a dataset consisting of 300 patient scans was used for model assessment; 150 patient scans were from the competition set and 150 were from an independent dataset. Both test datasets contained 50 patient scans with cancer and 100 without cancer. The reference standard was set by histopathological examination for cancer positive scans and imaging follow-up for at least 2 years for cancer negative scans. The test datasets were applied to the top three performing algorithms from the Data Science Bowl 2017 public competition (called grt123, Julian de Wit & Daniel Hammack [JWDH] and Aidence). Model outputs were compared with an observer study of 11 radiologists that assessed the same test datasets. Each scan was scored on a continuous scale by both the deep learning algorithms and the radiologists. Performance was measured using multireader multicase receiver operating characteristic analysis.

Results

The area under the receiver operating characteristic curve (AUC) was 0.877 (95% CI: 0.842, 0.910) for grt123, 0.902 (95% CI: 0.871, 0.932) for JWDH, and 0.900 (95% CI: 0.870, 0.928) for Aidence. The average AUC of the radiologists was 0.917 (95% CI: 0.889, 0.945), which was significantly higher than grt123 (P = .02); however, no significant difference between the radiologists and JWDH (P = .29) or Aidence (P = .26) was found.

Conclusion

Deep learning algorithms developed in a public competition for lung cancer detection in low-dose CT scans reached performance close to that of radiologists.

Show summary Read more

Clinical evaluation of a deep-learning-based computer-aided detection system for the detection of pulmonary nodules in a large teaching hospital

Authors: C.O.Martins Jarnalo, P.V.M.Linsen, S.P.Blazís, P.H.M.van der Valk, D.B.M.Dickerscheid

Aim

To evaluate a deep-learning-based computer-aided detection (DL-CAD) software system for pulmonary nodule detection on computed tomography (CT) images and assess its added value in the clinical practice of a large teaching hospital.

Materials and methods

A retrospective analysis was performed of 145 chest CT examinations by comparing the output of the DL-CAD software with a reference standard based on the consensus reading of three radiologists. For every nodule in each scan, the location, composition, and maximum diameter in the axial plane were recorded. The subgroup of chest CT examinations (n = 97) without any nodules was used to determine the negative predictive value at the given clinical sensitivity threshold setting.

Results

The radiologists found 91 nodules and the CAD system 130 nodules of which 80 were true positive. The measured sensitivity was 88% and the mean false-positive rate was 1.04 false positives/scan. The negative predictive value was 95%. For 23 nodules, there was a size discrepancy of which 19 (83%) were measured smaller by the radiologist. The agreement of nodule composition between the CAD results and the reference standard was 95%.

Conclusions

The present study found a sensitivity of 88% and a false-positive rate of 1.04 false positives/scan, which match the vendor specification. Together with the measured negative predictive value of 95% the system performs very well; however, these rates are still not good enough to replace the radiologist, even for the specific task of nodule detection. Furthermore, a surprisingly high rate of overestimation of nodule size was observed, which can lead to too many follow-up examinations.

Show summary Read more

The effect of CT reconstruction settings on the performance of a deep learning based lung nodule CAD system

Authors: Stephan P. Blazis, Dennis B.M. Dickerscheid, Philip V.M. Linsen, Carine O. Martins Jarnalo

Purpose

To study the effect of different reconstruction parameter settings on the performance of a commercially available deep learning based pulmonary nodule CAD system.

Materials and methods

We performed a retrospective analysis of 24 chest CT scans, reconstructed at 16 different reconstruction settings for two different iterative reconstruction algorithms (SAFIRE and ADMIRE) varying in slice thickness, kernel size and iterative reconstruction level strength using a commercially available deep learning pulmonary nodule CAD system. The DL-CAD software was evaluated at 25 different sensitivity threshold settings and nodules detected by the DL-CAD software were matched against a reference standard based on the consensus reading of three radiologists.

Results

A total of 384 CT reconstructions was analysed from 24 patients, resulting in a total of 5786 found nodules. We matched the detected nodules against the reference standard, defined by a team of thoracic radiologists, and showed a gradual drop in recall, and an improvement in precision when the iterative strength levels were increased for a constant kernel size. The optimal DL-CAD threshold setting for use in our clinical workflow was found to be 0.88 with an F2 of 0.73 ± 0.053.

Conclusions

The DL-CAD system behaves differently on IR data than on FBP data, there is a gradual drop in recall, and growth in precision when the iterative strength levels are increased. As a result, caution should be taken when implementing deep learning software in a hospital with multiple CT scanners and different reconstruction protocols. To the best of our knowledge, this is the first study that demonstrates this result from a DL-CAD system on clinical data.

Show summary Read more

Artificial intelligence for analysing chest CT images

Authors: C.O.Martins Jarnalo, P.V.M.Linsen, S.P.Blazís, P.H.M.van der Valk, D.B.M.Dickerscheid

Summary

The technologies described in this briefing are artificial intelligence (AI) technologies for chest CT. They are used for assisting with triaging, reporting, and identifying abnormalities.

The innovative aspects are that the software helps radiologists and radiographers detect abnormalities in chest CT images.

The intended place in therapy would be to support radiologists and radiographers when reviewing chest CT images in secondary care for people who have been referred for chest CT.

The main points from the evidence summarised in this briefing are from 2 retrospective studies. The best quality evidence came from 1 UK study showing that Veye Chest (Aidence) performed similarly to chest radiologists for lung nodule segmentation growth assessment. The studies were limited in quality and no studies were published in full.

Show summary Read more

Using machine learning in diagnostic services A report with recommendations from CQC’s regulatory sandbox

Summary

This report presents the findings from the Care Quality Commission’s (CQC’s) regulatory sandbox pilot. Regulatory sandboxing is a way of working proactively and collaboratively to understand new types of health and social care service, agree what good quality looks like, and develop our approach to regulation. We think this is particularly important for innovative and technology-enabled services, which are developing quickly, and where a response requires collaboration with other national bodies.

This sandbox round focused on the use of machine learning applications for diagnostic purposes in healthcare services. Part of this work involved building a consensus on what is needed to deliver high-quality care in services that use these applications. To do this, we worked with healthcare providers, technology suppliers, people who use services, clinicians, and other stakeholders. We have used the findings of this sandbox to identify and consider where we need to update our current regulatory methods, and what work we need to do to get this right, which will help us to regulate these services better.

Show summary Read more

Book a demo


    * Required fields.
    For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.