By VASANTH VENUGOPAL MD and VIDUR MAHAJAN MBBS, MBA
What can Artificial
Intelligence (AI) do?
simply put, do two things – one, it can do what humans can do. These are tasks
like looking at CCTV cameras, detecting faces of people, or in this case, read
CT scans and identify ‘findings’ of pneumonia that radiologists can otherwise
also find – just that this happens automatically and fast. Two, AI can do
things that humans can’t do – like telling you the exact time it would take you
to go from point A to point B (i.e. Google maps), or like in this case,
diagnose COVID-19 pneumonia on a CT scan.
on CT scans?
an infection of the lungs, is a killer disease. According to WHO statistics from
2015, Community Acquired Pneumonia (CAP) is the deadliest communicable disease
and third leading cause of mortality worldwide leading to 3.2 million deaths
be classified in many ways, including the type of infectious agent (etiology),
source of infection and pattern of lung involvement. From an etiological classification
perspective, the most common causative agents of pneumonia are bacteria
(typical like Pneumococcus, H.Influenza and atypical like Legionella,
Mycoplasma), viral (Influenza, Respiratory Syncytial Virus, Parainfluenza, and
adenoviruses) and fungi (Histoplasma & Pneumocystis Carinii).
radiologists find the infectious agent for pneumonia?
tests like sputum microscopy and culture, rapid antigen tests and gene
amplification tests are the cornerstone for the identification of causative
organisms in pneumonia, there have been attempts to do this differentiation
based on CT scans of the chest. The commonly referred to “typical” bacterial
pneumonias are known to present with findings such as classic lobar
consolidation, air bronchograms, centrilobular nodules and in some cases,
pleural effusions. There is a great degree of overlap between findings of the
less common “atypical” pneumonias including Mycoplasma pneumonia, Legionella
pneumonia and the plethora of viral pneumonias. The findings can include one or
more of the following: bronchopneumonia
pattern with centrilobular nodules, bronchial wall thickening, ground glass
opacities in diffuse, bilateral patchy distribution or in some cases like
varicella pneumonia randomly distributed nodules and also the more rememberable
“crazy paving pattern”. The CT findings of the COVID-19 pneumonia that have
been described in literature, mostly from hospitals in China, are a subset of
these findings with not a single specific diagnostic or “AuntMinnie” finding.
For even an experienced chest radiologist, these patterns are not enough to
suggest the causative etiology.
This inability to distinguish causative organism by looking at a CT scan, is seen by many – more commonly by proponents of deep learning – as a limitation of the pattern recognition capabilities of human readers (refer to above – a task AI can do, but humans can’t). As shown by a recent study titled “Artificial Intelligence Distinguishes COVID-19 from Community Acquired Pneumonia on Chest CT” published in the leading journal Radiology, Artificial Intelligence might be able to distinguish pneumonia caused by COVID-19 from CAP on Chest CT scans. The key strength of this study is the use of a large number of RT-PCR proven COVID-19 scans (1,165 scans), CAP (1,560 scans) and non- pneumonia scans (1,193 scans) for training a 3-dimensional convolutional neural network, aptly called ‘CovNet’. The results of CovNet on their test set, which comprised of a random sampling of 10% cases from the overall dataset, are very impressive with a sensitivity of 90% and specificity of 96% for diagnosing COVID-19 (with an Area Under the Curve (AUC) of 0.96) and a sensitivity of 87% and specificity of 92% for CAP diagnosis (AUC of 0.95). Naturally, a system with such high ability to diagnose COVID-19, should be used unequivocally everywhere. But as they say, the devil almost always lies in the details.
can’t it help?
There are two main problems with this study, the first pertains to the case mix of the data itself. The attached figure shows the breakdown of the training and test data by etiology – note that the COVID-19 cases (1,165) far outweigh the number of confirmed viral pneumonia (24) and confirmed bacterial pneumonia (72) cases. In fact, most of the CAP dataset is comprised of ‘unknown’ etiology, i.e. 1,212 cases. A review by Zhu et al. shows that viruses (in the pre-COVID-19 era) cause ~15% of CAP in China – implying that most of the 1,212 cases were probably of non-viral etiology. So, as pointed out in this Twitter post, is the algorithm simply learning to distinguish viral from non-viral pneumonia? If so, it raises serious concerns about the claim of AI being able to ‘diagnose COVID-19’.
The second issue is the lack of
testing on a ‘truly’ independent test set – a dataset from an separate
institution which has not been ‘seen’ by either the algorithm or the
researchers who have built it. Overfitting of algorithms to the training data,
leading to subsequent high performance on randomly sampled test data (from that
very same data pool) is a known problem within deep learning circles – and can
also be inferred from the fact that algorithms developed as part of numerous
competitions and challenges involving medical imaging have never seen the light
(or darkness) of a radiologist’s reporting room. This is not to take away
anything from the researchers who have toiled hard during the difficult times
to bring out this work – putting together such a massive dataset, then training
a deep learning algorithm on it, and subsequently publishing the results, when
the world around you is collapsing – is no easy feat.
For what its
worth, both ACR and CDC have advised against the use of CT as a first-line tool
for the diagnosis / screening of COVID-19. But does that mean CT has no role to
play? Of course not! CT is an essential tool in the armory of pulmonologists
and intensivists and serial CT scans will obviously be done for patients with
any kind of severe pneumonia – so it would be silly to assume that CT no role
to play in the clinical course of a COVID-19 patient. And this is precisely
where AI can help.
diagnosing COVID-19 on CT might be hard (as yet) for AI, it would be
interesting to see how the dynamics change when going through an epidemic of
COVID-19 – is every viral pneumonia suspected to be COVID-19? Does the
incidence of other viral pneumonias reduce? These are not easy questions to
answer. That said, picking up signs of viral pneumonia, or even pneumonia in
general, and quantifying them over serial CT scans of the same patient, would
be truly value additive, especially in resource-constrained locations or
situations. Another potential use case, which is again something that humans may
not be able to do, is predict prognosis and/or clinical course of certain
patients based on serial CT scans – this would be able to not only guide
treatment but also help triage cases in a situation where healthcare systems
The role of CT in COVID-19 is constantly evolving – when RTPCR kits were not available, it was part of the initial workup / screening of patients with suspected COVID-19, but now with the increased availability of RTPCR kits, CT’s role is moving to estimation of disease severity and progression. AI tools built to diagnose COVID-19 infection need to be evaluated robustly since current radiological knowledge does not have specific features that can help identify COVID-19 infection on CT. That said, AI tools have a vital role to play in quantifying disease progression and possibly predicting disease outcomes.
Vasanth Venugopal, MD is a radiologist who leads imaging research at CARING with expertise in developing novel validation and deployment methodologies for Artificial Intelligence algorithms for medical imaging.
Vidur Mahajan, MBBS, MBA is a physician-MBA who leads research & development for CARING who is passionate about bringing advanced technologies in imaging and genomics into the clinical workflow through a single unifying technology interface.