We used machine learning to develop a model that could assist doctors in diagnosing Coronavirus (COVID-19) with the help of an x-ray. If you are a doctor or nurse, you can contact us and get access to the model for free (firstname.lastname@example.org).
The article will explain the technology, the data used and the results.
Since the last article I wrote about visualizing data related to the Coronavirus, I couldn’t stop thinking about how I could help. Is there any way I could personally make or change something? I’m not the person to just talk about how bad things are and how the world goes bananas (all while I stockpile cans of beans and look at my neighbors like they have the plague).
I researched the applications of artificial intelligence (more specifically, machine learning) in cases similar to the Coronavirus, such as drug research and development, but that task requires an entire team of experts in the fields of medicine, artificial intelligence, and the pharmaceutical industry, and we don’t have such a team.
I turned to slightly simpler options. Can a diagnostic tool be created that can help doctors eliminate options faster?
I know there was a study where they tried to diagnose a Coronavirus with CT scans. Results on 1,119 CT shots are pretty good with 89.5% and 79.3% accuracy.
Through further thinking, it occurred to me that if something was visible on CT, there was probably a possibility that it could be detected on an “ordinary” X-ray as well.
When (unfortunately it is no longer “if”) the number of patients increases, it will be necessary to quickly diagnose people and separate people who need to be isolated. X-ray is pretty, it’s cheap, doesn’t require special care, and works as long as there is electricity. Good old x-ray.
While I was looking around for something similar, I came across an existing, well written and explained article.
What did I use?
Of course, I could have just copy-pasted the author’s solution and then say that I did something on my own, but I decided to play around and see what results I could get on my own.
For those who understand the ML ecosystem, I thought about using Keras (Tensorflow) or maybe this would be simple enough for fastai (Pytorch). I tried fastai first and got pretty good results.
I used the information available here (same information as in the article already mentioned).
Pictures of “normal” of the lungs, as well as pictures of the lungs diagnosed with pneumonia, can be found here.
After downloading everything, I checked out the data. Since we have X-ray images “front” – front to back (this type of imaging is called “Posterioranterior”, or “PA” abbreviated), then I only considered such pictures.
The dataset contains 69 x-ray images of confirmed COVID-19 patients. Now I needed a picture of healthy lungs and patients who may have another disease, such as pneumonia, to see how the algorithm behaves and whether it is possible to find some differences between those three classes of images – covid, normal, pneumonia (pneumonia).
I took these images from the Kaggle dataset, which contains 4265 x-ray images with pneumonia and 560 “normal” (healthy) images. I used 30% of the data for validation.
Final numbers, 560 healthy images (normal), 69 COVID-19, 4265 pneumonia.
As the architecture of the neural network itself, I first tried “Resnet 34”. This architecture is a bit smaller and although it has pretty good accuracy, I wanted to see what the maximum I could get and what results I would get to use the largest network that, “see the most things”. I ended up using “Resnet 152”, which is a pretty large network/architecture. Images were reduced to an input grid size. After a bit of experimentation, changing architecture, augmentation and fine-tuning, I got my results.
The first results shocked me. At first, I didn’t have as many examples of patients with healthy lungs and inflamed lungs and my accuracy was 100%!
When I informed the team, I had to tell them to curb their enthusiasm, because the news was too good to be true. Such a simple endeavor can’t bring about complete accuracy. After that, I included as many pictures as possible.
To sum up this picture.
Of the 16 patients, one who had pneumonia was diagnosed as having COVID-19 and one who had COVID-19 was diagnosed as having pneumonia.
No healthy patient was diagnosed as having COVID-19!
I still do not understand how an algorithm can be able to identify the features of a COVID-19 patient’s X-ray as opposed to a healthy lung?
I kept looking at the pictures and comparing them, but to be honest, I don’t understand what I’m looking for. Then I tried to ask the neural network to show me what it saw and to try and then took a closer look. Specifically, I wanted to see which part of the image most triggered/irritated the neural network and on what basis it concluded that it was one category and not the other.
Again, with complete honesty, I still don’t see much difference. It seems that based on the tiny spots that are all the same to my sight, the neural network can determine that it is a coronavirus or pneumonia or healthy lungs.
I think someone who understands x-rays is gonna have a good laugh at my interpretation. I mean, I see some small changes, but they don’t mean too much to me, but again, I’m not a doctor, I’m dealing with artificial intelligence and data.
How can you help?
Can you help? Yes you can.
What we’re still missing is a large number of X-ray images with the COVID-19 virus. What I plan to do is one web application (a website on the Internet) where hospitals can send patients x-rays. If possible, I invite hospitals that have such images to contact us (best at email@example.com)
I will give them access to the application, and they will be able to send an image from their computers themselves and get the result back, along with these images showing how the neural network “saw” that it was COVID-19. This may help to improve the algorithm itself since there are ways in which the neural network can learn to recognize individual elements of an image. If anyone interested in this is interested, I can clarify further. The more images we have, the more accurate the algorithm will be and will be able to better distinguish between diseased and healthy patients.
First, we must curb our enthusiasm. The results I got are impressive, but I’m skeptical of them, and there’s a good reason for that.
I do not sell or advertise anything (other than my expertise in machine learning and artificial intelligence), or try to tell people that this is a way to safely defeat the COVID-19 virus.
I don’t want to give people inaccurate information since it is also possible to cause confusion and the idea that we don’t need doctors, we just need x-rays (and pitchforks to eliminate those who have the virus).
These are human lives and should not be taken lightly. The errors of such systems are costly. The algorithm correctly recognizes 93% of COVID-19 cases.
One case was misclassified as pneumonia. In a phone call with one doctor, I learned that pneumonia is not that different from the COVID-19 virus image, since the COVID-19 virus result is, in a way, a kind of pneumonia (respiratory problems).
The model categorized one case of the COVID-19 virus as pneumonia. The model has never wrongly categorized a healthy person as having COVID-19. In other words, it may be possible, with sufficient examples, to eliminate individuals as candidates for COVID-19 viruses.
What bothers me a little is the fact that these results are visible after the symptoms progress. So there are people who have very little or no symptoms at all and can carry and transmit the virus. This test is not for such individuals as they are unlikely to even come close to the hospital. Those who already have (obvious?) symptoms will go to the hospital, and such a test would probably quickly eliminate the possibility of a COVID-19 virus. That would improve the capacity of the hospitals to diagnose and if needed separate the patients.
What is missing here is a large number of x-ray of COVID-19 patients. This way, I wouldn’t bet too much that this works representatively until we have at least 200 examples of x-rays.
Follow the instructions of the county and state headquarters and things will probably be better.