9,105 patient records,
cleaned before a single question.
Act like a data scientist.
No instructions.
This was a group project for the Applied Data Programming module of my MSc. Four of us were given SUPPORT2, a public dataset of 9,105 critically ill patients from five medical centres in the US, and one instruction. Act as data scientists.
I did the data cleaning, the outlier work and several of the analyses and plots, including deaths by sex, illnesses by ethnicity and the correlation matrix. Then we brought everyone's work together into one notebook.
There was no set question to answer. We had to explore the data ourselves, clean it, look for patterns and explain what we found, all in one Jupyter notebook. Our goal was to understand what affects the survival of critically ill patients. We got a high 2:1.
Most of the work
was cleaning.
The raw data had 48 columns, and a lot of them were full of gaps. I went through every column with missing values, worked out what it measured, and decided whether to keep it, fill it or drop it.
Income had nearly 3,000 blanks, so I labelled them as undisclosed rather than guess. Lab results with too many missing values were removed, and rows missing key fields like white blood cell count were dropped.
Then I used boxplots to find outliers, like hospital stays over 180 days and temperatures far outside the normal range, and removed the extreme ones.
| Stage | Rows | Columns |
|---|---|---|
| RAW DATASET | 9105 | 48 |
| AFTER MISSING VALUES | 8802 | 24 |
| AFTER OUTLIERS | 6962 | 23 |
print('There are', df[df['slos'] > 180].shape[0], 'outliers to be removed from the data frame')
print('There are', df[df['d.time'] > 1780].shape[0], 'outliers to be removed from the data frame')
# Deleting all the outliers using the drop() function
df = df.drop(df[df['slos'] > 180].index)
df = df.drop(df[df['d.time'] > 1780].index)
df = df.drop(df[df['charges'] > 400000].index)
df = df.drop(df[df['wblc'] > 24].index)
df = df.drop(df[df['hrt'] > 175].index)
df = df.drop(df[df['temp'] < 34].index)
df = df.drop(df[df['temp'] > 40].index)
df = df.drop(df[df['crea'] > 4].index)
print(f'As you can see the data frame has gone from 8802, to {df.shape[0]}')What the data
actually said.
Once the data was clean, I explored it with summary statistics and plots.
- 01.
Around two thirds of the patients, 68% of the 8,802 left after handling missing values, died during follow up.
- 02.
The typical patient was in their 60s. The average age was 62, the median 64 and the most common age 68.
- 03.
Acute respiratory failure and multiple organ failure with sepsis was by far the most common diagnosis, with 2,483 patients. Cirrhosis was the least common, with 405.
- 04.
40% of all the patients were men who died, and 30% were women who died. Men were more likely to die from their diagnosis, and for both men and women, far fewer patients survived than died.
- 05.
The doctors' own survival estimates lined up with what happened. Patients given a better chance of surviving six months were less likely to die.
- 06.
Patients with diabetes tended to have more conditions overall.
| Category | Value | Details |
|---|---|---|
| ARF/MOSF w/Sepsis | 2,483 | |
| CHF | 1,126 | |
| COPD | 784 | |
| Lung Cancer | 758 | |
| MOSF w/Malig | 536 | |
| Coma | 437 | |
| Colon Cancer | 433 | |
| Cirrhosis | 405 |
Out of all patients, 40% were men who died and 30% were women who died. Within each sex, the death rates are much closer.
| Category | Value | Details |
|---|---|---|
| Men | 71.5% | 2,812 of 3,933 |
| Women | 68.3% | 2,069 of 3,029 |
Rebuilt from our cleaned dataset of 6,962 patients.
What I'd do
differently now.
I really enjoyed this project. Looking at it again now, after working on real client data at Squarcle, there are a few things I'd do differently.
- 01.
Be careful what you call an outlier. We removed patients followed up for more than 1,780 days, but 362 of those 372 were still alive. They weren't errors, they were the survivors, and removing them made the death rate look higher than it really was.
- 02.
Don't fill money with zero. Filling missing hospital charges with 0 made some stays look free. Leaving them blank, or using the median, would have been more honest.
- 03.
Go beyond describing the data. Now I'd build a machine learning model, for example to predict hospital charges or which patients are most at risk, and test how well it works on patients it hasn't seen.
- 04.
Write less code, and reuse more of it. My notebook repeats the same plotting and filtering code again and again. After Squarcle, I'd wrap those steps into small reusable functions, so the notebook is shorter and easier to check.
- 05.
Give the notebook a clearer structure. Each section should lead into the next, so anyone reading can follow the story from raw data to findings without getting lost.
- 06.
Take the visuals further. Seaborn did the job, but now I'd build the final findings in Power BI, so the people reading them could filter and explore the data themselves.