Eka’s Weekly Roundup (16 June 2023)

Issue 36 l Healthcare LLMs

Jun 16, 2023

The paper “Health system-scale language models are all-purpose prediction engines” was published last week in Nature. It describes how a New York hospital developed a large language model using electronic health records covering >380k patients across four hospitals, resulting in a 4.1 billion-word dataset from January 2011 to May 2020. The LLM outperformed generalist LLMs in clinical tasks, and was more efficient in predicting hospital readmissions compared to a manual physician review.

On prediction tasks, NYUTron had an AUC of 79-95%, with improvements from traditional models. In a forward-looking readmission study, NYUTron predicted 83% of readmissions with 21% precision, with 61% being unplanned and 27% preventable according to a panel of physicians who reviewed cases.

Thinking about data access in healthcare LLMs 🗞️

Nature published an article last week about healthcare large language model (LLM) built off a dataset from a New York hospital, called NYUTron. The study evaluates the efficacy of NYUTron on three clinical tasks and two operational tasks, including readmission prediction, insurance claim denial prediction, and inpatient length of stay prediction. In the readmission prediction task, NYUTron was shown to be competitive with traditional models and other LLMs, and it was found to scale better with data than traditional structured models.

The study's approach of using a smaller (<1 billion parameters) encoder language model trained on highly tailored data represents a marked departure from the current trend in language model research that focuses on massive (>1 billion parameters), generative models pretrained on large, non-specific datasets. The study's results show that high-quality datasets for fine-tuning are more valuable than pretraining, and the study recommends that users locally fine-tune an externally pretrained language model when computational ability is limited.

The paper looked at retrospective performance on a number of variables, including estimates of length of stay in hospital and insurance denial. On both of these metrics, the model performed better than the baseline in predicting outcomes as shown below.

The study's prospective trial deployed NYUTron in a live healthcare environment and demonstrated its efficacy at predicting 30-day readmission while being integrated into clinical workflows. The model accurately predicted readmissions, and manual physician review suggested that some true positive predictions by NYUTron are clinically meaningful, preventable readmissions. The study concludes that NYUTron opens the door to translating the progress in modern natural language processing and deep learning to improving the quality and affordability of healthcare.

The economic implications are significant. As shown by the figure above, 61% of the predicted case were unplanned. Within the unplanned readmissions, 20% of patients experienced an adverse event or death on readmission, with 50% of these events considered preventable by the physician panel who conducted a manual review in parallel.