author

Jason Fries

Stanford University
Assistant Professor of Biomedical Data Science and of Medicine

I’m an Assistant Professor of Biomedical Data Science and of Medicine at Stanford University. My research focuses on training and evaluating foundation models for healthcare and is positioned at the intersection of computer science, medical informatics, and hospital systems. Much of my work explores using electronic health record (EHR) data to contextualize human health, leveraging longitudinal patient information to inform model development and evaluation. My work has appeared in NeurIPS, ICLR, AAAI, Nature Communications, and npj Digital Medicine.

The latest from Jason

Scalable Approach to Medical Wearable Post-Market Surveillance
Objective: We sought to develop a weak supervision-based approach to demonstrate feasibility of post-market surveillance of wearable devices that render AF pre-diagnosis. Materials and Methods: Two approaches were evaluated to reduce clinical note labeling overhead for creating a training set for a classifier: one using programmatic codes, and the other using prompts to large language models (LLMs). Probabilistically labeled notes were then used to fine-tune a classifier, which identified patients with AF pre-diagnosis mentions in a note. A retrospective cohort study was conducted, where the baseline characteristics and subsequent care patterns of patients identified by the classifier were compared against...
Research Paper
Scalable Approach to Medical Wearable Post-Market Surveillance

Objective: We sought to develop a weak supervision-based approach to demonstrate feasibility of post-market surveillance of wearable devices that render AF pre-diagnosis. Materials and Methods: Two approaches were evaluated to reduce clinical note labeling overhead for creating a training set for a classifier: one using programmatic codes, and the other using prompts to large language models (LLMs). Probabilistically labeled notes…

Sep 23, 2024 •

RM. Yoo, et al.

Learn more about Scalable Approach to Medical Wearable Post-Market Surveillance
Red Teaming Large Language Models in Medicine: Real-World Insights on Model Behavior
0. Abstract Background The integration of large language models (LLMs) in healthcare offers immense opportunity to streamline healthcare tasks, but also carries risks such as response accuracy and bias perpetration. To address this, we conducted a red-teaming exercise to assess LLMs in healthcare and developed a dataset of clinically relevant scenarios for future teams to use. Methods We convened 80 multi-disciplinary experts to evaluate the performance of popular LLMs across multiple medical scenarios. Teams composed of clinicians, medical and engineering students, and technical professionals stress-tested LLMs with real world clinical use cases. Teams were given a framework comprising four categories...
Research Paper
Red Teaming Large Language Models in Medicine: Real-World Insights on Model Behavior

0. Abstract Background The integration of large language models (LLMs) in healthcare offers immense opportunity to streamline healthcare tasks, but also carries risks such as response accuracy and bias perpetration. To address this, we conducted a red-teaming exercise to assess LLMs in healthcare and developed a dataset of clinically relevant scenarios for future teams to use. Methods We convened 80…

Sep 18, 2024 •

Crystal T. Chang, Hodan Farah, Haiwen Gui, Shawheen Justin Rezaei + 76 more

Learn more about Red Teaming Large Language Models in Medicine: Real-World Insights on Model Behavior
Recent Advances, Applications and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposium
The third ML4H symposium was held in person on December 10, 2023, in New Orleans, Louisiana, USA. The symposium included research roundtable sessions to foster discussions between participants and senior researchers on timely and relevant topics for the \ac{ML4H} community. Encouraged by the successful virtual roundtables in the previous year, we organized eleven in-person roundtables and four virtual roundtables at ML4H 2022. The organization of the research roundtables at the conference involved 17 Senior Chairs and 19 Junior Chairs across 11 tables. Each roundtable session included invited senior chairs (with substantial experience in the field), junior chairs (responsible for facilitating...
Research Paper
Recent Advances, Applications and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposium

The third ML4H symposium was held in person on December 10, 2023, in New Orleans, Louisiana, USA. The symposium included research roundtable sessions to foster discussions between participants and senior researchers on timely and relevant topics for the \ac{ML4H} community. Encouraged by the successful virtual roundtables in the previous year, we organized eleven in-person roundtables and four virtual roundtables at…

Sep 18, 2024 •

Hyewon Jeong, Sarah Jabbour, Yuzhe Yang, Rahul Thapta + 39 more

Learn more about Recent Advances, Applications and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposium
Merlin: A Vision Language Foundation Model for 3D Computed Tomography
The large volume of abdominal computed tomography (CT) scans coupled with the shortage of radiologists have intensified the need for automated medical image analysis tools. Previous state-of-the-art approaches for automated analysis leverage vision-language models (VLMs) that jointly model images and radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. Here to overcome these shortcomings for abdominal CT interpretation, we introduce Merlin, a 3D VLM that learns from volumetric CT scans, electronic health record data and radiology reports. This approach is enabled by a multistage pretraining framework that does not require additional manual annotations. We...
Research Paper
Merlin: A Vision Language Foundation Model for 3D Computed Tomography

The large volume of abdominal computed tomography (CT) scans coupled with the shortage of radiologists have intensified the need for automated medical image analysis tools. Previous state-of-the-art approaches for automated analysis leverage vision-language models (VLMs) that jointly model images and radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. Here to overcome these shortcomings…

Sep 18, 2024 •

Louis Blankemeier, Ashwin Kumar, Joseph Paul Cohen, Jiaming Liu + 36 more

Learn more about Merlin: A Vision Language Foundation Model for 3D Computed Tomography
Exploring the Potential of Large Language Models in Neurology, Using Neurologic Localization as an Example
An editorial in Neurology: Clinical Practice by Chia-Chun Chiang and Jason A. Fries on how large language models could be used in neurology. It discusses a study that tested GPT-4 on locating stroke lesions from 46 published case reports. GPT-4 performed well at identifying the side of the lesion and the brain region, but was less accurate for lesions in the cerebellum and the spinal cord.
Research Paper
Exploring the Potential of Large Language Models in Neurology, Using Neurologic Localization as an Example

An editorial in Neurology: Clinical Practice by Chia-Chun Chiang and Jason A. Fries on how large language models could be used in neurology. It discusses a study that tested GPT-4 on locating stroke lesions from 46 published case reports. GPT-4 performed well at identifying the side of the lesion and the brain region, but was less accurate for lesions in…

Sep 18, 2024 •

Chia-Chun Chiang, Jason Fries

Learn more about Exploring the Potential of Large Language Models in Neurology, Using Neurologic Localization as an Example
Characterizing the limitations of using diagnosis codes in the context of machine learning for healthcare
BACKGROUND: Diagnostic codes are commonly used as inputs for clinical prediction models, to create labels for prediction tasks, and to identify cohorts for multicenter network studies. However, the coverage rates of diagnostic codes and their variability across institutions are underexplored. The primary objective was to describe lab- and diagnosis-based labels for 7 selected outcomes at three institutions. Secondary objectives were to describe agreement, sensitivity, and specificity of diagnosis-based labels against lab-based labels. METHODS: This study included three cohorts: SickKids from The Hospital for Sick Children, and StanfordPeds and StanfordAdults from Stanford Medicine. We included seven clinical outcomes with lab-based definitions:...
Research Paper
Characterizing the limitations of using diagnosis codes in the context of machine learning for healthcare

BACKGROUND: Diagnostic codes are commonly used as inputs for clinical prediction models, to create labels for prediction tasks, and to identify cohorts for multicenter network studies. However, the coverage rates of diagnostic codes and their variability across institutions are underexplored. The primary objective was to describe lab- and diagnosis-based labels for 7 selected outcomes at three institutions. Secondary objectives were…

Sep 18, 2024 •

Lin Lawrence Guo, Keith E. Morse, Catherine Aftandilian, Ethan Steinberg + 7 more

Learn more about Characterizing the limitations of using diagnosis codes in the context of machine learning for healthcare
A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs)
1 Abstract Importance Large Language Models (LLMs) can assist in a wide range of healthcare-related activities. Current approaches to evaluating LLMs make it difficult to identify the most impactful LLM application areas. Objective To summarize the current evaluation of LLMs in healthcare in terms of 5 components: evaluation data type, healthcare task, Natural Language Processing (NLP)/Natural Language Understanding (NLU) task, dimension of evaluation, and medical specialty. Data Sources A systematic search of PubMed and Web of Science was performed for studies published between 01-01-2022 and 02-19-2024. Study Selection Studies evaluating one or more LLMs in healthcare. Data Extraction and Synthesis...
Research Paper
A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs)

1 Abstract Importance Large Language Models (LLMs) can assist in a wide range of healthcare-related activities. Current approaches to evaluating LLMs make it difficult to identify the most impactful LLM application areas. Objective To summarize the current evaluation of LLMs in healthcare in terms of 5 components: evaluation data type, healthcare task, Natural Language Processing (NLP)/Natural Language Understanding (NLU) task,…

Sep 18, 2024 •

Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash + 15 more

Learn more about A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs)
A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records
Foundation models are transforming artificial intelligence (AI) in healthcare by providing modular components adaptable for various downstream tasks, making AI development more scalable and cost-effective. Foundation models for structured electronic health records (EHR), trained on coded medical records from millions of patients, demonstrated benefits including increased performance with fewer training labels, and improved robustness to distribution shifts. However, questions remain on the feasibility of sharing these models across hospitals and their performance in local tasks. This multi-center study examined the adaptability of a publicly accessible structured EHR foundation model (FM SM ), trained on 2.57 M patient records from Stanford...
Research Paper
A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records

Foundation models are transforming artificial intelligence (AI) in healthcare by providing modular components adaptable for various downstream tasks, making AI development more scalable and cost-effective. Foundation models for structured electronic health records (EHR), trained on coded medical records from millions of patients, demonstrated benefits including increased performance with fewer training labels, and improved robustness to distribution shifts. However, questions remain…

Sep 18, 2024 •

Lin Lawrence Guo, Jason Fries, Ethan Steinberg, Scott L. Fleming + 5 more

Learn more about A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records
Language Models in the Loop: Incorporating Prompting into Weak Supervision
We propose a new strategy for applying large pre-trained language models to novel tasks when labeled training data is limited. Rather than apply the model in a typical zero-shot or few-shot fashion, we treat the model as the basis for labeling functions in a weak supervision framework. To create a classifier, we first prompt the model to answer multiple distinct queries about an example and define how the possible responses should be mapped to votes for labels and abstentions. We then denoise these noisy label sources using the Snorkel system and train an end classifier with the resulting training data....
Research Paper
Language Models in the Loop: Incorporating Prompting into Weak Supervision

We propose a new strategy for applying large pre-trained language models to novel tasks when labeled training data is limited. Rather than apply the model in a typical zero-shot or few-shot fashion, we treat the model as the basis for labeling functions in a weak supervision framework. To create a classifier, we first prompt the model to answer multiple distinct…

Aug 22, 2024 •
Learn more about Language Models in the Loop: Incorporating Prompting into Weak Supervision
1 2 … 6
 Illution Back
Illution Front

For models that need to be right. Not just good enough.