author
Chief Scientist
,
Snorkel AI
Assistant Professor @ University of Wisconsin-Madison
Frederic Sala is Chief Scientist at Snorkel AI and an assistant professor in the Computer Sciences Department at the University of Wisconsin-Madison. His research studies the fundamentals of data-driven systems and machine learning, with a focus on data-centric AI, foundation models, and automated machine learning. He and his group received the 2024 DARPA Young Faculty Award, a best student paper runner-up award at UAI ’22, the outstanding Ph.D. dissertation award from the UCLA Department of Electrical Engineering, the NSF Graduate Research Fellowship.
The latest from Fred
Efficient representation learning for higher-order data with simplicial complexes
Graph-based machine learning is experiencing explosive growth, driven by impressive recent developments and wide applicability. Typical approaches for graph representation learning predominantly focus on pairwise interactions, while neglecting the patterns of higher-order interactions common to complex systems. This paper explores many-body interaction models, centering on simplicial complexes. From a theoretical point of view, we offer a pair of insights illustrating why higher-order models are necessary, why non-graph-based models generally cannot generalize well, while graph-based models may be able to do so. We conduct experiments on synthetic data, co-citation networks, co-authorship networks and gene-disease associations and show that simplicial complexes with...
Research Paper
Efficient representation learning for higher-order data with simplicial complexes
Graph-based machine learning is experiencing explosive growth, driven by impressive recent developments and wide applicability. Typical approaches for graph representation learning predominantly focus on pairwise interactions, while neglecting the patterns of higher-order interactions common to complex systems. This paper explores many-body interaction models, centering on simplicial complexes. From a theoretical point of view, we offer a pair of insights illustrating…
Good Data from Bad Models: Foundations of Threshold-based Auto-labeling
Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Auto-labeling systems are a promising way to reduce reliance on manual labeling for dataset construction. Threshold-based auto-labeling, where validation data obtained from humans is used to find a threshold for confidence above which the data is machine-labeled, is emerging as a popular solution used widely in practice [SGT22, QCG20, SS22]. Given the long shelf-life and diverse usage of the resulting datasets, understanding when the data obtained by such auto-labeling systems can be relied on is crucial. In this work, we analyze threshold-based auto-labeling systems and derive...
Research Paper
Good Data from Bad Models: Foundations of Threshold-based Auto-labeling
Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Auto-labeling systems are a promising way to reduce reliance on manual labeling for dataset construction. Threshold-based auto-labeling, where validation data obtained from humans is used to find a threshold for confidence above which the data is machine-labeled, is emerging as a popular solution used widely in…
Domain Generalization with Nuclear Norm Regularization
The ability to generalize to unseen domains is crucial for machine learning systems deployed in the real world, especially when we only have data from limited training domains. In this paper, we propose a simple and effective regularization method based on the nuclear norm of the learned features for domain generalization. Intuitively, the proposed regularizer mitigates the impacts of environmental features and encourages learning domain-invariant features. Theoretically, we provide insights into why nuclear norm regularization is more effective compared to ERM and alternative regularization methods. Empirically, we conduct extensive experiments on both synthetic and real datasets. We show nuclear norm...
Research Paper
Domain Generalization with Nuclear Norm Regularization
The ability to generalize to unseen domains is crucial for machine learning systems deployed in the real world, especially when we only have data from limited training domains. In this paper, we propose a simple and effective regularization method based on the nuclear norm of the learned features for domain generalization. Intuitively, the proposed regularizer mitigates the impacts of environmental…
Automl for climate change: A call to action
The challenge that climate change poses to humanity has spurred a rapidly developing field of artificial intelligence research focused on climate change applications. The climate change AI (CCAI) community works on a diverse, challenging set of problems which often involve physics-constrained ML or heterogeneous spatiotemporal data. It would be desirable to use automated machine learning (AutoML) techniques to automatically find high-performing architectures and hyperparameters for a given dataset. In this work, we benchmark popular AutoML libraries on three high-leverage CCAI applications: climate modeling, wind power forecasting, and catalyst discovery. We find that out-of-the-box AutoML libraries currently fail to meaningfully surpass...
Research Paper
Automl for climate change: A call to action
The challenge that climate change poses to humanity has spurred a rapidly developing field of artificial intelligence research focused on climate change applications. The climate change AI (CCAI) community works on a diverse, challenging set of problems which often involve physics-constrained ML or heterogeneous spatiotemporal data. It would be desirable to use automated machine learning (AutoML) techniques to automatically find…
AutoML Decathlon: Diverse Tasks, Modern Methods, and Efficiency at Scale
The vision of Automated Machine Learning (AutoML) is to produce high performing ML pipelines that require very little human involvement or domain expertise to use. Competitions and benchmarks have been critical tools for accelerating progress in AutoML. However, much of the prior work on AutoML competitions has focused on well-studied domains in machine learning such as vision and language—these are domains which have benefited from several years of ML pipeline design by domain experts, which brings the usage of AutoML into question in the first place. Recently, AutoML for diverse tasks has emerged as an important research area that aims...
Research Paper
AutoML Decathlon: Diverse Tasks, Modern Methods, and Efficiency at Scale
The vision of Automated Machine Learning (AutoML) is to produce high performing ML pipelines that require very little human involvement or domain expertise to use. Competitions and benchmarks have been critical tools for accelerating progress in AutoML. However, much of the prior work on AutoML competitions has focused on well-studied domains in machine learning such as vision and language—these are…
Causal Omnivore: Fusing Noisy Estimates of Spurious Correlations
Spurious correlations are one of the biggest pain points for users of modern machine learning. To handle this issue, many approaches attempt to learn features that are causally linked to the prediction variable. Such techniques, however, suffer from various flaws—they are often prohibitively complex or based on heuristics and strong assumptions that may fail in practice. There is no one-size-fits-all causal feature identification approach. To address this challenge, we propose a simple way to fuse multiple noisy estimates of causal features. Our approach treats the underlying causal structure as a latent variable and exploits recent developments in estimating latent structures...
Research Paper
Causal Omnivore: Fusing Noisy Estimates of Spurious Correlations
Spurious correlations are one of the biggest pain points for users of modern machine learning. To handle this issue, many approaches attempt to learn features that are causally linked to the prediction variable. Such techniques, however, suffer from various flaws—they are often prohibitively complex or based on heuristics and strong assumptions that may fail in practice. There is no one-size-fits-all…
NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks
Most existing neural architecture search (NAS) benchmarks and algorithms prioritize well-studied tasks, e.g. image classification on CIFAR or ImageNet. This makes the performance of NAS approaches in more diverse areas poorly understood. In this paper, we present NAS-Bench-360, a benchmark suite to evaluate methods on domains beyond those traditionally studied in architecture search, and use it to address the following question: do state-of-the-art NAS methods perform well on diverse tasks? To construct the benchmark, we curate ten tasks spanning a diverse array of application domains, dataset sizes, problem dimensionalities, and learning objectives. Each task is carefully chosen to interoperate with...
Research Paper
NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks
Most existing neural architecture search (NAS) benchmarks and algorithms prioritize well-studied tasks, e.g. image classification on CIFAR or ImageNet. This makes the performance of NAS approaches in more diverse areas poorly understood. In this paper, we present NAS-Bench-360, a benchmark suite to evaluate methods on domains beyond those traditionally studied in architecture search, and use it to address the following…
Blog
4 new papers show foundation models can build on themselves
The surest way to improve foundation models is through more and better data, but Snorkel researchers showed FMs can learn from themselves.
Blog
Getting better performance from foundation models (with less data)
Getting better performance from foundation models (with less data)
