Evaluation

Discover how enterprises can leverage LLM-as-Judge systems to evaluate generative AI outputs at scale, improve model alignment, reduce costs, and tackle challenges like bias and interpretability.

It’s critical enterprises can trust and rely on GenAI evaluation results, and for that, SME-in-the-loop workflows are needed. In my first blog post on enterprise GenAI evaluation, I discussed the importance of specialized evaluators as a scalable proxy for SMEs. It simply isn’t practical to task SMEs with performing manual evaluations – it can take weeks if not longer, unnecessarily

GenAI needs fine-grained evaluation for AI teams to gain actionable insights.

Specialized GenAI evaluation ensures AI assistants meet business requirements, SME expertise, and industry regulations—critical for production-ready AI.

Discover how Snorkel AI’s methodical workflow can simplify the evaluation of LLM systems. Achieve better model performance in less time.

Learn about the obstacles faced by data scientists in LLM evaluation and discover effective strategies for overcoming them.

Snorkel AI has made building production-ready, high-value enterprise AI applications faster and easier than ever. The 2024.R3 update to our Snorkel Flow AI data development platform streamlines data-centric workflows, from easier-than-ever generative AI evaluation to multi-schema annotation.

Enterprises must evaluate LLM performance for production deployment. Custom, automated eval + data slices present the best path to production.

High-performing AI systems require more than a well-designed model. They also require properly constructed training and testing data.

We need more labeled data than ever, so we have explored weak supervision for non-categorical applications—with notable results.

To tackle generative AI use cases, Snorkel AI + AWS launched an accelerator program to address the biggest blocker: unstructured data.

The Snorkel Flow label model plays an instrumental role in driving the enterprise value we create. Here’s a peek at how it works.

Vision language models demonstrate impressive image classification capabilities, but LLMs can help improve their performance. Learn how.

Snorkel researchers devised a new way to evaluate long context models and address their “lost-in-the-middle” challenges with mediod voting.

See a walkthrough of how Snorkel Flow users build applications with production-grade RAG retrieval components.

Fine-tuning specialized LLMs demands a lot of time and cost We developed Bonito to make this process faster, cheaper, and easier.

ROBOSHOT acts like a lens on foundation models and improves their zero-shot performance without additional fine-tuning.

Snorkel AI placed a model at the top of the AlpacaEval leaderboard. Here’s how we built it, and how it changed AlpacaEval’s metrics.

As Snorkel AI prepares to build better enterprise LLM evaluations, we spoke with Yifan Mail from Stanford’s CRFM HELM project.

Snorkel AI helped a client solve the challenge of social media content filtering quickly and sustainably. Here’s how.

Microsoft infrastructure facilitates Snorkel AI research experiments, including our recent high rank on the AlpacaEval 2.0 LLM leaderboard.

Humans learn tasks better when taught in a logical order. So do LLMs. Researchers developed a way to exploit this tendency called “Skill-it!”

Enterprise GenAI 2024: applications will likely surge toward production, according to Snorkel AI Enterprise LLM Summit survey results .

Snorkel CEO Alex Ratner spoke with Douwe Keila, an author of the original paper about retrieval augmented generation (RAG).

Snorkel CEO Alex Ratner chatted with Stanford Professor Percy Liang about evaluation in machine learning and in AI generally.

A proof-of-concept project that combines predictive AI + generative AI to minimize LLM’s risks while keeping their advantages.

Data scientists can fine-tune Llama 2 to adapt it to specific tasks. The Snorkel Flow data development platform makes it easy to do so.

Insurance claims processing has long required a lot of tedious and expensive human labor, but artificial intelligence (AI) can help.

Gideon Mann, head of ML Product and Research at Bloomberg LP, chatted with Snorkel CEO Alex Ratner about building BloombergGPT.

Snorkel Flow makes it easy to fine tune LLMs like GPT-3.5 Turbo to work better for specific domain and enterprise requirements.