Wayfair accelerates product tagging with Snorkel Flow: 10x faster labeling

Using AI to supercharge online retail

Wayfair is a Boston-based e-commerce company specializing in home goods and furniture, serving ~22M customers and partnering with ~20K suppliers. To ensure that relevant products appear in customer searches (e.g., “blue outdoor pillows”), they rely on product tags (e.g., “blue” “outdoor” and “pillow”). With over 10,000 product tags across 40 million products, creating and managing labeled data is an enormous and time-consuming effort.

Challenge

Wayfair uses machine learning (ML) across the business to improve search and customer experience and organize its rich product catalog, but they wanted to innovate further to keep pace with growing inventory and changing consumer behavior. Wayfair was also looking to launch a computer vision initiative to unlock information from millions of product images using visual information extraction (VIE).

However, creating sufficient training and validation sets became a major bottleneck.

• Supplier labels were often insufficient, inaccurate, or inconsistent. With limited standardization and due to the subjective nature of the manually labeled data, datasets were noisy and unreliable. Relying on suppliers for this information could also cause unnecessary friction in key relationships.
• Manually labeling training data was prohibitively slow. Wayfair outsourced manual labeling using human-in-the-loop (HITL) data curation, but it can take 8 weeks to label just 6,000 images. Manually labeling the whole catalog would require person-years of effort, and HITL can struggle to keep pace with new inventory and evolving tags and searches.
• Rich information was buried within images and was challenging to extract and utilize. Wayfair has millions of product images that reveal the style, pattern, and theme to the viewer; however, they struggled to transform that information into tags that their ML could interpret.

Wayfair had experimented with an off-the-shelf foundation model, CLIP, for zero-shot image retrieval tasks. It worked well for about 5% of cases but made inaccurate high-confidence predictions and could not provide all necessary tags for their extensive product catalog. It was especially difficult to build datasets for high-value edge cases where sample data was limited.

Goal

Improve data labeling and reduce ML development hours with automated labeling to extract rich visual information accurately and with less manual intervention.

Solution

Programmatic labeling and computer vision for product tags

Wayfair partnered with Snorkel AI to speed the development of models designed to tackle key challenges in curating and managing their catalog data. These models help Wayfair reduce manual labeling costs and automate improvements across search, marketing, and customer experience.

Starting with millions of noisily-labeled images, Wayfair used Snorkel Flow to build 46 tag models within days instead of months. Armed with faster and more accurate modeling, Wayfair realized significant cost savings and major gains in search relevancy and key revenue predictors.

Auto-extracting visual information from product images

Wayfair was developing machine learning (ML) models to pull information from their product images and convert that data into tags used to return better, more accurate results to customer searches. To supply this algorithm with the best possible data to learn from, Wayfair collaborated with the computer vision research team at Snorkel to improve the quality of their training data, uncovering image quality issues.

Wayfair and Snorkel developed a workflow that incorporated data preprocessing, curation, and iterative development to extract and apply visual data to product labels. Using Snorkel Flow, Wayfair can clean data, remove outliers and duplicates, and quickly prepare training and evaluation datasets with strategic sampling and prompting.

By leveraging foundation models to automatically label data and iteratively looping that data to explore and improve their ML, Wayfair can unlock rich visual information from millions of product images to save years of manual labeling and improve accuracy over supplier-provided labels. Wayfair can build models ten times faster with Snorkel’s iterative data-centric workflows compared to their manual HITL workflows with the same or greater accuracy.

Model-guided error analysis

With Snorkel Flow, Wayfair’s data scientists can visually analyze the failure modes of the model on the validation data (e.g., “Geometric” pattern mistakenly tagged as “Chevron”). This allows them to obtain richer training data iteratively (e.g., add geometric accent chairs as negative labels) to create better models.

Wayfair can also generate higher-quality datasets with Snorkel Flow, achieving 20+ points accuracy over vendor-supplied labels. Improved datasets lead to improved model performance, all leading to customers finding the right products at the right time. Since engaging Snorkel, Wayfair has also seen a 7% increase in view-read, i.e., times when a customer clicks through to a product presented to them from a search. This is a leading indicator for add-to-cart rate and increased revenue.

In partnership with Snorkel Flow, Wayfair overcame its challenges with:

• Programmatic labeling: By feeding pre-trained foundation models relevant prompts (e.g., “Chevron area rugs”), Wayfair can generate thousands of labels through labeling functions (LFs). They can filter, denoise, and refine those labels via weak supervision, and quickly build high-quality training sets for their ML.
• Ensured adaptability: Tags and customer searches constantly evolve, and it’s time-consuming and expensive to manually update models. Wayfair can build iterative, data-centric workflows in Snorkel Flow to add or modify tags quickly and at scale.
• Validation-in-the-loop: Snorkel Flow allows data scientists and subject matter experts like category managers and suppliers to correct model errors and adjust validation datasets quickly while minimizing the manual efforts of their human agents.

Learn how to get more value from your PDF documents!

Transforming unstructured data such as text and documents into structured data is crucial for enterprise AI development. On December 17, we’ll hold a webinar that explains how to capture SME domain knowledge and use it to automate and scale PDF classification and information extraction tasks.

Sign up here!