How Wayfair accelerated product tagging automation with Snorkel Flow

Executive summary

Wayfair, the leading home goods and furniture e-commerce retailer, offers over 40M products to 22M customers. Wayfair relies on machine learning (ML) and product tagging to ensure customer searches result in relevant products. Tagging information like color, style, and pattern can make all the difference in a customer finding the right product at the right time. However, supplier-provided tags can be incomplete and inconsistent, and manually labeling products is time-intensive and difficult to scale.

With Snorkel Flow, Wayfair built a data-centric AI development workflow to help improve automated catalog tagging across their products. They also partnered with Snorkel to collaborate on a  computer vision use case that unlocks rich information from product images. Wayfair can now programmatically label product catalog data 10X faster than they did with manual labeling, and their end model performance improved by 20 points on average compared to previous models trained only on supplier-provided labels.

Context 

Wayfair is a Boston-based e-commerce company specializing in home goods and furniture, serving ~22M customers and partnering with ~20K suppliers. To ensure that relevant products appear in customer searches (e.g. “blue outdoor pillows”), they rely on product tags (e.g. “blue” “outdoor” and “pillow”). With over 10,000 product tags across 40 million products, creating and managing labeled data is an enormous and time-consuming effort.

Challenge

Wayfair uses machine learning (ML) across the business to improve search and customer experience and organize its rich product catalog, but they wanted to innovate further to keep pace with growing inventory and changing consumer behavior. Wayfair was also looking to launch a computer vision initiative to unlock information from millions of product images using visual information extraction (VIE).

However, creating sufficient training and validation sets became a major bottleneck.

• Supplier labels were often insufficient, inaccurate, or inconsistent. With limited standardization and due to subjective nature of the manually-labeled data, datasets were noisy and unreliable. Relying on suppliers for this information could also cause unnecessary friction in key relationships.
• Manually labeling training data was prohibitively slow. Wayfair outsourced manual labeling using human-in-the-loop (HITL) data curation, but it can take 8 weeks to label just 6,000 images. Manually labeling the whole catalog would require person-years of effort, and HITL can struggle to keep pace with new inventory and evolving tags and searches.
• Rich information was buried within images and was challenging to extract and utilize. Wayfair has millions of product images that reveal the style, pattern, and theme to the viewer; however, they struggled to transform that information into tags that their ML could interpret.

Wayfair had experimented with an off-the-shelf foundation model, CLIP, for zero-shot image retrieval tasks. It worked well for about 5% of cases but made inaccurate high-confidence predictions and could not provide all necessary tags for their extensive product catalog. It was especially difficult to build datasets for high-value edge cases where sample data was limited.

Goal

Improve data labeling and reduce ML development hours with automated labeling to extract rich visual information accurately and with less manual intervention.

Solution

Programmatic labeling and computer vision for product tags

Wayfair partnered with Snorkel AI to speed the development of models designed to tackle key challenges in curating and managing their catalog data. These models help Wayfair reduce manual labeling costs and automate improvements across search, marketing, and customer experience.

Starting with millions of noisily-labeled images, Wayfair used Snorkel Flow to build 46 tag models within days instead of months. Armed with faster and more accurate modeling, Wayfair realized significant cost savings and major gains in search relevancy and key revenue predictors.

Auto-extracting visual information from product images

Wayfair was developing machine learning (ML) models to pull information from their product images and convert that data into tags used to return better, more accurate results to customer searches. To supply this algorithm with the best possible data to learn from, Wayfair collaborated with the computer vision research team at Snorkel to improve the quality of their training data, uncovering image quality issues.

Wayfair and Snorkel developed a workflow that incorporated data preprocessing, curation, and iterative development to extract and apply visual data to product labels. Using Snorkel Flow, Wayfair can clean data, remove outliers and duplicates, and quickly prepare training and evaluation datasets with strategic sampling and prompting.

By leveraging foundation models to automatically label data and iteratively looping that data to explore and improve their ML, Wayfair can unlock rich visual information from millions of product images to save years of manual labeling and improve accuracy over supplier-provided labels. Wayfair can build models ten times faster with Snorkel’s iterative data-centric workflows compared to their manual HITL workflows with the same or greater accuracy.

Model-guided error analysis:

With Snorkel Flow, Wayfair’s data scientists can visually analyze the failure modes of the model on the validation data (e.g. “Geometric” pattern mistakenly tagged as “Chevron”). This allows them to obtain richer training data iteratively (e.g. add geometric accent chairs as negative labels) to create better models.

Wayfair can also generate higher-quality datasets with Snorkel Flow, achieving 20+ points accuracy over vendor-supplied labels. Improved datasets lead to improved model performance, all leading to customers finding the right products at the right time. Since engaging Snorkel, Wayfair has also seen a 7% increase in view-read, i.e. times when a customer clicks through to a product presented to them from a search. This is a leading indicator for add-to-cart rate and increased revenue.

In partnership with Snorkel Flow, Wayfair overcame its challenges with:

1. Programmatic labeling: By feeding pre-trained foundation models relevant prompts (e.g. “Chevron area rugs”), Wayfair can generate thousands of labels through labeling functions (LFs). They can filter, denoise, and refine those labels via weak supervision, and quickly build high-quality training sets for their ML.
2. Ensured adaptability: Tags and customer searches constantly evolve, and it’s time-consuming and expensive to manually update models. Wayfair can build iterative, data-centric workflows in Snorkel Flow to add or modify tags quickly and at scale.
3. Validation-in-the-loop: Snorkel Flow allows data scientists and subject matter experts like category managers and suppliers to correct model errors and adjust validation datasets quickly while minimizing the manual efforts of their human agents.

Results:

1. +10x faster development: compared to traditional HITL workflows.
2. +20 points accuracy: compared to supplier baseline.
3. +7% “view read,” a leading indicator for add-to-cart rate.

Learn more

If you'd like to learn how the Snorkel AI team can help you develop high-quality LLMs or deliver value to your organization from generative AI, contact us to get started. See what Snorkel can do to accelerate your data science and machine learning teams. Book a demo today.