SEARCH WITH PHOTO
A visual search tool that leverages a deep learning model to detect shoppable matches in any photo our customers upload
SEARCH WITH PHOTO: A Brief History
"You know what would be cool? If I could search your site with an image I found." That's a phrase we've heard time and time again
Search with Photo was the first feature on Wayfair to make use of our Visual Search algorithm. It was originally designed as a showcase of the algorithm's capabilities. However, since its release it has become a go-to tool for customers in the know. While visual search still isn't a widely know feature, those who do know rely on it to quickly and easily find exact or very close matches for the items that they're looking for.
Search with Photo wasn't always a great tool. When I first took over the project, it was cumbersome and slow to use. There were extraneous steps that forced the customer to painstakingly crop the photo they uploaded, and we were seeing high drop off rates.
However, a breakthrough from our data science team gave us the opportunity to change this. Archi Mitra and Will Lamond, our data scientists, had just developed a new algorithm they called Object Detection. With the advent of Object Detection, users wouldn't have to crop their photos anymore, because the algorithm could identify individual items within a larger photo. With this new framework in place, I set out to design an interface that would cut out the extraneous steps, and simplify the process of getting from upload to results. Here's the finished product:
UPDATE: Class Filtering
Insight:
Repeatedly in user testing sessions, we heard users wonder aloud why the rug they were looking at was producing results for wall art, or why a coffee table they uploaded was showing end tables. There's a simple answer for this: the visual search algorithm occasionally picks up multiple product classes when it focuses on an item. However, we weren't showing customers what classes Through this testing, we ran into a question:
When the Visual Search algorithm picks up on multiple product classes, how can we let customers narrow in on the right one?
I focused on interactive class filters as a non-intrusive way to let our users narrow down. The filters should be there for the customer when she wants them, but not interfere with her browsing.
During an ideation sprint, I produced the three above options. The first option showed all detected classes as active filters, which the customer could "X" out of to stop showing that class. The second used Wayfair's standard filter component. The third showed all detected classes as inactive filters, and let the user click one to filter down to just that class.
Ultimately, we decided to go with the third option. This option provides the user with visual feedback of what classes have been selected (which the 2nd option did not) and makes selecting the desired class as simple as one click (which the 1st and 2nd both didn't).
UPDATE: Object Detection Highlight
Insight:
After the release of object detection, we didn't see as much interaction with the selection dots as we had expected. To understand why, I decided to launch a few quick usability studies on usertesting.com. We noticed that users didn't really know what would happen when they tapped on a dot.
To fix this, we decided that we needed to clarify what would happen when the dot was pressed. I worked in Flinto to design a hover state and switch animation, that my development team was able to implement.