Background
In 2018 I spent 6 months as a contract product designer on Facebook's People Experiences team.
Facebook needed a way to identify bad actors on their advertiser system. Buyer feedback seemed like the best way to spot them, but the survey wasn't producing enough data. Leaving Facebook and advertisers in the dark.
My task was to design UX flows within the Facebook ecosystem to both identify bad actors as well as help the good actors better understand their customers. The first part of this goal was to fix the survey system.
Discovery
The old survey asked about shipping speed, product quality, customer service all on one screen, all in one go. With only a 2.8% completion rate, it was obvious this was too overwhelming for the users. On top of that, Facebook’s lack of data here meant that a scammer and a good advertiser looked practically the same.
The question became how to get more useful data from people who only give us a few seconds?
Challenges & Opportunities
Challenges
- Spotting bad actors takes a lot of responses, and the survey was losing nearly all of them
- Advertisers needed to see their own standing, and Facebook needed to see who to remove, from the same data
Opportunities
- Most of the survey's value was in answers we were already throwing away
- Free text could show patterns in what buyers actually said, which multiple choice can't
Ideation
I kept the questions mostly the same but changed how we asked them. Originally we asked 5 questions all on one screen and all questions needed to be answered before submitting. I knew it was asking too much and decided we needed to break it down. Eventually I settled on a single question approach. One question on the screen at a time. 3 answers to choose from. We now store the data as soon as a single question is answered. Then the last question was a text field they could write whatever they’d like about the company.
For the advertisers, I designed a dashboard where they could see their satisfaction metrics from the surveys. This included a word cloud (answers from the text field input) that would give quick signal beyond the metrics from the survey.
Results
What shipped
- A one-question-at-a-time survey with a free-text finish
- A word cloud built from buyers' own words
- An advertiser dashboard showing satisfaction metrics and what customers said
Results
Completion went from 2.8% to 19%.
The People Experience team now had a way to identify bad actors and remove them from the platform.
"Working with Scott saved our team last year. We underwent a reorg, had to shift priorities, and focus on many new projects. Our core product would have been rendered stale had it not been for him. Although remote, you could barely tell, he worked with our team as if he were here. I'd definitely work with Scott again given the chance."
Product Manager