Six weeks ago, I received a call asking if my wife and I wanted to become a Nielsen family, allowing the ratings company to track the TV and radio programs we watch and listen to. We gave it a little consideration and then said yes; it would provide us a chance to “vote” for the shows we typically watch and, we assumed, it would be something easy and transparent to do.
The company explained how it would provide us with tracking devices we would wear that would detect which programs we watched throughout the day. They worked by listening to an audio signature broadcast by each show. We also would be paid up to $50 per month. It seemed simple and straightforward.
A few days later, we each received a package that contained the device we’d carry with us. It was surprisingly bulky and archaic looking. In fact, it looked similar to one of the old SkyTel pagers from the mid 90s with its belt clip and small display. It came with a charging cradle and AC adapter, where it was to be put back each night to charge and to communicate the day’s results to Nielsen using its built-in cell modem.
We were told we’d need to wear the devices everywhere we went through the day, even when we left our home. Another module was provided for each of us that plugged into an outlet to detect when we were gone.
When the pager was put into the charging cradle, we’d be awarded points based on how long we wore it during the day. These points were converted to chances to win $15,000 in a drawing. In fact, nearly all of the company’s communication motivating us to wear the device was predicated on earning more chances to win their lottery. Little was communicated about the value of providing them and their clients with accurate viewing information.
We both took our responsibilities seriously, although we could care less about the drawings. Our focus was to be sure we used the units as much as possible. After all, with perhaps 40,000 households determining the content of TV, our choices of what we watched could help some of the programs we enjoyed.
But after a few days, my wife started getting calls and emails from Nielsen reminding her to wear the device more often. She often didn’t wear the pager, just placed it next to her on the couch when she watched TV. She had no belt to clip it on or large pockets to carry it in. But that created a problem: after 30 minutes of detecting no motion, its light would start blinking, requiring her to move it to keep it engaged.
Clearly, with its large belt clip, the device was not made for her nor for most women to conveniently carry. She surmised the product was likely designed by a man with little thought given to making it more convenient for women to use. I had to agree and wondered whether this might even bias the ratings.
It was also inconvenient for her to carry the device whenever she went out of the house. I could clip it to my belt for the day and forget about it but that was not an option she had.
Soon we learned that every time we went away for a few days, we would need to let Nielsen know by calling them and using a touch-tone phone to provide the start and stop dates, much like cancelling a newspaper delivery. Even though we did this, one of us would sometimes get a call after we returned asking why we weren’t using the device. After more than a month of being a Nielsen family, we each received a $7.50 check for our participation.
We finally decided participating was just too complicated and time consuming: the charging of the device, carrying it everywhere we went from waking up to going to bed, the incessant emails and calls, and signing off and on when going out of town.
What was so surprising to me was, in this era where there are so many technically innovative products being developed, the Nielsen solution was so archaic and technically deficient, particularly when it impacts the shows we watch. You have to wonder how accurate their ratings are when their measuring methodology requires so much effort.
And you would think Nielsen could be sampling a much larger and more diverse population using crowdsourcing through the use of an app on a smartphone, rather than using such primitive hardware. It would know exactly where we were at all times, could communicate the information to Nielsen’s cloud in real time, and we’d be more likely to have the phone with us all the time. Clearly this is an opportunity for another company to do a much better job.
Based on this experience, when I now look at Nielsen ratings, I have a lot less confidence in their numbers.
That’s funny. I also find participating in simple surveys very enlightening w/ similar conclusions: badly designed, badly sampled and proctored… Even political surveys with a stable subject, million-dollar budgets, and decades of track record are amusingly (or frighteningly) wrong; one-off surveys with a few K budget and a clear agenda are mostly ridiculous. Even more enlightening is being not simply one of the surveyees, but one for the surveyors: do they know what results they want, and not care about validity…
And then you get into not simply surveying, but forecasting stuff, and the situation gets insane. Last week, a forecast about phablets was making the rounds, that stated right on the chart it was statistical extrapolation, yet nobody (not the journalists/bloggers, and very few readers) stepped back with the mandatory “Wait, what ?”. That gets us the famous (and strangely hard to find) forecast about how MS and Nokia would quickly put paid to iPhone.
Re: Nielsen specifically, either they don’t care enough to come up with a bearable wearable (ah !) or they feel they’ve got a handle on the skew introduced by their current monstrosity, and are leery of modifying the skew by making it less monstrous.