The trouble with Nielsen, comScore, and sample-based measurement
Brian Stelter’s article “Hulu Questions Count of its Audience” in today’s New York Times does a great job of illustrating one of the central tensions in the world of online audience measurement: sampling vs. what I like to call comprehensive internal measurement or CIM.
No matter how you try to spin it, Nielsen and comScore are at a disadvantage. Both were born in a different media paradigm where sampling was really the only way that you could get good audience numbers. For one thing the media were largely broadcast which meant that there was no technical way of identifying the exact amount of time an individual watched or listened to a given program or channel. It was natural, however, that the industry came to accept standardized metrics when they were introduced because measurement both enables the capitalization on successful programming investments AND the ability to carefully plan development spending based on established market trends (or the appearance thereof).
Today however, advertisers and the established measurement firms are trying to apply these older measurement paradigms to a medium that in some ways defies standardized measurement (I have spoken about this at length in my recent lectures and hope to expand upon in the blog in due course). But more importantly – why would the advertising industry still rely on audience sampling as the primary means of measurement when most large websites now collect data on EVERY consumer of their content? More and more online content creators are finding ways to identify patterns in online audience behavior that are based on comprehensive sampling. This paragraph says it all:
While Nielsen reported 8.9 million visitors to Hulu in March, another measurement firm, comScore, counted 42 million. Exacerbating the confusion, Nielsen’s numbers for April show Hulu losing audience while still managing to add video views, also know as streams.
At Socialesque, we have developed a straightforward video analytics tool that can not only tell you how many viewers there were, but also how far into the video they watched, how often they clicked on various buttons, and even at what points they clicked on a banner ad that is integrated into the program.
Surely Hulu has at least this sort of numbers (if not, somebody at Hulu should contact us directly, because it would make our day). But this is perhaps not the point — instead, it might be more important to remember that at any given point in time it matters as much what is accepted as “truth” as what can be measured.
If they are wise, the big online content providers will start to push back against this piece-meal sampling by the established measurement firms. If we have the capability to do comprehensive internal measurement, which we do, then the focus would better be spent on figuring out how to implement open standards on this measurement so that people can trust the numbers coming out of firms. But I say sampling be damned, it is a tired remnant of an older media model that is no longer relevant.