Marketing teams like the predictability of an annual tracking study, which can become a content-marketing tentpole to plan their editorial calendar around. Before being acquired by Salesforce, Evergage conducted an annual survey for seven years about real-time marketing and personalization, generating coverage on an ongoing basis.
Journalists like tracking studies, too. “I prefer surveys that compare results to earlier periods because they are more than a snapshot in time,” said Michael Fitzgerald, long-time journalist and editor. “They are also conducted with a little more objectivity: you can say what you want to say with certain questions but now you have to at least be consistent.”
One of my mentors, Ray Boggs, the small business research guru, told me when I was in my twenties: “You don’t sneeze on a tracker without messing up the trends.” He worked hard to maintain the consistency of his annual tracking studies, worried that even small changes could distort year-over-year comparisons.
But small changes are impossible to resist. Sometimes large ones are as well.
The Only Constant is Change
I once had a client that set a gold standard for migrating a monthly tracker. Note, as a sign of their seriousness, that they had previously divided the year into a 13-month calendar of four weeks each, to avoid changes that could be explained by February having 9.7% fewer days than January, for example. It came time for a major methodological change. For this transition, each “month” they adjusted the sample mix, gradually decreasing the proportion allocated to the old methodology in favor of the new methodology.
That level of rigor is unusual. Despite the concerns, we’ve found most trackers to be robust enough to withstand changes. In fact, for many topics we’re surprised at how stable results are quarter after quarter, year after year.
Changing Existing Questions
Consistency is key, but having questions be 100% identical for each subsequent survey isn’t warranted.
Sometimes questions have to change, because the market has changed. Pity our long-suffering client tracking streaming usage: take the unicorn Quibi, which didn’t last the year; “CBS All Access” is now “Paramount+”; it’s “HBO Max,” no, it’s “Max,” no, it’s “HBO Max” again. Will it be “Paramount Max” next? And so on.
On AI usage, one of the “Other (please specify)” responses in our tracking study of students went from rare to commonplace and raised its hand to be added to the choice list. In fact, we flag new choices in our trend analyses to make it clear they weren’t asked in prior waves.
Adding choices to all-that-apply questions can be especially fraught. That’s because these act like “some that apply” questions for respondents, and with more choices, survey takers only select the most relevant choices that apply, rather than “all that apply” as instructed. So adding more choices results in lower percentages of respondents selecting each choice.
When it’s time to make changes, we might add two new items but retire the least-chosen item from the prior year, to minimize disruption.
But sometimes you just have to abandon trending a specific question, if it has had to change too much to stay up to date.
Changing Who’s Interviewed
As one of our clients grew rapidly, their US study mushroomed to be a global study. For trends that first year, we just trended the US results.
Another client gradually moved upmarket, with their annual study changing from those in firms with 100+ employees, to 250+ employees, to 500+ employees. We filtered historical results to match the new tighter qualifications when producing trends.
For surveys of specific lists, whether client lists or lists of opt-in panelists, you might think you’re keeping it constant. Unfortunately lists are like rivers. You can never wade in the same waters twice: people have unsubscribed; new people have flowed in. So the list isn’t exactly the same, even if you don’t think you’ve changed who you are targeting.
Adding and Retiring Questions
Part of the art of a tracking study is having a mix of static, trendable questions as well as topical questions that address current issues of the day. Some of our clients rotate topical sections back in on a biennial base, but most topical questions come and go, a select few working their way into the annual core. Typically you lead with the core questions then follow up with the topical questions.
The paradox at the heart of any good tracking study: change just enough to stay relevant, not so much that you lose valid comparisons. That’s nothing to sneeze at.
Author Notes:
Jeffrey Henning
