When is a survey not a survey?

It has been said (by whom, nobody's quite sure [1]) that there are three kinds of lies: "Lies, damned lies and statistics."

Surely, survey results of dubious objectivity (SRDO) must fall in there somewhere. People love surveys because they sound scientific, but it's easy for them to come out biased, especially with people who aren't trained in writing surveys. And because they're so common, it's a good idea to learn how to recognize SRDO when you see them.

This week's happen to be from Sitecore [2]. Not meaning to pick on them; it's just that they happened to be the company that sent out this week's SRDO.

"Developers, marketers, interactive agencies and IT professionals cite Search and Web analytics as key applications of an overall Web Content Management System (CMS) strategy, a recent survey conducted by Sitecore, has revealed," the company said in a press release. "In the survey of more than 100 decision makers, Sitecore found the number of organizations implementing Web analytics rose to nearly 90 percent."

The first question is going to be who that other 10 percent is, obviously, because what in the world are they doing with their website if they're not implementing Web analytics?

Aside from that, the first sign of SRDO is vagueness about the survey population. "100 decision makers," what is a decision maker? How many were surveyed to get this result? What sort of return rate did they get? Two percent is common, meaning Sitecore had to find 2000 decision makers to get 100 responses. How were the decision makers chosen? The reason these are important questions is to avoid survey bias. As much as possible in a survey, one wants to ensure that there is a random sample, or the answers can't really give an accurate representation. As it happens, the surveyed decision makers are from Sitecore's customer database, according to a company representative.

There's more vagueness. "[U]nifying Web Content Management, analytics and marketing automation has become a top priority to yield faster conversions. In addition, respondents noted that integrating Customer Relationship Management (CRM) systems and Web CMS software is a top priority," says the release. "Key applications," along with "top priority," "top drivers" and "key components" are all included in this press release.

Is a "top priority" more or less important than a "key application"? How were these terms defined in the survey? How are they different? Did the decision makers choose these terms and options themselves, or from a multiple-choice list?

The reason these are important questions is, again, to avoid survey bias. The way a question is phrased can influence the answers people provide.

That said, the results of the survey are interesting:

"Top drivers for implementing a Web CMS solution include: Improving productivity (19 percent), improving customer service (11 percent) increasing sales leads (10 percent) and building global brand awareness (10 percent)
73 percent of survey responses indicated online Search as the top third-party application with Portal (33 percent) and CRM (20 percent) following behind
More than 33 percent of respondents plan to integrate online video management as part of their 2010 WCM initiatives
More than two-thirds of respondents use LinkedIn the most for professional use with Facebook and Twitter closely behind
A majority of the respondents (52 percent) stated that Web Forms are key components of building an interactive website"

It would be interesting to see the entire results of the survey, and not key findings. Unfortunately, it doesn't appear to be on Sitecore's website.

For more:
- see the press release [3]