How To Evaluate Survey Findings and Reports With a Discerning Eye

Research surveys and survey reports have become important marketing tools for many types of B2B companies, including those that provide marketing techniques and various types of marketing-related services. Many B2B companies conduct or sponsor, and they feature survey reports in their marketing programs. As a result, many B2B marketers are now alike Producers And consumers Search based content.

Survey reports can be valuable sources of information on business trends, practices, emerging technologies, customer attitudes, and a wide range of other topics. But survey results and reports may also be unreliable and/or misleading.

The availability of free or inexpensive and easy-to-use survey tools has made it easy for marketers to create and conduct surveys. Unfortunately, these tools also facilitate the design and conduct of surveys that do not lead to reliable results.

As producers, marketers obviously want potential customers to view survey results and reports as reliable and trustworthy. And as consumers, marketers are increasingly using survey results when making important decisions. It is therefore important for them to make a careful assessment of the survey reports they encounter. As consumers, it is always a good idea to treat any survey report with a critical eye because as Mark Twain wrote, “There are three kinds of lies: lies, damned lies, and statistics.”

In my work, I review a lot from survey reports. I make extensive use of survey results and other research studies when I develop content for clients, and I frequently discuss survey results in this blog. Over the years, I’ve developed a mental checklist of things to look for when reviewing a survey report.

I plan to dedicate three posts to this topic. In this post, I will discuss some of the basic things I look for when I review a survey report. The following two messages will discuss issues that can affect the validity of survey results and/or the credibility of survey reports.

start mentality

When I begin to review a survey report produced or sponsored by a business, I assume that the survey was conducted to support the marketing agenda. Having a marketing purpose does not necessarily mean that the research is flawed, but it does put me on the alert for indications of bias in survey design and/or in the presentation of results.

Survey Methodology

Many survey reports will describe the research briefly in the introductory section of the report, but a comprehensive report will also include a detailed description of the methodology used in the research.

To conduct a survey of business professionals regarding a business topic, I am looking for a description of the methodology to include at most following:

  • Sample size (number responses scan received)
  • When survey responses were collected
  • How survey responses were collected (eg online, telephone)
  • How were the potential survey participants selected?
  • If relevant, how were the respondents qualified for the survey

Respondent Demographics

Most survey reports I encounter describe the demographics of survey participants at least to some degree. When I review a survey of business experts on a business topic, I usually want to see a breakdown of the following demographics.

  • Job role/functional job
  • Industry sectors/types of companies represented
  • Sizes of the companies represented
  • Respondents’ geographic locations
Responding demographics can be important for interpreting survey results and for assessing the significance of those findings. For example, I recently published a post describing the results of a survey conducted by Gartner regarding marketing data and analytics. In this survey, 83% of respondents were in companies that had $1 billion or more in annual revenue. So the results of this survey may be very relevant for marketers of large organizations, but somewhat less useful for marketers of small and medium-sized businesses.

Use of a “representative sample”

Surveys are frequently used to capture insights about a specific group of individuals by collecting data from a small sample of that demographic. This approach only works, however, if the survey participants make up representative sample The largest population.

Survey sampling is a complex topic, and it is impossible to fully describe it in a blog post. The most important thing to remember is this: if survey participants are not a representative sample, survey results cannot be “shown” to the larger target population. In essence, the results are only valid for the group of people who responded to the survey.

Therefore, when evaluating survey results, it is always important to determine whether survey participants constitute a representative sample. When the survey uses a representative sample, the survey report should include a detailed explanation of the sampling process in describing the survey methodology.

Very few of the surveys I review are based on representative samples. Such surveys can still be useful, but they can also be misleading because some report authors ignore this limitation. A well-prepared survey report will illustrate this limitation, as Gartner did in Survey Marketing Data and Analytics 2020 Include the following language:

“Disclaimer: The results of this study do not represent global results or the market as a whole but rather reflect the opinion of respondents and companies surveyed.”

coming

In my next two posts, I will discuss other issues that can affect the validity of survey results and the credibility of survey reports.

Leave a Comment