You work for the training-and-development department of a midsized company in the Midwest. You're not crazy about a particular vendor's software product that you use to help manage your company's training procedures.
Then one day you get an e-mail from a survey company, asking you to complete a confidential online satisfaction survey about your vendor's software product. You're told that customers of your vendor's competitors are being surveyed as well, and that a list of customer-satisfaction rankings will be published.
Bingo! Payback time! Now you can let your honest reactions surge forth. Your bosses will see how poorly your current vendor's product stacks up against the other software on the market and, quicker than you can say "flush my cache," they'll be dumping the old vendor.
That's exactly what happened to my reader C.P. The trouble was, her bosses never got to see how poorly their current software vendor did because, when the results were published a few months later, that vendor didn't show up in the rankings.
C.P. called the survey company to find out what was going on. The company told her that the vendor in question had chosen not to participate after seeing the survey results -- in other words, C.P. surmises, her vendor didn't want to see its name at the bottom of the published list.
"How valid is a survey if those ranked at the bottom can choose to bail out?" C.P. asks. "How fair is it to the vendor's customers, who took the time to respond, to not see the final results?"
Part of C.P.'s anger stems from righteous indignation at being misled into seeing this survey as an opportunity to prove to others -- and particularly to her bosses -- that her company's current software vendor isn't cutting it. And of course the survey company isn't in business to give C.P. and, presumably, other dissatisfied customers ammunition for their dissatisfaction, and is under no legal obligation to publish the full results.
Nonetheless, I understand C.P.'s resentment. Allowing poorly ranked companies to remove themselves from the published list may not affect the validity of the results that are published, since the companies who fared better did so by pleasing customers who responded to the survey. But dropping the names of companies that didn't perform well suggests that this survey will never be exhaustive enough to be truly useful for customers looking to compare software vendors. If the survey company allows drop-outs, after all, why would any company that didn't finish at the top in the rankings allow itself to appear in print?
Such limited surveys may well be useful to some companies, such as those that end up at the top of the list and want to use those findings to promote themselves. However, it's not fair for the survey company to give respondents the impression that they're participating in one type of survey when in fact they're participating in another. People put their time into responding out of certain expectations, and might not want to participate if their expectations were different.
The right thing for the survey company to do, in order to be fair to all involved, is to prominently place clear and detailed information about who is paying for the survey, whether and when companies have the ability to opt out, and exactly what information the survey taker will be able to see.
But that's not what happened. C.P. was misled into believing that, if she took the time to fill out the survey, the results would be tabulated and her vendor would be ranked against its competitors. Her assumption was a reasonable one, and it was up to the survey company to make clear what was actually going on.
The fault here does not lie in the published list of rankings, though its usefulness is at best suspect. So long as the numbers presented are correct, the list as a whole is incomplete but not unrepresentative.
The fault lies in the survey company's having taken advantage of C.P. and her dissatisfaction with her vendor to involve her in a survey which, as matters turned out, was useless for her purposes. She's right to be annoyed.
2 comments:
This is not untypical; many "surveys" are thinly-disguised marketing tools. Others are just slanted and invalid as my below response to one illustrates.
"The citizens of Anaheim have been sent "hit" pieces, including a "survey" from the so called Coalition to Defend and Protect Anaheim.
This "Coalition", whose backers are unnamed but financed by SunCal, a builder with an interest in building housing in the Resort Area, misrepresents the situation and fails to disclose that they have a vested interest in the outcome of the "survey".
Suncal, a private builder, wants to build housing, and is using low-income housing in the Resort Area to get Council support. Disney opposes that: the Resort Area, only 2.2 sq. mi., was zoned to exclude housing since it is not appropriate for the Resort Area, would conflict with tourism, would unduly burden the infrastructure (traffic, sewage, et al.), is not appropriate for children (not enough schools, playgrounds, etc), and not a cost-effective area for low-income workers.
The Resort engine funds Anaheim and has been very successful under the special zoning set up for this purpose by previous City Councils. It brings hundreds of millions of dollars in to the City of Anaheim. Local business and the Anaheim Chamber of Commerce support the current zoning.
Disney has proposed a ballot initiative that would require a public vote to change zoning to protect zoning from the vagaries of Council membership -- every time a new Councilperson is elected, it could change the zoning and harm/destroy the Resort area. This is so important that the public should make this decision, not every elected Council that panders to their lobbyists. That is all Disney is proposing -- public approval! The "Coalition" wants to exclude the public.
The "survey" poses questions that slant the issue, wording the issue so that it is one-sided. The first question states that Disney has filed a lawsuit to stop building homes on private property. That is misleading. Disney has filed action to prevent a zoning change to the long-term tourism zoning in the Resort Area. The third question asks if the reader supports building of new homes for local families, but that is not the issue. The new homes affected are only in the Resort Area and, in any case, these homes would not be reserved for local families (that is probably illegal). The Survey incorrectly states that Disney broke their agreement. One question characterizes Disney as restricting private property rights; that is a zoning issue and ALL zoning issues restrict private property rights. Do the Survey backers support NO zoning? Perhaps so since a private builder backs this survey. Another question asks if the reader believes that the City Council should "stand up to the Disney Corp." assuming a fact not in evidence.
Pandering to low-income families desire for housing is not in their best interest. Restricting tourism growth will restrict resort income, and jobs, which low-income people need. A private builder, for his own pofit, would actually cause loss of public funds (taxes) which would harm low-income families! Anaheim should not harm the golden goose. Some Council members think Disney should be in the housing business. Does that mean that ALL Anaheim businesses must be in the housing business, i.e., provide housing for their employees? This reeks of Socialism.
As a survey developer, I find it dispicable when politically motivated "surveys" are slanted to get the desired result. This is an unscientific disservice to the public. I wish there were room to show in detail how this "survey" is invalid; as a mathematician, I am offended that the public is being tricked by these "Coalition" (builder) charlatans and their stooges."
Arnold Vagts, PhD
avagts@gmail.com
(714) 533-6544
Anaheim, CA
CP's expectations were unfulfilled because they were unrealistic to start with. The survey company promised her nothing in exchange for her opinions. She -hoped- that she would be able to use the final survey to her own ends but they didn't promise her anything.
Allowing a vendor to opt out of published survey results after reviewing the results does introduce inherent bias to the results or at least limit the usefulness of the results but this, in itself, is not unethical. It does however reflect on the reputation of the surveyer, which was never mentioned. This smacks of "there are lies, damn lies, and statistics". Consider the source.
I'm not convinced that CP didn't actually get her unrealistic expectations fulfilled though. She was seeking evidence to present to her boss that their vendor's software was crummy. The survey alone didn't give that to her -BUT- the survey coupled with the surveyor's comments does give her the ammunition she was looking for, although perhaps not in four-color glossy.
Post a Comment