Even big survey companies create bad surveys
Today’s question makeover is special because it comes courtesy of a heavy hitter in the survey world: Qualtrics. I’ve been using the Qualtrics platform since 2012 — that’s over a decade! And, honestly — I love it.
That’s why it’s extra disappointing to see them put out low-quality survey questions. I mean, one of their services is helping people with survey design. This doesn’t exactly instill confidence.
Below is the question that Qualtrics shared to LinkedIn — that’s right ya’ll, public humiliation:
Now, I get why they are interested in this question. Qualtrics has been pushing its “employee experience” platform features lately. However, I don’t get why nobody on their team stopped whoever hit “post” on this question.
Any thoughts on why this may not be a good question? By “good,” I mean it isn’t a question that will get Qualtrics information they can use in marketing to make the case for employee experience surveys.
Let’s take a look.
Good survey questions are clear. When you read them, you should know exactly what you should think of or reflect on when you answer. This question is not clear. Because the word “asked” is super vague. Should you be thinking about a casual conversation? Formal reviews? A survey? An interview? You catch my drift? No good.
Good survey questions also have no overlap in response choices and cover all possible response options. Some people call this “MECE” (mee-see): mutually exclusive and collectively exhaustive. I call this common survey sense or “make sure everyone’s response fits somewhere and only in one somewhere.”
Ya’ll this question had me flabbergasted. This is QUALTRICS. Are they trying to be cute on social media? Don’t they know that bad survey questions (and therefore bad data) are never cute?
Let’s change it up for them:
Can you see the difference? The reworked questions more clearly state what one should reflect on when answering and have distinct answer choices that don’t overlap.
I’ll be fully transparent with ya’ll — I agonized over this rewrite a bit. For questions like this, I much prefer to discuss with a client what response choices make sense (e.g., if they don’t have HR send surveys, that wouldn’t be an option!).
I also don’t love splitting out the “when” and the “how” — a more efficient question would cover both simultaneously. Perhaps asking: When was the last time your employer asked you to take a survey about your experience as an employee? But alas, I erred on the side of being more all-encompassing because when I read Qualtrics’ original question — I couldn’t tell exactly what they wanted to know and I don’t know the inner workings of employee experience in corporate America. Were I working with them one-on-one I have no doubt we could make this question (or set of questions) even better!
On the one hand — I hate to see a question like this from a titan in the survey industry. On the other, it’s a good reminder that when it comes to survey design there is always room to learn and grow. Nobody is immune to asking bad questions.
If you want to get these makeovers and more in real-time, sign up for my weekly newsletter here.
