Attention checks necessary?

Question

Hi,

is it necessary to use attention checks like reverse coded items to make sure that participants actually read the questions? I forgot to include it into my questionnaire, but could still add them as I have just pretested it so far.

Best

Answer 1

That depends on a lot of factors. Finally, you must counterweight two things:

1. How likely is it that a respondent will click through your questionnaire, giving meaningless responses?

2. What will that do to your data? Will this cause type I or type II errors?

If you have a lottery for the latest iPhone, you'll definitely need attention checks. If it's a 40-minutes questionnaire that nobody will do, except when they is highly motivated, ... well.

Using inverted scale items (this is not really an attention check) has another (important) use: It will remove a general bias of people to agree oder disagree. However, the data looks like if you has a worse Cronbach's Alpha in your scales.

There actually is not soo much literature on data quality and attention checks. What you'll probably find is the IMC (works quite good, but also identifies "good" respondents who are just a bit unattentive), and maybe you'll step over my working paper on response times. The truth is, that it becomes more complicated, when you check for data quality and attention. This reveals problems in survey data, that many researchers prefer not to know about. If you'd like to make it right, then take good care -- but, honestly, you won't do yourself a favor.

Comments

Thanks for the insights! It should actually find ist application in my undergraduate thesis. Would your recommendation (obviously no imit or warranty from your side) still hold true in this context, or may my Approach be too Basic without the checks?

---

Well, most undergrade theses do not care too much about data quality (they are still engaged with all the other stuff). However, results suffers equally from poor data quality, no matter if it's done by a BA or PhD candidate.