Am I missing something in my questionnaire?

Question

Hi,

I am writing my undergraduate thesis and have finished my literature research and formulated my hypotheses. Next I gathered scales (from well ranked papers) that have already proven to be applicable.
But I am not sure, if I am missing something general, e.g. "do I have all manipulation checks or covariates necessary?", or "am I missing something entirely?". Can s/o give me a "hands-on"-source, which I can rely on when building my questionairre? Something like a construction kit or so.

Thanks in advance!

Answer 1

Depending on the perspective, you will always miss something ...

And, of couse, an experimental design requires a different questionnaire than a representative descriptive design.

If you'r about to test some hypotheses, the most important things - in my personal opinion - are that you have (1) your research question written down, (2) you have written all of your hypotheses written down, (3) for every hypothesis, you have written both concepts written down that the hypothesis connects, (4) you have written definitions of each concept, and (5) you have appropriate items and questions in the questionnaire to measure the concept.

Comments

Thank you for the reply!
Can you elaborate on (3) and (4)? This is actually sth that gives me a headache as well. One hypothesis of me is "Consumers perceived uncertainty will be higher if confronted with a highly-innovative product compared to a less innovative one". Therefore I apply scales from papers on that topic, but in this case it is just like one scale for innovativeness and one for uncertainty.
On the other hand when I read papers on diferent topics, the authors mostly divide these "broader" categories into several subcategories. Is it a sign of me being to shallow in my approach, as you e.g. mention in (4) that one should specify the definitions (and underlying concepts)?

---

If I understand your hypothesis correctly, then concept A is the uncertainty, that the respondent feels at the moment when confronted with a product. If you present a product in the questionnaire, then you can catch that moment - otherwise, you must rely on memory.

The concept B of your hypothesis is the product innovativeness. I assume that you'll manipulate that in an experimental setting, because your hypothesis is not about the perceived innovativeness, but on the actual innovativeness. That means, you may require a scale for the manipulation check, but not for the concept itself.

> On the other hand when I read papers on diferent topics, the authors mostly divide these "broader" categories into several subcategories.

If your hypothesis does not distinguish between different dimensions, you won't have to measure them separately. However, one criterion for validity is completeness. That means that you measures must cover every relevant aspect of the concept. And if concept A has 7 different subdimensions (which are probably highly correlated), you'll need at least 7 items.

---

Yes, you described my plan exactly right. It is only one of my hypotheses, but they are all very similar. In another e.g. I want to measure the willingness to buy the presented product. Actually I think that uncertainty is the mediator here and willingness to buy the dependent variable, but have to find out if that changes anything for my model.
So maybe you can give me a last advice on how I sum the insights up in my experimental design.

1.) I present the stimulus (in one case a high- and in the other case a low-innovative product).
2.) Participants fill out my questionaire with willingness to buy as the dependent variable, perceived uncertainty as the mediator, maybe a scale for personal preference for innovations to account for this as well and at the end a scale for perceived innovativeness as my manipulation check.

Is that ok so far?

And maybe on additional question: Let's say I find out that uncertainty is composed of the subcategories X, Y and Z and I implement a scale with 3 items, one for each subcategory. And now I have a case, where X, Y are confirmed by the participants, but Z is not. What does that mean then? Has the person perceived uncertainty?

Again thanks a lot for the help. Especially the advice to disassemble the problem has given me a much less "foggy" view.

---

The questionnaire itself does not distinguish what role a variable will take in the model. So I cannot give you an "okay" - it's up to you to ensure you can test your hypotheses based on the data that you collect. Make sure that you have written down *what* each of your concepts means (a short definition) and then double-check that the questions and items in the questionnaire fit that definition. Also make sure that you exactly know what you will computer when you have the data (which indices, which analyses).

> And maybe on additional question

When doind a quantitative experiment, you should know about the dimensions previously. Or better said: You should define it before collecting the data. There is no definition of how many dimensions a concept really has (actually, a lot things you hear about factor analyses are not true). If you find out that your items correlate not as strong as expected, than you will always have a problem. And in that case, you will be happy if you used more items. Because Cronbach's alpha is very sensitive to the number of items (you may want to read this one-pager: http://www.dominik-leiner.de/alpha.pdf).

There's one golden rule in collecting quantitative data: Shit in, shit out. So make sure that you assembled your questions and items in an appropriate way. If one or two items do not fit the main component of a concept, throw them out. If nothing fits, it's too late. Using tested scales reduced the risk of collecting trash data, but oft tested scales do not fit the reserch object or may contain obvious weaknesses. If was that simple to make a perfect study, everyone could do it - an we wouldn't have to learn this for years ;)

---

Ok thank you. I am a pretty sure that I will miss a lot, as e.g. uncertainty is defined by many authors in different ways (in different circumstances) an I am just applying a scale that has been used in a similar situation. But I guess thats fine..

Actually I tried to enter my scales in SPSS to calculate Cronbachs Alpha now. I am a little suprised that all the tutorials start a little further down the road and do not cover how to enter de items in the first place. I found one that shows how to enter items like "whats your gender" but am not really sure if I can apply all the concepts for my scales.
Is it just the same, like declaring a name and initiating e.g. 1=not at all etc. and I am good to go? Explanations for calculating alpha are actually fairly easy to find/use.

---

If you read the Cronbach's Alpha PDF I have linkes above, you already know that Alpha is nothing but a correlation coefficient (or rather an average coefficient). As soon as you have data, it's just as simple as computing a correlation in SPSS: You select the variables (items) of the scale and tell SPSS to calculate.

Of course, you must ensure that your data set knows about missing data, and that items are revesed if necessary. If you marked the inverted items in SoSci Survey, that's already done.

---

I am a little confused. I was told that I have to create a questionaire and calculate Cronbach's Alpha in advance. Maybe that's the reason, why I can't find any tutorials that fit my current situation :D

When you gave me the link to that PDF, I actually wanted to reply that, regardless of my limited knowledge in statistcs, it is incomprehensible to me how one can calculate the internal consistency without any data gathered so far.

---

Well, I guess, there was a misunderstanding. In general, you cannot calculate (anything) without any data. And you will definitely need some data (better a bit more) to compute Cronbach's Alpha.