Multifactorial vignette experiment with blocks and randomised order of vignettes and attributes

Question

Hi,

I'm looking to run a multifactorial survey experiment with a lot of randomization in SoSciSurvey and find myself a bit lost.

I have a factorial with 8 attributes with 2-4 values each (2x2x2x2x2x3x3x4) resulting in a total of 1152 vignettes. To be able to look at interactions I need to run the full factorial (all vignettes).

A vignette may look like this:

Please rate how likely you would be to buy a car with the following
attributes:
- Expensive
- White
- Automatic
- Four wheel drive
- No seat warmer
- Two doors
- Little storage space
- Electric

The vignettes are divided into 128 blocks of 9 vignettes each, with respondents being randomized to blocks.

Additionally, the order of the 9 vignettes should be randomized between respondents.

Finally, the order of the 8 attributes should be randomized between respondents.

I have already designed the blocks and vignettes and have that as a CSV file like below:

Block Run x1 x2 x3 x4 x5 x6 x7 x8
1 1 1 1 2 3 1 1 1 1
1 2 1 2 1 2 2 2 1 2
1 3 1 2 1 4 1 2 2 1
1 4 2 1 2 1 3 1 1 2
1 5 2 1 2 3 2 1 2 1
1 6 2 2 1 4 2 1 1 1
1 7 3 1 1 4 3 2 2 2
1 8 3 2 2 1 3 2 2 2
1 9 3 2 2 2 1 2 1 1
2 1 1 1 1 1 3 1 1 1
...
128 9 3 2 2 3 1 1 1 2

My first though was to use a random generator to divide respondents to a block between 1 and 128. But if I were to follow the instructions for randomization per block I would then need to create 128 additional random generators filled with the specific vignette combinations as shown in the guide for multifactorial experiments.

I feel like there should be a cleaner solution with php to first randomize to decks, then randomize the order of the attributes (example to x6x3x4x7x1x8x2x5), and then randomize the order of the vignettes (second column "Run"). This would then be three randomization per respondent, after which I would need to actually show these vignettes to the respondents. Alternatively, the order of the vignettes could just be a random draw without replacement if I can do it within the block only, and keeping the order of the attributes.

Any help is very much appreciated!

Comments

Sounds like you're facing some fun :)

> The vignettes are divided into 128 blocks of 9 vignettes each, with respondents being randomized to blocks.

First question: Are you sure you want to have 128 "fixed" blocks or would it be evenly (or even more) useful to randomly draw 9 out of the 1152 vignettes?

> the order of the 9 vignettes should be randomized between respondents.

If you decided to randomly draw 9 vignettes per respondents, this would not be an issue at all.

Second question: How many respondents will you have?

> I would then need to create 128 additional random generators

Thanks for seeing that this is a bad idea before doing a night-shift on creating those random generators :)

And also thank you for not overwhelming your respondents with a crazy number of vignettes per person. 9 may still annoy one person or another, but ther eis hope that a substanial part of the respondents will still read this number.

---

Thanks for the response!

> First question: Are you sure you want to have 128 "fixed" blocks or would it be evenly (or even more) useful to randomly draw 9 out of the 1152 vignettes?

Yes, it would be much easier to fully randomize but then I might get confounding between main effects and interaction effects. As part of the focus is on higher order interaction the blocking is a necessity.

> Second question: How many respondents will you have?
About 1500, so with each rating nine vignettes, each unique vignette is rated about 11 times.

---

> As part of the focus is on higher order interaction the blocking is a necessity.

Good, I just wanted to be sure that there's a valid plan :)

Next Question: How many questions do you want to ask for each vignette? And which question types? This is important to decide if you should create 8 pages, work with loopPage() or better work with a multi-level-structure: https://www.soscisurvey.de/help/doku.php/en:create:multilevel

---

>Next Question: How many questions do you want to ask for each vignette? And which question types? This is important to decide if you should create 8 pages, work with loopPage() or better work with a multi-level-structure: https://www.soscisurvey.de/help/doku.php/en:create:multilevel

We have five questions for each vignette. The first is a scaling slider 1-100, so question type slider, the other four are likert scales of 1-7 which I think is question type "Scale (extremes labeled)".

Answer 1

We have five questions for each vignette. The first is a scaling slider 1-100, so question type slider, the other four are likert scales of 1-7 which I think is question type "Scale (extremes labeled)".

Okay, in my personal opinion, a multi-level-structure is the better solution.

I would recommend the following procedure:

1. Give each vignette an ID (v1 to v1152) and

2. Write a table with this key (v1...) in the first colum and the vignette data in the subsequent colums:

   v1 1 1 1 1 2 3 1 1 1 1
   v2 1 2 1 2 1 2 2 2 1 2

3. Import this table (you can use an Excel file) inthe the database for contents.

4. Create a random generator with 128 rows, each having the IDs of the above vignette blocks, that is

   1=v1,v2,v3,...v8
   2=v9,v10,v11,...v12

Again, use a script or excel table to do that.

1. Create another random generator (for the vignette order) with only the numbers 1 to 8. Tell this random generator to draw 8 of those 8 codes per interview.
2. Now, you draw one block from the first random generator, use value(..., 'csv') to split that into an array, then
3. Draw the 8 codes from the second random generator an use some PHP to sort the array from above according to that order (you could also use shuffle(), but had to take care about page-reloads then).
4. And store these 8 values into 8 internal varibles.

Now you have 8 variables with vignette numbers from one block, randomly sorted in the 8 variables of a "internal variables" question. Now comes the multi-level part. Use a loop to go through the 8 values.

Where do you need the "database for contents" now? You give the vigentte code (v1) to the subordinate questionnaire, make it look up the proper line in the database, and show the vignette accordingly - along with your questions. I recommend writing the vignette values in another internal variable to make your analysis easier.

And should these trivial eight steps (<- irony) pose any troubles on you, just ask :)

Comments

Thanks for the extensive answer!

I have managed to assign respondents to blocks, randomise the vignette order, and additionally randomise the attribute order. However I'm a bit lost at the multi-level part, even after looking at the guide.

I use a code like below to loop over the vignette Ids, and send that together with the attribute order (in an array) to the subsurvey.


foreach ($keyVar as $Var) {
    // Add key to attribute order array
    $orderAtt[] = value($Var, 'free');
    // Start multilevel survey, sending attribute order and vignette key
    multiLevelDown('sub', $orderAtt);
};

However, while it runs on the subsurvey, when you reach the last page of the subsurvey it sends it back to the next page of the main survey rather than to the same page with the loop. Is there a command i'm missing here?

Additionally, I'm a little bit unclear on how I send the response data from the subsurvey back to the main survey.

---

You're very close. It does not yet work because the multiLevelDown() leaves the PHP code, and therefore, the foreach and the page at all. Use loopPage() instead:

$Var = loopPage($keyVar);
// Add key to attribute order array
$orderAtt[] = value($Var, 'free');
// Start multilevel survey, sending attribute order and vignette key
multiLevelDown('sub', $orderAtt);

> I'm a little bit unclear on how I send the response data from the subsurvey back to the main survey.

Usually, you do not need to do that. You have it stored in the dataset, one row per vignette.

---

Thank you very much for the help, that seems to have done the trick!

Maybe this should be a new question, but when the vignette is shown you continuously need to scroll up and down to see the text while answering the questions. Is there any functionality that would allow for the text to stay on the top half of the screen so that respondents are only scrolling through the questions?

---

Yes, that should be a new question, because it's a completely different topic :)

The answer ist, yes, you can use CSS to show the image at a fixed position. Try something like this:

<div style="position: sticky; top: 40px;">
    <img src="myimage.png" alt="">
</div>