Randomization of text stimuli

Question

Hello, this is my first time using SoSci.

In my 'experiment' i want to make participants answer whether they think a small text is written by a human or AI-written, and how certain they are of their choice.
I want to display 1 text at a time, maybe 10 in total (from a pool of many different text examples) , so that each participant randomly gets 5 human-written texts, and 5 ai-written texts.

In the data collection, besides getting the answer of the participant, i also need a column on whether the text they answered to was actually human or AI. I need this to do analysis about the accuracy of participants guesses later on.

How can i do this?

Answer 1

In the data collection, besides getting the answer of the participant, i also need a column on whether the text they answered to was actually human or AI.

Sounds like you want a multi level design. This will allow you to already store the data in one-row-per text. Of course, cou can collect the answers in a "normal" questionnaire as well, and then convert from wide- to long-format in your data analysis.

What you'll need, in any case, are two random generators. One with the IDs of the AI texts and one with the IDs of the human written texts.

To store the texts, you can either create a lot of texts in the list of questions, or you can use the database for contents. If we're talking about >20 texts, I recommend the second one.

For each random generator set to draw 5 codes per interview. And finally join the drawn codes via valueList(), array_values() and array_merge(), shuffle them, and then present them along the question(s).

Comments

Thank you for your very quick response! I have some follow up questions that i hope you can help me with. I have now created 2 random generators ,RG01 (for ai texts) and RG02 (for human texts), and set them each to 5 Codes drawn per interview. I have also uploaded a test csv in the database for contents. The csv has a key column with values A1-A20 (for ai texts) and HU1-HU20 (for human texts), a column containing the title of the story, and a column containing the actual text.

Where do i join the draws? Is that on the 'Ballot for the Random Drawing' , or on a page (using php code) in the questionnaire. Do you maybe have some example code on how to do it?
Thank you so much!

---

> Where do i join the draws?

If you mean to combine the 2 x 5 draws from the two random generators, it might look like this:

if (!isset($draws)) {
  $draws = array_merge(
    array_values(valueList('RG01', NULL, 'label')),
    array_values(valueList('RG02', NULL,'label')),
  );
  shuffle($draws);
  registerVariable($draws);
}

And let's say that the (text) contents of your random generators is A1 to A20 and HU1 to HU20, you can easily show all the contents.

$textID = loopPage($draws);
text($textID);

This is "flat" and you'll have to add some code for the multi-level-solution, but in principle...

---

Thank you, it now display 10 pages and then ends the questionnaire. But i am unsure on how to call and display the text data from the database for context. I figure i should use dbGet() and dbSet() but how?

And also, is it possible to just use the put() function to store the key for each text shown? Instead of doing  multi-level-design

---

If you have stored the texts in the database for contents, and their keys in the random generator, try this one

$textID = loopPage($draws);
$data = dbGet($textID);
html('<p>'.$data[0].'</p>');

> And also, is it possible to just use the put() function to store the key for each text shown?

You'll need 10 internal variables (let's call them IV01_01 to IV01_10 for the example) and a code such like ...

$i = loopPage(count($draws));
// Show the text
$textID = $draws[$i];
$data = dbGet($textID);
html('<p>'.$data[0].'</p>');
// Store the text ID
put(id('IV01', $i+1), $textID);

---

Hope that i understand correctly, but by storing the text in the database for contents, and their keys in the random generator, do you mean to upload the csv file to 'database for content' and to write A1, A2, ---- , A10 in the 'Ballot for the Random Drawing' for the ai random generator (and same procedure for human-texts)

I did that, and also made 10 internal variables like you wrote. When i run the last code you suggested, i get this error:

Warning (page 4)
The function dbGet() requires a valid key, but instead got %key%.
Questionnaire Error (page 4)
There is an error in the PHP code:
Questionnaire Error: Trying to access array offset on value of type bool
line: 14

PHP code

011
012 $textID = loopPage($draws);
013 $data = dbGet($textID);
014 html('<p>'.$data[0].'</p>');
015 return 'ok';
QE01 !

---

> The function dbGet() requires a valid key, but instead got %key%.

Seems to be an invalid error message - thank's for the bug report, we'll take care of that. That, however, won't fix the problem.

For debugging, please add debug($draws); before the loopPage() line, and report the debug information.

---

Okay, i did that and the debug information says:
[Information]    Interview number 39 is to be continued
[Information]    Questionnaire qnr2 from project test388761 will be used
[Information]    The retention period on page 4 has been additional 22 seconds
[Processing]    Reading answers from page 4
[Information]    Answer to QE01 = 1
[Information]    Answer to QE02 = 3
[Information]    100% of the questions have been answered so far
[Information]    According to the previously set page order, page 4 [] is shown
[Processing]    Create page 4 in questionnaire qnr2
debug()    Content: 0[false], 1[false], 2[false], 3[false], 4[false], 5[false], 6[false], 7[false], 8[false], 9[false]
[Information]    loopPage() i=3
[Information]    The internal variable DA01_04 gets a value of [false].
[Content]    Create question QE01
[Content]    Create question QE02


How do i go on from here?

---

> debug()    Content: 0[false], 1[false], 2[false], 3[false], 4[false], 5[false], 6[false], 7[false], 8[false], 9[false]

Okay, this looks wrong. To me it says: Something is not working in this PHP code:

if (!isset($draws)) {
  $draws = array_merge(
    array_values(valueList('RG01', NULL, 'label')),
    array_values(valueList('RG02', NULL,'label')),
  );
  shuffle($draws);
  registerVariable($draws);
}

However, I do not know what you made of this code. You included the two random generators somewhere on the page, and put a code like above below them, did you?

---

oh, i did just now, and i now see the the second column in my csv file which is the title for the story which is great! 3 questions:

1. How can i display both the title for each story (which is there now) and the story itself below (which is the third column in my csv file?
2. How can i do such that for each page shown the type and likert scale questions become 'blank' when you click next page? (right now, it has saved the last response and the boxes of the last response are ticked)
3. Is there anywhere i can see the file output of a finished response of the questionnaire? I would like to have a look at how the rows/columns are structured, and what variables are saved.

Thank you very much!

---

I figured out nr. 1 myself. However i would still like to know:
2. How can i do such that for each page shown the type and likert scale questions become 'blank' when you click next page? (right now, it has saved the last response and the boxes of the last response are ticked)
3. Is there anywhere i can see the file output of a finished response of the questionnaire? I would like to have a look at how the rows/columns are structured, and what variables are saved.

Thanks so much

---

> How can i do such that for each page shown the type and likert scale questions become 'blank' when you click next page?

That's an important one, because if it does not, then you show the same question over and over again, and respondents will delete their previous questions. Literally.

So, if you have 3 copies of the questions, let's say:

$questions = [
  1 => ['QE01', 'QE02'],
  2 => ['QE03', 'QE04'],
  3 => ['QE05', 'QE06']
];

the show another pair like this:

$i = loopPage(count($draws));
...
show($questions[$i][0]);
show($questions[$i][1]);

> Is there anywhere i can see the file output of a finished response of the questionnaire?

Look in "collected data" -> "view data".