Can SC-iAT be made optional

Question

Greetings,

I am including three single category IAT tests (SC-IAT) in my two day study. My IRB insists that all questions be optional but I have not found a setting that allows an SC-iAT to be skipped. Also, on whatever page I put an SC-IAT question on, the "next" and "back" buttons disappear. Is there an easy way to have the buttons appear at least until the space bar is pressed, or do I have to put a button on a prior page that will allow them to skip over the IAT page.

Answer 1

Is there an easy way to have the buttons appear at least until the space bar is pressed

You can use the following JavaScript code:

<script type="text/javascript">
window.addEventListener("load", function() {
  window.setTimeout(SoSciTools.submitButtonsDisplay, 500);
});
</script>

Or you can use the PHP function buttonCode() to add another button "Skip this test" to the page. This, however, would encourage the respondents to skip the question.

Comments

Thank you for the very fast reply.  Unfortunately neither solution worked.  The Javascript code created next and back buttons on the page.  

The 'Back' button worked perfectly but the 'Next' button did not advance to the next page.  Instead it seemed to just reload the existing page.  

Similarly when I added a button using the php code I could not get the page to advance to the specified page.  I took the code from an example of buttonCode in the documentation. It looked like the following:

html(
  '<div style="text-align: center; margin: 2em 0">'.
  buttonToPage('IAT 1, 'Skip Practice').
  '</div>'
);

Am I missing something or is there some issue with the SC-IAT question that somehow interferes with navigation ?

---

> The 'Back' button worked perfectly but the 'Next' button did not advance to the next page.  Instead it seemed to just reload the existing page.  

Please open the section containing your IAT question and make sure that the question does not demand a response (orange or blue box).

---

Thank you for the information.  I did not realize that the setting for responses could be set at the section level.  I have now tried it under all three conditions and found the following:

1.  Setting all the SC-IAT questions in the section to demand a response prevents anyone from advancing the page.  Unfortunately it does not display a dialog box demanding an answer as is typical for other types of questions.  Users may find this confusing as pushing the next button results only in a reloading of the page with no further information.

2.  Setting all the SC-IAT questions in the section to probe for an answer works correctly.  When the user pushes the next button a dialog is displayed saying that "One or more questions on this page remained unanswered." and offering a check box with the text "I do not want to change my answer".  I will use this option as it is exactly what I want.

3.  Setting all the SC-IAT questions in the section to not demand an answer does not work.  It behaves the same as setting them to demand an answer.   This will be very confusing for users also.

Overall I have four recommendations in decreasing order of importance.
1.  When the option is set to not demand an answer be sure that the next button will move to the next page and not reload the current one
2.  When the option is set to demand an answer be sure to post a dialog so that users know that they must answer and are not confused
3.  Allow the options to be set at the question level.  Currently SC-IAT questions display "N/A" (I assume meaning not applicable) in the box where the option would be set and do not allow any changes.  
4. Do not remove the next/back submit buttons from pages with SC-AIT questions by default.  
5.  Change the text in the dialog used when the option is set to probe for an answer.  Currently it says next to the option "I do not want to change my answer" as opposed to "I do not want to answer" as it does for other types of questions.

Thank you again for your quick responses and good advice.  This is truly an excellent platform and I love using it.  I am very happy now that I have an option that works.

---

ad 1) good point, this shall be changed in a future version.

ad 3) That is very interesting ... I will check what's wrong there.

> Overall I have four recommendations in decreasing order of importance.

Taken into consideration and made some new change requests for future version. However, I shall note, that 99% of the survey working with IAT simply do not allow to skip the question (it is often a central measure, and data without it is lost for analysis anyway). Therefore (and given a temporary workaround) we will only priotize these changes with medium priority.

---

This is a new issue but it is so highly related to the prior test that I thought it best to mention this as a reply.  I have a participant doing my study who has gotten stuck on my SC-IAT question.  It will not let them move forward and instead is constantly refreshing the page.  They sent me this link which shows the issue:

https://www.soscisurvey.de/DecisionSimularionSummer2020/index.php?i=LNZVCYK5AHCF&rnd=OOAK

They are totally stuck and would like to complete my study.  Can you advise ?

Thanks..

Incidentally, I am a former software product manager and I probably would have chosen the same course of action that you did int examining my issues.  But for this issue now I remain very happy with your support. You do a great job.

---

Thanks for the flowers :)

It will take me a bit to resolve what's wrong with the above case. But (A) if you accept the data as is stored at the moment - valid or not - and it is important that the respondent can skip to the next page soon, and (B) you can identify the CASE, for example by data given from the respondents (STARTED and LASTPAGE), then use the feature "Collected Data" -> "Restart an Interview Case".

This will provide you with an URL directing the respondent to whatever page you want.

---

PS: The problem probably is that you set the question to probing (instead of demanding an answer or ignoring a missing answer at all). In "normal" questions this means that a warning will be displayed if the answer is missing, and the respondent has to click "okay, I don't want to answer" (or similar).

As you (by now) know, the IAT was not really planned to accept missing answers. But it is designed to resume the procedure, if the data is sent after half if the test was completed, for example. I guess (am am quite sure on that) that the problem is as follows: The respondent gave too many wrong answers, rendering the IAT result invalid. SoSci Survey now recognized an "invalid answer" annd follows its instructions for them, given that "probing" is enabled. That means: It asks if the "response" shall be changed. At the same time, it recognizes that it has been completed, and automatically resumes to the next page (or tries to).

So I recomment disabling the probing, and generally allow for a missing answer in your case. That shall also be sufficient to allow the respondent resuming the interview. If I am wrong please let me know!

And, of course, this is a point for the TODO list. By the way ... do lists of bugs and requests for improvements grow larger and larger in "normal" software companies as well (SoSci Survey is not the normal case in many aspects)?

---

Thank you for the very quick reply.  Losing one data point on the IAT is not a problem for me so I will try to identify and restart the case.  Unfortunately I had already tried not demanding an answer and it did not work.  I think it resulted in the behavior we are seeing now where the page reloaded and would not allow the participant to proceed.

Also unfortunately, my IRB insists that every question be optional.  I can vary or or not provide payment but I can't force the participants to answer any particular question.  That was part of the feedback I received in the IRB ethics review so I really have no option.  

In answer to your other question, in my experience the lists of bugs and improvement requests do in fact grow larger and larger over time. It is life in software development.  You are in now way unusual in that respect.

---

> Unfortunately I had already tried not demanding an answer and it did not work.

Oh yes, I forgot that we had observed some non-consequential behavior with the IAT regarding the non-demanding options. Stil on my (growing) list... and thanks for the other answer :)

---

My apologies for brining this up again but a total of 5 people now have complained to me about this issue. This is out of a total of 250 or so possibilities but it is quite likely that many others are seeing it and have just not emailed me.  I was able to use the workaround for one of them but I don't think that everyone is comfortable giving me enough info to fine their case. I am also seeing a bunch of surveys that seem to be stuck on the first IAT page.  Unfortunately, I don't think it is as simple as people putting in lots of wrong answers as some people have gotten past the page without answering or having made many mistakes.   Is there any other advice you can offer to help people avoid this ?  Are there any ideas on when it might be fixed ?  I am hoping for a sample of 1000 so if I can't resolve this there will be lots of problems.

---

Yes, I also see a lot of error messages about "incomplete" IAT data in my server's log. I have this on my schedule for this evening or tomorrow. Most likely I will completely disable any response-checking, because it makes very little sense for the IAT questions...

---

So, I was able to replicate the problem of getting stuck, if I used the back-button after doing the SC-IAT. The problem actually was a mismatch of data after the "back" ... a bug that is solved now.

The problem of not accepting the question being skipped is not yet solved (you are using the "probe" setting for that). But the issue of an eternal loop after using the back button does not occur any more. Also (and this is where the bug actually was a substantial problem) no more IAT data is lost when using the back button.

Thanks for having brought the problem up again, because otherwise this bug would not have been found.

---

Thank you very much for the rapid attention to this issue.  This is really good news.  I have been launching the study in batches and was worried about launching more. I will keep going and let you know if I get any more complaints.

---

Yes, please. This issue was probably not dicovered earlier because the SC-IAT is the IAT variant used most rarely on SoSci Survey... I will see to take care of the "skip with no reminder" issue today and very much hope that no other unforseen problems disturb the data collection.

---

Unfortunately I am still getting reports that people are having this problem.  Four people have emailed me about this today the most recent at roughly 4:00 pm Central European time. One person seemed to have gotten past the issue by refreshing the browser.  If more people send me emails about this I will suggest that and see if it resolves it for them.  Any further advice is welcome.

---

Alright ... as I promised, I have taken care of the response checks for the SC-IAT. Skipping the question should now be possible, the SC-IAT does not handle results that we encode as invalid (-8) as invalid in the meaning that a value was sent that cannot be. This actually used to be an incorrect confusion of meaning.

That means two things: (1) After we had solved the automated reload-loop, I now strongly hope that nobody will get stuck in a manual loop any more. (2) It should be possible to disable probing for the question, and just set it to volontary.

I also found out, that these issues were specific for the SC-IAT, and did not occur (as far as I can see) in the IAT or BIAT... Please let me know if you volontary SC-IAT now works properly. And my apologies for the trouble involved with your survey.

---

Unfortunately, the problem seems not to have gone away.  I have had two reports of the problem this morning.  One of them was able to get around it by disabling an ad blocker and then reloading the page.  He also reported getting an error about javascript being required even though it was already enabled in his browser.  After the reload, however, the test worked.  I downloaded the data and looked at all the cases started after your last message reporting the fix and saw lots of cases stuck on the page with the SC-IAT test.

---

> One of them was able to get around it by disabling an ad blocker and then reloading the page.  He also reported getting an error about javascript being required even though it was already enabled in his browser.

Huh - that sounds like a dfferent issue. Possibly the Ad-Blocker recognized the IAT's JavaScript as advertising ... to my knowledge, that did not happen before.

> saw lots of cases stuck on the page with the SC-IAT test.

I would not exclude the possibility that doing such a relatively long and monotonous test causes people to drop out.

What about the other case that told you he or she had a problem? Did they disclose any details?

Btw.: Did you try and set the question to "no response check" in the respective section? I do not think that this is related to the problem, but it may be that somebody does not see the "probinge" message on the top of the page, if the question is skipped in probing mode.

---

I am pretty sure that this is the same problem.  The first person reported getting stuck on the page and seeing it reload every 5-10 seconds.  This was in google chrome. He also tried it on firefox and saw the same behavior - page reloads every few seconds.  In chrome he was able to use the browser to stop the page.  He just kept trying after this and I think one of the things he tried was disabling the ad blocker.

The test is only three minutes long so I don't that explains this level of dropout.  

The other person did not disclose any details.  Most of them do not.  These are Amazon Turk participants so I have limited contact with them.  Mostly they send an email with a short message such as I am stuck and please help.  

I have not changed the question at all. I would prefer not to make changes unless there is a good indication that it will solve the issue.

Incidentally, I don't think the back button was the only source of this problem but it may have been one cause. I am definitely seeing fewer data sets with -8 as the code.  For the 47 people (out of 54 possible) who's last page was one of the two first IAT tests (page 10 or page 11)  at noon today who started after 10pm last night I am seeing data for the bulk of them.  It looks more like they started and then when the test was over or somewhere during it started this cycling behavior.

---

> Mostly they send an email with a short message such as I am stuck and please help.  

If you are okay with that, I would screen the database for like such cases and try to hijack their interviews in order to test the (mis-)behavior? If the problem remains and I can replicate, I shall also be able to solve that.

---

I am ok with this as long as there is no possibility that the data will change.  The IAT data is a mediator in my study.  It is important but I can survive the loss of it if I have to.  If I lose the rest of the data or if it is corrupted in anyway it will delay my dissertation.  Obviously, I would like to avoid that as I am hoping to defend this academic year. In the meantime, I will slow down my launch rate.

---

I have more information.  5 people have asked me for the workaround where I restart their case.  I went into the data base and looked at the web pages displayed for those cases.  For 4 of them the screen for 1 or both of the first IAT tests had an error message telling them to please respond more quickly (which is a common error) and also a note about javascript being required and needing them to activate javascript and reload the page.  This includes a case that occurred after the most recent fix  The cases, FYI, are 702,801,2197,2459 and 2786. The study is Decision Simulation Summer 2020.

My guess here is that there is some issue with the IAT test that triggers a Javascript issue in some cases where people do not go fast enough.  The odd thing, however, is that more people seem to get stuck on the second IAT vs the first which is the practice one.  That data is inconsistent with my theory but I thought it was worth mention anyway.

---

Thank you for the CASE numbers. Actually, I was able to replicate the issue already with the first case (of course, I would not dare to change the data set in any way ... I am a researcher myself when I am not working for SoSci Survey).

The problem is the probing, as I have already described above. If the test was completed but invalid (limit of wrong responses exceeded) than SoSci Survey categorizes that as missing answer. Consequently the page is displayed again, but with the probing information. But at the same time, it is submitted, because the test was completed..

Long story short: Repeating the page after doing a test does not make sense. The test was completed and the result was invalid, that's it. So I went one step further than before and have SoSci Survey categorize an invalid result as valid answer (I elaborated on the mismatch of meaning for "invalid" above).

That prevents deadloops when the SC-IAT returns no valid result. And at the same time, it allows you to probe if people just want to skip the SC-IAT.

The good news is, that only those respondents have been trapped who did not do the SC-IAT properly (error rate > 20%). This is not in the sense of allowing them to skip the question, but something I can reconcile with my conscience.

---

Thank you for working on this until you found a solution. It is also good to know that the data would have been invalid anyway for the people who got stuck.  I hope that you will never hear from me again except to say thank you !

---

So I am sorry to report that I do not think this bug is fixed, in fact it may be worse.  I had another report of this issue this morning. The person sent me the following links to show the issue:
https://www.soscisurvey.de/DecisionSimularionSummer2020/index.php?i=I2ZU6SZBNLV2&rnd=UTZU
https://www.soscisurvey.de/DecisionSimularionSummer2020/index.php?i=I2ZU6SZBNLV2&rnd=QJFL
https://www.soscisurvey.de/DecisionSimularionSummer2020/index.php?i=I2ZU6SZBNLV2&rnd=YXJN
https://www.soscisurvey.de/DecisionSimularionSummer2020/index.php?i=I2ZU6SZBNLV2&rnd=PDMJ

When I saw this, I went into the data and looked at every case that was started after 11:00 pm on September 10, or roughly 30 minutes after I received your email reporting the most recent fix.  173 of a possible 283 are stuck on page 10 or 11 where the SC-IAT tests are.  That is over 60%.  

Given this data, I have taken your earlier advice and reset the IAT tests to not probe and simply allow no answers even though that was how the bug was first discovered.  I am hoping that the changes will allow that case to work.  

This is causing real problems for me.  If lots of people on Amazon Turk agree to do my study but can't complete it and thus don't get paid they will likely rate me low on Amazon. This could hinder my ability to use the platform for studies in the future.  Also, this is slowing my data collection down considerably.  Lastly, Amazon forces me to give them money to hold in reserve for all of the tasks I create. Since people aren't completing them this money is sitting in limbo and I won't be refunded until I can show that the tasks won't be used which requires various date limits to expire.  Hopefully this will work now that I have made the change but if it doesn't I really need a good working fix.

---

I just opened the URL (they are all the same except for a random string to prevent browser caching). I saw the IAT, then clicked on "next" and then saw a page

> Thank you for filling out these surveys. You are finished for today.

That may be because you have changed to non-probing. I could also set the interview one page back to replicate the issue.

> 173 of a possible 283 are stuck on page 10 or 11

Very strange ... I did several tests and was not able to provoke such a problem. Could you possibly download the question as XML (feature in the top of the question) and send this XML to info@soscisurvey.de? Then I can test this with your specific question...

I would, if you all right with that, also retrieve the IAT raw data from the case that you have linked above, and use this to test with your question.

> This is causing real problems for me.

I feel really bad for this any can only repeat my apologies. What is the current state without the probing? And more important: Is the IAT data in the cases that got stuck and what is the page time TIME*** stored for these cases? Are these people who actually did the test or did they just try to skip it?

---

Thank you again for you quick reply  The news is good.  Since I made the change to remove the probe, no one has written me to complain about getting stuck.  I probably would have heard from 1 or 2 by now.

 I went back to my downloaded data from yesterday and found the average time people spent on page 11 among those who's max page was 11.  It was much lower than I thought at only ~255.  The standard deviation was high however at roughly 508.  There were about 10 cases who spent 600 seconds or more on the page.  I am guessing that many fewer were stuck than I thought and so I am much less concerned today than I was yesterday.  I have an interrupt page on page 12 and so my assumption was that it would be the max page for anyone waiting on the interrupt.  My guess is that I had that wrong and that max page is 11 except for those who already got to the next page (page 13) which happens 18-24 hours later, typically.  If that is right than the vast majority of those 173 people were not stuck. It probably wasn't more than a dozen.  

Nevertheless I have mailed the XML of the question as per your request.  This is, however, the version with the probe off.

---

Thanks for sending me the question's XML file. I went into the database to CASE 4019 that you have sent me the links for.

And yes, I was able to identify the problem. The SC-IAT used a heuristic that any case with less than 4 valid trials in the last block (time > 350 ms and correctly answered) seems lacking data and was probably not even started.

But, really ... now I am quite sure that no useful data got lost. And I am also sure that nobody got stuck in the loop who did not earn that. Of course, this is not how research ethics work, so this (hopefully last) issue is also fixed now. The answer is either accepted (20+ trials of data sent, no matter if valid or not and counting all blocks) or there is no automatic re-submitting of the SC-IAT and more.

So, if you like, you could now re-enable the probing feature for the question. I assume you do not like, so this is only meant to be an update about the issue :)

---

Thank you very much for tracking this down and for the detailed explanation.  My data gathering is nearly complete so I won't change things now but if I use the SC-IAT again I will likely go back to the probe.  Incidentally I received no more complaints since my last post.