Home-made online surveys can make a monkey of us all

Online survey: clear, friendly, responsive

Responses from questionnaires are stored in our databanks

Pupils and students can still complete on paper, if you prefer

Questionnaires are fetched by secure courier and logged in the SHEU offices

Questions are composed collaboratively with clients and designed with the pupils and students in mind

Home-made online surveys can make a monkey of us all

I expect you are familiar with the admonishment from TV programmes, "Don't try this at home!".  What you are watching is for professionals, who know what they are doing, but if you try it, you might make a mess of it, and that is dangerous.  If you're watching Mythbusters or Brainiac, making a mess of it might involve some nasty burns at the very least.  But what about survey work, surely that's harmless enough?  Well, let me share some thoughts with you.

I was prompted recently to take part in an online survey hosted by SurveyMonkey.  The first question went:
 

Gender You Identify With:

Female
Male
Intersexed
.......

Now, lots of points must be awarded for considering people who do not identify as male or female, or who are not biologically male or female.  Yet here there is not only confusion of gender identity and biological sex, but I am pretty sure intersex people don't refer to themselves as 'intersexed'.   The very next question then asked for "sexual orientation", and offered as an option "transsexual"  -- a gender identity.  Not a good start.

I was shortly asked for the city and state where I lived.  If they wanted only US residents, they should have said so.

Later in the questionnaire we have:
 

When did [event] first start?

0-5
7-9
10-13
etc

A shame if it happened at 6, eh?

Online surveys allow designers a lot of extra gadgets, like filtering certain people into or out of a set of questions depending on their previous answers.  There was only one question that was compulsory -- where the computer would not allow you to continue if you didn't answer --
 

4. When you were [X] (assuming you were) did you ever feel [Y]?

Yes
No

Well, I don't think I was ever [X], and even if I was, I'd like an option to say 'Not sure'.

And further on, almost unbelievably:
 

If applicable, please explain how your partner affects your sex life.


And there was a large box for a reply.  By this time, I could only manage to type "accomplice" through laughing.

Now this was a survey written by someone who cites themselves as "Dr." and a colleague.  I don't know if they are a real medical doctor, or, like me, a Ph.D. (phony doctor).  It seems they thought they were perfectly well qualified to write their own survey.  Who knows, they may even have been on a course.  But the catalogue of errors and inaccuracies highlights some of the dangers of doing home-grown surveys. 

Dangers?  Surely this at most a nuisance?  Well, I'm concerned not just for this chap, whose subjects may not be answering as he would expect, but also the rest of us.  Every poorly designed questionnaire and every suspect report has the potential to reflect badly on the field as a whole. 

And, sometimes the most powerful weapon you have to create or resist change are the results of research.  I have in mind, for example, the work we did with Stonewall about people's experiences of homophobic bullying in schools

If confidence in social research is undermined, powerful groups or influential people will be more able to act as they want, ignoring the available evidence.  We have too much of that going on already, and so I would argue that faulty research can genuinely be harmful if it makes it harder for good research to have its say.

If you would like help with your own online survey, we can support with any or every part of the survey process: study design, question composition, survey hosting, results analysis, reporting and dissemination.  We have been doing online surveys at schoolsurveys.co.uk and healthsurveys.co.uk since 1994, and can not just offer all the technical facilities that you might want (filtering and branching, random allocation of questions, etc.) but also a bank of established questions and decades of experience in designing questions for young people.
 

Online Every Child Matters survey   Online Lifestyle Survey
Online FE student survey   Online Parent Survey

Contact me if you think we can help.
 

Author: 
DrDave