Speaker: ‘Kiwimeter’ is a methodological car crash and I still can’t look away
88 Responses
First ←Older Page 1 2 3 4 Newer→ Last
-
Sacha, in reply to
Thanks for commenting here, Peter.
-
In such a Semantic Differential scale, shouldn't there be a middle/neutral point akin to 'Neither agree nor disagree'? A non-committal response is a valid response. If UoA academics were involved in this, they are bloody incompetent.
-
Sacha, in reply to
a national broadcaster doing it essentially for entertainment
bet you they are doing it for business
#ad-targetting -
Sacha, in reply to
Likert scales force a choice. Standard practice.
-
Andrew Robertson, in reply to
I guess that is possible, but I really really doubt it.
-
There are arguments on both sides as to whether semantic-differential or Likert scales should have an odd number or an even number of options.
The former allows a neutral midpoint, which may well be a valid response, but can also be a popular choice if respondents are tired, or don’t want to think about the question (which is not the same as being neutral), or don’t understand the question.
The latter forces respondents to read and think about the question and choose an opinion in one direction, even if only by a slight margin; but it removes the ability to signal true neutrality on a single item. However, neutrality may still be allowed to emerge by including two opposite-polarity items asking for the same information. -
Sacha, in reply to
there is a trend away from using reverse scored items.
thank goodness
-
linger, in reply to
>#ad-targetting
I guess that is possible, but I really really doubt it.
It's hard to see any other meaningful purpose to the exercise, though.
-
linger, in reply to
Reverse scored items have their purpose. It is still recommended that at least one opposite-polarity item be included in each set of items attempting to measure the same thing – as an internal check that respondents are reading the questions rather than blindly choosing the same option, and as a way of minimising the inherent bias arising from forced choices (see previous comment). But wording such items takes some care, as they shouldn’t be made harder to understand than the corresponding positive statement.
-
Sacha, in reply to
90% of TVNZ's income is advertising.
why the disbelief? -
This whole project seems like a classic case of a failure of research expertise and oversight of the whole enterprise, from development to delivery, start to finish.
So you're saying the manager deserves a promotion then, right?
-
Sacha, in reply to
neutrality may still be allowed to emerge by including two opposite-polarity items asking for the same information.
which does not work if some items are removed.
#oops -
Am more excited in any case by the prospects of this project led by my ex-neighbour Dr Carla Houkamau.
-
Sacha, in reply to
Interested to hear what Carla thought of that question.
-
Geoff Lealand, in reply to
I think a neutral mid-point is essential, for it can include a range of possible motivations, such as 'this question is too complex to allow me to answer it' or 'this is a really stupid question' or 'I don't have an opinion on this issue' to 'fuck off with your dumb research' etc.
I blame management schools for bad research such as this, with their neglect of ethical considerations. -
I think too many people are taking the "research" angle of the Kiwimeter at face value. It surely is a simple promotional device for TVNZ and and any research value is very much a distant secondary priority. I bet the whole special treatment for Maori question is intentionally designed to provoke the reaction it has received. That's marketing not research. False forced choices never appeal in questionnaires of any kind.
-
linger, in reply to
a neutral mid-point is essential, for it can include a range of possible motivations
That's exactly why it's dispreferred: by design, it provides a response that is not easily given one single interpretation. (That said, a neutral option can be offered when those other competing responses are less likely: e.g. on uncontroversial topics, in short questionnaires that are easy for their respondents.)
-
Russell Brown, in reply to
I bet the whole special treatment for Maori question is intentionally designed to provoke the reaction it has received.
Well, perhaps not exactly the reaction it's received, but certainly a reaction. The difference between the preserve and Kiwimeter questions certainly suggests that.
-
Weird… I was in the middle of replying to Tinakori myself. What we do know is that the question was intentionally worded that way to mirror the phrasing of talkback discussion – which probably is intended to provoke reaction. So, yes, kind of, but not in that direct a fashion.
-
Okay wow, the Director of Vox replied, and confirmed that he does not accept the validity of any feedback from Maori so far, about their decreased likelihood of filling in the survey on ideological grounds, because feedback is "anecdotal" and "sampling on the dependent variable". He does not seem to understand that *any* qualitative evidence of groups selecting-out of the survey on these grounds is a problem for the survey, *precisely because* it cannot be quantified. This is like a stereotype of a quant guy who does not understand the purpose of qualitative research in the context of survey design, i.e. that often a survey will be useless without it. Also he does not seem to want to acknowledge the effect that the media coverage is also likely to have on selecting out of the survey. Also tries to cover himself by saying that cognitive testing is not common in "academic research", I guess this is why he hadn't heard of it oh dear. I'm getting the feeling that this guy is a freakin' amateur.
-
Tze Ming Mok, in reply to
The points made by Peter Davis are essentially what I covered in my first post on this matter, which is that we need to study the prevalence of racist attitudes by, inevitably, testing racist statement in surveys. I am not suggesting here that a research ethics committee needs to intervene into/block research like this; but that in this case the researchers did not think as hard as they normally should be expected to, about the likely impact of the reduced schedule of Kiwimeter questions .This is actually likely because they relied on their 'usual' approach despite the delivery of the survey being far removed from the context of say, the way the NZAVS is presented to respondents. These are duties researchers have to themselves and their own standards, not necessarily something that needs policing by an external REC.
-
linger, in reply to
One might politely enquire as to how the percentage of Maori respondents currently compares to the population proportion.
-
I saw the Kiwimeter survey on facebook, so I naturally assumed it was going to tell me that my "Kiwi IQ" was in the top 4% of the world and by the way - here is some more click-bait that might appeal to your narcissism. It didn't work on Firefox so I didn't persevere.
I guess all surveys are self-selecting to some extent, but I still have an issue with the sort of things I come across at work. My response is always the same: "Put your graphs and statistical models away. You got a 12.1% response rate. All we know is that those people are the freaks, the outliers. Stop pretending that this is now "evidence" for "evidence-based policy". -
Tze Ming Mok, in reply to
You would think that would have been their first line of defense if the numbers looked right. However, I suspect (just a hunch, not based on my experience of online self-selective surveying in NZ) they would be needing to upweight Maori participants in the first place, for any online survey.
-
A few thoughts: one of the things that seems clear here is that there are quite clear disciplinary expectations on how things should be done (including some I'm inclined to flip on myself, such as reverse scoring).
Reverse scoring — totally a classic move for the reasons previously outlined (agreement bias, check people are awake etc.), but also disadvantages those with lower English proficiency, and they still tend to cluster together as their own factor.
Having a mid point — I'm strongly in favour of having a midpoint (unless, Rasch, see later). Forcing people to have an opinion where they are actually neutral leads can in paper surveys to people circling between the two (creating their own midpoint), and people stopping the survey in electronic.
And we haven't really talked about clustering, which is less relevant for the main argument under discussion here, but are there really genuine clusters, or is the data pretty evenly scattered? In which case plotting in 2 or 3D space would make more sense, rather than applying inaccurate arbitrary boundaries.
So back to Rasch. As the NZAVS statement alludes, using adaptive Rasch or Mokken scaling would have been a great well to test this domain of interest. In essence, you start out with a statement on the midpoint of the scale, which is relatively uncontentious (e.g., Cultural diversity enriches New Zealand society). If you agree with that statement, you start to be shown progressively more, well progressive statements (e.g., New Zealanders need to do more to make up for past treatment of Māori.). Whereas if you disagree with the statement, you start see statements heading in the other direction ((e.g., Claims about Māori discrimination are exaggerated.), perhaps culminating in the "special treatment" question. This confers a couple of great benefits. (1) People don't have to answer as many questions and (2) People are less likely to see statements they don't agree with. This also swings both ways. It's not surprising that the special treatment rankles with many, but there is also the real potential to get non-completion with questions like Claims about Māori discrimination are exaggerated. I understand that the reason why the census ethnicity question reverted to using "New Zealand European" is due to the backlash/write-in responding when "Pākehā" was used (c. 2001).
Finally, in my nosing about today, the special treatment question could be a local adaption of "Irish, Italian, Jewish, and many other minorities overcame prejudice and worked their way up. Blacks should do the same without any special favors." (Symbolic Racism Scale; Henry & Sears, 2002)
Post your response…
This topic is closed.