Public Address | Cafe | Speaker: ‘Kiwimeter’ is a methodological car crash and I still can’t look away

Sacha

Mon, 21 Mar 2016 20:54:01 +1300

Sciblogs has a post about Kiwimeter.

Hilary Stace

Mon, 21 Mar 2016 17:26:32 +1300

As a member of ethics committees at various times, I like it when the research applicant comes along to the meeting. Then you can have a discussion about any unclear or controversial details and hear the researchers' justification for them, or decide how things might be amended to improve the…

Andrew Robertson

Mon, 21 Mar 2016 15:45:26 +1300

I've done five or six studies that asked similar sorts of questions. Not once was I asked by the Human Ethics Committee to change their wording.

My submission did give research justification for the statements I included.

Hilary Stace

Mon, 21 Mar 2016 12:39:17 +1300

Thank you, from a fellow ethics nerd. Do we know whether this survey actually went to an ethics committee?

I am not on a university one, but if this question came up when I was in a position to make a decision about it I would ask for justification for…

mpledger

Mon, 21 Mar 2016 09:58:19 +1300

@Andrew
I think it's going to take a pretty special question (and analysis) to separate out the people who agreed to that question but think "different treatment is ok when it balances out advantage and is not inherently special" and "any treatment I can't get is special treatment and that's…

TeAtaoTu

Sun, 20 Mar 2016 14:19:17 +1300

Hi Peter
Kicking ethics review committees is a bit rum. Especially given NZ and Australia's past use of social science to kick indigenous people, which is one, amongst other sobering reasons that we have these committees. As Chair of Canterbury University's ethics committee I think your characterisation of the committee's…

Andrew Robertson

Sat, 19 Mar 2016 06:31:41 +1300

It was a dumb question because it never defined special treatment. Person 1 could say Maori don’t deserve special treatment but believe that the Waitangi Tribunal and Whanau Ora are not special treatment but fair treatment. – that Maori should get those things because it balances out the unfairness that…

linger

Fri, 18 Mar 2016 23:14:59 +1300

Edwards does at least identify those mysterious "prominent academics"from Victoria University (Jack Vowles and Kaapua Smith) and the University of Auckland (Jennifer Lees-Marshment and Danny Osborne)Any of them reachable for comment?

Tze Ming Mok

Fri, 18 Mar 2016 22:59:12 +1300

I see Bryce Edwards's roundup completely missed my main point, which was about self-selective sampling bias posing a challenge to the validity of the results, but oh well. #nerdlife

mpledger

Fri, 18 Mar 2016 21:04:49 +1300

Academics should have academic freedom but that comes with the understanding that their work is open to criticism. And the criticism doesn't have to come from people with letters after their name.

It was a dumb question because it never defined special treatment. Person 1 could say Maori don't deserve…

Sacha

Fri, 18 Mar 2016 13:31:55 +1300

Bryce Edwards rounds up various articles about this topic. Surprisingly low level of understanding about ethnicity and framing.

Soper unironically uses the phrase "melting pot". Supposed professional researcher, a Mr Farrar, reckons Kelvin Davis can't challenge the notion of 'special treatment' because "He's an MP for an electorate where eligibility is…

Andrew Robertson

Fri, 18 Mar 2016 11:08:01 +1300

Yes, but the researcher's view is not always given as much weight as some might like, and what what has value to one person may not have value to another.

Also, in a competitive tender situation you have multiple companies telling you all sorts of things, so the final decision…

Angela Hart

Fri, 18 Mar 2016 10:58:30 +1300

Surely the researchers' role on such a panel is to be informed about the pros and cons of various methodologies and pass that information to the group. If the senior stakeholders then choose less accurate/reliable methods because of cost issues, at least they will be aware of the reduction in…

Andrew Robertson

Fri, 18 Mar 2016 09:35:50 +1300

You’d think they’d be able to see the effects of cognitive dissonance, which is rife in modern society (and politics), and want to exclude its skew to get the clearest results…

When it comes to the big research projects, the final procurement decisions are usually made by a panel of…

Ian Dalziel

Fri, 18 Mar 2016 07:35:42 +1300

pre-cogs unite!
I will often recommend and offer cognitive testing. You need to pick the right projects/clients though.
You'd think they'd be able to see the effects of cognitive dissonance, which is rife in modern society (and politics), and want to exclude its skew to get the clearest results...

Andrew Robertson

Thu, 17 Mar 2016 22:16:11 +1300

I work in the Social Research division of a NZ research company. I will often recommend and offer cognitive testing. You need to pick the right projects/clients though. It can be a really hard sell because it’s quite expensive when it’s done properly (and nearly all of the big research…

Tze Ming Mok

Thu, 17 Mar 2016 22:02:29 +1300

Case in point: If van der Linden had just Googled 'cognitive testing surveys' after I tweeted at him, this would have been the first hit:

Ipsos Mori's cognitive testing policy

Andrew Robertson

Thu, 17 Mar 2016 22:02:21 +1300

It’s expensive and slow to do this kind of surveying – it’s ‘oldskool’ in a time of wanting quick online fixes that are ‘good enough’. But I actually think it’s incredibly important – as the British election evidence and our Kiwimeter’ acceptability problem shows – that it’s worth holding up…

Tze Ming Mok

Thu, 17 Mar 2016 21:57:55 +1300

I'd say if academics look down their noses at market research, it's mainly because of the goals of the research, and possibly because there's not much quality control in the overall market. However, applied social researchers, whether in academia, research institutes or independent policy evaluation (I've been in all of…

Tze Ming Mok

Thu, 17 Mar 2016 21:48:14 +1300

Yo Andrew, thanks for bringing this up. Yep, as I implied in the original post, YouGov is the market leader in this kind of non-probability sample online polling, although their weighting and sampling of their panel is based on masses of data and their algorithm is a closely guarded industry…

Andrew Robertson

Thu, 17 Mar 2016 21:46:41 +1300

1. Social research objectives, market research method. Massively market oriented client (or at least, survey host) with zero demonstrable or stated interest in social goals.

Oh okay. In my view Kiwimeter doesn’t use market research methods.

– Self selecting sample – frowned upon by market researchers.
– Psycometrics – hardly…

Marc C

Thu, 17 Mar 2016 21:14:10 +1300

"Media organisations will want a survey with results reflecting their target disposable-income demographic, and biases in that direction are fine with them."

MSM and “bias”, oh, do you really think they are biased?

I have followed the MSM for years, and bias seems to rather be the rule rather than…

James Littlewood*

Thu, 17 Mar 2016 14:32:17 +1300

Although you could also label it as the new low in news research.

James Littlewood*

Thu, 17 Mar 2016 14:23:20 +1300

That’s marketing not research

Yeah, true. Not *even* market research.

James Littlewood*

Thu, 17 Mar 2016 14:19:02 +1300

1. Social research objectives, market research method. Massively market oriented client (or at least, survey host) with zero demonstrable or stated interest in social goals.

2. I thought that's what I said.

3. That's certainly true.

4. TVNZ certainly looks like the client. They haven't named any of the academics…

BenWilson

Thu, 17 Mar 2016 12:55:56 +1300

But of course, given that they fucked up the execution, you can’t even really call data coming from the ‘Kiwimeter’ itself ‘good’ – more like a ‘fatally compromised guess’.

It could be the "better than nothing" guess, if there really was nothing already. But there's not nothing...

st ephen

Thu, 17 Mar 2016 12:26:56 +1300

Perhaps it's more that people who hate everything about prostitution might also hate having anything to do with a mailed out survey on the topic, but will nevertheless answer questions politely when talking to the nice man on the phone. Likewise people who think prostitution is a complete non-issue may…

Andrew Robertson

Thu, 17 Mar 2016 12:19:37 +1300

Can you fill in the blanks a little – or are you just having a dig at people who think “nice pen” is a good way to bribe someone?

Postal self-complete surveys can have severe non-response bias because people can see all the questions before choosing whether to take part.…

Moz

Thu, 17 Mar 2016 12:03:56 +1300

I’d trust the results of the phone survey a lot more than I’d trust the results of the postal survey (which would have severe response bias directly related to the topic in question)

I wouldn't have guessed that, which is another reason you don't want amateurs designing these things. Can…

Alfie

Thu, 17 Mar 2016 11:50:15 +1300

Rabon Kan sums up the Kiwimeter nicely.
A giant selfie stick, Kiwimeter is an intrepid journey deep into the Kiwi heart, with you as the expert, and you as the star, but unfortunately via the colo-rectal passage.

Andrew Robertson

Thu, 17 Mar 2016 09:33:28 +1300

Hey, no worries at all.

I’m not for a second saying the Kiwimeter sample is representative. I’m just saying response rate is not the only indicator of sample quality.

For example, say we carried out a survey about legalising prostitution, in two different ways.

1. We post a self-complete questionnaire…

st ephen

Thu, 17 Mar 2016 09:21:18 +1300

Sorry, that reads as more snarky than intended. I was going for light-hearted cynicism, given that every profession can be accused of the same sort of thing.

st ephen

Thu, 17 Mar 2016 09:17:21 +1300

you can have a very representative sample with a 12.1% response rate
Purely by chance though, no? And only if it turns out that "willingness to fill in an on-line survey" is not correlated with the things you're trying to measure. I'd have thought that "willingness to fill in a…

Andrew Robertson

Thu, 17 Mar 2016 08:24:57 +1300

None of this is accurate. The best you can call it is a ‘good guess’ based on demographic weighting.

I dunno… we don’t know much about how they calculated their population estimates.

I’ve become much less critical of self-selecting surveys. Check out this one (PDF) for example. They show…

Tze Ming Mok

Thu, 17 Mar 2016 05:34:47 +1300

Final nerd point: though as linger points out, a pilot survey does not need to be a representative sample, in this case the '6 archetypes' that were based on that pilot/precursor survey, were accompanied within the 'Kiwimeter' by a percentage of the country they were meant to represent. But no…

Andrew Robertson

Thu, 17 Mar 2016 05:01:46 +1300

Put your graphs and statistical models away. You got a 12.1% response rate.

I hear this argument a lot, but you can have a very representative sample with a 12.1% response rate, and a very unrepresentative sample with a 50% response rate. The response rate is not the only indicator…

Andrew Robertson

Thu, 17 Mar 2016 04:53:28 +1300

One might politely enquire as to how the percentage of Maori respondents currently compares to the population proportion.

Not really. The methodolgy isn't designed to generate a representative sample in the first place.

With this sort of self-selecting design, modelling and adjustments are performed before results can be generalised to…

Sacha

Wed, 16 Mar 2016 21:48:05 +1300

I’m getting the feeling that this guy is a freakin’ amateur.

certainly avoided sounding like a genius on te radio..

James Green

Wed, 16 Mar 2016 21:25:55 +1300

A few thoughts: one of the things that seems clear here is that there are quite clear disciplinary expectations on how things should be done (including some I'm inclined to flip on myself, such as reverse scoring).

Reverse scoring — totally a classic move for the reasons previously outlined (agreement…

Tze Ming Mok

Wed, 16 Mar 2016 21:14:36 +1300

You would think that would have been their first line of defense if the numbers looked right. However, I suspect (just a hunch, not based on my experience of online self-selective surveying in NZ) they would be needing to upweight Maori participants in the first place, for any online survey.

st ephen

Wed, 16 Mar 2016 20:43:45 +1300

I saw the Kiwimeter survey on facebook, so I naturally assumed it was going to tell me that my "Kiwi IQ" was in the top 4% of the world and by the way – here is some more click-bait that might appeal to your narcissism. It didn't work on Firefox…

linger

Wed, 16 Mar 2016 20:41:50 +1300

One might politely enquire as to how the percentage of Maori respondents currently compares to the population proportion.

Tze Ming Mok

Wed, 16 Mar 2016 20:36:05 +1300

The points made by Peter Davis are essentially what I covered in my first post on this matter, which is that we need to study the prevalence of racist attitudes by, inevitably, testing racist statement in surveys. I am not suggesting here that a research ethics committee needs to intervene…

Tze Ming Mok

Wed, 16 Mar 2016 20:28:42 +1300

Okay wow, the Director of Vox replied, and confirmed that he does not accept the validity of any feedback from Maori so far, about their decreased likelihood of filling in the survey on ideological grounds, because feedback is "anecdotal" and "sampling on the dependent variable". He does not seem to…

linger

Wed, 16 Mar 2016 20:05:59 +1300

Weird… I was in the middle of replying to Tinakori myself. What we do know is that the question was intentionally worded that way to mirror the phrasing of talkback discussion – which probably is intended to provoke reaction. So, yes, kind of, but not in that direct a fashion.

Russell Brown

Wed, 16 Mar 2016 19:52:03 +1300

I bet the whole special treatment for Maori question is intentionally designed to provoke the reaction it has received.

Well, perhaps not exactly the reaction it's received, but certainly a reaction. The difference between the preserve and Kiwimeter questions certainly suggests that.

linger

Wed, 16 Mar 2016 19:35:15 +1300

a neutral mid-point is essential, for it can include a range of possible motivationsThat's exactly why it's dispreferred: by design, it provides a response that is not easily given one single interpretation. (That said, a neutral option can be offered when those other competing responses are less likely: e.g. on…

Tinakori

Wed, 16 Mar 2016 19:15:48 +1300

I think too many people are taking the "research" angle of the Kiwimeter at face value. It surely is a simple promotional device for TVNZ and and any research value is very much a distant secondary priority. I bet the whole special treatment for Maori question is intentionally designed to…

Geoff Lealand

Wed, 16 Mar 2016 18:58:57 +1300

I think a neutral mid-point is essential, for it can include a range of possible motivations, such as 'this question is too complex to allow me to answer it' or 'this is a really stupid question' or 'I don't have an opinion on this issue' to 'fuck off with your…

Sacha

Wed, 16 Mar 2016 18:15:00 +1300

Interested to hear what Carla thought of that question.

Sacha

Wed, 16 Mar 2016 18:08:11 +1300

Am more excited in any case by the prospects of this project led by my ex-neighbour Dr Carla Houkamau.

Sacha

Wed, 16 Mar 2016 17:59:24 +1300

neutrality may still be allowed to emerge by including two opposite-polarity items asking for the same information.

which does not work if some items are removed.
#oops

Bart Janssen

Wed, 16 Mar 2016 17:59:21 +1300

This whole project seems like a classic case of a failure of research expertise and oversight of the whole enterprise, from development to delivery, start to finish.

So you're saying the manager deserves a promotion then, right?

Sacha

Wed, 16 Mar 2016 17:57:50 +1300

90% of TVNZ's income is advertising.
why the disbelief?

linger

Wed, 16 Mar 2016 17:56:41 +1300

Reverse scored items have their purpose. It is still recommended that at least one opposite-polarity item be included in each set of items attempting to measure the same thing – as an internal check that respondents are reading the questions rather than blindly choosing the same option, and as a…

linger

Wed, 16 Mar 2016 17:50:24 +1300

>#ad-targetting
I guess that is possible, but I really really doubt it.
It's hard to see any other meaningful purpose to the exercise, though.

Sacha

Wed, 16 Mar 2016 17:47:44 +1300

there is a trend away from using reverse scored items.

thank goodness

linger

Wed, 16 Mar 2016 17:33:59 +1300

There are arguments on both sides as to whether semantic-differential or Likert scales should have an odd number or an even number of options.
The former allows a neutral midpoint, which may well be a valid response, but can also be a popular choice if respondents are tired, or don’t…

Andrew Robertson

Wed, 16 Mar 2016 17:30:47 +1300

I guess that is possible, but I really really doubt it.

Sacha

Wed, 16 Mar 2016 17:23:15 +1300

Likert scales force a choice. Standard practice.

Sacha

Wed, 16 Mar 2016 17:19:32 +1300

a national broadcaster doing it essentially for entertainment

bet you they are doing it for business
#ad-targetting

Geoff Lealand

Wed, 16 Mar 2016 17:11:23 +1300

In such a Semantic Differential scale, shouldn't there be a middle/neutral point akin to 'Neither agree nor disagree'? A non-committal response is a valid response. If UoA academics were involved in this, they are bloody incompetent.

Sacha

Wed, 16 Mar 2016 17:10:39 +1300

Thanks for commenting here, Peter.

izogi

Wed, 16 Mar 2016 17:02:58 +1300

But ... There is a problem if ethics committees start saying that you cannot ask questions about controversial and/or sensitive issues. You immediately get a whole lot of subjects taken off the agenda

I think it's premature to start suggesting that type of research couldn't take place under more ethical…

BenWilson

Wed, 16 Mar 2016 17:00:02 +1300

So this is like a mathemagical technomological version of Paul Henry on talkback?

Sacha

Wed, 16 Mar 2016 16:52:20 +1300

I’d say they took the data from the bigger set of questions, (used factor analysis as Cliff from Vox suggested) and worked out what the smallest number of questions they could ask was that yielded the same information. While this process is designed to decrease the boredom factor and make…

izogi

Wed, 16 Mar 2016 16:47:39 +1300

Fair enough! What happened here? I have no freakin’ idea.

TVNZ is subject to the Official Information Act. Perhaps there's room to find out some of the detail.

Carol Stewart

Wed, 16 Mar 2016 16:32:08 +1300

Not unrelated: methodological and ethical problems in a survey carried out by the Island Bay Residents Association on the Island Bay cycleway.

linger

Wed, 16 Mar 2016 16:26:29 +1300

Is Kiwimeter “research” at all?
Results of an online self-selected survey (which may be expected to be biased towards middle-class Pakeha) show, at best, groupings of beliefs that co-occur in subsets of that dominant set of respondents.
Use of factor analysis based on that dataset will further consolidate dimensions based…

Sacha

Wed, 16 Mar 2016 16:25:10 +1300

And every time the subject says “oh, f* you” the researcher ticks a box.

so that's what f-score** means :)

Sacha

Wed, 16 Mar 2016 16:21:53 +1300

I'm imagining TVNZ's advertisers are a beneficiary of this study. Psychometric targetting etc.

Lucy Telfar Barnard

Wed, 16 Mar 2016 16:08:03 +1300

my own mean instincts
They're not mean. They're definitely well above average.

Andrew Robertson

Wed, 16 Mar 2016 15:37:30 +1300

There are a lot of assumptions in your post.

1. Kiwimeter isn’t market research.

2. Cognitive testing is not standard in market research, but it is fairly common in robust social research.

3. Cognitive testing is not necessarily a suitable technique for constructing these sorts of psychometric measures (and psychometric…

Geoff Lealand

Wed, 16 Mar 2016 15:29:16 +1300

I am doing a lecture on Ethics in Research in my Media Research course tomorrow and will certainly use this.

James Littlewood*

Wed, 16 Mar 2016 14:54:30 +1300

It’s a standard step that credible survey research organisations build into the development phase. And it is not complicated or expensive shit to do.

I’ve spent many years in the market research biz. Still do occasionally. There’s no way cognitive testing is standard in market research (which 10:1 is how…

Jarno van der Linden

Wed, 16 Mar 2016 14:50:48 +1300

As a fellow but unrelated van der Linden, the 'v's and 'd's should be lower case.

linger

Wed, 16 Mar 2016 14:21:48 +1300

The whole purpose of a pilot is to improve the survey design so as to remove item ambiguity, maximise completion rates, and maximise interpretability of the results. (“Making the test shorter” is not an aim in itself: it is only useful if it leads to higher completion rates, without harming…

Felix_felix

Wed, 16 Mar 2016 13:49:36 +1300

Don't know about the UK, but the US recall is probably Gallup's "Do you approve or disapprove of marriage between blacks and whites" 1959- .

That question has been changed over time from referring to "white and colored people" and more recently (1968-1978) "whites and non-whites"; I'm guessing because of…

Andrew Robertson

Wed, 16 Mar 2016 13:43:06 +1300

Hi all.

The folks involved in the New Zealand Attitudes and Values study (me included) have written this open letter about the measurement of racism and prejudice.

This is not about the Kiwimeter survey. These issues come up from time to time on other research projects, so we thought it…

Moz

Wed, 16 Mar 2016 13:25:39 +1300

it is a really bad idea to include an item that is systematically more ambiguous or more stressful for one particular subset of respondents.... identify the problem by cognitive testing

I now have a vision of a lab with a white-coated researcher holding a clipboard while someone sits in front…

James Green

Wed, 16 Mar 2016 12:18:24 +1300

I’m still taking it in, but it looks like something very weird and wrong has gone on here.

On a preliminary look, I’d say they took the data from the bigger set of questions, (used factor analysis as Cliff from Vox suggested) and worked out what the smallest number of…

James Butler

Wed, 16 Mar 2016 11:00:30 +1300

a question along the lines of whether the respondent would be happy if their daughter/son married a person of another race (or maybe they specified black). On the face of it, that is racist.

Perhaps. Seems to me that “How would you feel if your daughter/son married someone of another…

Stephen Judd

Wed, 16 Mar 2016 10:57:16 +1300

It would be interesting to know how many people failed to complete the online survey after encountering that question...

Tze Ming Mok

Wed, 16 Mar 2016 10:40:31 +1300

EXACTLY, YO.

linger

Wed, 16 Mar 2016 10:37:35 +1300

…apologises for an entire blog about survey methodology
No apology necessary. Survey geeks represent!

On the cognitive testing/ factor analysis confusion:
Maybe the “special treatment” item does correlate reliably with many other items, mutually measuring some underlying dimension of belief (accepting/ rejecting the dogwhistle probably correlates quite highly with social…

Stephen Judd

Wed, 16 Mar 2016 10:31:23 +1300

I think there is a big difference between a university researcher asking these questions, and a national broadcaster doing it essentially for entertainment, without seemingly any safeguards.

Russell Brown

Wed, 16 Mar 2016 10:02:20 +1300

Stephanie Rogers, a respondent to the precursor survey, has written an extremely revealing post about where the questions came from – and what questions were picked up from the precursor survey for the ‘Kiwimeter’.

I’m still taking it in, but it looks like something very weird and wrong has gone…

Peter Davis

Wed, 16 Mar 2016 09:58:06 +1300

I must say I am not aware of the kiwimeter and I am generally pretty unhappy with the way in which the media are using so-called surveys (particularly those that are just write-in and online). But ... There is a problem if ethics committees start saying that you cannot ask…

Public Address

Wed, 16 Mar 2016 09:58:06 +1300

Discussion from blog post.