Speaker: The real problem with the ‘Kiwimeter’
74 Responses
First ←Older Page 1 2 3 Newer→ Last
-
Tze Ming Mok, in reply to
I was part of the initial survey, I think. Done through Horizonpoll, certainly not called Kiwimeter at the time.
There's so little information about the original survey - who conducted it, what the questions were, how long it was, how it was carried out... I'd be interested in how you thought the two compared.
-
izogi, in reply to
My own thought was that it certainly doesn’t seem too different from that dreadfully worded and highly criticised referendum question a few years ago, which did more to declare smacking kids was a good thing than to clearly ask if it should be legal.
How can you possibly answer a question like that, without feeling a certain amount of disgust or at least discomfort, unless you already agree with the assumptions of the questioner?
-
Giovanni Tiso considers the survey
http://bat-bean-beam.blogspot.co.nz/2016/03/what-colour-is-our-nationalism.html
-
Stephen Judd, in reply to
The survey I did (promoted on Waitangi Day!) was run by http://www.voxpoplabs.com/ who seem to specialise in these exercises. It was about 15 minutes to do, very long, and the questions were much as the present Kiwimeter has, including the "special treament" questions.
I complained in the feedback form at the end of the survey, and received an answer or two, which I wrote about here. There was no acknowledgement that they might have screwed up. They think their questions are neutral.
-
RNZ Morning Report had a couple of stories on KiwiMeter focusing on how Maori MPs are responding - the first featuring Kelvin Davis, Metiria Turei, TVNZ's head of news, and the Canadian survey company CEO - and in more depth later, Marama Fox.
-
From the 'some of my best friends' school of debate:
"TVNZ's Gillespie said 120,000 people had already participated in the survey, including Māori." (Stuff report)
So that's OK then.
Stuff has opened up their story for comments, so we'll soon get the response of self-appointed "typical Kiwis". Best avoid, if you want to keep enjoying your day.
-
izogi, in reply to
There was no acknowledgement that they might have screwed up. They think their questions are neutral.
I s'pose an argument would go that they're more interested in how people respond, not in what they mean.
But where is the value in that? What's being learned, if anything, at the cost of antagonising some people by intentionally giving them an emotionally loaded question that's impossible to cleanly answer? That's the type of thing that some classes of researchers start requiring ethics approval for. Was there any here?
-
More generally, this fits in with the overall TVNZ perspective, and its ingrained assumptions about "us" that go beyond racism. Most days you'll hear One News say something like:
"Oil prices are up - how will that affect your family at the petrol pump?"
"Interest rates down - what will happen to your mortgage?"
By trying to be relatable, to include us as their faux-friends, they exclude many.
-
I just completed the “Kiwimeter” survey. Despite misgivings about the framing of certain questions I was able to answer most of them in a way that (somewhat) represented my views.
Where I did have trouble was in the later stages of trying to choose between say rugby and the All Blacks as a NZ symbol. I know both of those do represent NZ but not the actual NZ I live in. However the other options in those sections were even more constrained. So when presented with a limited range of “closed end” answers like that I would have preferred to answer ‘none of the above’ or ‘something else’. That is the difficulty. Which answer do I dislike the least?
There were also 2 questions that seemed like huge outliers in the context. One of them was about NZ’s economy and the other about the armed forces. I couldn’t answer either question meaningfully so I didn’t.
I get that the survey is probably some form of social engineering but if you ask closed questions you will only get answers that people hate the least. If as others have mentioned this is part two of a process which was already reduced to some kind of litmus test then maybe it is useful but?
The short answer is that the way the questions were framed made me quite uncomfortable and feeling manipulated by a process that thinks I should care about rugby or some other cliched attitudes. The reality is much more nuanced than that.
-
Howard Edwards, in reply to
Geoff - please point your students to Andrew Gelman's recent blog post about reporting published research - or at least quote him where he says:
Journalists who are writing about quantitative research should not hold the default belief that a published analysis is correct.
-
izogi, in reply to
Where I did have trouble was in the later stages of trying to choose between say rugby and the All Blacks as a NZ symbol. I know both of those do represent NZ but not the actual NZ I live in.
That part struck me as very similar to a decision-making tool that a former colleague of mine went off to co-develop. https://www.1000minds.com/about/how-it-works You feed in a heap of things that you want to prioritise, then in generates countless combinations of them and asks survey-takers to prioritise them against each other. Chances are people's preferences between the input isn't transitive and there will be a heap of contradictions, but the system takes all that data and ranks things into priorities based on whatever algorithms it uses. (I'm not an insider on this -- it's more my understanding from having seen it in use.)
It's quite taxing on people taking the survey, though. It results in lots of questions and comparisons aren't always obvious. I'm not entirely sure what they're trying to achieve by doing it here, if that's what it is. On the face it's to slot people into arbitrary categories, I suppose. Deeper down, who knows? Does this info get sold?
-
As far as I’m concerned Kiwimeter is not proper research. It is “entertainment” intended to increase audience participation for commercial gain, and should be evaluated as such. The approach might be defended in terms of “freedom of speech” but I think xkcd has the right answer to that claim:
http://imgs.xkcd.com/comics/free_speech.png
I would embed the image itself here but I don’t see how.
Since I occasionally get involved in these methodological and social research issues, I’ve actually put up my cv so you can see that when I say "not proper research" I have a bit of experience to back up my opinion:
-
Hilary Stace, in reply to
An ethical requirement of research is that it is well designed and will credibly address a well formulated research question. Doesn't seem like any of this applies here.
-
Jason Kemp, in reply to
Thanks Izogi,
There are definitely a few filters being applied in order to get a pre-determined result.
I actually I just completed the survey again on the same setup so there appears to be no system blocking on it. To me that seems like anyone can skew the results by doing the same thing.
Since I knew what to expect I made my results go in a different direction because I could. Not really on purpose but the reality is when “beach based holidays” is a choice it just seems bizarre that the NZ template is really so narrowly defined.
Way back in 2007 there was a survey on the NZ class system called 8 Tribes Tribal Demographics (my post at the time) that was ( in my view) a genuine attempt to try and get a better idea of what the NZ demographics really are and to identify NZ archetypes ( if such a thing exists). As I recall Russell also did that survey and wrote about it at the time.
The Kiwimeter feels more like a party trick than a well designed sociology tool. I wonder if the 8 Tribes people have done anything more recently.
I just had a quick look. The 8 Tribes survey is still running.
"8 Tribes is a twenty first century reaction to the myth of the “typical New Zealander”. Perhaps we were once a genuinely homogeneous society but now we have become a rich blend of profoundly different people who cannot be defined by a single set of values and attitudes.
We dream different dreams, we value different virtues, we judge each other by different standards, and because of that we create lives that look very different.
8 Tribes is our attempt to make sense of this rich diversity.
This is a snapshot of New Zealand that explores our unspoken class system and the hidden social boundaries that separate us from each other.”
Link is: 8 Tribes
-
Euan Mason, in reply to
The wording was "Māori should not receive any special treatment." I can't simply disagree with that statement because I take issue with the way it's phrased.
Completely agree. Meeting treaty obligations does not constitute "special treatment". This was a loaded question.
As for why the survey is being done by TVNZ, probably:
1) For marketing, to sell more advertising; and
2) To give Mike Hosking something to inanely pontificate about. -
Rob Stowell, in reply to
This is beautiful :)
Imagine your HR department running a survey at your workplace, where one question was "Susan should not receive special treatment." I don't think you would feel very good about that, even though it does not logically imply that you receive special treatment.
This is what I am concerned about when I say this was a loaded question.
-
Rich of Observationz, in reply to
Oh, the joys of meaningless cluster analysis
-
Rob Stowell, in reply to
The wording was “Māori should not receive any special treatment.” I can’t simply disagree with that statement because I take issue with the way it’s phrased.
Yeah. It's a reasonable assumption Māori are discriminated against - we have all manner of evidence for this. Discrimination is 'special treatment'.
This is a very poorly thought-out question. It may well bring out the 'shitty attitudes' - but how would you have any idea what people meant when they agreed or disagreed?
Without even starting to think about whether 'having a tiny proportion of what was unjustly seized from your iwi returned a century or so later' is 'special treatment'?
Meh. -
Moz, in reply to
Without even starting to think about whether 'having a tiny proportion of what was unjustly seized from your iwi returned a century or so later' is 'special treatment'?
We have much better questions in Oz, because we have honest-to-bob Libertarians in parliament. Oddly, he is very strongly against property rights and when I asked him about that he started talking "all lives matter" type nonsense.
we think the property rights of remote and rural Aborigines should be upgraded to the same as everyone else, allowing them to fully participate in the general economy.
Note the failure to mention that if he was serious this would mean invalidating all the squatter rights and going back to square one over most of Australia, negotiating a treaty to base the new land rights (for white people) on.
In an Australian context what the above means, BTW, is extinguishing collective land rights and forcing tribal land to be divided up between individuals, granted to them as freehold title, and allowing them to sell it. Which is not-very-coded for "sold to mining companies".
-
Morgan Nichol, in reply to
Where I did have trouble was in the later stages of trying to choose between say rugby and the All Blacks as a NZ symbol.
I found that to be a very interesting challenge.
-
linger, in reply to
I think xkcd has the right answer to that claim:
http://imgs.xkcd.com/comics/free_speech.png
I would embed the image itself here but I don’t see how.Download; convert to jpg; attach to comment … et voila
-
It seemed to have echoes, unintentionally or not, of old American citizenship tests that owed more to pop culture values than actual civics. Namely, those tests that asked questions about famous baseball players and other stuff.
An even more infamous variant in the Jim Crow-era US South cranked it up to 11, and was designed to prevent blacks from voting.
-
This is a very poorly thought-out question. It may well bring out the 'shitty attitudes' - but how would you have any idea what people meant when they agreed or disagreed?
I agree the 'scale' is wrong (and while it's not completely 'useless' as a way to gather opinions it does actually make people think the question is insulting, predetermined and racist), but the exact statement is the right one to be testing opinions to. As I mentioned in comments, a better scale would have been 'how acceptable or unacceptable do you find this statement?' or maybe something like 'how likely is it that I would say something like this?' or 'how alike am I to a person who says this?'
The whole thing is poorly thought out. The more I think about many aspects of it, the worse and worse it gets. They're really boasting now about 130,000 respondents? Congratulations, that's 130,000 respondents worth of worthless data that doesn't mean anything!
-
Moz, in reply to
echoes, unintentionally or not, of old American citizenship tests
That may be history in the US but it's very much alive today in Australia. The questions are not as bad as you might fear, they have been cleaned up a lot since the Howard-era version which was insane ("name the state flower of Victoria" type questions). There are practice tests online if you want to play (some better than others).
My father-in-law found them tricky despite living here longer than I have, because a lot of them were inane and seemingly designed to filter out people who are not native English speakers and those who haven't done Australian multi-choice tests before. Lots of odd wording and the usual "one obviously silly, one probably wrong, guess between the other two" type stuff.
-
Tze Ming Mok, in reply to
The survey I did (promoted on Waitangi Day!) was run by http://www.voxpoplabs.com/ who seem to specialise in these exercises. It was about 15 minutes to do, very long, and the questions were much as the present Kiwimeter has, including the "special treament" questions.
I complained in the feedback form at the end of the survey, and received an answer or two, which I wrote about here. There was no acknowledgement that they might have screwed up. They think their questions are neutral.
If this was the total of their piloting, that's a sad-ass yet totally typical level of corner-cutting. Credible social research organisations routinely do cognitive piloting for large-scale surveys. This is where you recruit a broad range of people, sit down with them and essentially do qualitative interviewing about what is going through their heads when they fill out a survey. This is where you find out whether your survey *really* works, or whether it's going to look shady and kinda racist. It's not even expensive to do. Ugh, the incompetence.
Post your response…
This topic is closed.