Speaker by Various Artists

Read Post

Speaker: The real problem with the ‘Kiwimeter’

74 Responses

First ←Older Page 1 2 3 Newer→ Last

  • Tze Ming Mok, in reply to Aaron Dick,

    I was part of the initial survey, I think. Done through Horizonpoll, certainly not called Kiwimeter at the time.

    There's so little information about the original survey - who conducted it, what the questions were, how long it was, how it was carried out... I'd be interested in how you thought the two compared.

    SarfBank, Lunnin' • Since Nov 2006 • 154 posts Report

  • izogi, in reply to EJ,

    My own thought was that it certainly doesn’t seem too different from that dreadfully worded and highly criticised referendum question a few years ago, which did more to declare smacking kids was a good thing than to clearly ask if it should be legal.

    How can you possibly answer a question like that, without feeling a certain amount of disgust or at least discomfort, unless you already agree with the assumptions of the questioner?

    Wellington • Since Jan 2007 • 1142 posts Report

  • Hilary Stace,

    Wgtn • Since Jun 2008 • 3229 posts Report

  • Stephen Judd, in reply to Tze Ming Mok,

    The survey I did (promoted on Waitangi Day!) was run by http://www.voxpoplabs.com/ who seem to specialise in these exercises. It was about 15 minutes to do, very long, and the questions were much as the present Kiwimeter has, including the "special treament" questions.

    I complained in the feedback form at the end of the survey, and received an answer or two, which I wrote about here. There was no acknowledgement that they might have screwed up. They think their questions are neutral.

    Wellington • Since Nov 2006 • 3122 posts Report

  • Sacha,

    RNZ Morning Report had a couple of stories on KiwiMeter focusing on how Maori MPs are responding - the first featuring Kelvin Davis, Metiria Turei, TVNZ's head of news, and the Canadian survey company CEO - and in more depth later, Marama Fox.

    Ak • Since May 2008 • 19745 posts Report

  • simon g,

    From the 'some of my best friends' school of debate:

    "TVNZ's Gillespie said 120,000 people had already participated in the survey, including Māori." (Stuff report)

    So that's OK then.

    Stuff has opened up their story for comments, so we'll soon get the response of self-appointed "typical Kiwis". Best avoid, if you want to keep enjoying your day.

    Auckland • Since Nov 2006 • 1333 posts Report

  • izogi, in reply to Stephen Judd,

    There was no acknowledgement that they might have screwed up. They think their questions are neutral.

    I s'pose an argument would go that they're more interested in how people respond, not in what they mean.

    But where is the value in that? What's being learned, if anything, at the cost of antagonising some people by intentionally giving them an emotionally loaded question that's impossible to cleanly answer? That's the type of thing that some classes of researchers start requiring ethics approval for. Was there any here?

    Wellington • Since Jan 2007 • 1142 posts Report

  • simon g,

    More generally, this fits in with the overall TVNZ perspective, and its ingrained assumptions about "us" that go beyond racism. Most days you'll hear One News say something like:

    "Oil prices are up - how will that affect your family at the petrol pump?"

    "Interest rates down - what will happen to your mortgage?"

    By trying to be relatable, to include us as their faux-friends, they exclude many.

    Auckland • Since Nov 2006 • 1333 posts Report

  • Jason Kemp,

    I just completed the “Kiwimeter” survey. Despite misgivings about the framing of certain questions I was able to answer most of them in a way that (somewhat) represented my views.

    Where I did have trouble was in the later stages of trying to choose between say rugby and the All Blacks as a NZ symbol. I know both of those do represent NZ but not the actual NZ I live in. However the other options in those sections were even more constrained. So when presented with a limited range of “closed end” answers like that I would have preferred to answer ‘none of the above’ or ‘something else’. That is the difficulty. Which answer do I dislike the least?

    There were also 2 questions that seemed like huge outliers in the context. One of them was about NZ’s economy and the other about the armed forces. I couldn’t answer either question meaningfully so I didn’t.

    I get that the survey is probably some form of social engineering but if you ask closed questions you will only get answers that people hate the least. If as others have mentioned this is part two of a process which was already reduced to some kind of litmus test then maybe it is useful but?

    The short answer is that the way the questions were framed made me quite uncomfortable and feeling manipulated by a process that thinks I should care about rugby or some other cliched attitudes. The reality is much more nuanced than that.

    Auckland • Since Nov 2006 • 368 posts Report

  • Howard Edwards, in reply to Geoff Lealand,

    Geoff - please point your students to Andrew Gelman's recent blog post about reporting published research - or at least quote him where he says:

    Journalists who are writing about quantitative research should not hold the default belief that a published analysis is correct.

    Albany • Since Apr 2013 • 66 posts Report

  • izogi, in reply to Jason Kemp,

    Where I did have trouble was in the later stages of trying to choose between say rugby and the All Blacks as a NZ symbol. I know both of those do represent NZ but not the actual NZ I live in.

    That part struck me as very similar to a decision-making tool that a former colleague of mine went off to co-develop. https://www.1000minds.com/about/how-it-works You feed in a heap of things that you want to prioritise, then in generates countless combinations of them and asks survey-takers to prioritise them against each other. Chances are people's preferences between the input isn't transitive and there will be a heap of contradictions, but the system takes all that data and ranks things into priorities based on whatever algorithms it uses. (I'm not an insider on this -- it's more my understanding from having seen it in use.)

    It's quite taxing on people taking the survey, though. It results in lots of questions and comparisons aren't always obvious. I'm not entirely sure what they're trying to achieve by doing it here, if that's what it is. On the face it's to slot people into arbitrary categories, I suppose. Deeper down, who knows? Does this info get sold?

    Wellington • Since Jan 2007 • 1142 posts Report

  • steve black,

    As far as I’m concerned Kiwimeter is not proper research. It is “entertainment” intended to increase audience participation for commercial gain, and should be evaluated as such. The approach might be defended in terms of “freedom of speech” but I think xkcd has the right answer to that claim:

    http://imgs.xkcd.com/comics/free_speech.png

    I would embed the image itself here but I don’t see how.

    Since I occasionally get involved in these methodological and social research issues, I’ve actually put up my cv so you can see that when I say "not proper research" I have a bit of experience to back up my opinion:

    http://black.net.nz/misc/blackcv-online.pdf

    sunny mt albert • Since Jan 2007 • 116 posts Report

  • Hilary Stace, in reply to izogi,

    An ethical requirement of research is that it is well designed and will credibly address a well formulated research question. Doesn't seem like any of this applies here.

    Wgtn • Since Jun 2008 • 3229 posts Report

  • Jason Kemp, in reply to izogi,

    Thanks Izogi,

    There are definitely a few filters being applied in order to get a pre-determined result.

    I actually I just completed the survey again on the same setup so there appears to be no system blocking on it. To me that seems like anyone can skew the results by doing the same thing.

    Since I knew what to expect I made my results go in a different direction because I could. Not really on purpose but the reality is when “beach based holidays” is a choice it just seems bizarre that the NZ template is really so narrowly defined.

    Way back in 2007 there was a survey on the NZ class system called 8 Tribes Tribal Demographics (my post at the time) that was ( in my view) a genuine attempt to try and get a better idea of what the NZ demographics really are and to identify NZ archetypes ( if such a thing exists). As I recall Russell also did that survey and wrote about it at the time.

    The Kiwimeter feels more like a party trick than a well designed sociology tool. I wonder if the 8 Tribes people have done anything more recently.

    I just had a quick look. The 8 Tribes survey is still running.

    "8 Tribes is a twenty first century reaction to the myth of the “typical New Zealander”. Perhaps we were once a genuinely homogeneous society but now we have become a rich blend of profoundly different people who cannot be defined by a single set of values and attitudes.

    We dream different dreams, we value different virtues, we judge each other by different standards, and because of that we create lives that look very different.

    8 Tribes is our attempt to make sense of this rich diversity.

    This is a snapshot of New Zealand that explores our unspoken class system and the hidden social boundaries that separate us from each other.”

    Link is: 8 Tribes

    Auckland • Since Nov 2006 • 368 posts Report

  • Euan Mason, in reply to EJ,

    The wording was "Māori should not receive any special treatment." I can't simply disagree with that statement because I take issue with the way it's phrased.

    Completely agree. Meeting treaty obligations does not constitute "special treatment". This was a loaded question.

    As for why the survey is being done by TVNZ, probably:

    1) For marketing, to sell more advertising; and
    2) To give Mike Hosking something to inanely pontificate about.

    Canterbury • Since Jul 2008 • 259 posts Report

  • Rob Stowell, in reply to Stephen Judd,

    This is beautiful :)

    Imagine your HR department running a survey at your workplace, where one question was "Susan should not receive special treatment." I don't think you would feel very good about that, even though it does not logically imply that you receive special treatment.

    This is what I am concerned about when I say this was a loaded question.

    Whakaraupo • Since Nov 2006 • 2120 posts Report

  • Rich of Observationz, in reply to Jason Kemp,

    Oh, the joys of meaningless cluster analysis

    Back in Wellington • Since Nov 2006 • 5550 posts Report

  • Rob Stowell, in reply to Euan Mason,

    The wording was “Māori should not receive any special treatment.” I can’t simply disagree with that statement because I take issue with the way it’s phrased.

    Yeah. It's a reasonable assumption Māori are discriminated against - we have all manner of evidence for this. Discrimination is 'special treatment'.
    This is a very poorly thought-out question. It may well bring out the 'shitty attitudes' - but how would you have any idea what people meant when they agreed or disagreed?
    Without even starting to think about whether 'having a tiny proportion of what was unjustly seized from your iwi returned a century or so later' is 'special treatment'?
    Meh.

    Whakaraupo • Since Nov 2006 • 2120 posts Report

  • Moz, in reply to Rob Stowell,

    Without even starting to think about whether 'having a tiny proportion of what was unjustly seized from your iwi returned a century or so later' is 'special treatment'?

    We have much better questions in Oz, because we have honest-to-bob Libertarians in parliament. Oddly, he is very strongly against property rights and when I asked him about that he started talking "all lives matter" type nonsense.

    we think the property rights of remote and rural Aborigines should be upgraded to the same as everyone else, allowing them to fully participate in the general economy.

    Note the failure to mention that if he was serious this would mean invalidating all the squatter rights and going back to square one over most of Australia, negotiating a treaty to base the new land rights (for white people) on.

    In an Australian context what the above means, BTW, is extinguishing collective land rights and forcing tribal land to be divided up between individuals, granted to them as freehold title, and allowing them to sell it. Which is not-very-coded for "sold to mining companies".

    Sydney, West Island • Since Nov 2006 • 1233 posts Report

  • Morgan Nichol, in reply to Jason Kemp,

    Attachment

    Where I did have trouble was in the later stages of trying to choose between say rugby and the All Blacks as a NZ symbol.

    I found that to be a very interesting challenge.

    Auckland CBD • Since Nov 2006 • 314 posts Report

  • linger, in reply to steve black,

    Attachment

    I think xkcd has the right answer to that claim:
    http://imgs.xkcd.com/comics/free_speech.png
    I would embed the image itself here but I don’t see how.

    Download; convert to jpg; attach to comment … et voila

    Tokyo • Since Apr 2007 • 1944 posts Report

  • Kumara Republic,

    It seemed to have echoes, unintentionally or not, of old American citizenship tests that owed more to pop culture values than actual civics. Namely, those tests that asked questions about famous baseball players and other stuff.

    An even more infamous variant in the Jim Crow-era US South cranked it up to 11, and was designed to prevent blacks from voting.

    The southernmost capital … • Since Nov 2006 • 5446 posts Report

  • Tze Ming Mok,

    This is a very poorly thought-out question. It may well bring out the 'shitty attitudes' - but how would you have any idea what people meant when they agreed or disagreed?

    I agree the 'scale' is wrong (and while it's not completely 'useless' as a way to gather opinions it does actually make people think the question is insulting, predetermined and racist), but the exact statement is the right one to be testing opinions to. As I mentioned in comments, a better scale would have been 'how acceptable or unacceptable do you find this statement?' or maybe something like 'how likely is it that I would say something like this?' or 'how alike am I to a person who says this?'

    The whole thing is poorly thought out. The more I think about many aspects of it, the worse and worse it gets. They're really boasting now about 130,000 respondents? Congratulations, that's 130,000 respondents worth of worthless data that doesn't mean anything!

    SarfBank, Lunnin' • Since Nov 2006 • 154 posts Report

  • Moz, in reply to Kumara Republic,

    echoes, unintentionally or not, of old American citizenship tests

    That may be history in the US but it's very much alive today in Australia. The questions are not as bad as you might fear, they have been cleaned up a lot since the Howard-era version which was insane ("name the state flower of Victoria" type questions). There are practice tests online if you want to play (some better than others).

    My father-in-law found them tricky despite living here longer than I have, because a lot of them were inane and seemingly designed to filter out people who are not native English speakers and those who haven't done Australian multi-choice tests before. Lots of odd wording and the usual "one obviously silly, one probably wrong, guess between the other two" type stuff.

    Sydney, West Island • Since Nov 2006 • 1233 posts Report

  • Tze Ming Mok, in reply to Stephen Judd,

    The survey I did (promoted on Waitangi Day!) was run by http://www.voxpoplabs.com/ who seem to specialise in these exercises. It was about 15 minutes to do, very long, and the questions were much as the present Kiwimeter has, including the "special treament" questions.

    I complained in the feedback form at the end of the survey, and received an answer or two, which I wrote about here. There was no acknowledgement that they might have screwed up. They think their questions are neutral.

    If this was the total of their piloting, that's a sad-ass yet totally typical level of corner-cutting. Credible social research organisations routinely do cognitive piloting for large-scale surveys. This is where you recruit a broad range of people, sit down with them and essentially do qualitative interviewing about what is going through their heads when they fill out a survey. This is where you find out whether your survey *really* works, or whether it's going to look shady and kinda racist. It's not even expensive to do. Ugh, the incompetence.

    SarfBank, Lunnin' • Since Nov 2006 • 154 posts Report

First ←Older Page 1 2 3 Newer→ Last

Post your response…

This topic is closed.