Speaker: ‘Kiwimeter’ is a methodological car crash and I still can’t look away
88 Responses
First ←Older Page 1 2 3 4 Newer→ Last
-
Surely the researchers' role on such a panel is to be informed about the pros and cons of various methodologies and pass that information to the group. If the senior stakeholders then choose less accurate/reliable methods because of cost issues, at least they will be aware of the reduction in accuracy/value of the resulting information.
What the research will help them achieve may well be compromised when poor methodologies are employed. They need to know that the money spent may be entirely wasted if poor decisions are made about methods. -
Andrew Robertson, in reply to
Yes, but the researcher's view is not always given as much weight as some might like, and what what has value to one person may not have value to another.
Also, in a competitive tender situation you have multiple companies telling you all sorts of things, so the final decision will rarely come down to something like cognitive testing alone.
-
Bryce Edwards rounds up various articles about this topic. Surprisingly low level of understanding about ethnicity and framing.
Soper unironically uses the phrase "melting pot". Supposed professional researcher, a Mr Farrar, reckons Kelvin Davis can't challenge the notion of 'special treatment' because "He's an MP for an electorate where eligibility is dependent on your blood ancestry".
-
Academics should have academic freedom but that comes with the understanding that their work is open to criticism. And the criticism doesn't have to come from people with letters after their name.
It was a dumb question because it never defined special treatment. Person 1 could say Maori don't deserve special treatment but believe that the Waitangi Tribunal and Whanau Ora are not special treatment but fair treatment. - that Maori should get those things because it balances out the unfairness that happened to them before. But person 2 might think that the Waitangi Tribunal and Whanau Ora are special treatment and think Maori should not have them. So both people would answer "agree" but their perspectives would be radically different ... and that defeats the purpose of the question.
The other reason the question was dumb was it's point of view (and that of the whole survey) was from a (male) European NZer perspective. I have done the NZAV Survey twice and I bridled each time because of the othering and demeaning nature of the questions about women.
There are ways to ask questions about sensitive topics without reinforcing prejudice. Or subjecting the victims of that prejudice to it.
-
I see Bryce Edwards's roundup completely missed my main point, which was about self-selective sampling bias posing a challenge to the validity of the results, but oh well. #nerdlife
-
Edwards does at least identify those mysterious "prominent academics"
from Victoria University (Jack Vowles and Kaapua Smith) and the University of Auckland (Jennifer Lees-Marshment and Danny Osborne)
Any of them reachable for comment?
-
Andrew Robertson, in reply to
It was a dumb question because it never defined special treatment. Person 1 could say Maori don’t deserve special treatment but believe that the Waitangi Tribunal and Whanau Ora are not special treatment but fair treatment. – that Maori should get those things because it balances out the unfairness that happened to them before. But person 2 might think that the Waitangi Tribunal and Whanau Ora are special treatment and think Maori should not have them. So both people would answer “agree” but their perspectives would be radically different … and that defeats the purpose of the question.
This is why we have psychometric measures in the first place. The discipline exists out of recognition that you can’t measure an internal construct with a single attitude statement.
I’m not saying your criticism is wrong or invalid Megan (not at all). And psychometricians do need to think more about context, and how some statements can make people feel. (For example, I’ve heard about a personality test being used in employment that asks ‘Do you prefer a bath or a shower?’. The question may be a valid measure of some personality trait, but it’s inappropriate in that context.)
Many of the criticisms I’ve read are not about Kiwimeter at all. They’re about the way social science measures internal constructs. Seems to me the discussion/debate should be about that, rather than Kiwimeter. It would be a good debate to have.
-
TeAtaoTu, in reply to
Hi Peter
Kicking ethics review committees is a bit rum. Especially given NZ and Australia's past use of social science to kick indigenous people, which is one, amongst other sobering reasons that we have these committees. As Chair of Canterbury University's ethics committee I think your characterisation of the committee's is reasonably offensive, though perhaps you have had unfortunate experiences in the past. Whatever else those who sit in the committees to examine social science research are your peers and colleagues and community members whose work (without pay) makes your research possible. Come to Canterbury if you strike problems again, we'll be only too happy to have you come through our conversational approach to ethics review, where we attempt to model the best of collaborative and trust making ethics processes. In the meantime, would you know anyone who could justify, from the academic literature, the use of the special treatment question? Because no one seems to want to reply to the original point: if there are one thousand ways to ask that question without echoing that politicized phrase, and given how much scholarly work has gone into criticising the use of that phrase, what possible scholarly reason could justify it's use? I just wish someone would answer with empathy and a respect for the human dignity of those questioning the use of special treatment.
Cheers Lindsey Te Ata o Tu MacDonald -
@Andrew
I think it's going to take a pretty special question (and analysis) to separate out the people who agreed to that question but think "different treatment is ok when it balances out advantage and is not inherently special" and "any treatment I can't get is special treatment and that's unfair and shouldn't be allowed".And because this is all probably analysed with factor analysis or principal component analysis, these two questions will have low correlation so it's unlikely they'll feature together anyway.
-
Hilary Stace, in reply to
Thank you, from a fellow ethics nerd. Do we know whether this survey actually went to an ethics committee?
I am not on a university one, but if this question came up when I was in a position to make a decision about it I would ask for justification for it to be asked this way. I come from a perspective that (almost) all NZ research is relevant to Maori so would expect evidence of consultation with Maori about the whole survey as well as the wording of questions which will obviously be controversial. Peter Davis implies that ethics committees are risk averse, but as a person with some experience I am strongly in favour of doing research which might be controversial or with participants who are considered 'vulnerable' (a term defined by the National Ethics Advisory Committee), but it is important that they are 'researched with' rather than 'researched on'. This survey doesn't seem to be an example of good ethical practice.
-
Andrew Robertson, in reply to
I've done five or six studies that asked similar sorts of questions. Not once was I asked by the Human Ethics Committee to change their wording.
My submission did give research justification for the statements I included.
-
Hilary Stace, in reply to
As a member of ethics committees at various times, I like it when the research applicant comes along to the meeting. Then you can have a discussion about any unclear or controversial details and hear the researchers' justification for them, or decide how things might be amended to improve the project and better answer the research question. It is when the applicant cannot justify something, or hasn't considered why it might be problematic, that issues arise.
-
Sciblogs has a post about Kiwimeter.
Post your response…
This topic is closed.