OnPoint by Keith Ng

Read Post

OnPoint: #WTFMSD: "Damning"

68 Responses

First ←Older Page 1 2 3 Newer→ Last

  • nzlemming, in reply to Keith Ng,

    Boyle talked about not finding any "download patterns" - i.e. People leeching large volumes of data, like I did.

    Did they identify your downloads? And from that, which files you accessed? If not, then they know precisely nothing about what anyone else may or may not have done.

    Waikanae • Since Nov 2006 • 2933 posts Report Reply

  • Idiot Savant, in reply to Keith Ng,

    My understanding is that there's no audit trail to determine *who* accessed information, but that there *were* network logs. Boyle talked about not finding any "download patterns" - i.e. People leeching large volumes of data, like I did. That seems like a reasonable way to detect intrusion, unless it was someone who covered their own tracks (in which case no audit trail would help).

    That would certainly get you mass-downloads. But it wouldn't e.g. spot a debt-collector using a kiosk to access information on a few people they were looking for.

    Palmerston North • Since Nov 2006 • 1716 posts Report Reply

  • Sacha,

    The full report is interesting reading.

    Ak • Since May 2008 • 19719 posts Report Reply

  • Martin Lindberg, in reply to Sacha,

    Some person or group has made a decision about the cost/benefit of acting on concerns raised.

    Never attribute to cost/benefit analysis that which is adequately explained by stupidity.

    Stockholm • Since Jul 2009 • 802 posts Report Reply

  • Sacha, in reply to Martin Lindberg,

    Not a formal economic analysis. And stupidity, yes. Report mentions project management defects and inadequate approach to risk.

    Ak • Since May 2008 • 19719 posts Report Reply

  • Kyle Matthews,

    I would imagine that they reported it, or were aware it had been reported, by making an entry in the project risk register, on the meeting minutes, or, if using Agile, in crayon on a piece of brightly coloured paper stuck to the wall of the meeting room, which will subsequently have fallen on the floor and been hoovered up by the cleaner*

    Either way it was reported. Management would have then failed to understand it and ignored it.

    Well to quote Keith [referring to Boyle]:

    But he did make clear that the decisions didn’t get escalated properly – i.e. Senior managers weren’t involved. He also said that it simply “dropped off the radar” – that it wasn’t a matter of cost-cutting, it was a matter of WTF.

    That may be incorrect, but if it is, pushing all the blame up the chain clearly isn't valid. What 'escalated properly' means... quite possibly there's blame above and below.

    Since Nov 2006 • 6243 posts Report Reply

  • James George,

    I stand corrected Mr Ng, but if you are right that there was only one large unprotected and unaudited national network, rather than hundreds of smaller localised sub-networks, the decision to eschew both auditing and security controls defies belief.
    Even back in the early 90's when big departments moved from 'dumb' terminals on a mainframe to 'intelligent' nodes capable of accessing both mainframe and smaller local apps, the idea of hooking all the nodes into one big network would have been kicked outta consideration immediately. Not just from a security standpoint, which even back then would have created opposition, but because local chiefs want to be able to maintain some opacity from centralised oversight or worse, from their peers, rivals on the career ladder. Of course statistical collation and other essential forms of data mining by central authorities, was possible on these local networks, but HQ examination of nuts n bolts required co-operation from the local site.

    One network - that makes the call not to insulate kiosks irrational. It would have been relatively simple and inexpensive.
    I guess I better read the report.

    Since Sep 2007 • 96 posts Report Reply

  • nzlemming, in reply to Martin Lindberg,

    Never attribute to cost/benefit analysis that which is adequately explained by stupidity.

    Word.

    Waikanae • Since Nov 2006 • 2933 posts Report Reply

  • Dave Marks, in reply to izogi,

    Based on my reading of the review report I think you are correct. You really have to question how thorough that review was.

    - The Logs - How far back in time do the logs go? The reviewer used the logs to find out what information was accessed from the kiosks in October 2012, but presumably wasn't able to go back to logs dated October 2011.

    - The Beneficiary Advocates Did the reviewers attempt to identify or make contact with the Beneficiary Advocate with Systems Administration knowledge who pointed out a problem to Kay Brereton in 2011?

    1) The Logs:

    Page 23 - "Events and the Ministry's responses"

    10 October 2011:
    Ms Brereton, a Beneficiary Advocate, raises an access to information issue with Work and Income.

    08 October 2012:
    Mr Bailey calls the Ministry, indicating that he knows of vulnerability in the Ministry’s systems.

    14 October 2012:
    Mr Ng alerts the media and Office of the Privacy Commissioner to a security vulnerability in the Ministry’s systems.

    The latter two events are related, with Mr Bailey and Mr Ng collaborating to an extent. Our understanding of the event involving Ms Brereton is limited to an interview conducted with her and reviews of emails that were sent in relation to the event. No technical information to confirm technical details of the event is available.

    Our understanding of the latter two events is based on interviews with Mr Bailey and Mr Ng, as well as reviews of network logs.

    If logs were available for October 2012 but not for October 2011 then clearly there's a period of time where logs haven't been analysed. How long is this - 11 months?

    2) Beneficiary Advocates
    Kay Brereton raises an issue about information access but is unable to provide clear information about the nature of the problem. Another "Beneficiary Advocate" with systems administration knowledge is mentioned.

    Ms Brereton was attending a session to familiarise Beneficiary Advocates with the new “kiosks” when another Beneficiary Advocate, a volunteer, who also worked as a systems administrator in another role, uncovered a way to gain access to computer names and internet protocol address information which Ms Brereton considered sensitive.(Page 23)

    It would be interesting to find out what attempts the reviewers made to speak to the system admin who spotted the problem in 2011. If not why not - and why doesn't the report address this?

    Waterview • Since Nov 2012 • 3 posts Report Reply

  • Hamish,

    While network logs might help detect the sort of mass extract that you're talking about, there is no way you could claim that it's evidence of a "low risk" to security. For all we know, people have been popping in and out of the back end for years, establishing a standard pattern of unauthorized access.

    The A.K. • Since Nov 2006 • 155 posts Report Reply

  • Rich of Observationz, in reply to Kyle Matthews,

    Obviously the process didn't work.

    But if you are trying to find low-level scapegoats for sacking, then I'm sure they would argue that the existence of a document with a statement that "non-separation of networks was an urgent issue" circulated to appropriate levels of management equates to communication. Management would in turn argue that it wasn't their job to understand low-level project documents.

    I suspect the Employment Tribunal would agree.

    Back in Wellington • Since Nov 2006 • 5550 posts Report Reply

  • Keith Ng, in reply to nzlemming,

    Did they identify your downloads? And from that, which files you accessed?

    Yes (I think).

    Auckland • Since Nov 2006 • 543 posts Report Reply

  • Keith Ng, in reply to Idiot Savant,

    That would certainly get you mass-downloads. But it wouldn't e.g. spot a debt-collector using a kiosk to access information on a few people they were looking for.

    Can't. They were scanned PDFs with no metadata and sequential file names. It is impossible to look for anything. Scouring for personal information is possible, but you'd need to sort them by hand or OCR them first, which would require a data dump.

    Auckland • Since Nov 2006 • 543 posts Report Reply

  • Matthew Poole, in reply to Keith Ng,

    Did they identify your downloads? And from that, which files you accessed?

    Yes (I think).

    I certainly see in the report that they saw your accessing the servers, and the transfer of data. I don't get from there to an audit trail of what you accessed, were they to be absent your USB key.

    Auckland • Since Mar 2007 • 4097 posts Report Reply

  • Matthew Poole, in reply to Keith Ng,

    It is impossible to look for anything.

    Orly? From the original post:

    And then there were file server logs. Normally, they aren’t that exciting. Except that WINZ name their files quite well. For example:

    s:\SharedData\wi_wites\Waikato\HAM\Fraud Investigations\[Name of investigator]\[Name of WINZ client] 23 Jun 2011 Case 640026-10.WMA

    That looks pretty damned searchable to me, if one had a spot of inside info.

    Auckland • Since Mar 2007 • 4097 posts Report Reply

  • Matthew Poole,

    OK. I've skimmed the Deloitte report with some degree of thoroughness. I think it is reasonable to conclude that there was no mass transfer of data in the same manner was was done by Keith, based on what's in the report. The requirements of the Public Records Act 2005 make it certain that there will be long-term backups available, and network logs take up so little space when compressed that trying to keep them out is just not worth the effort. Hell, it's entirely possible that the logs back to the installation of the kiosks are still on hard disk. Recovering them and checking for bulk transfers could easily have been accomplished in the time since Keith went noseying.

    That said, there's still no certainty that there hasn't been a wider compromise of the MSD network. There's no audit trail, as confirmed by the report, there are only logs that will identify connections and transfers of data. Even if there were an audit trail it would still be unlikely to deliver up the identity of a miscreant. It is very unlikely that widespread compromise has happened, given that there's apparently no evidence of bulk data transfers (and most people inclined to take serious advantage of this vulnerability would be more likely to want data than access), but it's not impossible. What is impossible is saying so for certain without complete examination of every computer on the network; or "nuking from orbit".

    Auckland • Since Mar 2007 • 4097 posts Report Reply

  • Kyle Matthews,

    But if you are trying to find low-level scapegoats for sacking, then I’m sure they would argue that the existence of a document with a statement that “non-separation of networks was an urgent issue” circulated to appropriate levels of management equates to communication. Management would in turn argue that it wasn’t their job to understand low-level project documents.

    What's been outlined above, if it's accurate, is that low level people received DD's report, which highlighted the problem. It was not then escalated properly.

    If that's accurate...

    It will somewhat turn on what 'escalated properly' means. However it it means "not escalated at all" then the guys at the bottom are in trouble. You're going to struggle to blame upper management when they weren't told about the problem.

    If it means escalated but not in the correct manner, then you could probably say that it was bad enough that it doesn't matter how it was escalated, anyone that knew about it should have dealt to it.

    The existence of the document isn't disputed. It's who had it and what they did with it that will start to cause problems for people up the chain. I struggle to reconcile "not escalated properly" with your "circulated to appropriate levels of management" however Rich.

    Since Nov 2006 • 6243 posts Report Reply

  • izogi, in reply to Matthew Poole,

    The requirements of the Public Records Act 2005 make it certain that there will be long-term backups available, and network logs take up so little space when compressed that trying to keep them out is just not worth the effort.

    In practice, do all departments keep all network log files as if they're records under the Public Records Act? They probably should for certain kinds of information able to be logged, but short of being prompted to think about it, I could imagine a situation where IT staff aren't thinking in terms of the Public Records Act while Records staff don't realise the information even exists, and are more focused on the obvious information used by the department is treated as records.

    Wellington • Since Jan 2007 • 1139 posts Report Reply

  • Russell Brown, in reply to Kyle Matthews,

    What’s been outlined above, if it’s accurate, is that low level people received DD’s report, which highlighted the problem. It was not then escalated properly.

    If that’s accurate…

    It will somewhat turn on what ‘escalated properly’ means.

    This is interesting in the context of this week's Media3 discussion of whistleblowing. There are indications that concerns were raised internally and not acted upon. In theory the Protected Disclosures Act 2000 provides a structure and process for staff to safely escalate concerns above their immediate managers, and even to the Ombudsman or the minister.

    In practice, not so much.

    Auckland • Since Nov 2006 • 22834 posts Report Reply

  • Matthew Poole, in reply to izogi,

    In practice, do all departments keep all network log files as if they’re records under the Public Records Act? They probably should for certain kinds of information able to be logged, but short of being prompted to think about it, I could imagine a situation where IT staff aren’t thinking in terms of the Public Records Act while Records staff don’t realise the information even exists, and are more focused on the obvious information used by the department is treated as records.

    I wouldn't consider them to be public records, but my observation was more that it's generally easier to have a single backup retention policy for the entire organisational network than to split things up based on information servers vs management servers. The logs are very likely to be backed up as part of the backup of the server on which they reside, based on whatever the internal backup policies are, and then held for the necessary period to comply with the PRA.

    Auckland • Since Mar 2007 • 4097 posts Report Reply

  • nzlemming, in reply to Matthew Poole,

    I wouldn't consider them to be public records, but my observation was more that it's generally easier to have a single backup retention policy for the entire organisational network than to split things up based on information servers vs management servers. The logs are very likely to be backed up as part of the backup of the server on which they reside, based on whatever the internal backup policies are, and then held for the necessary period to comply with the PRA.

    Yeah, nah.

    While I agree that it's trivial and prudent to backup your network and systems logs (in fact, it's more work to ensure they're deleted, if that's how you roll) and stuff, records management is something else again. I agree that most organisations won't class logfiles as public records at this time (bet there's some review of that situation going on in Welly right now), but that doesn't mean they fall under the purview of the PRA. Network backup has nothing to do with records management. It's more likely that the online records are backed up with the system data, rather than the reason for the backup of the system data. Hairsplitting? Not if your wife's a records manager for a government agency ;-)

    On the Archives site, edit - link they have a guide to the PRA which says :

    What is a record?

    * The Public Records Act defines 'records' as any information that is compiled, recorded or stored in any format. Electronic records (email, etc.) should be treated according to content not format, and should be managed in a recordkeeping system;
    * 'Public records' are defined as records that are created or received by a public office in the conduct of its affairs. (N.B. this does not mean records that are available for public inspection, but rather records created in the course of a public office's normal business practice.)
    * Local government records are defined as records that are created or received by a local authority in the conduct of its affairs. 'Protected records' are the classes of local government records declared to be protected by the Chief Archivist, as outlined in the Local Government Schedule.

    That might __seem to cover network logs but is not generally construed to do so and I don't think I've ever seen a retention and disposal schedule that mentions them, but I'll ask Mrs Lemming when she gets back from walking the dog.

    Waikanae • Since Nov 2006 • 2933 posts Report Reply

  • Ds,

    TV3
    Read more: http://www.3news.co.nz/Junior-staff-blamed-for-WINZ-privacy-breach/tabid/1607/articleID/275225/Default.aspx#ixzz2B2wNIMlq

    A report on the breach released today blames staff at the ministry, and says they should have told senior managers.

    But senior management would have known about the security check.
    Why would the senior manager NOT ask, hey what did the security report say.
    Systems seemed to be slack.

    wellington • Since Sep 2012 • 8 posts Report Reply

  • nzlemming,

    I retract and withdraw. Lovely Wife advises that system logs are considered records on the basis that they act as metadata for the actual content. How things have changed.

    Waikanae • Since Nov 2006 • 2933 posts Report Reply

  • BenWilson, in reply to Keith Ng,

    Yes (I think).

    Next time (:-)) keep a hold-off set of unimportant/irrelevant data that you don't tell them you ever got. Then you can check whether they are lying about being able to detect what you took. Two scoops for the price of one.

    Auckland • Since Nov 2006 • 10653 posts Report Reply

  • Sacha, in reply to BenWilson,

    keep

    rather different ethical/legal implications

    Ak • Since May 2008 • 19719 posts Report Reply

First ←Older Page 1 2 3 Newer→ Last

Post your response…

Please sign in using your Public Address credentials…

Login

You may also create an account or retrieve your password.