Hard News by Russell Brown

Read Post

Hard News: The United States of Surveillance?

143 Responses

First ←Older Page 1 2 3 4 5 6 Newer→ Last

  • BenWilson,

    Heh, Big Data to find criminals and terrorists. It's hilarious how silly the idea is. How much money could be expended on computers, taken away from actual coppers and intelligence agents, how much it will actually reduce their chances of finding what they're looking for. It's just the kind of thing technocrats would think of.

    Auckland • Since Nov 2006 • 10657 posts Report Reply

  • Bart Janssen, in reply to nzlemming,

    I said “I can read your files and email. How do you know you can trust me?”

    The look on his face have been a thing to behold.

    Auckland • Since Nov 2006 • 4461 posts Report Reply

  • Bart Janssen, in reply to BenWilson,

    It’s just the kind of thing technocrats would think of.

    Stupid technocrats. Next thing they'll be promising computers predicting crimes before they happen

    Auckland • Since Nov 2006 • 4461 posts Report Reply

  • Rich of Observationz, in reply to Bart Janssen,

    There are architectural solutions (involving end-end encryption) that do avoid giving some or even all of the sysadmins the ability to read the CEOs email.

    Back in Wellington • Since Nov 2006 • 5550 posts Report Reply

  • BenWilson, in reply to Bart Janssen,

    Stupid technocrats. Next thing they’ll be promising computers predicting crimes before they happen

    That's not really the same thing. It's just distributing coverage of resources, exactly like the cops already have to do. That is actually a somewhat appropriate job for a computer. To some extent, obviously there still needs to be coordinating humans on the job, and they really do have to know what to do if the computer starts stuffing things up (or just goes down, as software frequently does). I wouldn't be surprised if most of the real value in the system isn't the production of the "heat map" visualizations, which enable the humans to see their problem in a more convenient way.

    Auckland • Since Nov 2006 • 10657 posts Report Reply

  • nzlemming, in reply to Bart Janssen,

    The look on his face have been a thing to behold.

    It was.

    Waikanae • Since Nov 2006 • 2937 posts Report Reply

  • Russell Brown, in reply to Bart Janssen,

    Stupid technocrats. Next thing they’ll be promising computers predicting crimes before they happen

    Ah. I was just about to mention the LAPD project.

    Auckland • Since Nov 2006 • 22850 posts Report Reply

  • nzlemming, in reply to Rich of Observationz,

    There are architectural solutions (involving end-end encryption) that do avoid giving some or even all of the sysadmins the ability to read the CEOs email.

    There weren't in the mid-90's on NT4 and Exchange.

    Waikanae • Since Nov 2006 • 2937 posts Report Reply

  • Bart Janssen, in reply to BenWilson,

    That’s not really the same thing.

    Agreed, but 20 years ago even that would have been a laughable suggestion.

    I've worked with big datasets and modern biology is now a mixture of combining large datasets with in depth detailed knowledge to make progress. As important as the in depth understanding of the detail is, there is now real value in the patterns identifed by the algorithms used to examine the big data.

    It's easy to dismiss the value of the large datasets because it is currently hard to extract information from those datasets. But that is changing rapidly. It doesn't diminish the value of the individual with detailed understanding (in my case someone who looks at the plants, but in your case the beat cop). But to argue that it won't happen because it's too hard now is something I've learned not to do.

    Auckland • Since Nov 2006 • 4461 posts Report Reply

  • Russell Brown,

    The DOJ says it is in “the initial stages of an investigation into the unauthorized disclosure of classified information by an individual with authorized access.”

    Which appears to say that he had authorised access to what he has leaked.

    Booz Allen has acknowledged that Snowden was their guy, but says he worked in the Hawaii NSA office for less than three months. There is a suggestion in that story that the Feds were on to him before the story broke -- tipped off, perhaps, by the Washington Post?

    Auckland • Since Nov 2006 • 22850 posts Report Reply

  • Angus Robertson,

    On a lighter note Obama has just completed a summit with Xi Jinping where the American complained loudly about an on-going violation of cyber security.

    Auckland • Since May 2007 • 984 posts Report Reply

  • Russell Brown, in reply to Angus Robertson,

    On a lighter note Obama has just completed a summit with Xi Jinping where the American complained loudly about an on-going violation of cyber security.

    I ran out of time to work in a Huawei mention.

    Auckland • Since Nov 2006 • 22850 posts Report Reply

  • BenWilson,

    But to argue that it won’t happen because it’s too hard now is something I’ve learned not to do

    I'm sure it will happen. 20 years ago I was one of the people designing resource allocation systems and getting laughed at. I make my comment entirely because I actually remember what the kind of things were that the technocratic management wanted out of the systems, and what actual value they really generated. It was not nothing, but it also wasn't anywhere near the hype.

    Maps were extremely valuable. Put a mapping tool in front of someone who needs to manage things in real time, and you've got an instant friend. Put a computer in there making the actual decisions and you've got a massive job ahead of you. Not just in terms of convincing them it's better, but also in actually getting it to be better.

    10 years later I was mining big data in the form of massive spam and ham sets to enable to automatic blocking of spam. I spearheaded a conversion in our organization to Bayesian filtering (against the wishes and beliefs of management) which made the system finally viable, designed the system from the ground up. I found ways to collect reliable data that had never yet been implemented. This was an appropriate use of data to make decisions, because no one actually wants the job of filtering spam, you're not replacing some specialist. However, I can tell you for sure that if getting it right is really important, you should sort your spam by hand, something that's hardly a big job. The commercial viability of anti-spam systems is mostly predicated around how little it matters if you don't get it exactly right.

    Also, ironically, I was eventually replaced by a large group of people who sort the spam by hand, in an outsourced situation, writing hand rules to block the latest trick. It worked better. Turns out that it's easier to get a super high block rate by having approximately as many people blocking spam as are writing it, which turns out to be remarkably few people for most of it.

    Just laying my own credentials on the table to be having a conversation about statistical data mining of internet information. I'm not the world's foremost expert, but I do have an idea what I'm talking about.

    Auckland • Since Nov 2006 • 10657 posts Report Reply

  • Russell Brown, in reply to BenWilson,

    Also, ironically, I was eventually replaced by a large group of people who sort the spam by hand, in an outsourced situation, writing hand rules to block the latest trick. It worked better. Turns out that it’s easier to get a super high block rate by having approximately as many people blocking spam as are writing it, which turns out to be remarkably few people for most of it.

    Holy shit that's interesting.

    Just laying my own credentials on the table to be having a conversation about statistical data mining of internet information. I’m not the world’s foremost expert, but I do have an idea what I’m talking about.

    Not doubting it for a second. But I did have the impression that the LAPD data project is showing results. It helps guide patrols on where to patrol, so it's a helping decide where your by-hand policing is done.

    Auckland • Since Nov 2006 • 22850 posts Report Reply

  • BenWilson, in reply to Russell Brown,

    Yes, that's just the kind of thing it should be used for. We have similar things in operation here in Auckland, routing ambulances. It's standard operations research stuff, backed up by a big robust set of map data. I interviewed for a job there a few years ago. Gotta scoot now, will see if I can remember who they were. Some crowd in Parnell.

    Auckland • Since Nov 2006 • 10657 posts Report Reply

  • slarty, in reply to BenWilson,

    I love it when you get all operational. Far too much CSI driven policy nowadays... good to remember most of this stuff is about analysis and graft...

    Since Nov 2006 • 290 posts Report Reply

  • Russell Brown,

    WaPo reporter Bart Gellman's side of the story is quite interesting. Snowden's communications asserted a strong moral and intellectual basis for his actions.

    Otoh, Glenn Greenwald is now blowing up on Twitter and saying:

    Bart Gellman's claims about Snowden's interactions with me - when, how and why - are all false.

    Auckland • Since Nov 2006 • 22850 posts Report Reply

  • Kumara Republic, in reply to Russell Brown,

    Ah. I was just about to mention the LAPD project.

    Minority Report could be de-fictionalising sooner than we think.

    The southernmost capital … • Since Nov 2006 • 5446 posts Report Reply

  • slarty, in reply to Russell Brown,

    Google "intelligence led policing". And look at popcenter.org
    Its not about the data so much as applying some thought. Extreme surveillance is lazy and counterproductive: 9/11 happened despite all the data being known to the FBI. Giving more to organisations that cannot handle what they already have just makes it worse...

    Since Nov 2006 • 290 posts Report Reply

  • Sacha, in reply to slarty,

    Agreed. Lack of respectful relationships with other agencies seems a bigger barrier in cases like Boston and 9/11 than lack of data.

    Ak • Since May 2008 • 19745 posts Report Reply

  • slarty, in reply to Rich of Observationz,

    public sector CEs are deemed to have TS clearance by default. have a look at SIGS

    Since Nov 2006 • 290 posts Report Reply

  • B Jones,

    in pursuit of what DARPA called Total Information Awareness.

    I read something about this recently, but unfortunately my retention of ideas is better than my ability to dig up a proper citation - maybe it was Gladwell's thing about thin-slicing. It was a story about some US army war games, in which they were running some Total Information Awareness system through its paces, against one of their more experienced generals. The general came up with a bunch of simple low-tech workarounds so that his side weren't reliant on the technology the Total Information Awareness was all over, and ran circles around it. One of the problems was the huge buckets of data needed so much processing. What you can do with the data is more important than how much of it you can catch.

    Wellington • Since Nov 2006 • 976 posts Report Reply

  • BenWilson, in reply to slarty,

    I love it when you get all operational

    Who, me?

    good to remember most of this stuff is about analysis and graft…

    Mind you, I wouldn't know if there aren't thousands of people charged with reading the outputs of the various data trolling surveillance algorithms, and doing footwork to follow up the leads. And it's not like they could get worse at detecting terrorist plots. Inherently, it's a very hard problem, possibly insoluble, to know when a cell of terrorists is planning something major. I'd think it wouldn't be far off, just on account of the fact that such people would be super paranoid and would avoid the grid as much as possible for their actual dealings.

    Auckland • Since Nov 2006 • 10657 posts Report Reply

  • Craig Ranapia,

    In part, this may be down to Greenwald himself. His dreadful invocation of a rape metaphor to respond to his critics over the weekend fell well short of the conduct we might expect of a public-interest journalist.

    Yeah, but it wish it was at all surprising from someone who went rancid at people suggesting he might hold fire on Zero Dark Thirty (and some rather acidic personal attacks on the director and writer) until, you know... he'd actually seem the (IMO nuts-deeply problematic) film.

    North Shore, Auckland • Since Nov 2006 • 12370 posts Report Reply

  • andin,

    IBM would sell this knowledge to corporate customers with an interest in knowing what was about to happen before it actually happened.

    Only stupid dicks in corporations need to pay for this kind of information

    raglan • Since Mar 2007 • 1891 posts Report Reply

First ←Older Page 1 2 3 4 5 6 Newer→ Last

Post your response…

Please sign in using your Public Address credentials…

Login

You may also create an account or retrieve your password.