The other day I had a conversation with a friend about the the failures leading to the Christmas Bomber. I wasn't especially smart about the issue, except that I'm very aware that the unbelievably monstrous amount of data come into relevant government offices means that eventually some terrorist will slip through the net. My assumption is that there's no perfect filter. We'll get the first 1,000 pieces of data, but the next piece will slip through. My friends in security tell me that my assumption is correct. We don't like the assumption but it's inevitable.
Then I stumbled on a great deal more clarity from a post by Lewis Shepherd. When you want the details on such issues, Shepherd's blog is one place to go.
Shepherd is the analytical guru, not the political guru. He points out that to fix the governmental data mining issues you have to figure out what's broken.
What we heard ad nauseum was about stovepiping. Organizations are not talking to each other, so the data got lost in the connections. Although some reforms are already in place for data sharing, that job is a long way from being finished. It will take five years to implement that sharing.
The second issue is not about organizational turf, but about the formal differences of the data. Intelligence analysts must gather and analyze information from a wide variety of forms, data sets that include: general references, news, technical journals and reports, geospatial data, entity databases, internal reports and more. The different terminologies, formats, data models, and contexts make it difficult to perform advanced analytic tasks across different data sets.
Shepherd summarizes the issues as information sharing or getting wheat out of chaff. Difficult to mine. You bet.