Jeff Jonas

Black Unicorn Hunting

The word Unicorn is often used to describe a start-up that quickly becomes worth over a billion dollars, no matter how profitable. Billion dollar valued start-ups being almost as rare as unicorns.

The term Black Swan, popularized by Nassim Taleb’s book entitled The Black Swan: The Impact of the Highly Improbable, describes the extreme impact of certain kinds of rare and unpredictable events and humans' tendency to find simplistic explanations for these events retrospectively.

Smashing these concepts together, I have coined the words “Black Unicorn” to be those rare circumstances which severely impacts a company by causing a billion dollars or more in damage (e.g., fraud, fines).

Regulatory fines in banking are one example.

Imagine a bank ran off a customer for suspected money laundering activity. Months later, the very same bad actor worked his way back into the very same bank he was tossed out of. Cleverer the second time around, to avoid detection, this customer has opened many seemingly unrelated accounts – just to ensure they remain under the bank’s radar this go-around. Unbeknownst to the bank, the known money launderer is again enjoying an excellent banking relationship.

If the regulators stumble upon this situation before the bank does, and it’s a big enough deal or reveals a larger systemic problem, a billion-dollar consequence is not out of the question.

Black Unicorns - bad news.

Flushing-out Black Unicorns requires one can overcome channel separation. Channel separation is a form of deception tradecraft used to prevent adversaries from piecing together the big picture. Only the idiots use the same credit card to rent a moving truck, purchase a large supply of fertilizer and diesel fuel; or in one email spell out the entire plan including who, what, when, where, and which credit card he'll use.

Bad actors alter their identity information to keep their channels separated. They do not use their real name and date of birth if they can help it, especially if they know their name and date of birth are on your watch list. They will alter their name, maybe swap the month and day on their date of birth, use a different passport, and so on. BTW: If you have ever emailed someone an encrypted document then called them to share the password – yes, you too have used channel separation.

If you cannot see through (defeat) channel separation, you will never catch the clever bad guys.

While defeating channel separation is non-trivial, there is a big data technique we call entity-centric learning – a specialized form of entity resolution – designed to resolve identities despite your adversary’s attempts to maintain channel separation. [More about entity-centric learning at a future time.]

No matter how many Black Unicorns are lurking in your organization, my new company (still in stealth mode) is going to make hunting for Black Unicorns a whole lot easier – flushing them out, sooner not later – before they mature into a full-fledged billion-dollar consequence.

More fun soon.

 

RELATED POSTS:

Prediction: Channel Consolidation

How to Use a "Glue Gun" to Catch a Liar

 

Data Visualization: Outing Hype

Contrary to hype, hopes and dreams, big data visualization is generally not helping humans make novel discoveries. Data visualization has two primary purposes: exploration and storytelling.

Hype. Imagine, for a moment, a beautiful, big ass graph – all four walls in the room covered with icons of people, companies, cars, boats, planes with lines between these entities depicting how they are related. Some icons are blinking. Lines of different colors and widths are undulating at varying rates to signify volume and pace of the flows (e.g., fast moving money). So much information is encoded in this magnificent representation that it took hours to study the legend in order to fully understand what all the color and commotion mean. The big graph can be re-scaled down to the size of a postage stamp fitting nicely on one screen (where at this density it appears as if a solid). Then with one click, the visualization instantly expands back to its previous form; consuming the four walls of screens. Reset to a specific date/time; play events forward or backward, fast or slow; all at a whim. Maybe this visualization system can even recognize hand gestures and voice commands to crawl through the picture – or better yet, this can all be experienced from within an immersive virtual reality environment.

Visualizations like this are magnificent and impressive; delivering true shock and awe.

Question: What are the odds this form factor and experience will help someone find novelty – true weak signal; that proverbial ‘needle in the haystack?’

Answer: Slim to none.

Such claims are likely pure hype.

Data visualization is best suited for exploration and storytelling, not ‘discovery of novelty.’

The most common use of data visualization is “exploration.” In this use case, someone has a specific place to start already in mind; an entrance point. The inspiration for the starting point may have been an alert from a fraud detection system, the boss has asked a question, a personal epiphany followed by a hypothesis, and so on. In each case there is an entrance point in hand before one enters the visualization.

Starting from this entry point one thoughtfully drills in and out, navigates up this branch or down that, etc. This exploration process helps one get their hands around what is known and how things are oriented (context). In conjunction with human expertise and intuition, this visual exploration assists investigations, audits, forensics, diagnostics, etc. Simulations e.g., 3D structural models that help imagine the house before it is constructed make for another good example that falls into the exploration category; the starting point being the front door of your house.

The next most common use of data visualization is “storytelling.” In this case visualization is used to deliver on the “picture is worth 1,000 words” axiom. For example, following much research one has an exciting finding worth sharing; maybe a call to action is in order. What better way to share the findings than an artistically finessed and professionally packaged visual aide. Visualizations used this way help make a case more compelling; whether one is presenting evidence to the judge or enlightening their boss as they ask for more resources.

What to do if one accidentally discovers important novelty when gazing at a big visualization: First, it is probably a good day to go buy a lottery ticket. Next, find out how long this condition was knowable before it was accidentally stumbled upon. Had this been knowable for years? Has this been discovered too late to matter? Also ponder how long it might take a future ‘visualization gazer’ to miraculously stumble upon such a discovery, if it were to happen again?

NOTE: It is for these reasons that systems which allow the data to find the data and the relevance to find you are so essential. Such systems detect actionable insights (new starting points) the moment they become knowable, e.g., supplying an eager fraud analyst that next lead worth exploring in a beautiful data visualization system.

RELATED POSTS:

Data Finds Data

What Came First, the Query or the Data?

The Data is the Query

Context: A Must-Have and Thoughts on Getting Some …

G2 | Sensemaking and its 7 Secret Sauces

 

G2 is 4

Following two and a half years of incubation, G2 was revealed four years ago yesterday.  You ask, “What is G2?” Well, it is specialized Context Computing software, of course. “What is Context Computing?” you now ask. Well, check out this six minute video I made to explain Context Computing!

G2 is seeing some exciting successes. Successes that my team and I are quite proud of. For example:

• In conjunction with the Singaporean government and in partnership with Singapore Technologies, we used G2 to help better focus human attention. In this case, G2 is helping the government decide which vessel traveling through the Malacca and Singapore Straits is of the greatest interest, right this second. With half the world’s oil supply and one-third of the world’s commodities passing through these water ways, this is a non-trivial, mission critical task (see video). One reason we picked this project to “sea trial” G2 is because of the substantial amount of geospatial data – data about where vessels are, and when and how they move. Combining geospatial data with other structured and unstructured sources dramatically improves the ability to discover truly interesting insights.

• In conjunction with The Pew Charitable Trusts, we used G2 to help modernize voter registration in America, enabling several states to get hundreds of thousands of new voters registered, as well as ensuring that their voter lists are more accurate and up-to-date. Using “Selective Anonymization,” one of our baked-in Privacy by Design (PbD) features, the system keeps private data private as member states are not sharing any human readable Personally Identifiable Information (PII) when they send their data to the data center. The ability to perform Context Computing over anonymized data to discover insights was essential to this system’s adoption and subsequent success (see video). When organizations can share anonymized data and get a materially similar result, why would they ever share data any other way? It is for this reason I am very optimistic that Selective Anonymization is going to become rather popular.

• Over 1,000 customers have downloaded G2 via our “Entity Analytics” and/or “Entity Analytics Unleashed” feature that ships inside of SPSS Modeler Premium, a data mining tool (see video and paper). This super easy-to-use feature helps organizations quickly determine who is who and who is related to whom. The enhanced context that comes with reconciling and relating entities helps organizations discover higher quality models. Higher quality models means better business outcomes. One example: At a recent conference someone walked up to me thanked me for the technology – he said G2 discovered over a hundred thousand falsely enrolled students, unraveling a huge scam in his country’s education system.

• Most recently, we have been heads down developing and deploying a new end-to-end solution that will soon hit the market called “Sensemaking for Anti-Money Laundering (AML).” This G2-based system helps banks better triage the massive number of AML leads that their existing transaction monitoring engines generate – primarily false leads that misdirect analyst attention. G2 insights help inform the analyst of critical data points. More informed analysts are then able to be more productive (higher quality work) and more efficient (significantly less time) on each case they investigate. While the solution is very lightweight and easy to attach to existing infrastructure and investments; it is nonetheless a heavily logged and fully reconciling system, as one would expect from such a highly regulated activity. We have seen fantastic results at customer one. Other financial institutions are now lining up to achieve similar gains.

A few technical details for my techie friends.

While we can run on a SQL engine (Oracle, DB2) we are optimized for Key Value data stores. We have a new Key Value data store we are testing with that runs on Remote Direct Memory Access (RDMA). This is scaling linearly and due to its ultra-low latency, it is the first data store we have ever seen where the bottleneck moves from I/O to CPU! Damn exciting.

Thanks to the underlying high speed streaming engine called InfoSphere Streams, we are able to compute Space Time Boxes (STB) and Hang Outs (places things dwell) at a rate of 200k per second per core. I’ve long been fascinated with geospatial data. Recently, we used our STBs to forecast asteroid vs. asteroid interactions over the next 25 years – an astronomy paper is forthcoming. More details here in this speech: Asteroid Hunting & Other Stealthy Things. Geospatial data related to people is going to be extraordinarily useful; but also will come with major privacy issues. We are envisioning that anonymized STBs (using our Selective Anonymization feature) will greatly reduce the risk of unintended disclosure and/or misuse of geospatial data too.

Near term focus is now on reducing CPU in the G2 core and getting data in, out, and onto Hadoop File Systems (HDFS) for those organizations landing more and more data and analytics on Big Data – on premise and/or in the Cloud.

Many more exciting things to come including features I refer to as Selective Curiosity and the Hint Service.

2015 will be a big year for G2.

FYI: My new title over here at IBM is: Chief Scientist, Context Computing. Fitting, as this does happen to be my obsession.

RELATED POSTS:

G2 | Sensemaking – One Year Birthday Today. Cognitive Basics Emerging.

G2 | Sensemaking – Two Years Old Today

G2 | Sensemaking and its 7 Secret Sauces

 

G2 | Sensemaking and its 7 Secret Sauces

Once a year, on its birthday, I say a few words about my G2 project.  Three years ago today, G2 was made public on International Privacy Day in 2011.  With two years of secret “skunk works” development before that; G2 is now five years in the making.  It is no small endeavor.

More background on G2 | Sensemaking can be found here.

I get asked from time-to-time to describe what makes G2 different.  So this year, I’m going to highlight seven rather unique features of G2 – aka seven not so secret sauces.

1. Principle-based Context Accumulation

Systems that make decisions about “when things are the same or related” generally require lots of data for training and/or experts to dictate very domain specific rules.  Lots of data for training is easier said than done, unless you are Google.  And domain experts can be hard to come by when you need them; and after they come and go, one finds themselves being encumbered by an ever growing and complex set of rules – rules few if any will ever fully understand.

Principle-based context accumulation is a new technique to determine “when things are the same or related” that requires neither training data nor experts.  As new data sources (e.g., the company’s asset management system), entity types (e.g., vehicles) and features (e.g., VIN numbers) are introduced to G2, there is no need to first see training data or extend an already unwieldy set of rather brittle rules.

Preparing G2 to make sense of new data sources, new entity types and new features will often take less than ten minutes.  Immediately thereafter, one can start pumping vehicle related data into G2 and G2 will be able to differentiate when two vehicles are the same or related, without any other configurations/complexity.

How? Adding new data sources and entity types is nearly effortless – so simple that this should take no more than two minutes to introduce G2 to these concepts.  Describing the new kinds of features to be expected (e.g., for a vehicle these might include VIN, license plate, make, model, year, color) is the time-consuming part … might even take eight minutes.  This configuration step requires that the user define three behavior options for each added feature:

Frequency: How many entities generally share the same value? One, Few, Many?

Exclusivity: Does an entity generally have only one such value? Yes, No?

Stability: Is this value generally stable over the lifetime of the entity? Yes, No?

For example, a VIN number is simply introduced to G2 as an “F1ES” (Frequency one, Exclusive and Stable) because a VIN number typically refers to exactly one vehicle, a vehicle generally have only one VIN number (i.e., Exclusive), and a VIN number remains the same over its lifetime (i.e., Stable).  The “Make” feature is “FMES” (Frequency Many, Exclusive and Stable).  Color would be “FM” (Frequency Many, and not Exclusive or Stable) because a car can have multiple colors and can be repainted anytime.  Keep in mind these are guidelines, not hard and fast behaviors.  For example, G2 would detect that a VIN of “00000000” is not behaving as expected and take this into account, auto-magically.

Because all features are described as having expected behaviors; a small, manageable number of abstracted rules (aka principles) rely on only these feature behaviors to assert, persist and manage context (determining same and related).  As a result, in most cases, G2’s default principles will be all you will ever need.  So an organization can start with people and organizations and then pivot to vehicles, vessels and routers without training data and without the need for experts to build upon ever expanding, elaborate, rules.

2. Sequence Neutrality at Scale

I use the term “Sequence Neutrality” to mean: Did the records arrive in the order of [A, B, C] or some other order like [C, A, B]?  Regardless of the arrival order of the data, the end-state should be the same.  It took my team and me roughly 20 years of building and deploying entity resolution systems before we stumbled into this most subtle, essential behavior.

Imagine if you learned of a [Mark, 111-22-333] first and a [Mark, [email protected]] second.  With just these two records there is no basis to claim they are the same (based on just the same first name) so they become Entity1 and Entity2.  Some time later you learn of a [Mark, 111-22-3333, [email protected]]; with this information, one could reasonably assert that Entity1 and Entity2 are the same.  As such, this third record should cause Entity1 and Entity2 to collapse into a single Entity – an entity now containing all three observations (records).

Many systems cannot do the above; something we call “re-resolve.”

Few systems can do the reverse, using a new observation to fix previous false positives; something we call “un-resolve.”

And no other technology, to my knowledge, can perform such re-resolve and un-resolve events while managing the relationship graph, in real-time, and at scale.  The fastest we can do this today in our existing industrial strength engine is roughly 2k/second over a database containing >10B records and >500M entities.  G2 is specifically designed to smoke these numbers.

Sequence Neutrality is non-trivial.  Essentially, this concept means that new observations can reverse earlier assertions.  Imagine that.  When ingesting the 10 billionth observation, not only does G2 figure out how it relates to what is known, it also asks the question “Had I known this in the beginning, over the 10B previous observations and assertions I have already made, should I have made any of these assertions differently?”  And if so, these Sequence Neutral algorithms fix the past while integrating the new observation.  G2 has been specifically designed to do this at extraordinary scale, far beyond anything the world has seen yet.

How?  We have fundamentally re-engineered the underlying schemas to work on a distributed number of computers (call it “optimized for the elastic cloud” if you like).  We also have introduced a slight increase in compute during ingestion that pays off big time later when dealing with entities containing very large numbers of conjoined records.  That is about all I am willing to share at this time.

I think this will be the single most difficult aspect of G2 for others to replicate.  And its importance is paramount.  Without Sequence Neutrality new data can cause databases to drift from the truth – data drift.  One common remedy for data drift involves periodically tearing down and reloading the entire database; obviously, the larger the database the less fun this is.  As well, the exciting “Big Data. New Physics.” phenomenon I have described, whereby a system becomes more accurate and faster as more data is loaded cannot be attained without this Sequence Neutrality behavior.

How hard is this?  Now with over 10 years of engineering experience in Sequence Neutral algorithms, my G2 team and I continue to learn, tune, weep, fix and dream.  Nonetheless, I think we are years ahead – if not a decade ahead.

3. Privacy by Design (PbD)

We, just three of us at the time, spent the first year designing G2 on paper, drafting detailed design specifications.  From day one, we felt it was important to bake-in as many privacy and civil liberties protecting features as we could fathom.

All seven of our Privacy by Design features can be read about here: Privacy by Design in the Era of Big Data.  We are proud to say that G2 may have more baked-in privacy features than any other, even remotely similar, technology.

Features like Selective Anonymization (a capability that now ships with SPSS Modeler Premium V16) will allow an organization to perform rich analytics without using human readable forms of Personally Identifiable Information (PII).  This PbD feature reduces the risk of unintended disclosure – and in this day and age given all of the data breaches – this one feature may very well become a business imperative.

4. High Tolerance for Uncertainty

 

“The test of a first rate intelligence isthe ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.”

F. Scott Fitzgerald, "The Crack-Up", Esquire Magazine (February 1936)

 

There is a lot of uncertainty, ambiguity and conflicting information in the real world.  Traditional systems and processes spend considerable effort trying to remedy uncertainties and errors … as they yearn to establish a single version of truth.  Not G2.    There are stark differences between the intentions of Master Data Management (MDM) vs. Sensemaking.

The G2 technology has a high tolerance for uncertainly (e.g., they might be the same or related), ambiguity (e.g., this Pat record could be this Patrick or that Patricia) and information in conflict (e.g., a person who reportedly has eleven reported dates of birth).  In fact, we find that all this natural variability in data – albeit disconcerting at times – is valuable and makes our system smarter.

Turns out: Sometimes bad data is good.  Before you pooh-pooh this crazy talk, might I remind you that you have already seen and benefitted from similar nonsense.

When you search Google and it says “Did you mean this?”

Google is not using a dictionary. 

Rather, it has remembered all the errors. 

If Google had not remembered all the errors (natural variability), it would not be so smart.

G2 is well suited to manage millions and even billions of uncertainties, ambiguities, and contradictions.  And there is no compelling need for humans to review all these maybes.  Instead G2 is quite comfortable letting all these uncertainties just fester – waiting for new information to bring clarity, or not.

G2 is smarter for this reason.  For example, because G2 has such a high tolerance for information in conflict (e.g., noting someone has eleven different dates of birth) G2 comes to realize on its own that it is confused.  Why is this important?  Well, if you are looking to analytics to make important decisions, wouldn’t you want to know during the decision-making process if there was any related confusion before action is taken?

5. Selective Curiosity

Now imagine all of those maybes, uncertainties, ambiguities and dissent, floating around as described above.  Most of those maybes don’t matter and never will.

Imagine a system that routinely comes upon maybe conditions.  And with each such occurrence the system asks itself “If this was true, would it matter?”  Of course, most of the time the answer would be no.  In this case G2 moves on, as it lets this uncertainty fester.  But every now and then, as you might imagine, one stumbles upon a maybe whereby if it turned out to be true … Holy Crap!  Summon the police – Billy the Kid is in the house!

The G2 Selective Curiosity feature, is just that, selectively curious.  It finds a maybe that would matter.  It figures out what it wishes it knew (e.g., If I just knew the work address …).  It figures out where it should go to ask for such a data point (e.g., Google, LexusNexus or … wait for it … a Jeopardy! champion).  Then G2 asks.  And if lucky, the next inbound observation from this inquiry confirms or denies.  It won’t always be so lucky, of course, but such is life.

6. Geospatial Awareness

I firmly believe geospatial data – data about where things are when – will prove to be the highest order bits when it comes to Sensemaking systems.

The power of geospatial data and the privacy ramifications are mind-numbing.  If you want to get excited and creeped-out at the same time about all the potential of geospatial data, check out this session Jennifer Lynch of EFF and I delivered at the 2013 SxSW Conference entitled “I Know Where You're Going: Location as Biometric.”

With G2 I have introduced the notion of Space-Time-Boxes (STB’s).  A STB is used to group nearby coordinates with nearby time.  The conversion of geospatial data to STB’s enables exceptionally fast correlation of space and time.  The technique also helps account for rather large errors in precision, although these details are beyond the scope of a happy birthday G2 blog post.  [We have a few techie papers.  If you want to know more and are willing to agree to no further redistribution, drop me a note and make your case.]  In short, STB’s will allow G2 to contribute to new and exciting domains ranging from better identity theft protection and maritime domain awareness to the hunt for asteroids.

Note: While asteroids don’t have privacy, when dealing with geospatial data about consumers I first strongly recommend you let them opt-in.  And then if you are going to perform compute over their geospatial data I would encourage you to use STB’s in conjunction with G2’s Selective Anonymization feature (as discussed above in the Privacy by Design section).  This means rich geospatial analysis can be performed while at the same time reducing the risk of unintended disclosure.

7. Diverse Perspective

Imagine a rather large pile of messy puzzle pieces (some pieces missing, some duplicates, some professionally fabricated lies, etc).   If we gave exactly this same set of puzzle pieces to two people, and gave them each some finite amount of time to make the most of it.  Do you think both assemblies would be exactly the same?  Of course not.  Now, while much of the puzzle assembly is likely to be in agreement between these two people, one may have missed a few obvious connecting pieces (false negatives).  The other may have inadvertently connected two pieces in error (false positives).

Why is this?  Despite having the same observation space presented to them … they each used their own strategy and biases to assert which pieces belonged where.

For example, one analyst considers name and date of birth sufficient evidence to claim someone is the same (rarely a good idea by the way).  Another analyst believes this should be treated as just a potential match – at least until more evidence emerges such as a similar address.  Maintaining both perspectives allows G2 to notice there may be different opinions about the contextual interpretation of things, despite the fact the observation space is the same.

Under the covers, G2 utilizes the notion of a “Lens.”  Raw observations are maintained in a single place and in a single state.  Interpretations about how these observations relate to each other are managed and maintained through one or more “Lenses.”  This Lens construct allows varying perspectives to co-exist over a single observation space.

The ability to recognize Diverse Perspectives is essential to Sensemaking as this enables, among other things, one to notice there is a Minority Report when considering critical decisions.

Disclaimer

Of the seven capabilities described above, items 1, 2, 3, 4 and 6 are in relatively decent working order.  We have not yet started working on items 5 (‘Selective Curiosity’) and 7 (‘Diverse Perspectives’) – these are on our road map.  I have lots of things on the G2 roadmap – at least a few more years’ worth of exciting features.

In Closing

G2 is not trying to deliver Artificial Intelligence or Cognitive Computing.  That said, I think G2’s capacity to deliver “incremental context accumulation” may be a fundamental stepping stone for such things as Artificial Intelligence and Cognitive Computing in the future.

To be clear about what G2 is and is not:  G2 does not extract structure from unstructured data.  G2 does not discover new patterns (more about the relationship between what G2 does versus “Deep Reflection” here in this video).  G2 does not express itself in any human consumable, visual form (think XML in, XML out).  While G2 does not do these things, one day G2 may help these technologies do these things better.

So what does G2 do?  Put in the simplest terms: G2 helps finds the obvious. 

 

 “All truths are easy to understand once they are discovered; the point is to discover them.”

Galileo Galilei (1564-1642)

 

So what is so great about G2?  Well … yes, it finds the obvious … at the moment it becomes knowable, over billions and billions of historical observations, in real-time, responding fast enough to do something about it while it is still happening.

Availability, you ask … How do you get some?  Well, this is not open source, my friend.  And if you are waiting for anything close to appear in open source, I suggest you do not hold your breath.  Fear not, there is a relatively inexpensive way to get your hands on G2.  In fact, there is some chance if you work for a big company you may already own it.  A light-weight version of G2 is now commercially available via SPSS Modeler Premium V16.  This very easy to use version of G2, manifesting itself as the “Entity Analytics” node, which is included at no charge.  If you own this, you can get real business done today.  It comes with Selective Anonymization and Space-Time-Boxes (STB’s) too.  And, if you have more than 10M records and want to benefit from degrees of separation (how entities are related) there is an “Entity Analytics Unleashed” version that will cost you a few more bucks.

As you can probably tell, this project is my “MAIN THING” and is very exciting.  I would like to thank my team that works tirelessly as they perform their engineering feats of strength.  And, last but not least, I would like to thank IBM, my employer, for their faith in this work and the significant investment they have made to date and continue to be making.

 

MORE ABOUT SPSS MODELER WITH ENTITY ANALYTICS (G2):

Using Entity Analytics to Greatly Increase the Accuracy of Your Models Quickly and Easily (Video and paper)

 

RELATED SPEECHES:

Analytics Education Series.  Case Study: Entity Resolution

I Know Where You're Going: Location as Biometric (SxSW Conference 2013)

 

RELATED POSTS:

G2 | Sensemaking – One Year Birthday Today. Cognitive Basics Emerging.

G2 | Sensemaking – Two Years Old Today

Big Data. New Physics

Entity Resolution Systems vs. Match Merge/Merge Purge/List De-duplication Systems

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

Privacy by Design in the Era of Big Data

To Anonymize or Not Anonymize, That is the Question

Your Movements Speak for Themselves: Space-Time Travel Data is Analytic Super-Food!

Structuring Unstructured Data

 

NSA Superheroes Help Fix Everyone’s Internet #WildIdeas #SpyReboot

I do believe it is game over for business as usual for those in the intelligence business.  Radical reinvention of the US intelligence community is in order.  What we need now are big, creative, wild ideas as we re-create our next generation, world-class, trusted, hyper-efficient, and globally admired intelligence apparatus.  I explained this thinking here #SpyReboot.

With this in mind … how about this crazy idea for starters:

Maybe one of the NSA’s new roles would include: timely and vigilant disclosure of every zero-day exploit, design weakness or other discovered vulnerability they find in the world’s information and communications technology infrastructure.  These discreet disclosures would be made to manufacturers globally – US and foreign companies alike, including for example the Chinese company Huawei.  Why?  Well, most of the world is friend, not foe … and lots of our friends use other people’s stuff.

While patching these hardware and software deficiencies is the responsibility of the manufacturer; just because they are notified is no guarantee they act.  So maybe one way to compel manufacturers to take action would involve publishing some specific statistics about how many discovered/disclosed vulnerabilities remain un-answered- all in the name of transparency, of course.  If manufacturers competed for security and trust around these public statistics, maybe everybody wins.

The NSA will still be sneaky.  And if this sneaky work uncovers new vulnerabilities being eyed by foreign governments or criminal organizations, these too will be revealed.

True, we will be shooting ourselves in the foot with respect to our ability to penetrate some significant, hard to reach adversaries.  But balance this cost with the many upsides: (1) The world’s infrastructure is more secure; 2) Global banking is less at risk; (3) There are fewer data breaches as it becomes harder for bad guys to steal data; (4) Average citizens world-wide will have more confidence in the security of their private communications; (5) Dissidents in oppressive countries will be less at risk when speaking up; (6) The amount of stolen US intellectual property over cyber channels will see all-time lows; (7) US technology manufacturers will recover much of the world’s trust in its ability to produce safe and secure products.  Outcomes of such significance may well be worth the loss of some intelligence collection.  Oh and by the way, I think foreign intelligence organizations will be at least as disadvantaged as the US, as their exploited foreign (and domestic) pipes that they currently enjoy also begin to dry up.

This one shift in mission would help level the cyber playing field in a way that the US could lead with credibility – yes, in part because the world now knows (thanks to the leaked documents) just how superior the NSA is, offensively.

The word ‘loved’ might be too strong, but do not underestimate the goodwill and trust towards the USA that would come from the free, voting people of the world as the NSA takes on the mission to help secure the Internet for everyone.

The NSA will still be expected to carry out offensive espionage, including SIGINT collection against foreign targets.  And just because they have discretely disclosed the vulnerability to a manufacturer does not mean they won’t exploit the hell out of it until someone patches it.  The NSA will also surely be investing in and using other clever (secret) methods – methods other than weakened, commercially available technology.

This might actually be a horrible idea.  I don’t know.  The devil is in the details.  How would this be implemented?  What is the new oversight mechanism?  What kind of transparency would accompany this mission?  Would publishing the total number of disclosed vs. fixed vulnerabilities by manufacturer and product really compel companies to make fixes faster?

Oh, and while I am thinking about it: to maximize this Jiu-jitsu move, maybe this white hat team of NSA geniuses should be moved from the NSA to a different organization, e.g., National Institute of Standards and Technology (NIST), National Telecommunications & Information Administration (NTIA) at the Department of Commerce, or Federal Communication Commission (FCC).

This idea might be too wild, or God forbid, not wild enough.  One thing is clear – we need to break out of the old ways of thinking.  Instead, let’s all start focusing on wild ideas in general, ideally hundreds of such ideas, as we envision a next generation, trusted, world-class, hyper-efficient, and globally admired intelligence apparatus.

Lots of law and policy have to be fixed too – wait till you hear my wild idea about what to do about the problematic “3^rd Party Doctrine.”

RELATED POSTS:

#SpyReboot

 

 

#SpyReboot

When the captain of the Titanic learned five compartments of its hull had been breached by the iceberg, he knew what was going to happen next.  When thinking about the scope of Snowden’s disclosed documents (e.g., NSA’s ANT Division Catalog of Exploits), it seems to me one could come to a pretty quick conclusion about what happens next: Game over for business as usual for the US intelligence community.  It is time to reboot the US intelligence community – doing this will require big, creative, wild ideas.

The intelligence community is already in a monster pickle and the world has only seen a fraction of the documents, if the reported number of stolen documents is true.

From the 80,000 foot view I can foresee so many second and third order effects in the future – it’s quite mind numbing.  Sure, we now have a bigger chunk of the US population that does not trust our government; a growing concern over the erosion of our 4^th Amendment in the name of national security; and a global outcry saying ”foul.” Important debates around these things are happening now.  But this is not even remotely half the problem for the US intelligence community.

Foreign manufacturers of information and communications technology (ICT) are surely going to use these revelations to undermine US products in their sales pitches.  US industry is at risk of losing billions in business over coming years, whether they were called out in the Snowden documents or not.

Foreign intelligence organizations may look upon these documents as roadmaps and best practices.  If these organizations point these new capabilities inward, especially in less democratic countries, the citizens of these countries will be subject to unprecedented levels of government surveillance.  So the US becomes more transparent and our 300 million US citizens benefit from less wholesale surveillance all the while several billion people experience exactly the opposite.  Unremedied, we, the US, will bear responsibility for this.

On the current trajectory, trust at home and abroad is going to shake the rafters, economically and otherwise.  All the while our intelligence apparatus is going to see ever-shrinking yields on the money we spend.  Said another way, on the current heading, given the vessel, current weather and coming forecast, the risk to the crew (our democracy) is unacceptably high while at the same time the previous well-traveled route will now prove to be an epic waste of energy.

While the recent recommendations by the President’s Review Group on Intelligence and Communications Technologies cover a wide range of things worth considering and/or fixing, these are incremental tweaks (e.g., where the metadata lives).  Even if all of these recommendations are followed, a major ‘methods of operations’ reboot is still in order.  From my point of view, making incremental tweaks at this point is akin to the Titanic captain proposing to preserve ship power by re-wiring the electrical system, while the vessel will still, no matter what, be going nose-down.

Radical reinvention of US intelligence community is in order.  What we need now are big, creative, wild ideas as we re-create our next generation, world-class, trusted, hyper-efficient, and globally admired intelligence apparatus.

I plan on posting some of my own wild ideas soon.

I encourage you to propose your own wild ideas too – use the #SpyReboot #WildIdeas hash tags so folks can find them.

Structuring Unstructured Data

When discussing the importance of weaving together an organization’s diverse observation space (data), often this question comes up: “How does one related structured data to unstructured data?”

My shortest answer is: “Shared features.”  Let me explain.

Observations are only useful to the extent you can extract features from them.  A picture taken at night without a flash, that is pitch black, is nearly useless as it contains no extractable features.  The cell phone call that suddenly goes bonkers and becomes all garbled is equally useless because you are unable to extract any useful features from the noise.  On the other hand, a video from the parking garage – from which the license plate number, location and time features are easily extracted – is easily related to other data.

This point of view that “observations are only useful if features can be extracted from them” levels the playing field between different kinds of data sources, at least from a systems architecture point of view.  Simplified architecture?  Check.

Unfortunately, getting high quality features extracted from unstructured observations – at scale without humans – is easier said than done. 

Beneficiary vs. Victim

When I talk to the hard-working folks building and deploying feature/entity extraction algorithms I often first say “My context accumulating engines are beneficiaries of your work.”  They typically respond with a grin as they hold their head high.  Then I say “I am also the victim.”  Buzz kill, no more grin, as they know exactly what I am talking about as the accuracy rates of feature/entity extraction leave much to be desired.  A common counterclaim: It is better than doing nothing at all!  So true, today these feature extraction algorithms may be the only viable option when there are not enough people to do this by hand anyway.

Eventually, I think breakthroughs in feature/entity extraction algorithms will mean no more need to manually map structured data (e.g., to common field names).  Instead, we will just forklift in boatloads of structured data and it will auto map – knowing the difference between first name vs. middle name fields and home address vs. work address fields merely by evidence found in the data.

As features find features, data will find data.

Whether the observation space is structured or unstructured is going to matter less and less.

 

RELATED POSTS:

Data Finds Data

Context: A Must-Have and Thoughts on Getting Some …

 

G2 | Sensemaking – Two Years Old Today

My latest big, bold, new invention, is two years old today!

What is G2?

When I speak about Context Accumulation, Data Finds Data and Relevance Finds You, and Sensemaking I am describing various aspects of G2.

In simple terms G2 software is designed to integrate diverse observations (data) as it arrives, in real-time.  G2 does this incrementally, piece by piece, much in the same way you would put a puzzle together at home.  And just like at home, the more puzzle pieces integrated into the puzzle, the more complete the picture.  The more complete the picture, the better the ability to make sense of what has happened in the past, what is happening now, and what may come next.  Users of G2 technology will be more efficient, deliver high quality outcomes, and ultimately will be more competitive.

Early adopters seem to be especially interested in one specific use case: Using G2 to help organizations better direct the attention of its finite workforce.  With the workforce now focusing on the most important things first, G2 is then used to improve the quality of analysis while at the same time reducing the amount of time such analysis takes.  The bigger the organization, the bigger the observation space, the more essential sensemaking is.

About Sensemaking

One of the things G2 can already do pretty darn well – considering she just turned two years old – is ”Sensemaking.”  Imagine a system capable of paying very close attention to every observation that comes its way.  Each observation incrementally improving upon the picture and using this emerging picture in real-time to make higher quality business decisions; for example, the selection of the perfect ad for a web page (in sub-200 milliseconds as the user navigates to the page) or raising an alarm to a human for inspection (an alarm sufficiently important to be placed top of the queue).  G2, when used this way, enables Enterprise Intelligence.

Of course there is no magic.  Sensemaking engines are limited by their available observation space.  If a sentient being would be unable to make sense of the situation based on the available observation space, neither would G2.  I am not talking about Fantasy Analytics here.

From Insight to Relevance

Not often does a single observation fully contain sufficient information to trigger a high quality, immediate reaction. 

Imagine looking out your kitchen windows only to witness your neighbors in an epic fight.  A few days later you witness the husband purchasing a handgun.  A few days later, while trying to fall asleep, you hear a somewhat muffled ”bang” sound from next door.  The next morning, as you leave the house for work, you see the husband dragging a sleeping bag loaded up with something very heavy toward his pickup truck.

Insights add up.

From a G2 | Sensemaking perspective, the notion is ”Insight to Relevance.”  Of course, determining what is a bona fide insight and combinations of insights become relevant (for action) and requires domain knowledge, modeling, and other real work.  However, unlike brittle rule-based systems, G2 (like any good Sensemaking engine) performs best when taught principles.

General Purpose Context Accumulation

After recently sizing up G2 in action, I pondered what set of features make General Purpose Context Accumulation so unique. 

Complete Context: General purpose context accumulation systems must be indifferent to diverse observations e.g., originating from such sources as structured, semi-structured, unstructured text, social media, pictures, and video each containing events and transactions with temporal, geospatial, identifiers, biographics, biometrics and other features.  To the extent features can be extracted from an observation and delivered to such engines, they contextualize these diverse observations at the same time, in the same data space – every individual observation having an equal opportunity to find and benefit from all other observations.

Current Context: New observations are incrementally integrated with the whole (puzzle) in real time.  As such, G2 is fully capable of recognizing opportunity and risk at the split-second such relevance becomes knowable.

Conflicting Context: There is no such thing as a single version of truth when it comes to general purpose context accumulation.  Context accumulating engines let dissent fester.  When you search Google and it says “Did you mean ____?”,  the same principle is at work.  Google is not looking in a static dictionary.  No, Google is remembering everyone’s errors.  If Google did not remember the errors, it would not be so smart.  In the same way, context accumulating systems let disagreement coexist; otherwise new emerging trends and weak signal will never have a chance to add up to anything.

Self-Correcting Context: This is the most essential ingredient of context accumulation.  I believe few, if any, analytics in the world can do this – and especially at our scale.  Imagine having already seen and contextualized billions of historical observations, and now the next record arrives.  At this moment one must not only decide where to place this new observation, one must also decide if this new observation (had it been known in the beginning of time i.e., first), warrants the reversal of any of the billions of previous assertions.  And if so fix them.  The effect being: new observation can reverse earlier assertions.  Smart Systems Flip-Flop.  Doing this in real-time over billions of observations is non-trivial.  This happens to be the single most technically sophisticated aspect of context accumulation and well worth the last 10 years of effort we have spent researching, tweaking, and tuning our technique.  The difference between our previous method and the new G2 method should be more than an order of magnitude more efficient when dealing with some of the nastiest scenarios that one stumbles upon when implementing algorithms that can change their minds about the past.  In any case, it is this behavior alone that delivers what I call “Big Data. New Physics.”  Essentially, as the observation space widens natural variability in the data (errors) start to become your friend, the false positives and false negatives both begin to self-correct, and at the same time the computation effort required to make sense of the next observation begins to DECREASE as the observation space grows!  Translation: As the database grows, predictions become more accurate while computational effort decreases.  Absolute magic.

To tell you the truth, one of the most exciting things for me is this: when considering the wide range of unrelated domains I have explored with G2 (from maritime domain awareness and anti-money laundering to genealogy work using the 1880 census) – believe it or not, I think G2 could perform context accumulation over all of them at the same time, on the same computer, in the same database schema, using the same configuration.  Not that anyone would want to, mind you, or should.  Nonetheless, this idea that G2 is so general purpose quickens my pulse simply because this has never been possible before.  You see, I am hopeful that this means G2 will (one day) be able to readily make almost anything smarter – from managing my personal calendar (while better protecting my privacy) to helping researchers head-off some of the most debilitating diseases like Alzheimer’s.

Exciting Times Ahead

There is a lite-version of G2 on the market now, deeply embedded in an off-the-shelf predictive modeling product.  I continue to hand select special projects for the big Sensemaking version of G2 to press and stress this technology in new ways – using these “sea trials” to drive her development.   I can’t share details about these efforts just yet but boy she is growing up quickly.  And just to be clear: I am not saying this is easy, far from it, especially in these formative years.  Make no mistake; this is very hard work requiring the team and I to work crazy long hours

I am a dreamer with a long list specific engineering tasks to further advance G2 – things like ‘selective curiosity’ where G2 thinks of its own questions and knows whom to ask (e.g., a Jeopardy! Champion) and a ”hint service” that among other things may help feature extraction algorithms perform much better (without traditional training data).

G2 is different than other software, very different.  Another example, saving the best for last, we have built a series of features into G2 for enhanced privacy and civil liberties protections.  These socially responsible features factored into the original blueprints – from conception – a design approach called Privacy by Design (PbD). 

Finally: While I may be driving the G2 vision, I am certainly not building it.  I have an amazing group of engineers who actually do all the real work.  We also have some very patient customers willing to tolerate all the ups and downs that come with being early adopters.  And without both of them none of this would be possible.  So to them I say … thank you.

 

[TECH NOTES]

A few technical comments for my more technically minded friends.  Entirely coded in C++.  Schemas designed specifically for primary key row stores (although capable of running against a SQL back-end).  One index per table, never two.  Application-aware sharding (supporting heterogeneous data stores, table partitioning, and uniform distribution of records over the available grid nodes).  Most tested on DB2.  Eager to finish testing against other popular SQL back-ends.  Hoping to be able to demonstrate near linear scale against a NoSQL row store very soon.  G2 does not run on Hadoop Map/Reduce (because it is not batch), although Hadoop-based systems will certainly benefit from G2’s accumulated context.

Talk to G2 via HTTP, flat file, SPSS Modeler stream jobs, or compile it into InfoSphere Streams for stream computing over big data.  No user interface work being done at this time.  Using our standard and very simple XML-based Universal Message Format (UMF), G2 eats inbound observations and spits out interesting chunks of the puzzle, aka resumes, when it has something useful to say.

G2 does not do entity/feature extraction on unstructured data.  G2 does not do pattern discovery or anomaly detection.  Hopefully G2 will one day help such things perform better.

Just to be clear, G2 is far from perfect – as upon close inspection she clearly has some moles and warts and even still makes poopy pants from time-to-time.  And I am already fretting the teen years.  Nonetheless, despite these distractions, I can say with certainty this is a journey I am deeply committed to, it is the right direction, hence my ongoing and relentless focus and motivation.

 
 

RELATED VIDEOS (In my own words, Courtesy of Redbooks)

Enterprise Amnesia versus Enterprise Intelligence

Using Entity Analytics to Greatly Increase the Accuracy of Your Models Quickly and Easily

 

 

RELATED POSTS:

On A Smarter Planet … Some Organizations Will Be Smarter-er Than Others
What Came First, the Query or the Data?

The Data is the Query

Data Finds Data

Puzzling: How Observations Are Accumulated Into Context

Accumulating Context: Now or Never
Asserting Context: A Prerequisite for Smart, Sensemaking Systems

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems
Sensemaking on Streams – My G2 Skunk Works Project: Privacy by Design (PbD)
G2 | Sensemaking – One Year Birthday Today. Cognitive Basics Emerging.

General Purpose Sensemaking Systems and Information Colocation
Smart Systems Flip-Flop

Big Data. New Physics.
There Is No Such Thing As A Single Version of Truth

Master Data Management (MDM) vs. Sensemaking

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

Federated Discovery vs. Persistent Context – Enterprise Intelligence Requires the Later

When Federated Search Bites

Enterprise Intelligence – My Presentation at the Third Annual Web 2.0 Summit

Self-Correcting False Positives/Negatives: Exonerate the Innocent

Privacy by Design in the Era of Big Data

Responsible Innovation: Designing for Human Rights

 

Fantasy Analytics

 

Sometimes it just amazes me what people think is computable given their actual observation space.  At times you have to look them in the eye and tell them they are living in fantasyland.

 

Here is how an example conversation:

Me:                 “Tell me about your company.”

Customer:       “We are in the business of moving things through supply chains.”

Me:                 “What do you want to achieve with analytics?”

Customer:       “We want to find bombs in the supply chain.”

Me:                 “COOL!”

Me:                 “Tell me about your available observation space.”

Customer:       “We have information on the shipper and receiver.”

                       “We also know the owner of the plane, train, truck, car, etc.”

                       “And the people who operate these vehicles too.”

Me:                 “Nice.  What else do you have?”

Customer:       “We have the manifest – a statement about the contents.”

Me:                 “Excellent.  What else you got?”

Customer:       “That’s it.”

Me:                 “WHAT?!"

                       "YOU ARE NEVER GONNA FIND A BOMB!”

                       “NO ONE WRITES ‘BOMB’ ON THE MANIFEST!”

 

The problem being; often the business objectives (e.g., finding a bomb) are simply not possible given the proposed observation space (data sources).

[Unless, in this case, the perpetrator writes the word “BOMB” on the manifest.  And only idiots do that.  And luckily we don’t have to worry much about the idiots as they run out of gas on the way to the operation and take wet matches to their fuses.]

When we software engineering folks get overly excited and run off and build systems with little forethought about the balance between the mission objectives and the observation space, there is a risk the system will be a useless piece of crud – so many false positives and false negatives – the value  of the system not worth the cost.

As I have no interest in spending intense chunks of my life building pointless systems … when initially scoping a project I recommend first qualifying the available observation space to determine if it is sufficient to deliver on the mission objectives.  And if the available observation space is insufficient, then one must first figure out if/how the observation space can be appropriately widened.

In case you are interested, here are the some of the ways I try to mitigate these risks: 

Qualifying Observation Spaces

1. Ask for real examples from the past of things they would like to detect (opportunity or risk), and then look in the real data to see if, upon human inspection, it is discoverable.
2. If real examples from the past cannot be detected in the provided data sources, I tell the them “not even a sentient being could discover this.” 
3. Have them name their data sources and the data elements (key features). 
4. Then, just because they say a data source has certain features, go look yourself – I can’t tell you how many times I go take a look and find key columns are empty or so dirty that the value of that data source is negligible.
5. If the data sources share common features between them (e.g., customer number, address, phone number, etc.) then generally more is good.
6. For those data sources that have no (or few useful) shared features (e.g., one data source has name and address and the other data source has stock symbol and stock price) then generally this is not so good.

Widening Observation Spaces

1. There will be many cases where it becomes necessary to help the customer think about widening their observation space … if they are ever to realize their hopes and dreams (business objectives).
2. Conjuring up additional data to expand the observation space is quite an art and requires real-world understanding of what and how data flows inside the walls and outside the walls as well the legal and policy ramifications.
3. Generally one starts looking for new data sources in this order: 1) other stuff inside the walls that you already collect (e.g., product returns), 2) external data that can be purchased (e.g., marketing flags like “presence of children” and “income indicators” as routinely sold by data aggregators).  Of course there are other options like collecting more data themselves (e.g., adding a field to a web page so their customers can express sentiment, capturing the fingerprint of the device during on-line transactions, etc.)
4. If you are trying to catch bad guys, hope that some of the data sources would be unknown or non-intuitive to their adversary (if the bad guys know you have cameras on these four streets, then they will take the fifth street).
5. Beware of social media: There is much allure to the idea that one can computationally map Twitter statements (about your company/brand) to which customer said it.  Go take a look yourself and see how often the Tweet account contains sufficient features to make an accurate identity assertion.  I think you will see the frequency in which an identity can be asserted is underwhelming.  Different countries and different kinds of social sites will have different statistics.  In any case, be wary and look for yourself first.
6. Now let’s say one has a list of potentially new data sources to use.  Then the next question is how to prioritize all these possibilities.  Again, there are a lot of ways to think about this – but here are a few common ways I think about this: A) Data that improves the ability to count or relate entities (e.g., a source that may contains new identifiers like email addresses) so that one can discover that two customers you thought were different are more likely the same customer; B) Data that brings more facts (e.g., what, where, when, how many, how much); C) Diverse data potentially containing identifiers and facts in disagreement (e.g., this fact indicates they are here, but that fact shows they really may be over there – helpful if trying to keep strangers from using your credit card).
7. Finally, don’t forget there will be plenty of times that the mission objectives cannot be achieved because the necessary observation space is not available.  In which case, punt.

The above list is somewhat off the cuff, certainly incomplete.  So … please consider it a starter kit and hack at it any which way you like …

 

OTHER RELATED POSTS:

Data Beats Math

Context: A Must-Have and Thoughts on Getting Some …

More Data is Better, Proceed With Caution

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

 

 

 

 

 

Privacy by Design in the Era of Big Data

Ann Cavoukian, Information and Privacy Commissioner, Ontario, Canada and I released a joint paper June 8, 2012 entitled “Privacy by Design in the Era of Big Data” available here.

In this paper, Ann describers her aspirations for Privacy by Design (PbD) to which I call out specific PbD inspired engineering decisions the team and I made during the development of my latest (software) invention.

Internally code named “G2,” this now 3.5+ year effort started with a full year first working on paper only.  No different than first rendering detailed architectural plans to paper before building a house, our first year was dedicated to overcoming what we thought were our performance limitations of the past while simultaneously weaving in as many advances in privacy and civil liberty protections as we could fathom at the time.

As a result, I can say that this new technology (something you may come to hear about one day called Sensemaking) has more privacy and civil liberties protecting features than any technology my team and I have ever created in the past.  In fact this technology may have more baked-in privacy and civil liberties enhancing features of any advanced analytic software ever engineered.  I would love to be wrong about this – starting a fierce competition over “I have more privacy features than you” is going to be a good thing for planet Earth.

Here are a few highlights from the report that you may find interesting:

While organizations have practical incentives to make the most of their ever growing observation space (the data they have access to), they also have a pressing need to embed in these systems enhanced privacy protections.  We outline in this paper just such an example — how an advanced Big Data sensemaking technology was, from the ground up, engineered with privacy-enhancing features. Some of these features are so critical to accuracy that the team decided they should be mandatory — so deeply baked-in they cannot be turned off.                                           

                                                                                                                ~ page 2

But as technological advances improve our ability to exploit Big Data, potential privacy concerns could stir a regulatory backlash that would dampen the data economy and stifle innovation.

                                                                                                                ~ page 3

A new class of analytic capability is emerging that one might characterize as “general purpose sensemaking.” These sensemaking techniques integrate new transactions (observations) with previous transactions — much in the same way one takes a jigsaw puzzle piece and locates its companions on the table — and use this context-accumulating process to improve understanding about what is happening right now. Crucially, this process can occur fast enough to permit the user do something about whatever is happening while it is still happening. Unlike many existing analytic methods that require users to ask questions of systems, these new systems operate on a different principle: the data finds the data, and the relevance finds the user.

                                                                                                                ~ page 5

However, in these new systems the task of ensuring data security and privacy becomes harder as more copies of information are created. Large data stores containing context-accumulated information are more useful not only to their mission holders but also to those with interests in misuse. That is, the more personally identifiable information Big Data systems contain, the greater the potential risk. This risk arises not only from potential misuse of the data by unauthorized individuals, but also from misuse of the system itself.

                                                                                                                ~ page 7

PbD prescribes that privacy be built directly into the design and operation, not only of technology, but also how a system is operationalized (e.g., work processes, management structures, physical spaces and networked infrastructure.)  Today, PbD is widely recognized internationally as the standard for developing privacy compliant information systems. As a framework for effective privacy protection, PbD’s focus is more about encouraging organizations to both drive and demonstrate their commitment to privacy than some strict technical compliance definition.

In short, in the age of Big Data, we strongly encourage technologists engaged in the design and deployment of advanced analytics to embrace PbD as a way to deliver responsible innovation.

                                                                                                                ~ page 9

In late 2008, Jeff Jonas embarked on an ambitious journey to create a sensemaking-style system. This effort started with overall architecture planning and design specifications. Over the first year of this project, while drafting and redrafting these blueprints, his team worked to embed properties that would enhance, rather than erode, the privacy and civil liberties of data subjects.

To engineer for privacy, his team weighed performance consequences, default settings, and which, if any, PbD features should be so hard wired into the system they literally cannot be disabled.

Over the year that spanned the preliminary and detailed design, the team created a robust suite of PbD features.

                                                                                                                ~ page 9

The dynamic pace of technological innovation requires us to protect privacy in a proactive manner in order to better safeguard privacy within our societies. In order to achieve this goal, system designers should be encouraged to practice responsible innovation in the field of advanced analytics.

With this in mind, we strongly encourage those designing and building next-generation analytics of any kind to carry out this work while being informed by Privacy by Design as it relates to personally identifiable data.

                                                                                                                ~ page 13-14

One thing is for sure: our PbD efforts are a work in progress – much has yet to be done.  Comments and critiques are most welcome.

In the meantime, if you are an engineer consider becoming a student of privacy yourself and then begin engineering with PbD in mind.  And if you are a privacy/civil liberties advocacy type, find yourself an engineer to take under your wing – and be gentle as you nurse these newbies along.

 

RELATED MATERIAL:

Privacy by Design (PbD)

 

RELATED POSTS:

Big Data Q&A for the Data Protection Law and Policy Newsletter

Sensemaking on Streams – My G2 Skunk Works Project: Privacy by Design (PbD)

G2 | Sensemaking – One Year Birthday Today. Cognitive Basics Emerging.

Responsible Innovation: Designing for Human Rights

Responsible Innovation: Some Things are Best Left Un-invented

Responsible Innovation: Staying Engaged with the Privacy Community