Jeff Jonas

G2 | Sensemaking – Two Years Old Today

My latest big, bold, new invention, is two years old today!

What is G2?

When I speak about Context Accumulation, Data Finds Data and Relevance Finds You, and Sensemaking I am describing various aspects of G2.

In simple terms G2 software is designed to integrate diverse observations (data) as it arrives, in real-time.  G2 does this incrementally, piece by piece, much in the same way you would put a puzzle together at home.  And just like at home, the more puzzle pieces integrated into the puzzle, the more complete the picture.  The more complete the picture, the better the ability to make sense of what has happened in the past, what is happening now, and what may come next.  Users of G2 technology will be more efficient, deliver high quality outcomes, and ultimately will be more competitive.

Early adopters seem to be especially interested in one specific use case: Using G2 to help organizations better direct the attention of its finite workforce.  With the workforce now focusing on the most important things first, G2 is then used to improve the quality of analysis while at the same time reducing the amount of time such analysis takes.  The bigger the organization, the bigger the observation space, the more essential sensemaking is.

About Sensemaking

One of the things G2 can already do pretty darn well – considering she just turned two years old – is ”Sensemaking.”  Imagine a system capable of paying very close attention to every observation that comes its way.  Each observation incrementally improving upon the picture and using this emerging picture in real-time to make higher quality business decisions; for example, the selection of the perfect ad for a web page (in sub-200 milliseconds as the user navigates to the page) or raising an alarm to a human for inspection (an alarm sufficiently important to be placed top of the queue).  G2, when used this way, enables Enterprise Intelligence.

Of course there is no magic.  Sensemaking engines are limited by their available observation space.  If a sentient being would be unable to make sense of the situation based on the available observation space, neither would G2.  I am not talking about Fantasy Analytics here.

From Insight to Relevance

Not often does a single observation fully contain sufficient information to trigger a high quality, immediate reaction. 

Imagine looking out your kitchen windows only to witness your neighbors in an epic fight.  A few days later you witness the husband purchasing a handgun.  A few days later, while trying to fall asleep, you hear a somewhat muffled ”bang” sound from next door.  The next morning, as you leave the house for work, you see the husband dragging a sleeping bag loaded up with something very heavy toward his pickup truck.

Insights add up.

From a G2 | Sensemaking perspective, the notion is ”Insight to Relevance.”  Of course, determining what is a bona fide insight and combinations of insights become relevant (for action) and requires domain knowledge, modeling, and other real work.  However, unlike brittle rule-based systems, G2 (like any good Sensemaking engine) performs best when taught principles.

General Purpose Context Accumulation

After recently sizing up G2 in action, I pondered what set of features make General Purpose Context Accumulation so unique. 

Complete Context: General purpose context accumulation systems must be indifferent to diverse observations e.g., originating from such sources as structured, semi-structured, unstructured text, social media, pictures, and video each containing events and transactions with temporal, geospatial, identifiers, biographics, biometrics and other features.  To the extent features can be extracted from an observation and delivered to such engines, they contextualize these diverse observations at the same time, in the same data space – every individual observation having an equal opportunity to find and benefit from all other observations.

Current Context: New observations are incrementally integrated with the whole (puzzle) in real time.  As such, G2 is fully capable of recognizing opportunity and risk at the split-second such relevance becomes knowable.

Conflicting Context: There is no such thing as a single version of truth when it comes to general purpose context accumulation.  Context accumulating engines let dissent fester.  When you search Google and it says “Did you mean ____?”,  the same principle is at work.  Google is not looking in a static dictionary.  No, Google is remembering everyone’s errors.  If Google did not remember the errors, it would not be so smart.  In the same way, context accumulating systems let disagreement coexist; otherwise new emerging trends and weak signal will never have a chance to add up to anything.

Self-Correcting Context: This is the most essential ingredient of context accumulation.  I believe few, if any, analytics in the world can do this – and especially at our scale.  Imagine having already seen and contextualized billions of historical observations, and now the next record arrives.  At this moment one must not only decide where to place this new observation, one must also decide if this new observation (had it been known in the beginning of time i.e., first), warrants the reversal of any of the billions of previous assertions.  And if so fix them.  The effect being: new observation can reverse earlier assertions.  Smart Systems Flip-Flop.  Doing this in real-time over billions of observations is non-trivial.  This happens to be the single most technically sophisticated aspect of context accumulation and well worth the last 10 years of effort we have spent researching, tweaking, and tuning our technique.  The difference between our previous method and the new G2 method should be more than an order of magnitude more efficient when dealing with some of the nastiest scenarios that one stumbles upon when implementing algorithms that can change their minds about the past.  In any case, it is this behavior alone that delivers what I call “Big Data. New Physics.”  Essentially, as the observation space widens natural variability in the data (errors) start to become your friend, the false positives and false negatives both begin to self-correct, and at the same time the computation effort required to make sense of the next observation begins to DECREASE as the observation space grows!  Translation: As the database grows, predictions become more accurate while computational effort decreases.  Absolute magic.

To tell you the truth, one of the most exciting things for me is this: when considering the wide range of unrelated domains I have explored with G2 (from maritime domain awareness and anti-money laundering to genealogy work using the 1880 census) – believe it or not, I think G2 could perform context accumulation over all of them at the same time, on the same computer, in the same database schema, using the same configuration.  Not that anyone would want to, mind you, or should.  Nonetheless, this idea that G2 is so general purpose quickens my pulse simply because this has never been possible before.  You see, I am hopeful that this means G2 will (one day) be able to readily make almost anything smarter – from managing my personal calendar (while better protecting my privacy) to helping researchers head-off some of the most debilitating diseases like Alzheimer’s.

Exciting Times Ahead

There is a lite-version of G2 on the market now, deeply embedded in an off-the-shelf predictive modeling product.  I continue to hand select special projects for the big Sensemaking version of G2 to press and stress this technology in new ways – using these “sea trials” to drive her development.   I can’t share details about these efforts just yet but boy she is growing up quickly.  And just to be clear: I am not saying this is easy, far from it, especially in these formative years.  Make no mistake; this is very hard work requiring the team and I to work crazy long hours

I am a dreamer with a long list specific engineering tasks to further advance G2 – things like ‘selective curiosity’ where G2 thinks of its own questions and knows whom to ask (e.g., a Jeopardy! Champion) and a ”hint service” that among other things may help feature extraction algorithms perform much better (without traditional training data).

G2 is different than other software, very different.  Another example, saving the best for last, we have built a series of features into G2 for enhanced privacy and civil liberties protections.  These socially responsible features factored into the original blueprints – from conception – a design approach called Privacy by Design (PbD). 

Finally: While I may be driving the G2 vision, I am certainly not building it.  I have an amazing group of engineers who actually do all the real work.  We also have some very patient customers willing to tolerate all the ups and downs that come with being early adopters.  And without both of them none of this would be possible.  So to them I say … thank you.

 

[TECH NOTES]

A few technical comments for my more technically minded friends.  Entirely coded in C++.  Schemas designed specifically for primary key row stores (although capable of running against a SQL back-end).  One index per table, never two.  Application-aware sharding (supporting heterogeneous data stores, table partitioning, and uniform distribution of records over the available grid nodes).  Most tested on DB2.  Eager to finish testing against other popular SQL back-ends.  Hoping to be able to demonstrate near linear scale against a NoSQL row store very soon.  G2 does not run on Hadoop Map/Reduce (because it is not batch), although Hadoop-based systems will certainly benefit from G2’s accumulated context.

Talk to G2 via HTTP, flat file, SPSS Modeler stream jobs, or compile it into InfoSphere Streams for stream computing over big data.  No user interface work being done at this time.  Using our standard and very simple XML-based Universal Message Format (UMF), G2 eats inbound observations and spits out interesting chunks of the puzzle, aka resumes, when it has something useful to say.

G2 does not do entity/feature extraction on unstructured data.  G2 does not do pattern discovery or anomaly detection.  Hopefully G2 will one day help such things perform better.

Just to be clear, G2 is far from perfect – as upon close inspection she clearly has some moles and warts and even still makes poopy pants from time-to-time.  And I am already fretting the teen years.  Nonetheless, despite these distractions, I can say with certainty this is a journey I am deeply committed to, it is the right direction, hence my ongoing and relentless focus and motivation.

 
 

RELATED VIDEOS (In my own words, Courtesy of Redbooks)

Enterprise Amnesia versus Enterprise Intelligence

Using Entity Analytics to Greatly Increase the Accuracy of Your Models Quickly and Easily

 

 

RELATED POSTS:

On A Smarter Planet … Some Organizations Will Be Smarter-er Than Others
What Came First, the Query or the Data?

The Data is the Query

Data Finds Data

Puzzling: How Observations Are Accumulated Into Context

Accumulating Context: Now or Never
Asserting Context: A Prerequisite for Smart, Sensemaking Systems

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems
Sensemaking on Streams – My G2 Skunk Works Project: Privacy by Design (PbD)
G2 | Sensemaking – One Year Birthday Today. Cognitive Basics Emerging.

General Purpose Sensemaking Systems and Information Colocation
Smart Systems Flip-Flop

Big Data. New Physics.
There Is No Such Thing As A Single Version of Truth

Master Data Management (MDM) vs. Sensemaking

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

Federated Discovery vs. Persistent Context – Enterprise Intelligence Requires the Later

When Federated Search Bites

Enterprise Intelligence – My Presentation at the Third Annual Web 2.0 Summit

Self-Correcting False Positives/Negatives: Exonerate the Innocent

Privacy by Design in the Era of Big Data

Responsible Innovation: Designing for Human Rights

 

Fantasy Analytics

 

Sometimes it just amazes me what people think is computable given their actual observation space.  At times you have to look them in the eye and tell them they are living in fantasyland.

 

Here is how an example conversation:

Me:                 “Tell me about your company.”

Customer:       “We are in the business of moving things through supply chains.”

Me:                 “What do you want to achieve with analytics?”

Customer:       “We want to find bombs in the supply chain.”

Me:                 “COOL!”

Me:                 “Tell me about your available observation space.”

Customer:       “We have information on the shipper and receiver.”

                       “We also know the owner of the plane, train, truck, car, etc.”

                       “And the people who operate these vehicles too.”

Me:                 “Nice.  What else do you have?”

Customer:       “We have the manifest – a statement about the contents.”

Me:                 “Excellent.  What else you got?”

Customer:       “That’s it.”

Me:                 “WHAT?!"

                       "YOU ARE NEVER GONNA FIND A BOMB!”

                       “NO ONE WRITES ‘BOMB’ ON THE MANIFEST!”

 

The problem being; often the business objectives (e.g., finding a bomb) are simply not possible given the proposed observation space (data sources).

[Unless, in this case, the perpetrator writes the word “BOMB” on the manifest.  And only idiots do that.  And luckily we don’t have to worry much about the idiots as they run out of gas on the way to the operation and take wet matches to their fuses.]

When we software engineering folks get overly excited and run off and build systems with little forethought about the balance between the mission objectives and the observation space, there is a risk the system will be a useless piece of crud – so many false positives and false negatives – the value  of the system not worth the cost.

As I have no interest in spending intense chunks of my life building pointless systems … when initially scoping a project I recommend first qualifying the available observation space to determine if it is sufficient to deliver on the mission objectives.  And if the available observation space is insufficient, then one must first figure out if/how the observation space can be appropriately widened.

In case you are interested, here are the some of the ways I try to mitigate these risks: 

Qualifying Observation Spaces

1. Ask for real examples from the past of things they would like to detect (opportunity or risk), and then look in the real data to see if, upon human inspection, it is discoverable.
2. If real examples from the past cannot be detected in the provided data sources, I tell the them “not even a sentient being could discover this.” 
3. Have them name their data sources and the data elements (key features). 
4. Then, just because they say a data source has certain features, go look yourself – I can’t tell you how many times I go take a look and find key columns are empty or so dirty that the value of that data source is negligible.
5. If the data sources share common features between them (e.g., customer number, address, phone number, etc.) then generally more is good.
6. For those data sources that have no (or few useful) shared features (e.g., one data source has name and address and the other data source has stock symbol and stock price) then generally this is not so good.

Widening Observation Spaces

1. There will be many cases where it becomes necessary to help the customer think about widening their observation space … if they are ever to realize their hopes and dreams (business objectives).
2. Conjuring up additional data to expand the observation space is quite an art and requires real-world understanding of what and how data flows inside the walls and outside the walls as well the legal and policy ramifications.
3. Generally one starts looking for new data sources in this order: 1) other stuff inside the walls that you already collect (e.g., product returns), 2) external data that can be purchased (e.g., marketing flags like “presence of children” and “income indicators” as routinely sold by data aggregators).  Of course there are other options like collecting more data themselves (e.g., adding a field to a web page so their customers can express sentiment, capturing the fingerprint of the device during on-line transactions, etc.)
4. If you are trying to catch bad guys, hope that some of the data sources would be unknown or non-intuitive to their adversary (if the bad guys know you have cameras on these four streets, then they will take the fifth street).
5. Beware of social media: There is much allure to the idea that one can computationally map Twitter statements (about your company/brand) to which customer said it.  Go take a look yourself and see how often the Tweet account contains sufficient features to make an accurate identity assertion.  I think you will see the frequency in which an identity can be asserted is underwhelming.  Different countries and different kinds of social sites will have different statistics.  In any case, be wary and look for yourself first.
6. Now let’s say one has a list of potentially new data sources to use.  Then the next question is how to prioritize all these possibilities.  Again, there are a lot of ways to think about this – but here are a few common ways I think about this: A) Data that improves the ability to count or relate entities (e.g., a source that may contains new identifiers like email addresses) so that one can discover that two customers you thought were different are more likely the same customer; B) Data that brings more facts (e.g., what, where, when, how many, how much); C) Diverse data potentially containing identifiers and facts in disagreement (e.g., this fact indicates they are here, but that fact shows they really may be over there – helpful if trying to keep strangers from using your credit card).
7. Finally, don’t forget there will be plenty of times that the mission objectives cannot be achieved because the necessary observation space is not available.  In which case, punt.

The above list is somewhat off the cuff, certainly incomplete.  So … please consider it a starter kit and hack at it any which way you like …

 

OTHER RELATED POSTS:

Data Beats Math

Context: A Must-Have and Thoughts on Getting Some …

More Data is Better, Proceed With Caution

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

 

 

 

 

 

Privacy by Design in the Era of Big Data

Ann Cavoukian, Information and Privacy Commissioner, Ontario, Canada and I released a joint paper June 8, 2012 entitled “Privacy by Design in the Era of Big Data” available here.

In this paper, Ann describers her aspirations for Privacy by Design (PbD) to which I call out specific PbD inspired engineering decisions the team and I made during the development of my latest (software) invention.

Internally code named “G2,” this now 3.5+ year effort started with a full year first working on paper only.  No different than first rendering detailed architectural plans to paper before building a house, our first year was dedicated to overcoming what we thought were our performance limitations of the past while simultaneously weaving in as many advances in privacy and civil liberty protections as we could fathom at the time.

As a result, I can say that this new technology (something you may come to hear about one day called Sensemaking) has more privacy and civil liberties protecting features than any technology my team and I have ever created in the past.  In fact this technology may have more baked-in privacy and civil liberties enhancing features of any advanced analytic software ever engineered.  I would love to be wrong about this – starting a fierce competition over “I have more privacy features than you” is going to be a good thing for planet Earth.

Here are a few highlights from the report that you may find interesting:

While organizations have practical incentives to make the most of their ever growing observation space (the data they have access to), they also have a pressing need to embed in these systems enhanced privacy protections.  We outline in this paper just such an example — how an advanced Big Data sensemaking technology was, from the ground up, engineered with privacy-enhancing features. Some of these features are so critical to accuracy that the team decided they should be mandatory — so deeply baked-in they cannot be turned off.                                           

                                                                                                                ~ page 2

But as technological advances improve our ability to exploit Big Data, potential privacy concerns could stir a regulatory backlash that would dampen the data economy and stifle innovation.

                                                                                                                ~ page 3

A new class of analytic capability is emerging that one might characterize as “general purpose sensemaking.” These sensemaking techniques integrate new transactions (observations) with previous transactions — much in the same way one takes a jigsaw puzzle piece and locates its companions on the table — and use this context-accumulating process to improve understanding about what is happening right now. Crucially, this process can occur fast enough to permit the user do something about whatever is happening while it is still happening. Unlike many existing analytic methods that require users to ask questions of systems, these new systems operate on a different principle: the data finds the data, and the relevance finds the user.

                                                                                                                ~ page 5

However, in these new systems the task of ensuring data security and privacy becomes harder as more copies of information are created. Large data stores containing context-accumulated information are more useful not only to their mission holders but also to those with interests in misuse. That is, the more personally identifiable information Big Data systems contain, the greater the potential risk. This risk arises not only from potential misuse of the data by unauthorized individuals, but also from misuse of the system itself.

                                                                                                                ~ page 7

PbD prescribes that privacy be built directly into the design and operation, not only of technology, but also how a system is operationalized (e.g., work processes, management structures, physical spaces and networked infrastructure.)  Today, PbD is widely recognized internationally as the standard for developing privacy compliant information systems. As a framework for effective privacy protection, PbD’s focus is more about encouraging organizations to both drive and demonstrate their commitment to privacy than some strict technical compliance definition.

In short, in the age of Big Data, we strongly encourage technologists engaged in the design and deployment of advanced analytics to embrace PbD as a way to deliver responsible innovation.

                                                                                                                ~ page 9

In late 2008, Jeff Jonas embarked on an ambitious journey to create a sensemaking-style system. This effort started with overall architecture planning and design specifications. Over the first year of this project, while drafting and redrafting these blueprints, his team worked to embed properties that would enhance, rather than erode, the privacy and civil liberties of data subjects.

To engineer for privacy, his team weighed performance consequences, default settings, and which, if any, PbD features should be so hard wired into the system they literally cannot be disabled.

Over the year that spanned the preliminary and detailed design, the team created a robust suite of PbD features.

                                                                                                                ~ page 9

The dynamic pace of technological innovation requires us to protect privacy in a proactive manner in order to better safeguard privacy within our societies. In order to achieve this goal, system designers should be encouraged to practice responsible innovation in the field of advanced analytics.

With this in mind, we strongly encourage those designing and building next-generation analytics of any kind to carry out this work while being informed by Privacy by Design as it relates to personally identifiable data.

                                                                                                                ~ page 13-14

One thing is for sure: our PbD efforts are a work in progress – much has yet to be done.  Comments and critiques are most welcome.

In the meantime, if you are an engineer consider becoming a student of privacy yourself and then begin engineering with PbD in mind.  And if you are a privacy/civil liberties advocacy type, find yourself an engineer to take under your wing – and be gentle as you nurse these newbies along.

 

RELATED MATERIAL:

Privacy by Design (PbD)

 

RELATED POSTS:

Big Data Q&A for the Data Protection Law and Policy Newsletter

Sensemaking on Streams – My G2 Skunk Works Project: Privacy by Design (PbD)

G2 | Sensemaking – One Year Birthday Today. Cognitive Basics Emerging.

Responsible Innovation: Designing for Human Rights

Responsible Innovation: Some Things are Best Left Un-invented

Responsible Innovation: Staying Engaged with the Privacy Community

 

Self-Correcting False Positives/Negatives: Exonerate the Innocent

This blog entry is dedicated to false positives and false negatives, specifically why it is so essential that systems find and fix them.  A false negative occurs when an assertion about something true, is missed e.g., the true perpetrator of a crime is overlooked.  A false positive occurs when an assertion is made about something being true when, in fact, it is not true, e.g., when a court convicts someone for a crime he/she did not commit.

Imagine for a moment a DNA database.  What if a subject’s second DNA profile does not match the subject’s previously collected DNA profile? Would anyone notice this false negative?  Or, what if a DNA profile from a new criminal investigation is submitted to the DNA database and it matches someone who has been in prison for 15 years.  Is this proof of a false positive?

How could such errors happen?  Contamination?  Incompetence?  Criminal negligence?  Regardless, such tragic things really do happen.

Las Vegas police reveal DNA error put wrong man in prison

“It did not catch the mistake in Jackson's case, however, because the mistake was not merely a mislabel -- it was the wrong DNA in the wrong vial.”

DNA EVIDENCE: Officials admit error, dismiss case

“… acknowledging that the police lab accidentally had placed Sotolusson's name on another man's DNA sample.”

Asheville area murder case turns on missteps

“… revelations DNA evidence that might have cleared the men never made it to defense attorneys.”

Now the million dollar question: If a DNA database like the Combined DNA Index System (CODIS) were to get new evidence that brought doubt to an earlier assertion; can the system itself detect it?  Is anyone notified?  Better yet, is the defendant notified?

Systems with the ability to use new observations to reverse earlier assertions are able to self-correct false negatives and self-correct false positives – in essence, changing their minds about the past.  Yes … smart systems flip flop.

Traditional systems are generally unable to use new observations to reverse earlier assertions and thus end up with internal inconsistencies – the evidence exists right before your eyes (in the database) while an errant assertion from the past lives on.  Some call this “database drift.”

One common remedy to database drift involves periodically re-processing all the data.  Which begs the question: How long do you want to wait for a right answer, when the correct answer is in hand, and right before your eyes?  The other big issue is that if your system requires periodic reprocessing to correct for this database drift, how long is that going to take?  Point being, in very large databases this can take a very long time.

I have been extremely interested in how to prevent this type of database drift – detecting previous false negatives and false positives at the split second each new observation arrives – doing this over billions of rows of historical data while sustaining thousands of transactions a second.

In essence, this kind of algorithm works like this: Now that I know this, had I known this first, are there any assertions I have made that would have been made differently – in real-time at high transaction rates over ALL historical data?

From a privacy and civil liberties perspective, systems incapable of correcting previous false negative and false positives are problematic e.g., an innocent person remains harmed (incarcerated) despite the fact that late-arriving evidence clears his/her name.

Fortunately for some wrongly convicted, the Innocence Project is addressing this problem.  This organization uses DNA evidence to exonerate the wrongfully convicted.  They have nearly 300 examples (like Michael Morton) of folks wrongly imprisoned – in some cases these innocent folks have been incarcerated for many decades.  This amazing group does this by hand, it takes a lot of work, and they have an enormous backlog of cases worth reviewing.

I encourage those working on identity-based assertion systems to add self-correcting false negatives and self-correcting false positives to your systems.

[TECHNICAL NOTES]

Technical Note #1: A Mini-Demonstration of a Self-Correcting False Positive

I first stumbled across the need for self-correcting false positives about a decade ago when we discovered a particular data set in which three out of every eight million people, a Patrick and a Patricia, lived at the same address with the same last name.

Imagine what might happen in this scenario.  You already have this one record in your database:

Entity #1

Record 1, Patrick Smith, Male, 123 Main Street, 555-1212

And today you get Record 2 containing:

Pat Smith, 123 Main Street, 555-1212 and date of birth 03/17/1986

Let’s say for the sake of argument that same name, address and phone number are deemed sufficient evidence to assert that an entity is the same (not always a great idea by the way).  The results:

Entity #1

Record 1, Patrick Smith, Male, 123 Main Street, 555-1212

Record 2, Pat Smith, 123 Main Street, 555-1212 and date of birth 03/17/1986

This is fine and dandy and all … until Record 3 appears on the scene:

Patricia Smith, Female, 123 Main Street, 555-1212, and date of birth 03/17/1986

At this moment, one has an opportunity to recognize the earlier error … in this case, the Pat Smith record is now most likely to be Patricia, NOT Patrick.  At this split second, a system capable of self-correcting false positives fixes the past replacing it with these new assertions:

Entity #1

Record 1, Patrick Smith, Male, 123 Main Street, 555-1212

Entity #2

Record 2, Pat Smith, 123 Main Street, 555-1212, and date of birth 03/17/1986

Record 3, Patricia Smith, Female, 123 Main Street, 555-1212, and date of birth 03/17/1986

 

Technical Note #2: A More In-depth Look at CODIS Processes

Over the course of my research, I was fortunate to come across background information about CODIS.  Some may find this interesting or helpful.

CODIS has three tiers:  LDIS (local lab databases), SDIS (statewide databases), and NDIS (USA database).  Selective LDIS records roll-up to SDIS and selective SDIS records roll-up to NDIS.

The criteria to enter the LDIS tier are the least stringent while the criteria to reach the NDIS level are the most stringent.  A profile in the NDIS database will have come from a SDIS database.  And entries in a SDIS database may have come from a LDIS database below it, possibly varying by each state’s lab structure/organization.

Each tier is broken down further into several indices:  Forensic, Convicted Offender, Arrestees (depending on state legislation), Missing Persons, Relatives of Missing Persons, and Unidentified Human Remains (LDIS, SDIS, NDIS).  Some LDIS/SDIS databases also have a Volunteer and or Suspect Index (depending on respective local/state legislation).  

DNA Profile Submission

Each sample’s profile is generally only eligible for submission to a single index.  If a forensic profile matches an individual and is uploaded to the forensic index, that individual’s reference profile can’t simply be added to the convicted offender index later after conviction.  Also, the reference sample collected for comparison to the evidence profile cannot be retested to re-obtain the individual’s profile for convicted offender submission either—an entirely new sample is collected upon conviction for submission to the convicted offender index. 

The only information contained in each of these databases is:

a) the DNA profile

b) the lab submitting the profile

c) the lab employee responsible for submitting the profile

d) some numerical identifier for the submitting lab/employee to track the profile

CODIS Search Procedures

Only certain index combinations are compared against each other.  For example, the Relatives of Missing Persons Index is only compared to the unidentified Human Remains Index while the Convicted Offender Index is compared against all indices except the Relatives of Missing Person’s index.

• If a profile only meets LDIS criteria, it resides in the LDIS database and is only compared within the appropriate indices to other LDIS profiles  (profiles belonging to the individual laboratory)
  • For example, if a profile meets Miami’s LDIS upload criteria, but not Florida’s SDIS criteria, the profile would simply reside in and be compared against the appropriate indices within Miami’s LDIS.
  • If the profile meets SDIS criteria, it is compared within the appropriate indices to other profiles in both the respective LDIS and SDIS databases but not outside LDIS databases within the state (Put differently, a profile that has reached SDIS will not be compared against other states’ LDIS or SDIS profiles.)
    • For example, if a profile meets both Miami’s LDIS and Florida’s SDIS upload criteria, but not NDIS’s, the profile would reside in both the LDIS and SDIS and be compared against the appropriate indices at both levels. It would not, however, be compared against any profiles contained in NDIS, nor would it be compared against the LDIS and SDIS of other states, say New Orleans’s LDIS and Louisiana’s SDIS.
    • If the profile meets NDIS criteria, it is compared to other profiles within the appropriate indices in the respective LDIS, SDIS, and NDIS databases, but not outside SDIS/LDIS databases
      • For example, if a profile meets Miami’s LDIS, Florida’s SDIS, and NDIS upload criteria, it would reside at all three levels and be compared against appropriate indices at each.  It would not, however, be compared against profiles residing only in other states’ LDIS or SDIS systems.

CODIS Matches and DNA Identification Errors

The CODIS software performs comparisons between the appropriate indices/tiers of the databases and then reports back any matching profiles with their associated identifiers to the submitting lab(s).  Then the lab(s) must follow up on their end to retest their matching sample(s) and report back the confirmation results to each other (the exception being forensic profiles are NOT re-tested to avoid consumption of evidence).  If the source (DNA donor) of the matching DNA profiles is known, this info is exchanged between the individual submitting lab(s) and it is at this time a “false-positive” match is discovered.  

Basically, this type DNA identification error can only be caught after a match has occurred – meaning the falsely matching profiles have to wind up in an index/tier where they will be compared to each other … an erroneous LDIS profile residing in the SDIS database will not be detected if the falsely matching profile resides in the NDIS database, a different SDIS database, or even an unsearchable index in the same SDIS!  Note: this can take an unreasonably long time, if it ever happens at all. 

Additionally, since the database contains minimum identifying attributes for each entity or DNA profile, and the attributes mostly contain information that can only be utilized by the submitting lab, the CODIS software doesn’t have any capability to detect “false negative” non-matches.  It’s important to note that this inherent limitation was developed intentionally to protect individuals’ privacy … but as a consequence this may lead to erroneous identification and wrongful incarceration. 

Here is another interesting weakness in CODIS.  When a subject’s DNA becomes crime scene evidence, this DNA profile may possibly appear in the databases up to four times over the entire course of a single investigation:  First as a forensic sample (the evidence profile attributed to him), second as a volunteer (the reference specimen he consented to database entry according to local/state law), third as an arrestee (the reference specimen collected from him when arrested for a qualifying offense according to local/state law), and fourth as a convicted offender (the reference specimen collected from him when convicted of a qualifying offense according to local/state/federal law). 

Matches should occur at each of these stages of submission if there are no DNA identification errors, and multiple matches would definitely validate each individual testing result.  However, if a match does not occur at one or more of these submission stages, then an error has occurred.  After the forensic profile was uploaded, each subsequent related submission provided an opportunity to detect an identification error because each subsequent match reported by CODIS would be returned to the lab for confirmation.  CODIS’s inability to detect a falsely-negative non-match at any and each of the submission stages means opportunities to detect errant results are missed – this faulty evidence means the wrong people are convicted or freed. 

Even if CODIS had the capability to detect/correct a false positive, waiting for that false-positive match to occur is problematic because it takes longer to occur/detect than a false-negative non-match, or even worse, might never occur at all.  To obtain the false positive association, the true DNA donor to the forensic profile must somehow get uploaded to an index/tier that will be compared to the forensic profile that was erroneously attributed to the original subject.  The match might occur immediately if the true donor’s profile is already there, but might never occur if it’s residing in an ineligible index/tier or the true donor’s profile is forever absent altogether.

Basically, detecting/correcting the false-positive match prevents a subject from being wrongfully convicted for the second crime, but might only minimally exonerate him for the first wrongful conviction (he may have already served his sentence, be deceased, etc.).  A false-negative non-match actually precludes a false-positive match unless the falsely matching profile is immediately available for comparison when the erroneous profile is uploaded to the database. 

 

RELATED POSTS:

Smart Systems Flip-Flop

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems

 

Big Data Q&A for the Data Protection Law and Policy Newsletter

I have been given permission to re-publish an interview I did with the Data Protection & Law Policy newsletter.  Also to appear in the e-Finance and Payments Law & Policy newsletter.

May be of interest to some of my readers.

[INTERVIEW]

1. Data protection challenge of the future: what is Big Data?

The three V’s - Volume, Velocity, and Variety – are the essential characteristics of “Big Data”. While data protection and privacy laws are still busy catching up with technologies of yesterday, Big Data is growing at a lightning speed on a daily basis. How can companies deal with the data protection challenges brought about by Big Data in order to truly benefit from the opportunities introduced by Big Data? First, one must truly grasp what is Big Data. We interview Jeff Jonas, Chief Scientist at IBM Entity Analytics, to obtain his perspectives and definition of Big Data, and his experience handling Big Data. 

2. When did data become big?

Big Data did not become big overnight.  What I think happened is data started getting generated faster than organizations could get their hands around it.  Then one day you simply wake up and feel like you are drowning in data.  On that day, data felt big.

3. Please explain and elaborate on the characteristics of Big Data?

Big Data means different things to different people.

Personally, my favorite definition is: “something magical happens when very large corpuses of data come together.”  Some example of this can be seen at Google, for example Google flu trends and Google translate.  In my own work, I witnessed this first in 2006.  In this particular system, the system started getting higher quality predictions and faster as it ingested more data.  This is so counter intuitive.  The easiest way to explain this though is to consider the familiar process of putting a puzzle together at home.  Why is it do you think the last few pieces are as easy as the first few – even though you have more data in front of you then ever before?  Same thing really that is happening in my systems these days.  It’s rather exciting to tell you the truth.

To elaborate briefly on the new physics of Big Data, I pinpointed the three phenomena of Big Data physics in my blog entry - Big Data. New Physics – drawing from my personal experience of 14 years of designing and deploying a number of multi-billion row context accumulating systems:

1. Better Prediction.  Simultaneously lower false positives and lower false negatives

2. Bad data good.  More specifically, natural variability in data including spelling errors, transposition errors, and even professionally fabricated lies – all helpful. 

3. More data faster.  Less compute effort as the database gets bigger. 

Another definition of Big Data is related to the ability for organizations to harness data sets previously believed to be “too large to handle.” Historically, Big Data means too many rows, too much storage and too much cost for organizations who lack the tools and ability to really handle data of such quantity. Today, we are seeing ways to explore and iterate cheaply over Big Data.

4. When did data become big for you? What is your “Big Data” processing experience?

As previously mentioned, for me, Big Data is about the magical things that happen when a critical mass is reached.  To be honest, Big Data does not feel big to me unless it is hard to process and make sense of.  A few billion rows here and a few billion rows there – such volumes once seemed a lot of data to me. Then helping organizations think about dealing volumes of 100 million or more records a day seemed like a lot.  Today, when I think about the volumes at Google and Facebook, I think: “Now that really is Big Data!”

My personal interest and primary focus on Big Data these days is: how to make sense of data in real time, that is fast enough to do something about the transaction while the transaction is still happening. While you swipe that credit card, there is only a few seconds to decide if that is you or maybe someone pretending to be you.  If an unauthorized user is inside your network, and data starts getting pumped out, an organization needs sub-second “sense and respond” capabilities. End of day batch processes producing great answers is simply late!

5. What are the technologies currently adopted to process Big Data?

The availability of Big Data technologies seems to be growing by leaps and bounds and on many fronts.  We are seeing a large corporate investments resulting in commercial products – at IBM two examples would be IBM InfoSphere Streams for Big Data in motion and IBM InfoSphere Big Insights for pattern discovery over data at rest.  There are also many Big Data open source efforts under way for example HADOOP, Cassandra and Lucene.  If one were to divide these into types one would find some well suited for streaming analytics and others for batch analytics.  Some help organizations harness structured data while others are ideal for unstructured data.  One thing is for sure – there are many options, and there will be many more choices to come as Big Data continues to get investment.

6. How can companies benefit from the use of Big Data?

I’d like to think consumers benefit too, just to be clear.  To illustrate my point, I find it very helpful when Google responds to my search with “did you mean ______”.  To pull this very smart stunt, Google must remember the typographical errors of the world, and that I do believe would qualify as Big Data.  Moreover, I think health care is benefiting from Big Data, or let’s hope so.  Organizations like financial institutions and insurance companies are benefitting from Big Data also by using these insights to run more efficient operations and mitigate risks.

We, you and I, are responsible in part for generating so much Big Data.  These social media platforms we use to speak our mind and stay connected are responsible for massive volumes of data.  Companies know this and are paying attention.  For example, my friend’s wife complained on Twitter about a specific company’s service.  Not long thereafter they reached out to her because they too were listening.  They fixed the problem and she was as happy as ever.  How did the company benefit? They kept a customer.

7. What is the trend of processing Big Data?

I think a lot of Big Data systems are running as periodic batch processes, for example, once a week or once a month.  My suspicion is as these systems begin to generate more and more relevant insight, it will not be long before the users say: “Why did I have to wait until the end of the week to learn that?  They already left the web site.”; or, “I already denied their loan when it is now clear I should have granted them that loan.”

8. What are the complications dealing with the privacy implications brought about by Big Data compare to average sized data?

There are lots of privacy complications that come along with Big Data.  Consumers, for example, often want to know what data an organization collects and the purpose of the collection.  Something that further complicates this: I think many consumers would be surprised to know what is computationally possible with Big Data.  For example, where you are going to be next Thursday at 5:35pm or your three best friends, and which two of them are not on Facebook. Big Data is making it harder to have secrets. To illustrate using lines from my blog entry - Using Transparency As A Mask – ‘Unlike two decades ago, humans are now creating huge volumes of extraordinarily useful data as they self-annotate their relationships and yours, their photographs and yours, their thoughts and their thoughts about you … and more. With more data, comes better understanding and prediction.  The convergence of data might reveal your “discreet” rendezvous or the fact you are no longer on speaking terms your best friend.  No longer secret is your visit to the porn store and the subsequent change in your home’s late night energy profile, another telling story about who you are … again out of the bag, and little you can do about it.  Pity … you thought that all of this information was secret.’

9. What are the privacy concerns & threats Big Data might bring about - to companies and to individuals whose data are contained in 'Big Data'?

My number one recommendation to organizations is “Avoid Consumer Surprise.” 

That said, my concern is many consumers don’t seem to give a hoot. When is the last time you actually read the privacy statement or terms of use on your favourite social media site? I think in the future we’ll see Big Data being used to make the services offered even more irresistible.  Your Internet searches will become custom crafted lenses.  As a student of privacy and someone building Privacy by Design (PbD) into my inventions, I think about these things all the time.

10. How are companies currently applying privacy protection principles before/after Big Data has been processed?

I think there are many best practices being adopted.  One of my favorites involves letting consumers opt-in instead of opting them in automatically and then requiring them to opt-out.  One new thing I would like to see become a new best practice is: a place on the web site, for example my bank, where I can see a list of third parties whom my bank has shared my data with.  I think this transparency would be good and certainly would make consumers more aware.

11. What is “Big Data”, according to Jeff Jonas?

Big Data is a pile of data so big - and harnessed so well - that it becomes possible to make substantially better predictions, for example, what web page would be the absolute best web page to place first on your results, just for you.

 

G2 | Sensemaking – One Year Birthday Today. Cognitive Basics Emerging.

Following a quiet two-plus year gestation period, G2 came to life January 28^th, 2011 – one year ago today – when I formally announced its existence here: Sensemaking on Streams – My G2 Skunk Works Project: Privacy by Design (PbD).

“This new technology, something that might be characterized as a “big data analytic sensemaking” engine, is designed to make sense of new observations as they happen, fast enough to do something about it, while the transaction is still happening.”

Oh, the difference a year makes.  Over the last twelve months G2 has evolved in many ways. Three of the more exciting characteristics to emerge are:

Orders of Magnitude, appreciation of

G2 now has the ability to consider proportions when determining the best way to count things … evolving counting methods in real-time, without external influence (humans).  There is a big difference between zero, one, two, three, etc.  But at some point why take the effort to increment a counter from 4,090,223 to 4,090,224?  Why is this important?   Imagine how much attention it would take if you had to keep an exact count of everything you had ever seen.  What a waste.  In the interest of saving its energy for more important tasks, G2 starts using orders of magnitude to optimize its resources.  Of course, G2 can be told to keep exact counts (despite scale); if an exact count is needed from time to time, G2 will count them, one-by-one.

Geospatial and Temporal Reasoning, use of

G2 is showing early promise in the area of geospatial and temporal reasoning.  Using the basic physics principle “Two different things cannot occupy the same space at the same time,” G2 can more accurately assert when things are the same.  Also, G2 can now tell the difference between something happening “here and now” versus “then and there.”  If you were reading a book about the 1906 fire that swept through San Francisco, you would not be freaking out about the danger, filling the bathtub and watering the roof of your house.  No, you understand that this is information about “then and there,” not “here and now.”  Why is this important?  When it comes to making recommendations about a “next best action,” if one doesn’t take into account geospatial and temporal factors, be prepared for a slew of bogus recommendations.

Confusion, awareness of

And, just today, G2 suddenly developed the ability to notice when it becomes confused about something, making “note to self” about this important fact.  This awareness of confusion is triggered when facts appear to disagree.  If someone has three phone numbers – no big deal.  On the other hand, if someone has five different dates of birth, that just doesn’t seem quite right does it?  That would be confusing.  Why is this important?  Well, if you are looking to analytics to make important decisions, wouldn’t you want to know during the decision making process if there was related confusion … before action is taken?

While G2 is showing no signs of evolving any motor function any time soon … the cognitive advances we are seeing and the benefits we hope this will bring society … make these exciting times.

Happy Birthday G2!

 

Sincerely,

 

Jeff Jonas and the entire

G2 Development Team

 

RELATED POSTS:

Sensemaking on Streams – My G2 Skunk Works Project: Privacy by Design (PbD)
Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems
On A Smarter Planet … Some Organizations Will Be Smarter-er Than Others
Data Finds Data
Accumulating Context: Now or Never
General Purpose Sensemaking Systems and Information Colocation
Big Data. New Physics.
Master Data Management (MDM) vs. Sensemaking

 

Master Data Management (MDM) vs. Sensemaking

I get asked from time to time how Master Data Management (MDM) relates to my work in Sensemaking Systems.  There is sufficient confusion to warrant a blog post on this subject, so here it is.

Different missions, different tools.  Some organizations will use one or the other; most organizations will want both.

MDM in a nutshell

MDM is about helping companies gain control over business information by enabling them to manage and maintain a complete and accurate view of their master data.  MDM deals with a known problem.  Master data is “intentional” business data that is structured and flows from systems under one’s control.  Common master data domains include customers, products, and accounts.  The MDM outcome: organizations have higher quality, reliable and consistent master data records.

Sensemaking Systems in a nutshell

Sensemaking is about helping companies make sense of their diverse observational space, ranging from data they own and control (e.g., structured master data) to data they do not or cannot control (e.g., externally-generated and less structured social media).  Sensemaking deals with uncertainty around an unknown and every changing domain – as if operating over an arbitrary number of puzzles, each unclear, incomplete and riddled with inconsistencies.  Common Sensemaking domains include customers, watch lists, social circles and investigations.  The Sensemaking outcome: organizations make better decisions, faster.

Both MDM and Sensemaking Systems make organizations more competitive.  Both can serve real time or batch missions.  And both play together quite nicely.

Now to highlight some stark differences between these two missions and hence the reason the algorithms used in each domain are fundamentally so different.

Rich Data vs. Poor Data 

Because master file data is owned and controlled by the organization, this data tends to be more complete (feature rich).  For example, customer records often contain a name, address, and some form of further identification like a date of birth or tax identification number.

While high fidelity data is better, Sensemaking Systems must routinely attempt to associate low fidelity observations e.g., connecting the dots between a third party watch list containing only name, nationality and date of birth fields and an internal cyber investigation database containing only email and IP addresses.

The algorithms required to deal with low fidelity data require union/set-based entity resolution algorithms which are fundamentally different than the record linking algorithms used when dealing with rich data.

Bad Data Bad vs. Bad Data Good

Organizations that deploy MDM seek to improve data quality at collection, e.g., detection and elimination of duplicate accounts during data entry – the purpose being clean records.  This kind of control over internally generated master data means less bad data gets in the front door.  And in this pursuit of “golden” master data records … bad data is bad.

Contrast this with Sensemaking Systems.  In this class of system, errors such as misspellings and numeric transpositions are valuable regardless of whether these errors have been generated by accident or are professionally fabricated lies created by sophisticated criminals.  When a criminal uses different alias and date of birth combinations, Sensemaking Systems meticulously record this untrusted information, permit this ambiguity to fester, and monitor numerous versions of truth.  This “bad data” is sometimes the only source of an important clue. 

The algorithms used to harness an out of control observation space are quite different than those algorithms used to manage internally controlled, higher quality records.

Manageable Data vs. Unimaginable Big Data

For most organizations, master data will not exceed 500M records.  At these scales, the automated triage of changes to master data results in a small to modest number of records that must be arbitrated following human review.  Fortunately, as an organization’s master data is a closed domain … there is a real chance humans can keep up as “uncertainty” is minimal and manageable. 

Sensemaking Systems must be able to deal with extremely large data sets – potentially involving tens to hundreds of billions of records – being generated from an ever more diverse range of data sources (e.g., from Twitter to OpenStreetMap).  At this volume and diversity the “uncertainty” is beyond the capacity of virtually any human review.

The algorithms used to deal with the enormous ambiguity tend to be very different than algorithms that routinely face smaller, more controlled data sets that routinely benefit from human participation.  Sensemaking algorithms have also been seen to actually get smarter and faster over ever growing datasets – an exciting feature for customers staring information overload in the face.

Firm Facts vs. Predictions

MDM is about being in control of the master data that an organization generates and about the truth of this data.  Master data becomes so reliable that its assertions are treated as fact – and these facts are so firm they are rarely reversed or invalidated.

Because Sensemaking Systems are attempting to harness an out of control observation space, assertions about context are constantly postulated, reevaluated and retroactively readjusted.  This accumulating context is viewed more like a prediction that is expected to evolve as more observations are considered.

The algorithms required to constantly reassess and context correct the historical observations with every new observation, in real-time, are fundamentally different than algorithms that do not have this requirement.

Event Triggered vs. Everything a Trigger

MDM processes are triggered during master data events.  MDM is used to guide the collection and management of master data.  For example, real-time duplicate detection during:

• Customer on-boarding at a financial services company
• Account set-up at a telecommunications company
• New product creation at an insurance company
• Patient admission at an emergency room or doctor’s office

Sensemaking Systems are more akin to “intuition support systems” – treating every piece of data that falls within the organization’s observational space as the potential to learn something new, detect something relevant, or as the basis for an alarm.  As each new observation arrives in the enterprise, the enterprise needs to know if it just learned something new, confirmed something it already knew, or has something wrong.  And with each new observation it must answer the hardest question of all (for a computer), “How does this relate to what I already know?  Does this matter and, if so, to whom?”  For example:

• As fast as the consumer joins the insurance company’s Facebook group, the insurance company recognizes the consumer’s significant social influence and is now instantly prepared to respond accordingly during the next business interaction
• The customer’s on-line retail purchase this morning is used to render a more relevant “next best action” recommendation later that morning when they visit the physical store front, e.g., the coupon on the back of the receipt no longer includes the item they already purchased
• While processing new deceased persons’ data from the Social Security Administration, notification is send to the marketing department suggesting they “inactivate” those persons in the bank’s prospect database
• During the creation of a casino arrest report, an alert is generated when the subject provides the same home address as that provided by the dealer on his employment application

The algorithms necessary to publish insight at ingestion based on incremental context accumulation and complex event processing are fundamentally different than algorithms that do not have this requirement.

MDM + Sensemaking

If an organization has a well constructed MDM system, it has a good handle on its own data and processes. This MDM data being a great starting point for Sensemaking Systems.  Nice, but not required.

Ironically, when Sensemaking Systems are used in weak signals domains e.g., the detection of criminal activity, irregularity and errors in the data being presented by adversaries happens to be extremely useful.  So while MDM systems (by design) clean up data entry, Sensemaking Systems should be provided the good, bad, and ugly data – pre- and post- cleaning.  Point being: if you polish every piece of data to perfection, one may never find the weak signal.

Different technologies.  Different missions.   And a simple integration story.

RELATED POSTS:

 

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems

Sensemaking on Streams – My G2 Skunk Works Project: Privacy by Design (PbD)

Entity Resolution Systems vs. Match Merge/Merge Purge/List De-duplication Systems

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

Smart Systems Flip-Flop

The Data is the Query

Data Finds Data

Big Data. New Physics

Data Beats Math

 

“It is a capital mistake to theorize before one has data”

~ Sir Arthur Conan Doyle

 

Over the years, folks have often asked me what kind of math am I using to create large scale, real-time, context accumulating systems (e.g., NORA).  Some fond of Bayesian speculate I am using Bayesian techniques.  Some ask if I am using neural networks or heuristics.  A math professor said I was doing advanced work in the field of Set Theory.

My answer is always, “I don’t know any math.  I didn’t finish high school.  But I can explain how it works, step-by-step, and it is really quite simple.”

That reminds me of a related funny story.  After IBM acquired my SRD company in 2005 I began touring IBM’s impressive research facilities around the world.  During a visit to one of IBM's research labs I explained my techniques to a room full of researchers.  A few months later, to my surprise, they sent me a technical paper to express my work … using math.  Fascinating I thought.  The idea that my algorithms are now expressed in math terms was really exciting.  Could it be?  I was so curious.  So I asked them to humor me and take me through the paper very slowly via a conference call.  It was actually a bit embarrassing.  I started out by asking the question what does an equal sign mean when a colon is in front of it?  Symbol by symbol I asked for an explanation.  Then I asked about this thing shaped like the letter “U” … what does that mean?  (Union as it turns out).  Anyway, I was able to follow the math and it all made sense until about halfway through the paper when I spotted an obvious error.  So I said, “um, the math here is inconsistent with my technique.”  I suggested a fix.  The phone went quiet for a minute and then about 45 days later they came back with a new and improved paper.  Continuing where we left off, I found a similar discrepancy further down the page and then provided some more specifics about my technique.  Unfortunately, I never received another draft.  Clearly, they could have.  But honestly, I suspect they simply lost interest in having to teach me math.

I wish we would have finished that paper, as then folks trained in formal methods would better understand what I am doing and seeing.

One of the things demonstrated by this mathy paper might have been the notion that “data beats math” – at least when it comes to Assertion Algorithms.  Based on the available observation space, can an assertion be made?  Yes or no.  In short, there comes a point where sufficient evidence exists such that an assertion can be made as a “no-brainer” without feeling compelled to split hairs with probability math.

Here is practical example.  Imagine being presented with two identity records?

Record #1

Name: Mark Smith

Date of Birth: 05/12/1987

SSN: 555-00-1122

 

Record #2

Name: Mark Smith

Date of Birth: May 1987

D/L: 0099912334

Are they the same person?  It is certainly possible.  Using population statistics and some math someone could compute a reasonably accurate probability.  I say heck with using math to guess.  I’d say where can I find some glue around here?  For example, a record like this:

Record #3

Name: Mark K Smith

Date of Birth: May 12, 1987

D/L: 0099912334

SSN: 555-00-1122

So the point is: I’d rather look for corroborating and/or dissenting evidence than look to math for estimated probabilities.  And if a really important outcome might come from such an assertion, I would continue to seek observations until it was so obvious you could show the board of directors and they would say “duh.”  If you run out of available observations and you are still not sure … then you have a few choices: 1) locate and collect the kinds of observations you need, 2) wait until you luck into a future observation related to the assertion in question (letting the existing ambiguity fester), or 3) pound on it with math.  But I say only pound on it with math if it is going to be worth the additional effort/compute (e.g., you are playing high-stakes poker in Vegas).

My gripe, if any, is that way too many people are chipping away at hard problems and making no material gains in decades (e.g., entity extraction and classification) … when what they actually need is more data.  Not more of same data, by the way.  No, they more likely need orthogonal data – data from a different sensor sharing some of the same domain, entities and features (e.g., name and driver’s license number).

When the quality of mathematical predictions start to flatten out, I recommend increasing your observation space.  Hence the above reference to this awesome quote:

 

“It is a capital mistake to theorize before one has data”

~ Sir Arthur Conan Doyle

 

RELATED POSTS:

Accumulating Context: Now or Never

Smart Systems Flip-Flop

Algorithms At Dead-End: Cannot Squeeze Knowledge Out Of A Pixel

How to Use a Glue Gun to Catch a Liar

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems

The Data is the Query

I have been talking about the notion that "the data must find the data and the relevance must find the user" for some time now.

Another way to think about this is "the data is the question."

As each new piece of data arrives in the enterprise, the enterprise just learned something.  And with each new observation one should ask, “How does this relate to what I already know?  Does this matter and, if so, to whom?"

This is the world of sense and respond, situational awareness, sensemaking or whatever you want to call it.

Example: An employee in the bank’s credit department changes his home address in the payroll system.  What if the employee's new address is the same address currently under investigation by the bank's own fraud department?  How would the bank know?

They wouldn't.

When the data is the query, a change to a home address in the payroll system is determined, at that split second, to be the same address under investigation by the fraud department.  And, at that split second this is determined to be relevant, so the fraud department is notified.

Real-time, sub-second.

When organizations can process arriving observations for relevance … organizations will be more competitive, or might even, for the first time, seem to be “awake.”

Note: The systems/technologies that are going to do this are very different than what organizations have in place today.  Existing operational systems cannot do this.  Neither can master data management systems, data warehouses, operational data stores, data mining engines … not even Hadoop Map/Reduce.

 

RELATED POSTS:

Data Finds Data

General Purpose Sensemaking Systems and Information Colocation

On A Smarter Planet … Some Organizations Will Be Smarter-er Than Others

 

Sensemaking on Streams – My G2 Skunk Works Project: Privacy by Design (PbD)

Over the last twenty-eight months I have been quietly running a skunk works effort that I’ve code named “G2.”  To my delight, on January 28th, 2011 this system became officially viable and will be entering something akin to a “sea trial” phase through 2011.

I believe this system will prove to be my most innovative work to date.  I also believe it is the most responsible technology I have created to date.

This new technology, something that might be characterized as a “big data analytic sensemaking” engine, is designed to make sense of new observations as they happen, fast enough to do something about it, while the transaction is still happening.  This engine brings to life many of the principles I have been openly sharing on my blog, ranging from Sensemaking Systems Must be Expert Counting Systems, Data Finds Data, Context Accumulation, Sequence Neutrality and Information Colocation to new techniques to harness the Big Data/New Physics phenomenon.  That said, as this is version 1.1, there remain many things to do to realize my full vision.  It is a very ambitious effort, but more about that some other day. 

In terms of responsible innovation, I am even more proud to report that my team and I have baked in, from conception, more privacy and civil liberties enhancing technologies than any other product I am aware of to date. 

Friday, January 28^th, 2011 – my official launch date – also happened to be the international Data Privacy Day.  And on this day, internationally recognized privacy commissioner, Ann Cavoukian hosted a few hundred privacy executives and practitioners from around the world in Toronto Canada at her Privacy by Design: Time to Take Control conference.  During my keynote entitled “Confessions of an Architect” I highlighted seven (7) exciting features that have been baked into this new technology (Privacy by Design), specifically:

1. Full Attribution

2. Data Tethering

3. Analytics in the Anonymized Data Space

4. Tamper-Resistant Audit Logs

5. False Negative Favoring Methods

6. Self-Correcting False Positives

7. Information Transfer Accounting

The full presentation is here.

Here is a summary of the above seven PbD features:

1. FULL ATTRIBUTION: Every observation (record) needs to know from where it came and when.  There cannot be merge/purge data survivorship processing whereby some observations or fields are discarded.  Why is this so important?

A. If received data does not contain its data source and transaction pedigree, then system-to-system reconciliation and audit are virtually impossible, especially in large information sharing environments.

B. If the system merges and purges observations, only later to discover the wrong observations were merged or purged, then without full attribution correcting these earlier mistakes can be difficult if not impossible.  The typical alternative being periodic batch re-processing.

C. The Universal Declaration of Human Rights has four articles containing the word “arbitrary” e.g., Article 9 reads “No one shall be subjected to arbitrary arrest, detention or exile.”  If you don’t know where the data came from or when, how can any resulting action be anything but arbitrary?

2. DATA TETHERING: Adds, changes and deletes occurring in systems of record must be accounted for, in real-time, in sub-seconds.  Why is this so important?

A. Data currency in information sharing environments is important, especially if one is using data to make important, difficult to reverse decisions that affect people’s freedoms or privileges.

B. When derogatory data is removed or corrected in a system of record, it is vital to reflect such corrections immediately.  For example, if someone is removed from a watch list, how long should they have to wait before their name is cleared?

3. ANALYTICS ON ANONYMIZED DATA: The ability to perform advanced analytics (including some fuzzy matching) over cryptographically altered data means organizations can anonymize more data before information sharing.  Why is this so important?

A. With every copy of data, there is an increased risk of unintended disclosure.

B. Data anonymized before transfer and anonymized at rest reduces the risk of unintended disclosure.

C. If organizations can now share information in an anonymized form and still get a materially similar result, why would organizations want to share information any other way?

[Technical Note: As every anonymized value maintains full attribution, re-identification is by design to support Data Tethering as well reconciliation and audit.]

4. TAMPER-RESISTANT AUDIT LOGS: Each record of who searches for what should be logged in a tamper-resistant manner – even the database administrator should not be able to alter the evidence contained in this audit log. Why is this so important?

A. Every now and then people with access and privilege take a look at records without a legitimate business purpose, e.g., should an employee at a financial services institution take a peek into their roommate’s file.

B. Tamper-resistant logs make it possible to audit user behavior.

C. And, when the word gets out to the work force that such accountability exists, this can cause a chilling effect on misuse.

5. FALSE NEGATIVE FAVORING METHODS: The ability to more strongly favor false negatives is of critical importance in systems that could be used to affect someone’s civil liberties.  Why is this so important?

A. In many business scenarios, it is better to miss a few things (false negatives) than inadvertently make claims that are not true (false positives).  False positives can feed into decisions that adversely affect people’s lives – e.g., the police find themselves knocking down the wrong door or an innocent passenger is denied the ability to board a plane.

[Technical Note: Sometimes a new observation can lead to multiple conclusions.  Systems that are not false negative favoring may select the strongest conclusion and ignore the remaining conclusions.  But had the strongest candidate not existed, the second strongest conclusion would be asserted.  One false negative favoring method involves remedy such a condition, for example by reversing an earlier conclusion should a future observation bring to light that fact that multiple possible conclusions now exist.]

6. SELF-CORRECTING FALSE POSITIVES: With every new observation presented, prior assertions are re-evaluated to ensure they are still correct, and if no longer correct, these earlier assertions can often be repaired – in real-time, not end of month.  Why is this so important?

A.  False positives occur when an assertion (claim) is made, but is not true.  If relied upon to make a decision, false positives can adversely affect people’s lives e.g., consider someone who cannot board a plane because he or she shares a similar name and date of birth as someone else on a watch list.

B. Without self-correcting false positives, databases start to drift from the truth and become provably wrong (even to the naked eye) – necessitating periodic (batch) reloading to true-up the database.

C. Periodic monthly reloading to correct for false positives means wrong decisions are possible all month until the next reload, even though the system had everything it needed to know beforehand.

[Technical Note: Reversing earlier assertions in real-time at scale, as new observations present themselves, is computationally non-trivial.  Imagine making an assertion that two people are the same because they share exactly the same name, address and home phone number – only later to learn through another series of observations that these are really two different people (a junior and a senior).  Our “self-correcting false positives” feature self-corrects for these rare cases, in real-time.  We consider our ability to perform sequence neutrality at scale one of several breakthrough aspects of our work.]

7. INFORMATION TRANSFER ACCOUNTING: Every secondary transfer of data, whether to human eyeball or tertiary system, can be recorded to allow stakeholders (e.g., data custodians or the consumers themselves) to determine how their data is flowing.  Why is this so important?

A. It is often cumbersome to learn who has seen what records, or what records have been shared with tertiary systems.

B. Much like a US credit report that contains an inquiries section exposing the list of recent inquiring parties, now so can your medical or financial file.

C. Users can now be easily provided with such disclosures, increasing transparency and control e.g. enabling a consumer in some cases to request an information recall.

D. When there is a series of leaks, information transfer accounting makes discovery of who accessed all records in the series quite trivial.  This can narrow an investigation when looking for criminals within.

What has me most excited is that where some features above would typically be an extra priced option in my new system so many are built in (e.g. this tamper-resistant audit logs).  And some of our privacy and civil liberties enhancing features cannot even be turned off!

Yes, there is an official name for my new technology.  And no, I’m not telling you, because this is not a sales pitch.  Rather, I am simply trying to inspire other technologists to consider Privacy by Design as they innovate.

I’ve had two most great days at IBM.  The first great day was in January 2005 when IBM bought my company, SRD.  And the second greatest day came six years later on January 28^th, 2011.

RELATED MATERIAL:

Privacy by Design (PbD)

 

RELATED POSTS:

Smart Sensemaking Systems, First and Foremost, Must be Expert Counting Systems

Data Finds Data

Accumulating Context: Now or Never

Sequence Neutrality

General Purpose Sensemaking Systems and Information Colocation

Big Data. New Physics.

Source Attribution, Don’t Leave Home Without It

Data Tethering: Managing the Echo

To Anonymize or Not Anonymize, That is the Question

Immutable Audit Logs (IAL’s)

Out-bound Record-level Accountability in Information Sharing Systems

Big Data Flows vs. Wicked Leaks

Decommissioning Data: Destruction of Accountability