I have been talking about the notion that "the data must find the data and the relevance must find the user" for some time now.
Another way to think about this is "the data is the question."
As each new piece of data arrives in the enterprise, the enterprise just learned something. And with each new observation one should ask, “How does this relate to what I already know? Does this matter and, if so, to whom?"
This is the world of sense and respond, situational awareness, sensemaking or whatever you want to call it.
Example: An employee in the bank’s credit department changes his home address in the payroll system. What if the employee's new address is the same address currently under investigation by the bank's own fraud department? How would the bank know?
They wouldn't.
When the data is the query, a change to a home address in the payroll system is determined, at that split second, to be the same address under investigation by the fraud department. And, at that split second this is determined to be relevant, so the fraud department is notified.
Real-time, sub-second.
When organizations can process arriving observations for relevance … organizations will be more competitive, or might even, for the first time, seem to be “awake.”
Note: The systems/technologies that are going to do this are very different than what organizations have in place today. Existing operational systems cannot do this. Neither can master data management systems, data warehouses, operational data stores, data mining engines … not even Hadoop Map/Reduce.
RELATED POSTS:
General Purpose Sensemaking Systems and Information Colocation
On A Smarter Planet … Some Organizations Will Be Smarter-er Than Others
I agree that the systems/technologies that an org will need to create situational awareness with their data will be very different than what they have today. I also agree that data management environments (big data, rdbms, etc) in of themselves will not solve this problem, however, I believe the beginning steps in the right direction are establishing a robust scalable data layer not bound by the constrains of a typical rdbms, it is this scalable data layer that will enable the advanced analytics (which also do not exist in most orgs) to begin to answer some of the questions you pose.
Posted by: P | April 04, 2011 at 03:32 PM
Hi Jeff,
Just discovered your blog, I'm a long time data jockey--sold a data warehouse/BI stack to Cisco about 4 years back.
Just curious if there are any products on the market that approach your vision, or if only the upper echelon at IBM have been blessed with a glimpse of the future at this point in time :)
Cheers
Jason
Posted by: Jason Kolb | April 04, 2011 at 06:33 PM
Hi,
I applaud your efforts to elevate this topic to a business discussion. Creating meaningful use cases will help bridge what has been, for too long, localized and academic efforts to solve parts of this problem.
Architectural patterns like pub-sub/ESB, data warehouse/datamart, and classification/association have been used to attack this problem "in the small" - not always at the enterprise level (as in, scope of business and range of data) as you suggest.
I don't think it's a problem of lack of tools and methods, as there has been a lot of technical work on this in the KDD (Knowledge Discovery and Data Mining) community, as well as for text the IR (Information Retrieval) community.
The problem is in the way businesses (and IT) think about and organize their approach to data management. More attention to the possibilities from the business might drive better solutions.
Keep beating the drum!
Posted by: Dude | April 09, 2011 at 09:16 AM
Just discovered your blog, I'm a long time data jockey--sold a data warehouse/BI stack to Cisco about 4 years back.
Posted by: EVE ISK | September 06, 2011 at 12:31 AM
Architectural patterns like pub-sub/ESB, data warehouse/datamart, and classification/association have been used to attack this problem "in the small" - not always at the enterprise level (as in, scope of business and range of data) as you suggest.
Posted by: EVE ISK | September 06, 2011 at 12:34 AM