Algorithms At Dead-End: Cannot Squeeze Knowledge Out Of A Pixel

Have you ever heard the expression “climbing trees to get to the moon”?  This means that while one can make progress inching along (up a tree in this case) … if one stands back to look at the big picture … no matter how successful one is at climbing trees … it is quite obvious they are never going to get to the moon.

I have news … for those waiting for big breakthroughs in algorithms to better make sense of transactional data (e.g., to lower false positives/negatives), they are going to be waiting a long time (actually, it's worse, forever).

Using algorithms to analyze transactions is like analyzing a single pixel.  No matter how much computing power, time and sophistication of algorithms … determining if the “red pixel” is a fire versus a fire engine simply isn’t a precision activity.

Next generation, real-world aware systems are going to FIRST apply inbound transactions (i.e., pixels) to earlier observations in order to construct context (i.e., pictures).  Then, and only then, will algorithms be able to more accurately determine the relevance of new transactions.  There are no short cuts.

Assembling pixels into pictures is the story of Context Accumulation. 

But how do pixels get assembled into pictures?  There is only one way to accomplish this and that starts with “feature extraction” and I wrote a bit about this in my post called “Context: A Must-Have and Thoughts on Getting Some …”  Just to be clear, algorithms are, of course, required on pixels to perform “feature extraction” and there is room for extraordinary improvement in this area (e.g., see point #5 in the above blog post).

Persistent Context and Perpetual Analytics are the most significant hurdles necessary to deliver the next generation of intelligent systems.

AND FOR THE RECORD: Don’t expect Google’s search analytics to get much better as this technology is near the end of its road as well.  The reason being is; they are applying analytics to documents … which are really just context-less pixels.  Quantum leaps forward in search are going to come from context accumulation.

RELATED POSTS:

Context: A Must-Have and Thoughts on Getting Some …

Accumulating Context: Now or Never

Sensing Importance: Now or Never

Enterprise Intelligence – My Presentation at the third Annual Web 2.0 Summit

Federated Discovery vs. Persistent Context – Enterprise Intelligence Requires the Later

Streaming Analytics vs. Perpetual Analytics (Advantages of Windowless Thinking)

Enterprise Intelligence: Conference Proceedings from TTI/Vanguard (December 2006)

Intelligent Organizations – Assembling Context and The Proof is in the Chimp!

It’s All About the Librarian! New Paradigms in Enterprise Discovery and Awareness

To Know Semantic Reconciliation is to Love Semantic Reconciliation

More Data is Better, Proceed With Caution

Comments

[Alexandre Queiroz]

Hi Jeff,

I definitely agree with you. However, when you mention "Next generation, real-world aware systems are going to FIRST apply inbound transactions (i.e., pixels) to earlier observations in order to construct context (i.e., pictures)...."
Is it not already happening?? CEP (Complex event processing) engines applying queries on real-time and historical data to find context and relevance.

P.S I really like your blog, you should blog more -:) I know time is a constraint..but....

[Sarven Capadisli]

or Quantum leaps will come from Quantum computing ;)

On Digital computing, "meaningful" representation for data is the best way forward.

[Sreenath Chary]

Jeff,
Excellent short article! Very insightful. One question....Context is something that is defined by us as humans. How can we determine context automatically by software, even if we are analysing a stream of data....past and current? Cheers,