Jeff Jonas

Smart Systems Flip-Flop

Certainty often shifts with observations over time. And this is good.

The smartest an organization can be is directly related to the net sum of its perceptions. These perceptions come in the form of observations – observations collected across the various enterprise sensors (e.g., transactional/monitoring systems).

The second most important factor for smarts is the ability to determine how these observations relate to each other – for lack of a better word – contextualization.

But smarts requires much more than just available data and good correlation. Two additional critical elements of smart systems are:

1. An ability to make assertions based on new data points

2. An ability to use new data points to reverse earlier assertions

If everything you ever learned was held in your head as a probability, your ability to think quickly on your feet would be drastically reduced. Deciding that two objects are the same (aka Semantic Reconciliation) is an example of an assertion. Like human thinking, smart systems make assertions based on previous experience.

Smart systems also have to be able to undo earlier assertions made in error. If a new observation is in fact evidence that invalidates earlier assertions, these earlier incorrect assertions must be corrected (there are some caveats, more on this at another time).

Once presented with compelling new data, systems that cannot flip-flop on previous certainties … are dumb. The same goes for humans.

I thought my youngest son’s birthday was the 5th of the month for the first five years of his life – celebrated his birthday on the 5th, filled in forms using this date, and so on. This went on until I saw his birth certificate where to my surprise I discovered his birthday was really on the 2nd of the month! Having to explain the change of date to him and others made for an interesting conversation. When revealed this to my boy, I used the good news and bad news model – the good news being he was a little older than we thought – kids love that. Should I someday get evidence the birth certificate was in error … well then … I’ll have to revise the ground-truth yet again.

With this point of view, when politicians accuse each other of “flip-flopping,” I always laugh to myself. As it is the person who refuses to change his/her point of view – despite ANY and ALL new evidence presented – that scares me the most.

When I speak of Sequence Neutrality I am referring to exactly this characteristic whereby new data points remedy earlier assertions. Basically, had one known the new data point earlier, would the outcome of any earlier assertions be different or were there assertions that could have been made that were not? While this is hard to do on real-time transactional streams at high data rates, nonetheless this is essential behavior and hence something I have spent years thinking about and working on.

Incremental learning systems must have a high degree of sequence neutral processing, or there is little chance they will be smart.

On a more technical note: As most systems are not smart, sequence neutral, flip-flopping systems … downstream systems getting fed data (e.g., data marts and case management systems) have not been designed for the ever-changing recontextualization of previous reports. A new breed of systems – with different design parameters – are going to be needed to take full advantage of upstream smart systems.

RELATED POSTS:

Big Breakthrough in Performance: Tuning Tips for Incremental Learning Systems

Context: A Must-Have and Thoughts on Getting Some …

Accumulating Context: Now or Never

USC School of Cinematic Arts, "Imagine the World in 2050"

In April, 2008 five IBM scientists including myself appeared at the USC Film School on a panel discussing what the world would look like in 2050. Picture here.

The general topic was “putting the science back in science fiction.”

We each had five minutes to articulate how technologies might affect us in the year 2050. The rest of the session was dedicated to Q&A. I presented a newly crafted hand-drawn PowerPoint deck entitled “Macro Trends and What They Mean for 2050.” 

In summary, I made these observations about the future:

1. Big advances are generally not the result of any single technology but rather the result of combinations of technologies.

2. Good news: The world continues to become less dangerous. Today, we live longer than any time in the history of man – and this trend will continue. More here: The World is Not a More Dangerous Place.

2050 Prediction: Your doctor is 102 and this is not weird.

3.0 Fewer people can create much more, much faster … and more easily.

3.1. The bad news is faster death: Today, it takes less than 50 people and less than $100,000 to manufacture – and infest humans with – the reanimated 1918 Spanish Influenza virus. Some estimates place the death toll of such an event at 160 million people. That is an awful lot of death for relatively little cost and effort. More here: More Death Cheaper in Future.

3.2. The good news is faster wealth: On the other side of the coin, wealth can now be created faster than ever. For example, Marc Zuckerberg of FaceBook went from zero to over a billion in net worth in under three years. More here: Ludicrous Speed Billionaires.

2050 Prediction: Your 14-year-old neighbor makes $10B from their bedroom.

4.1. Surveillance societies are not only inevitable, irreversible … but more importantly they are irresistible! You love location-based services on your phone and you will love RFID chips in your sunglasses so you can find them if you lose them. More here: Six Ticks till Midnight: One Plausible Journey from Here to a Total Surveillance Society. 

4.2. Sensors become ubiquitous … not due to governments … rather this is caused by commercial enterprises as they compete for consumers who are eager to adopt any and all technologies that help them optimize their lives. More here: Ubiquitous Sensors? You Have Seen Nothing Yet.

4.3 Ubiquitous sensors result in countless piles of data. But to make real sense of this data it must be commingled. As such, these piles of data will have a tendency to converge into one pile of data. More here: More Data is Better, Proceed with Caution.

5. When disparate information is stitched together context emerges – much in the same way individual puzzle pieces are difficult to make sense of in isolation. Point being, the more puzzle pieces that can be associated, the greater the overall picture emerges. This stitched-together data will be stored in a pre-assembled form, in very large databases, ready for use. But unlike today, the enormous power of such “information in context” will not be in the hands of a few privileged (e.g., multi-billion dollar organizations) … in the future, such capabilities will live in the “network cloud” and be available to the masses. With such, the future will not be like Tom Cruise in the movie “Minority Report” where he stands in front large screens and searches for data using his hands to navigate. Nope, that would be old school.

2050 Prediction: Collective intelligence will locate what you need to know … and tell you!

To bring this prediction to life I presented the following example: The collective intelligence system in the network cloud will know the precise geo-coordinates of where you are physically standing right now. As well, imagine if it is also amassing data related to the current activities of migratory birds. While at one moment these two data sets may have no nexus … the moment a sensor presents some wind speed data to the collective intelligence system … this new data point being the puzzle piece that connects your location data and the birds’ migratory data. The cloud then will immediately push real-time relevant insight to you, the consumer. In this case, you are told to “jump to the right one foot.” Seconds later, to the your delight, a bird poop falls from the sky … splats on the ground … exactly where you were standing! 

6. When collective intelligence serves you and your doctor … you are going to love it. But when it serves the police looking at you … you are going to hate it. 

7. And that is the truth about the future … it’s going to be love/hate. 

Here is a related news story:

C|Net – Imagining the Tech World in 2050

Here is a related 7 minute YouTube video:

Imaging the World of 2050

Rapid Application Development: Fast Car No Map and 50 Laps to a “Who House”

[If you have never been involved in a custom software engagement (i.e., on the software builder side, customer buyer side, contract lawyers, etc.), there is no need to read this post.]

Rapid application development and prototyping have their place. One of these places is not the creation of complex, mission critical, ultra-high performance systems (i.e., space shuttles not mash-ups) – where it is essential that in hindsight the system appears well thought out, explainable, maintainable, etc. The simplest post-project assessment being, “Knowing what I know now, I would have built it exactly this way!”

The only method I am aware of that has a reasonable chance of making the above claim involves building a system based on stable, pre-defined, well-conceived blueprints. When operating off tight specifications, what you end up creating is virtually no different that what you were envisioning at the start … including the very first brick laid.

If you start building something without a precise road map, every time you have a design revelation, resource conservation drives one to make compromises. It is in this way that the air conditioning unit ends up on the roof of the car, the stereo mounted to the back of the driver’s seat, and so on. What rapid application development environments can lead to is the fast, iterative (50 laps) construction of a  Dr. Suess "Who House" or a Rube Goldberg-like contraption.

My primary issue with rapid prototyping, rapid application development, iterative development, and the like is the creation of valuable artifacts that one is hesitant to later discard at a whim. On this path, it becomes more prudent to perform a least-effort modification of A to become B. This compromise is compelled by the urge to preserve earlier investment … even if the best tact involves utterly discarding A and constructing B from the ground up.

It is therefore my belief that the most valuable prototyping tools, rapid application development environments, etc. are those that leave the designer with a sense of “as close to zero investment.” Radical changes to the design, complete rethinking of a whole component, big shifts in database schema, wholesale rework to the GUI, etc., need to be absorbed in the design phase without an iota of grief.

[One of these days maybe I’ll blog about Computer Aided Software Engineering (CASE) technology. During one era of my life I spent eight years building such technology to automate the blueprint process; hence this subject remains very dear to my heart.]

RELATED POSTS:

Super Consultants

Custom Software Scope Changes (Not)

Scalability and Sustainability in Large Information Sharing Systems

Custom Software Scope Changes (Not)

[If you have never been involved in a custom software engagement (e.g., on the software builder side, customer buyer side, contract law side, etc.), there is no need to read this post.]

I spent roughly 22 years of my life building somewhere between 150-200 custom software systems ranging from accounting systems, llama genealogy systems, labor reporting systems, airport profitability systems, huge consolidation consumer data warehouses, and sewer flow modeling systems to the first versions of NORA.

There was one particularly interesting lesson from these days:

“Custom software change orders CLAIMED by builders … are actually more often changes in understanding that come to light as the visions between parties are forced to reconcile.”

Here is how this goes: The buyer describes what they want to the builder.  The builder captures this vision in the form of requirements, features, design layouts, etc.  This exchange of vision continues until the builders and buyers think they are in synch.  The builder starts to build.  The buyer starts to see the system come to life.  The buyer notes that what he sees is not quite what he had in mind and calls for some adjustments.  The builder calls this a change order.

What is really happening is the visions between parties are being forced to reconcile as the system takes shape in the real world.

In my book, this is not a scope change.  This only demonstrates that both parties had something different in mind on day one.

There is only one way to safely avoid this all too frequent problem and that is blueprints.  And I am not talking about a few screen sketches and report headers.  I am talking about robust architectural plans: complete screen layouts with error messages, full report mock-ups containing realistic data and sub-totals.  I am talking about a degree of design completeness that the specifications could be used to simulate (e.g., in paper form) the use of the whole system by each type of user.

No one builds a high-rise building let alone a doghouse without blueprints.  Otherwise, there is an extraordinary risk of a) an abandoned project or b) the creation of a Dr. Seuss “Who House.”

On a related note: some think that rapid prototyping of big complex systems can shortcut a comprehensive design phase.  With few exceptions, I think this is ridiculous.  [I will elaborate on this is a coming post.  But in the mean time, as a teaser, I have a single test question that determines if a custom software system has been well-conceived.  Once the system is all done, knowing what you know now, would you have built it the same way?]

Is the custom software world still this way?

RELATED POSTS:
Super Consultants

Scalability and Sustainability in Large Information Sharing Systems

Malaysia Ironman versus South Africa Ironman – “Tastes like Mango”

In February 2008 I did the Malaysia Ironman, which took me nearly 15 hours. This last Sunday, seven weeks later, I did the South Africa Ironman, which took me just over 12.5 hours, missing my best time by just five minutes. In hindsight I now realize I should not have stopped to take a 10 minute shower in the middle of the run. My bad.

You learn something in every race. And some things are never as they appear. Take the guy I ran past last Sunday in South Africa. I could tell that he had a bit of a hygiene problem going on. So I said, “Hey, I think I should tell you that you have some diarrhea seeping through the back of your run trunks.” He says “You mean it looks like I ’shat’” my pants?” I nod. He then proceeds to take his right hand to scrape the back of his shorts. He can now see it as he has this brown gunk on his hand. He looks at it. Smells it. Then licks it! He looks at me and says “Tastes like mango.” Apparently some mango-flavored supplement ended up in the wrong place. However, had I been thinking more quickly I would have said “Did you have mangos for dinner last night?” In any case, he thanks me, shakes my hand … mango poopy material and all … and then we part ways.

Here is another big lesson that I seem to learn over and over. “Sighting” during open water swims is important. In this most recent race I did the first half in 38 minutes, yet the second half took me an extra seven minutes. Why? Well, my big mistake was selecting a cloud that was hovering way out over the Indian Ocean as a sighting point. About 10 minutes later while I was basking in how lucky I was not to be kicked and scratched by other swimmers – it dawned on me that I had not seen anyone in quite some time. So I poked my head out of the water to locate the next race buoy only to discovered I was actually en route to India itself.  Great white shark territory, I was thinking – and without all the feeding options (the other athletes) … just me.

And on that note, here are a few new triathlon tips for those like me who don’t actually have the time to train properly:

• Go long. If you can only ride 2-5 times a month, make each ride super long (75 to 200 miles) and hard – hills are good. Same goes with runs.
• Try a full marathon on a treadmill. What is great training about this is the monotony.
• Swim training. Never. The training time it takes to get faster is not worth the effort when you are in my hacker category. (In my first triathlon I did the stroke known as the “backfloat” which by the way, on a related note, was shortly after I asked triathlete legend Scott Tinley just before the race if fast people can actually swim the whole distance freestyle. I recall he looked at me in a disgusted fashion.)
• Brick workouts (practicing going from bike to run) are hard and take more time. There is probably a reason they are called bricks. I think I have only done three bricks in my entire life. Try to avoid these.
• Swimming. Don’t sight on clouds, boats, or other things that move.
• Get a time trial bike and stay down on those aero bars the whole time (unless out of your saddle climbing). I bought a Cervelo P3C bike about a month before my last race. Had four rides on it before the South Africa race getting used to the new position (two rides at 38 miles, one a 58 mile ride, and one 200 miler). This bike gave me my best bike time ever in an Ironman race, shaving nearly 45 minutes off of my previous best time. I almost broke 5.5 hours (5h33m) averaging 19.6mph!
• Drinking (alcohol) the night before the race is approved. It helps take the stress off the night before the race. Tell your friends you are “carb loading!”
• During the race don’t drink weird stuff you have not trained with on the race course. If you do try something new. Only take a sip or two and then see how you feel in 30 minutes before ingesting any more.  Stomach cramps suck.
• Shave (the undercarriage) the morning of the race. Otherwise, if you shave the night before your morning micro stubble will serve as sandpaper. Ouch-O-rama.
• After shaving the same morning of the South Africa race, I boldly slipped on some brand new, untested, tri shorts for the race. Ouch-O-rama times 100. A very bad idea. They caused such severe chafing my huevos were bleeding by race end. I could barely walk. Girlfriend says it is one of the grossest things she has ever seen. I’m simply hoping there is no permanent scarring. Reminder: don’t try untested things race day.
• Water and nutrition will make or break your race. I continue to remain uncertain about the importance of mangos.

Now a few race specifics.

Malaysia Ironman (Langkawi), February 2008. Beautiful country. Extraordinarily nice people. The swim is a “no wet suit” swim. Aside from this messing up the swim times of hackers like myself … the real problem was the stinging sea lice (some called it plankton). I did not know what was stinging me. But whatever it was … it was getting caught in my chest hair (which was behaving like a little net). Key point: It is hard to maintain a decent swim stroke while trying to beat these little critters out of your chest hair. Tip: In such conditions, maybe try shaving all that hair off so that less of these critters get stuck on you. The bike course was smooth. Some portions were closed to cars. Others portions were not closed. Watch out for the begging kids on the bike race course – there were a few – they appeared to think one of us might pull over to give them money! Anyway, by the time you get to the run, the temperature and humidity are killing you. On my race day it was something like 32 degree C (90 degrees F) and 40-50% humidity. To add to the suffering, some parts of the race course are right next to heavy traffic (smog) and, what appears to be, open sewers. I spotted some athletes (mid run) lying down in bus stops, others being assisted by ambulances, and quite a few simply puking their guts up. Looked to me like they were all from Japan! (This is a popular race if you live in Japan). The Singaporeans, on the other hand, seemed to fare very well (to know humidity is to love humidity – because I live in Las Vegas, you can guess how I feel about humidity). Finally, this race course itself is also not even remotely spectator friendly. You will hardly ever see anyone. Needless to say, this was my least favorite Ironman race so far.

South Africa Ironman (Port Elizabeth), April 2008. Very smooth waters on race day (I heard this was not that common). Extraordinary bike and run course – all roads in excellent shape and 100% closed to traffic! Simply fantastic. Very professionally organized. And this race course is the most spectator friendly I’ve seen – in part because you see your friends at least three times on the bike ride and up to six times on the run! This was my favorite race of the eight Ironman-sanctioned events I’ve done. Only one word of caution:  South Africa is suffering from a power shortage, which is evidenced by the roaming (scheduled) blackouts. The point being if you decide to get out and cycle the course before the race, pay special attention when passing through intersections with disabled traffic signals! And finally, if you see the beautiful little shower facility in the middle of the run course … and you are positioned to possibly beat your best time … don’t take off your socks, shoes, hat and shirt … for a dreamy little 10 minute shower (asking around for soap) in the middle of your race … unless it is really going to be worth it … which for me, it was!

 

RELATED POSTS:

Hacking the 2007 Brazil Ironman Triathlon in Florianopolis (May 27, 2007) – Strategy, Tragedy and 100% Pure Agava Tequila

Preparing for the 2007 New Zealand Ironman in Singapore?

Handicapped at the 2006 Arizona Ironman

Surviving the 2006 France Ironman and How Intelligent are Chimpanzees?

Dumb and Dumber: Consequences of the 2006 Silverman Triathlon

What sharks? Reflections on the 2005 Western Australia Ironman