PERSPECTIVES

Big Impact from Small Data – Part 2

In this, the second part of the latest in a series exclusive to Data Makes Possible, Dr. Kirk Borne, Principal Data Scientist for Booz Allen Hamilton, adds onto his explanation of the value proposition of small data points with a set of real-world examples. If you missed Part 1, read it here.


When we are looking at instances where small data can have a big impact, we need look no further than the far-reaching consequences of such cases in scientific research. We can build beautiful theories to explain the universe, but those theories must stand up against all evidence and new data, no matter how small. Here is an example that I can personally relate to: my academic lineage includes the 19th-century scientist and philosopher Thomas Huxley (who was my academic great-great-great-great-great-great-great grandfather) – he summarized the above scientific dogma in this colorful way:

“There is nothing so tragic as a beautiful theory destroyed by an ugly fact.”

Below are a few cases where small data had a big impact:

Hyper-personalizing Customer Experience

In digital marketing, one of the most important current trends is hyper-personalization and one-to-one marketing to the individual customer. A large collection of customer data is important to find the one thing that will most delight the customer: the right offer, at the right time, in the right context, at the moment of impact. That one small thing (evidence of a specific customer’s unique preference) is the golden ticket to successful marketing campaigns. Small data yields big value!

A Zero Data Tragedy

Sometimes we forget that the smallest unit of data is not one byte. The smallest unit is zero data! My second example is a case where zero data was enormously significant. The tragic story of the explosion of the Space Shuttle Challenger on an extremely cold morning in January 1986 has been used in statistics, decision science, and business management classes for three decades.

One reason for this extraordinary level of academic attention is one of the primary precursors that led to the tragic deaths of all seven astronauts aboard. On the night before launch, the engineers convinced the launch decision-makers that a cold-weather launch should not affect the key engineering component of this enormously complex system: the O-rings that sealed the hot explosive gas propellants within the shuttle propulsion systems.

What they said was that there was no evidence of a failure of the O-rings in cold weather conditions. What they didn’t say was that they never tested the O-rings in cold weather conditions. In this case, the absence of evidence (of O-ring failure in freezing conditions) was misconstrued as evidence (proof) of the absence of failure in such conditions. Zero data had huge consequences!

Big Impact from Small Data - Part 2

Interstellar Data

My third example is from NASA’s deep space probe Voyager 1. The Voyager I spacecraft was launched in 1977 and subsequently conducted fly-by studies (i.e., scientific data collection) at both Jupiter and Saturn from 1979 through 1980. The Voyager I spacecraft trajectory was intentionally programmed to align with those massive planets in such a way that it received a strong gravitational boost – or “slingshot effect” – during the flybys that ultimately sent the probe on its way at the speed of 38,000 miles per hour into deep space.

Big Impact from Small Data – Part 2

NASA scientists have monitored data coming from Voyager 1 for over four decades now. Specifically, there is a very low-bandwidth transceiver on the probe that sends back measurements of the high-energy particle flux that the spacecraft measures in its space environment several times each day. The data flow from that magnetometer instrument is very small, probably much less than one kilobyte per day. In August 2012, those tiny data revealed that Voyager I’s situation had suddenly changed. The magnetometer readings dropped dramatically from approximately 25 energetic particles detected per second (that had been the norm for nearly 35 years) to just one or two particles detected per second!

What happened? Well, one of the most significant events in the history of technology and perhaps in all of human history! The sudden decline in particle flux (albeit from very small data) proved without a doubt that the Voyager I probe has exited the heliopause, the region whose plasma physics interactions are dominated by our Sun. The probe has entered true interstellar space, the region between the stars where the Sun has little, if any, influence.

This is not quite as dramatic as “Elvis has left the building”, but it is far more significant – a minuscule stream of data has proven that something of human design and invention has left our Sun’s sphere of influence entirely and is now truly traveling through interstellar space. Small data had a significant impact on our understanding of our place in the universe!

Data makes possible countless innovations, insightful discoveries, and impactful actions (usually beneficial, though not always). The value associated with those results is not dependent on the size of the data. Small data, even zero data in some cases, may represent the long tail of your big data distribution, but it can a leave a long trail of impact that is amplified across organizations through the years. The real key to success from data of any size is to ask the right questions. That’s the essence of cognitive analytics – generating new questions from your data. So here is my response when I am asked about where is big data’s biggest impact:

“When it comes to big impact, Small Data is the new Big Data.”

Now, take your thoughts on the Big Impact from Small Data conversation to Twitter! Agree or disagree with Kirk’s ideas and examples? Want to tell us your story? Tweet @KirkDBorne using the hashtag #datamakespossible right now!


Which example of small data impact was the most interesting to you?

View Results

Loading ... Loading ...

FORWARD-LOOKING STATEMENTS:

Certain blog and other posts on this website may contain forward-looking statements, including statements relating to expectations in the market for our products and applications of our products. These forward-looking statements are subject to risks and uncertainties that could cause actual results to differ materially from those expressed in the forward-looking statements, including development challenges or delays, changes in markets, demand, global economic conditions and other risks and uncertainties listed in Western Digital Corporation’s most recent quarterly and annual reports filed with the Securities and Exchange Commission, to which your attention is directed. Readers are cautioned not to place undue reliance on these forward-looking statements and we undertake no obligation to update these forward-looking statements to reflect subsequent events or circumstances.

9 Comments

  1. With regards to a the Challenger disaster, your assertion that the engineers convinced those in charge that a launch was safe despite the cold conditions is incorrect. From Wikipedia: “The Rogers Commission found NASA’s organizational culture and decision-making processes had been key contributing factors to the accident,[4] with the agency violating its own safety rules. NASA managers had known since 1977 that contractor Morton-Thiokol’s design of the SRBs contained a potentially catastrophic flaw in the O-rings, but they had failed to address this problem properly. NASA managers also disregarded warnings from engineers about the dangers of launching posed by the low temperatures of that morning, and failed to adequately report these technical concerns to their superiors.” https://en.m.wikipedia.org/wiki/Space_Shuttle_Challenger_disaster

  2. As a matter of fact, at least one ignored engineer raised an alarm about the cold temperatures before the shuttle launch and catastrophe. Please don’t help NASA evade responsibility for ignoring him.

  3. Huxley wasn’t that long ago! He was 19th century, dying I think in 1895. (My grandfather was already an adult.)

    The science that anyone now much over 50 was taught at school was mostly the science of Huxley’s contemporaries.

Leave a Reply