Gold Strike vs. Striking Out – Extracting Unstructured Data from the Enterprise Backyard

- Kon Leong of ZL Technologies, says:

A “Big Data” gold rush is sweeping the enterprise world, complete with the hype and anything-goes attitude of the original 49ers. Information is increasingly described as the new currency of business, and companies are driven into a frenzy when it comes to staking their claim and “mining” it. But just like the prospectors driven West with the promise of easy fortune, we are often mistakenly overlooking the value of materials that have been in front of us all along. Software and services that claim to instantly process and mine data are flying off the shelves before the companies buying them even bother to assess their own internal data landscape. And as it turns out, there’s quite a bit of valuable business information to be unearthed right in the enterprise backyard.

Why haven’t corporations been quick to leverage data that’s been available for years in knowledge management (KM) systems? The digitization of communications means that the internal business process is driven by an endless effluent of documents, emails, IMs, and other forms of “unstructured” data that are potentially rich in human insight but poor in the polished nuggets of traditional BI outputs. In the race to numerical data veins, we’ve just been trampling over equally valuable deposits of human information that are simply more difficult to refine. The raw ore of communications may take more processing, but it promises to yield additional information we need to drive businesses forward, especially as the benefits from traditional “structured” analytics start to reach an upper limit.

Internal sources can offer a treasure trove of information that rivals anything collected from customers or “outside” sources such as data brokers. Regulatory and legal requirements already mandate that unstructured data of emails and documents be archived and managed, and yet today this information often sits unused or filed away. We hoard unstructured data defensively rather than for proactive analysis. But just like those early prospectors, our methods of extracting value are improving with time.

The next frontier of KM is to refine the information right under our noses, and to add a uniquely human perspective to strategy. Our business communication patterns reveal workflows, strengths, weaknesses, efficiency, and even collective sentiment. With the rapid advancement of text analytics, the possibilities of mining human-generated material have staggering potential. The challenge is having centralized architecture capable of processing an entire corpus of unstructured information in one platform: “silos” of information irreparably hinder the aggregation required for true “Big Data” analytics. Moving forward, the infrastructure used to handle enterprise unstructured data will largely determine whether the business will obtain true glimmers of insight or simply end up with fool’s gold.