Search vs. Directory

Data classification or content catalogues on the Internet have died long time ago. Lycos, Excite, Altavista, Yahoo, MSN.... Long list of unsuccessful attempts to classify the Internet content into a meaningful list of categories. Then comes Google and it promotes idea of simple search box. Everybody hooks up.

In the same time, plenty of enterprises are looking into Knowledge Management projects and put lot's of efforts into locking the collective knowledge within the company. How to secure your employees knowledge assets within the boundaries of your digital empire? There are many ways to tackle this quest - roles based information push (based on information classification), enforcing document management procedures, data warehousing, knowledge directories, etc... None of them actually worked and here is the answer why:

Employes are not willing to share their knowledge for free. Current compensation schemes are mostly oriented on individual performance. Forced curves in bonus distributions usually increase competition among the employees. So, extra effort in making your knowledge public is not incentivized. Systems supporting this process are very complex and still aren't properly integrated with operational systems where most of the digital assets are created (mostly email). New knowledge categories are emerging every day so instability of the system causes lot's of troubles in managing them. It's basically a never ending software development story with very few static elements. Cost benefit analysis would tell you don't go there.

This is all valid in case that we really want to classify knowledge and capture it for later use. If we scale the scope down, it brings everything to a very basic level of operational process descriptions what is far away from knowledge management. It means than only possible information possible to manage is process description and it is far away from knowledge. It operating manual maintenance.

When considering other structural approaches to information consolidation, first idea are big data warehouses. Complex ETL procedures are loading terabytes of data for later use. When you start comparing what you get in the reports with what you have in transactional systems, disillusionment kicks in. The change dynamics of originating systems is too fast to manage the process along the way. Complexities keep increasing and makes it almost impossible to reflect those changes in consolidated systems.

Let's get back to the beginning of the post. Search engines have made it possible to reach the Internet content as you need it. It seems to me that same approach should taken with structured enterprise data, residing in many heterogenous transactional systems. Information discovery based on search is becoming more popular within the enterprise IT solutions. Have a look at Authonomy and Fast. I am not sure that these companies are getting the full potential of enterprise data consolidation. They are still too much focusing on the Internet scenarios. IMHO, in the future, traditional approach to data consolidation using ETL and Cubes will be squeezed out by more advanced search technologies. Indexing structured data seems to work lot easier than long lasting projects of data warehousing. Analytical power based on search makes every IT user an ad hoc data analyst without any predefined data mining scenarios. Scenarios will emerge by themselves when the appropriate technology will be unleashed.

Recently, Wolfram launched something that aims to be an ultimate knowledge repository software. It is basically a search engine. Fact that the directories are dead and search rules the Internet is not new. Now is the time to transfer this trend into the enterprises and index structured operational systems. For sure, a whole new revolution in information management and use will be started.