Emerging KM Playbook Powered by Knowledge Engineers
Let me define an intelligent enterprise as one that can learn from its past and where the knowledge workers can learn from each other all enabled by knowledge management (KM) capabilities. In this picture, KM plays a key role in assessing the processes that create knowledge in the first place and then help folks leverage them throughout the enterprise. In the new era of AI and ML, two kinds of knowledge workers are appearing in the organization that are enhancing current KM capabilities; the data scientists and data engineers. They work in tandem to build knowledge from complex recipes that involve data processing pipelines (e.g. build, train, QC), algorithms, training sets, and statistics-based models (i.e. ML). These new knowledge engineers apply their skills to support decisions throughout the enterprise, creating a new kind of KM playbook along the way.
Let’s focus on the knowledge that enables key decisions that is usually shared at leadership meetings (dept./group/ project) where the facts, hypothesis, analysis, scenarios, and proposals are presented. Some of this knowledge may be coined an “insight” if it brings to bear a unique piece of information or intel around a new opportunity, risk or threat. Currently that knowledge sits with your staff and mostly gets transmitted via conversations, training, or captured at one of the “decision meeting”. Many companies use blogs/wiki/ messaging capabilities (e.g. Sharepoint, Confluence, Slack) along with document repositories and the usual email to share knowledge. That is all good but the challenge we all face is to be able to find, extract, and unravel the thought process and the data provenance/lineage that was actually used to support those decisions. This limits our ability to share the “how we got here?” as a company and “why certain decisions were made?” We simply lack the ability to learn from the past.
These new knowledge engineers apply their skills to support decisions throughout the enterprise, creating a new kind of KM playbook along the way
Today we have new means to address those challenges in both skills and technologies (both evolving rapidly) that are powering a new data-driven KM. This new breed of data-centric roles called data engineer and data scientist entered the enterprise with the big data wave. Their main job is to mine for knowledge, extract new correlations or even causations (if they get so lucky) out of existing data/knowledge/information available throughout their company and their industry. In essence, data engineers are doing ELT instead of ETL spending more time on complex transformation pipelines and data scientist spend their time building statistical models not reports.
These new skillsets are driving an emerging KM playbook that can be represented at a high-level by five distinct steps. It all starts with identifying the key questions that need answering in order to drive the business forward (step-1) and look at the KPIs to assess the impact/value of answering each question. Those questions then feed into your data strategy (step-2), which defines the data you need, both internal and external in order to answer the questions. This step captures the what/when/where and why you need this data and is usually tied to your business processes. Then comes the aggregation ETL/ELT pipelines (step-3) of cleansing, merging, and transforming various sources of information (structured and unstructured) into readily available normalized and question-specific datasets (e.g. KNIME, XCalar). These pipelines contain the recipe of how the data came together and the various assumptions and merging/matching models/algorithm that were used.
At this point the data is ready for the analytics pipelines (step- 4) and appropriate building statistical models. Statistical models come in different shapes and forms and usually don’t come with a “user guide” (what it is good for and what to avoid) or a list of “ingredients” (stats used and training set bias). That is where the pipeline comes to play describing the recipe of how the insight or knowledge was put together and the various assumptions and models/algorithm that were used (e.g. KNIME, Databricks). It is worth mentioning that Gartner created a new magic quadrant called “Insight platform” which essentially covers step 3 and 4 for enterprise content. These platforms are the natural evolution of the enterprise search platform that now embeds “cognitive” capabilities à la Spark-ML by training statistical models using the search index created from the enterprise corpus (e.g. Sinequa, LucidWorks, Coveo). These platform help solve a variety of regulatory and legal use cases (e.g. GDPR, e-Discovery).
The grand finale is all about storytelling and sharing the thought process (step-5). These knowledge engineers are inherently focused on exploring new hypothesis and need to be able to demonstrate their logic in order to collaborate, share their results and ultimately convince the stakeholders. The computable notebook is just the tool as it combines live code, narrative, math, graphs/charts that helps them make their case while sharing their thought process (e.g. Jupyter, Wolfram CDF, SageMath). The key point is being explicit about what makes up a proposal (facts, drivers, hypothesis, assumptions, context) to make it easier to understand where/when/why you were right or wrong and learn from it.
This new KM playbook does generate many assets (question/ use case, data, pipeline, models) that will need to be catalogued and managed. The asset catalog is the cornerstone of knowledge sharing and re-use by helping you navigate to the original analysis, understand its context and see if the data, the analytics and the thought process applies to your current challenge. Getting this catalog right along with the right governance is probably the biggest challenge in this playbook. We can turn to master data management, enterprise asset management playbooks and emerging FAIR (go-fair.org) principlesto help solve this one.
In order to share knowledge and most importantly for a company to learn from its past it must understand why and when decisions were made and be able to reflect back on their original hypothesis and expected outcome. Imagine having captured in various notebooks the series of events and detailed thought processes that led to where the company is today. Now imagine the “lessons learned” sessions happening throughout the company leveraging those notebooks. This is what the KM playbook powered by knowledge engineers is enabling today.