About Better BI

The Nutshell Version

Better BI is a synthesis of over twenty five years experience in computer-based business data analysis coupled with the best modern tools and technologies into a coherent approach ruthlessly committed to eliminating the barriers between real human people and their data. Closing this gap provides the best possible opportunity for obtaining information and achieving insights from the data.

The past couple of decades have seen advances in technological capability and capacity harnessing the phenomenal increases in computing power along with stunning increases in the volume of business data.

Sadly, however, the evolution of Business Intelligence has not improved the lot of a business person needing to obtain information from and insights into their data. Twenty five years ago, in the age of mainframes and COBOL data processing shops it would commonly take weeks or months for a new report to be created. Modern Big BI, with its foundation in consolidated dimensionally modeled marts and warehouses fed by complex ETL processes and fronted by monolithic mega-BI platforms, is today's data processing. It can take weeks, commonly months, from the onset of a BI program until the first delivery of useful analytics to real human business people. Big BI also fiendishly difficult and expensive, consuming vast amounts of time, energy, effort, resources, and money.

Better BI is a Better Way

Better BI provides an alternative approach to Business Intelligence. Taking advantage of the best new direct data access and visualization tools coupled with professional skills, Better BI provides the opportunity to change the paradigm of BI delivery. Better BI considers all business data suitable for analysis.
With Better BI business people are able to obtain meaningful, actionable, high value, high quality, timely analyses of their data immediately, frequently on the spot, or within hours. There's no longer any need to wait until their business data is assimilated into the enterprise data stores before any analysis can be performed.

A Brief History

The Early Years

Growing up in the early days of computing, I was fascinated by the idea that computers could be used to help people acquire and make sense of information.

In the 1970s I began my computing experience with mainframe FORTRAN and COBOL programming at the University of Guelph. Our master COBOL project was bird banding data analysis—it wasn't what I had hoped for; the barrier to get meaningful insights into the data was extremely high. The difficulty of this approach led me to pursue better ways of analyzing data.

The increasing presence of non-mainframe computers in the late 70s and early 80s provided the rich environment for the widespread adoption of human-oriented approaches to computing, and the time was right to pursue my interest in computer support for human intellectual efforts.

In 1985, as a newly minted Bachelor of Business Administration, majoring in Computer Information Systems, I went to work for JCPenney convinced that the future of business excellence lay in improving the ability of business decision makers to access, analyze, and interpret their business data, and in communicating the essential insights they achieved.

FOCUS

My early career was graced with tremendous good fortune. My first job at JCPenney was to use a new(ish) fourth generation programming language (4GL)—FOCUS—to implement a variety of business systems. FOCUS was the creation of brilliant programmers who recognized that writing COBOL reports against mainframe data was horribly inefficient, cumbersome, and slow. Their insight was that reporting was (and still is) a structured activity whose elements could be captured in English terms. FOCUS was their English language and interpreter for specifying reports and the data access and processing engine to retrieve the appropriate data and render the desired report from it.

With FOCUS, the barriers between business people and their data dropped away. No longer would it be necessary to make the pilgrimage to the DP shop to ask, plead, or beg for a COBOL program to be written, and then to wait the days, weeks, even months it took for the overworked DP programmers to get through their backlog.

In 1986 I had the opportunity to join Information Builders, Inc., FOCUS' creator and vendor, where I spent the next eight years, first as a consultant helping clients achieve the rapid access and high quality reporting of their business data they needed, and then as a product manager where I was able to work on the FOCUS products for the increasingly important PC and Unix platforms.

The Middle Years

The proliferation of non-mainframe computing in business led to a number of significant changes in approaches to data processing. Relational databases became the norm, and as experience with them grew so did the realization that they were not well suited for business analysis of the data they contained.

The struggles with relational databases led to the development of alternate data storage designs better suited for business analysis. Dimensionally models emerged in the form of data marts and warehouses. While well suited for their intended purpose data warehousing evolved into large scale operations.

The data warehouse-based paradigm of business data analysis has its virtues. An organization's data is consolidated with the attendant opportunities for harmonization, governance, management, and access control. Centralized analytical tools such as Cognos, Business Objects, MicroStrategy, and their cousins evolved to become capable of providing a full range of business analytics—tables, charts, graphs, reports, dashboards, scorecards, strategy maps, etc. once the warehouse is populated and the requisite semantic layer mapping data to business objects is in place.

The data warehouse-based paradigm has severe flaws, to the detriment of BI's main purpose of delivering actionable, high quality information and insights to real human people.

Some of them are structural. Building a consolidated data warehouse and the necessary analytics platform necessarily involves many activities and specialized technologists just to get the data processing and analysis machinery in place and working. The data processing—ETL—effort to understand relational source systems, design the appropriate dimensional model, and design, implement, and test the ETL processes turned out to be tremendously complex and expensive, consuming the bulk of the resources required in BI efforts. Similarly, designing and implementing the analytic tool's semantic layer must be done before actual business analytics can be created and delivered.

Some of them are organizational. Managing the multiplicity of processes and activities involved in constructing the complete technology stack is a real challenge. One of the derivative effects of undertaking traditional BI projects is that the management of the internal technical aspects tends to dominate the landscape, drawing time and attention away from the only part that really matters to the business: providing information and insights into the business data.

The worst of them are cognitive. The framing of – Business Intelligence – as enterprise scale "Big BI" pollutes the entire concept of delivering meaningful, actionable, high quality, timely information and insights into their data for business people. People who need and can use information have become accustomed tremendous amounts of friction involved. Business sponsors have become callused with the notion that it will take many weeks, even months, after the start of a BI project before analytics are delivered (and experience shows that these are likely not what the business really needs).
The worst cognitive problem is that the world of data analysis has been segmented into the Big BI approach with all of its heavy weight and high ceremony, and the purely technical where developers write SQL queries.

The inevitable outcome is that Big BI became a very large scale industrial undertaking with all of its attendant problems, not the least of which is the assumption that it is subject to the same forces as and amenable to the same approaches as any other industrial effort. This has been terrible. Big BI costs far too much and delivers far too little real value.

Better BI Emerges

The roots of Better BI go back decades, to the age of mainframes, COBOL, and data processing shops. As noted above, FOCUS was invented to facilitate the burden of developing reports. It was tremendously successful as organizations and business managers learned that they could get reports directly from their data without the delays involved in going to their DP shops, FOCUS adoption soared. One example: in 1987 one of my clients was the Accounting manager for a very large company, responsible for delivering financial reports. I was able to work with him daily, each morning reviewing the reports I'd built and deciding on the necessary adjustments and new reports. Together we created his complete set of reports in a very short time, to his complete satisfaction.

In the recent past a number of innovative data analysis tools have emerged including: Tableau, Spotfire, QlikView, Advisor, and Omniscope. Each has its strengths; most significantly they share the premise that analyzing data should be as straightforward and easy as possible. These tools were largely adopted by individuals looking for understanding of their own data.

My experience with the new generation tools began when I was asked to work on an ongoing Big BI project. A major city was implementing an Agency accountability program wherein the Mayor could assess agency effectiveness in providing services to constituent. Each of several dozen city agencies was to provide data feeds from their service systems, which would then be integrated into the Citywide Data Warehouse and made available to the various stakeholders. It was a pretty standard undertaking: the data feeds would load into a staging area, be ETLed into an Oracle data warehouse, which in turn was fronted by Business Objects. The project was two-plus years old, had spent a lot of money, and had no business users.

The best approach to understanding complex data-based systems is to gain an understanding of the full spectrum of data across all of the systems involved. With this goal I looked for the best tools for the job and found those mentioned above.

Using the new tools, and rapidly settling on Tableau as the best for my purposes, I was able to rapidly understand the various systems' data. Just as importantly, I realized that there were tremendous gaps in the existing project that could be filled with the rapid analytics development I was able to perform. The project staff had poor insight into the business nature of the Agencies' data. The analytics being developed in Business Objects had no input from the business stakeholders; this is common in Big BI projects where analytics can't be created until the end. The internal technical staff were hand writing SQL queries to understand and assess the data flows across the various systems.

These gaps are endemic to Big BI projects.

Most often, the business analytics are simple sketches or schematics, e.g. Visio diagrams and Word documents, and are frequently created as the output of an interview of a business stakeholder by someone unskilled in the business domain, business data analysis or analytical information design.

Business analysis of source data is rarely conducted well. Even when collaboration between the technical and business stakeholders occurs high impedance impedes understanding, largely because of the difficulty involved, and/or in the use of low-tech and low fidelity SQL querying (either hand written or via Toad or similar query tool.

Internal processes in Big BI projects suffer from inefficiencies in understanding the data flows across the project's components. ETL developers rely on SQL queries to assess the source and target data and evaluate the correctness of the transformations. These assessments are suboptimal in both gaining insight into the processes quality and in communicating those insights to stakeholders.