Capco Institute Blog

Massive centralized data warehouses that could presumably contain all the data within a financial institution have been the holy grail of information management architecture ever since storage costs plummeted to immaterial levels. In this “supply-side warehousing” school of thought, the prevailing rule is that anything that is available should be harnessed because someone, somewhere, at some point in time may have some use for it. This has led to the development of ever-larger and more complex systems, as diversified organizations have attempted to harmonize detailed models of a large array of disparate instruments, transactions and analytic measures.

As budgets become increasingly tighter, regulatory demands more numerous, and the maintenance costs of such systems grow exponentially (along with an increase in the amount of workarounds), many financial institutions are seeing the pendulum swing in the opposite direction: Avoid centralizing the data and rely on direct inter-system messaging instead. In a recent online article on Wall Street & Technology, well-known experts in information management systems argue that “[...] firms are no longer spending money on gigantic warehouses and agreeing on a single data model that would fit all the data in one place [and instead] are looking at more messaging-oriented data management, where the reference and infrastructure lie in the systems of record, and they then pass that information to subsequent systems [...].”

This approach, however, is applicable only for firms that have spent considerable amounts of resources over the past several years to improve and standardize their primary source systems at least to some reasonable extent — in other words, the largest commercial and investment banks. For the vast majority of second- and third-tier banks that are now experiencing the risk management and regulatory pressures previously reserved for the top tier, this kind of approach is not feasible. Their systems are simply too numerous and too disparate and their processes involve too many manual adjustments to allow for such a decentralized approach.

For example, an assessment of requirements to build fairly basic credit risk reporting at a large super-regional bank resulted in the need to harvest information from more than 30 source systems — whose data was stored in proprietary formats, various flavors of relational and Lotus Notes databases, myriad spreadsheets, text files and other “repositories” of manual overrides and adjustments. Customer information existed in no fewer than nine customer relationship management systems.

This case study, by no means unique, is representative of the state of infrastructure within institutions that are just below the “too-big-to-fail” radar. Clearly, direct communications among those systems is out of the question. But what is the answer? One solution is a demand-side-driven approach to data warehousing, where a centralized repository still provides a harmonized view of the data and acts as the “traffic cop” between all these disjointed systems but is built under much more carefully defined guidelines. Financial institutions interested in developing a successful demand-side-driven data warehouse should:

• Design an initial, generic, structurally flexible and parsimonious model that can easily accommodate new requirements.
• Enrich the warehouse, as the term “demand side” implies, only with data required for a specific business use (outside of the original source system), instead of the old “just in case” approach.
• Ensure that even with the addition of specific business requirements, the structure of the warehouse remains “foundational,” that is, includes only data and business logic likely to be enterprise-wide and which can be accessed only through properly designed handles.
• Build a secondary layer of specialized data marts, where additional data and logic relevant to one or a few “consumers” (which can be a business unit, a particular type of reporting package, etc.) can enrich or transform the data to suit its own needs.
• Establish a responsible governance system, in which the development costs are borne, at least in part, by the “consumer” of the data, thus ensuring that any new requirements are carefully evaluated.

This kind of pragmatic approach retains most of the benefits of a centralized, harmonized and carefully managed data warehouse that is essential for institutions with a large number of legacy systems, while avoiding the costs and pitfalls of the universal, monolithic behemoths that have been created over the years. Banks that have lagged behind in infrastructure spending should learn these lessons from the past as they struggle to cope with the growing transparency, auditability and reporting requirements that will only increase in the future.

What benefits could your organization reap from a demand-side-driven approach to data warehousing? Join the discussion.