🗄️Data Management: Master Data Management (Part I: Understanding Integration Challenges) [Answer]

Data Management Series

Answering Piethein Strengholt’s post [1] on Master Data Management’s (MDM) integration challenges, the author of "Data Management at Scale".

Master data can be managed within individual domains though the boundaries must be clearly defined, and some coordination is needed. Attempting to partition the entities based on domains doesn’t always work. The partition needs to be performed at attribute level, though even then might be some exceptions involved (e.g. some Products are only for Finance to use). One can identify then attributes inside of the system to create the boundaries.

MDM is simple if you have the right systems, processes, procedures, roles, and data culture in place. Unfortunately, people make it too complicated – oh, we need a nice shiny system for managing the data before they are entered in ERP or other systems, we need a system for storing and maintaining the metadata, and another system for managing the policies, and the story goes on. The lack of systems is given as reason why people make no progress. Moreover, people will want to integrate the systems, increasing the overall complexity of the ecosystem.

The data should be cleaned in the source systems and assessed against the same. If that's not possible, then you have the wrong system! A set of well-built reports can make data assessment possible. 

The metadata and policies can be maintained in Excel (and stored in SharePoint), SharePoint or a similar system that supports versioning. Also, for other topics can be found pragmatic solutions.

ERP systems allow us to define workflows and enable a master data record to be published only when the information is complete, though there will always be exceptions (e.g., a Purchase Order must be sent today). Such exceptions make people circumvent the MDM systems with all the issues deriving from this.

Adding an MDM system within an architecture tends to increase the complexity of the overall infrastructure and create more bottlenecks. Occasionally, it just replicates the structures existing in the target system(s).

Integrations are supposed to reduce the effort, though in the past 20 years I never saw an integration to work without issues, even in what MDM concerns. One of the main issues is that the solutions just synchronized the data without considering the processual dependencies, and sometimes also the referential dependencies. The time needed for troubleshooting the integrations can easily exceed the time for importing the data manually over an upload mechanism.

To make the integration work the MDM will arrive to duplicate the all the validation available in the target system(s). This can make sense when daily or weekly a considerable volume of master data is created. Native connectors simplify the integrations, especially when it can handle the errors transparently and allow to modify the records manually, though the issues start as soon the target system is extended with more attributes or other structures.

If an organization has an MDM system, then all the master data should come from the MDM. As soon as a bidirectional synchronization is used (and other integrations might require this), Pandora’s box is open. One can define hard rules, though again, there are always exceptions in which manual interference is needed.

Attempting an integration of reference data is not recommended. ERP systems can have hundreds of such entities. Some organizations tend to have a golden system (a copy of production) with all the reference data. It works for some time, until people realize that the solution is expensive and time-consuming.

MDM systems do make sense in certain scenarios, though to get the integrations right can involve a considerable effort and certain assumptions and requirements must be met.

Previous Post <<||>> Next Post

References:
[1] Piethein Strengholt (2023) Understanding Master Data Management’s Integration Challenges (link)

⛏️Data Management: Master Data (Definitions)

"Data describing the people, places, and things involved in an organization’s business. Examples include people (e.g., customers, employees, vendors, suppliers), places (e.g., locations, sales territories, offices), and things (e.g., accounts, products, assets, document sets). Master data tend to be grouped into master records, which may include associated reference data." (Danette McGilvray, "Executing Data Quality Projects", 2008)

"Master data is the core information for an enterprise, such as information about customers or products, accounts or locations, and the relationships between them. In many companies, this master data is unmanaged and can be found in many, overlapping systems and is often of unknown quality." (Allen Dreibelbis et al, "Enterprise Master Data Management", 2008)

"Data that describes the important details of a business subject area such as customer, product, or material across the organization. Master data allows different applications and lines of business to use the same definitions and data regarding the subject area. Master data gives an accurate, 360° degree view of the business subject." (Tony Fisher, "The Data Asset", 2009)

"The set of codes and structures that identify and organize data, such as customer numbers, employee IDs, and general ledger account numbers." (Janice M Roehl-Anderson, "IT Best Practices for Financial Managers", 2010)

"The data that provides the context for business activity data in the form of common and abstract concepts that relate to the activity. It includes the details (definitions and identifiers) of internal and external objects involved in business transactions, such as customers, products, employees, vendors, and controlled domains (code values)." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"The critical data of a business, such as customer, product, location, employee, and asset. Master data fall generally into four groupings: people, things, places, and concepts and can be further categorized. For example, within people, there are customer, employee, and salesperson. Within things, there are product, part, store, and asset. Within concepts, there are things like contract, warrantee, and licenses. Finally, within places, there are office locations and geographic divisions." (Microsoft, "SQL Server 2012 Glossary", 2012)

"Data that is key to the operation of a business, such as data about customers, suppliers, partners, products, and materials." (Brenda L Dietrich et al, "Analytics Across the Enterprise", 2014)

"The data that describes the important details of a business subject area such as customer, product, or material across the organization. Master data allows different applications and lines of business to use the same definitions and data regarding the subject area. Master data gives an accurate, 360-degree view of the business subject." (Jim Davis & Aiman Zeid, "Business Transformation: A Roadmap for Maximizing Organizational Insights", 2014)

"Informational objects that represent the core business objects (customers, suppliers, products and so on) and are fundamental to an organization. Master data must be referenced in order to be able to perform transactions. In contrast with transaction or inventory data, master data does not change very often." (Boris Otto & Hubert Österle, "Corporate Data Quality", 2015)

"The most critical data is called master data and the companioned discipline of master data management, which is about making the master data within the organization accessible, secure, transparent, and trustworthy." (Piethein Strengholt, "Data Management at Scale", 2020)

⛏️Data Management: Reference Data (Definitions)

"Reference data is focused on defining and distributing collections of common values to support accurate and efficient processing of operational and analytical activities." (Martin Oberhofer et al, "Enterprise Master Data Management", 2008)

"Sets of values or classification schemas referred to by systems, applications, data stores, processes, and reports, as well as by transactional and master records. Examples include lists of valid values, code lists, status codes, flags, product types, charts of accounts, product hierarchy." (Danette McGilvray, "Executing Data Quality Projects", 2008)

"Data that describe the infrastructure of an enterprise. These comprise the 'type' entity classes that provide lists of values for other attributes." (David C Hay, "Data Model Patterns: A Metadata Map", 2010)

"Data characterized by shared read operations and infrequent changes. Examples of reference data include flight schedules and product catalogs. Windows Server AppFabric offers the local cache feature for storing this type of data." (Microsoft, "SQL Server 2012 Glossary", 2012)

"Corporate data that has been defined externally and is uniformly changed across company boundaries, such as country codes, currency codes and geo-data." (Boris Otto & Hubert Österle, "Corporate Data Quality", 2015)

"Reference data is commonly used to link and give additional details to the data. It is the data used to classify, organize, or categorize other data. Reference data can also contain value hierarchies, for example, the relationships between product and geographic hierarchies. It is escorted by the discipline Reference Data Management, which makes sure the reference data is consistent and that different versions are managed and distributed properly." (Piethein Strengholt, "Data Management at Scale", 2020)