📦Data Migrations (DM): Conceptualization (Part II: Plan vs. Concept vs. Strategy)

Data Migrations Series

A concept is a document that describes at high level the set of necessary steps and their implications to achieve a desired result, typically making the object of a project. A concept is usually needed to provide more technical and nontechnical information about the desired solution, the context in which a set of steps are conducted, respectively the changes considered, how the changes will be implemented and the further aspects that need to be considered. It can include a high-level plan and sometimes also information that typically belong in a Business Case – goals,objectives, required resources, estimated effort and costs, risks and opportunities.

A concept is used primarily as basis for sign-off as well for establishing common ground and understanding. When approved, it’s used for the actual implementation and solution’s validation. The concept should be updated as the project progresses, respectively as new information are discovered.

Creating a concept for a DM can be considered as best practice because it allows documenting the context, the technical and organizational requirements and dependencies existing between the DM and other projects, how they will be addressed. The concept can include also a high-level plan of the main activities (following to be detailed in a separate document).

Especially when the concept has an exploratory nature (due to incomplete knowledge or other considerations), it can be validated with the help of a proof-of-concept (PoC), the realization of a high-level-design prototype that focuses on the main characteristics of the solution and allows thus identifying the challenges. Once the PoC implemented, the feedback can be used to round out the concept.

Building a PoC for a DM should be considered as objective even when the project doesn’t seem to meet any major challenges. The PoC should resume in addressing the most important DM requirements, ideally by implementing the whole or most important aspects of functionality (e.g. data extraction, data transformations, integrity validation, respectively the import into the target system) for one or two data entities. Once the PoC built, the team can use it as basis for the evolutive development of the solution during the iterations considered.

A strategy is a set of coordinated and sustainable actions following a set of well-defined goals, actions devised into a plan and designed to create value and overcome further challenges. A strategy has the character of a concept though it has a broader scope being usually considered when multiple projects or initiatives compete for the same resources to provide a broader context and handle the challenges, risks and opportunities. Moreover, the strategy takes an inventory of the current issues and architecture – the 'AS-IS' perspective and sketches the to 'TO-BE' perspective by devising a roadmap that bridges the gap between the two.

In the case of a DM a strategy might be required when multiple DM projects need to be performed in parallel or sequentially, as it can help the organization to better manage the migrations.

A plan is a high-level document that describes the tasks, schedule and resources required to carry on an activity. Even if it typically refers to the work or product breakdown structure, it can cover other information usually available in a Business Case. A project plan is used to guide both project execution and project control, while in the context of Strategic Management the (strategic) plan provides a high-level roadmap on how the defined goals and objectives will be achieved during the period covered by the strategy.

For small DM projects a plan can be in theory enough. As both a strategy and a concept can include a high-level plan, the names are in praxis interchangeable.

Previous Post <<||>> Next Post

📦Data Migrations (DM): Conceptualization I (Goals, Objectives & Requirements)

Data Migrations Series

One of the nowadays’ challenges is finding the right mix of technologies that allows building a solution for a business need. There are so many choices and the responsible person is easily tempted to use one of the trending technologies just because he wants to learn something new or the technologies seem to fit into the bigger picture, which probably in many cases it would be acceptable. Unfortunately, there’s also the tendency of picking a technology without looking at what functionality it provides, respectively whether the functionality meets intended solutions’ requirements. Moreover, the requirements are sometimes barely defined at the appropriate level of detail, fact that makes from the implementation project a candidate for failure. Sometimes even the goals and objectives aren’t clearly stated, fact that can make a project’s success easily questionable from the beginning. 

A goal is a general statement that reflects the desired result toward which an organization’s effort needs to be directed. For example, a Data Migration (DM)’s primary goal can be formulated as 'to make available all the master and transactional data needed by the business from the legacy systems to the target system(s) within expected timeline and quality with a minimal disruption for the business'. 

An objective is a break down of the goal into several components that should foster a clear understanding on how the goal will be achieved. Ideally the objectives should be SMART (specific, measurable, attainable, relevant, time-bound), even if measurable objectives are sometimes hard to define properly. One can consider them as the tactics used in achieving the goal. For example, the above formulated goal can be broken down into the following objectives:

• Build a DM concept/strategy
• Build a flexible and performant infrastructure for DM that can be adapted to further requirements
• Provide a basis for further DMs
• Align DM and main project’s requirements and activities
• Provide an interface and support for the Data Management areas
• Foster trust, transparency and awareness 
• Address internal/external compliance requirements
• Document and communicate accountability for the various activities
• Cleanse and enrich the data needed by the target system 
• Archive the DM and project data 

One can attempt defining the objectives directly from the goal(s), though unless one is aware of all the implication a DM has, more likely one will be forced to define and evaluate the individual functional and nonfunctional requirements for the DM first, and attempt consolidating the requirements into a set of objectives. In the end it can be a combination of both, in which some objectives are first formulated, the requirements are defined and evaluated, respectively the objectives are refined to accommodate the requirements. 

ISO 9126, an international standard for the evaluation of software quality, defines about 45-50 attributes that can be used for addressing the requirements of software solutions, attributes that reflect functionality, reliability, usability, efficiency, and maintainability characteristics. One can start with such a list and identify how important are the respective attributes for the solution.  The next step would be to document the requirements into a consolidated list by providing a short argumentation for their use, respectively how they will be addressed as part of the solution. The process can prove to be time-consuming, however it is a useful exercise that usually needs to be done only once and be reviewed occasionally.

The list can be created independently of any other documentation or be included directly into a concept or strategy. The latter will assure in theory that the document provides a unitary view of the migration, considering that each new or obsolete requirement can impact the concept. 

Previous Post <<||>> Next Post

𖣯Strategic Management: Strategy Design (Part VIII: Quality Acceptance Criteria for Strategies and Concepts)

Quality acceptance criteria for concept documents in general, and for strategies in particular, are not straightforward for all, behind the typical request of completeness hiding other criteria like flexibility, robustness, predictability, implementable, specificity, fact-based, time-boundedness, clearness, comprehensibility or measurability.

Flexible: once the strategy approved, one must be able to change the strategy as may seem fit, especially to address changes in risks, opportunities, goals and objectives, respectively the identification of new facts. A strategy implies a roadmap on how to arrive from the starting point to destination. As the intermediary or final destinations change, the strategy must reflect these changes (and it’s useful to document these changes accordingly). This also implies that the strategy must be periodically reviewed, the new facts accordingly analyzed and decided whether they must be part of the strategy.
Robust: a strategy must handle variability (aka changes) and remain effective (producing the desired/intended results).

Predictable: the strategy needs to embrace the uncertainty and complexity of the world. Even if one can’t predict the future, the strategy must consider the changes foreseen in the industry and technologies. Is not necessarily about imagining the future, even if this would be ideal, but to consider the current trends in the industry.

Implementable: starting with the goals and ending with the roadmap, the strategy must be realistic and address organization’s current, respectively future capabilities. If the organization need to acquire further capabilities, they need to be considered as well.

Specific: the strategy must address the issues, goals and objectives specific for the organization. As long these are not reflected in it, the strategy is more likely to fail. It is true that many of the issues and goals considered can be met in other organizations, however there are always important aspects that need to be made explicit.

Fact-based: the strategy must be based on facts rooted in internal or market analysis, however the strategy is not a research paper to treat in detail the various concepts and findings – definitions, summaries of the findings with their implications, and references to further literature are enough, if needed.

Time-bound:  in contrast to other concepts, the strategy must specify the timeframe considered for its implementation. Typically a strategy addresses a time interval of 3 to 5 years, though upon case, the interval may be contracted or dilated to consider business specifics. The strategy can further break down the roadmap per year or biannually.

Complete: the strategy must be complete in respect to the important topics it needs to address. It’s not only about filing out a template with information, the reader must get a good understanding of what the strategy is about. Complete doesn’t mean perfect, but providing a good enough description of the intent.

Clear: especially when there are competing interests, the strategy must describe what is in scope and what was left out. What was left out is as important as what is considered, including the various presumptions. A test of clearness is whether the why, how, who, when and by what means were adequately considered.

Comprehensible: the targeted audience must be able to read and understand the strategy at the appropriate level of detail or scope.

Measurable: the progress of a strategy must be measurable, and there are two aspects to consider. On one side the goals and objectives considered must be measurable by definition (see SMART criteria), while on the other, one must be able to track the progress and various factors related to it (e.g. implementation costs, impact of the changes made, etc.). Therefore, a strategy must include a set of metrics that will allow quantifying the mentioned aspects.

#️⃣☯Software Engineering: Concept Documents (The Good, the Bad and the Ugly)

A concept document (simply a concept) is a document that describes at high level the set of necessary steps and their implications in order to achieve a desired result, typically making the object of a project. In other words, it describes how something can be done or achieved, respectively how a problem can be solved.

The Good: The main aim of the document is to give all the important aspects and to assure that the idea is worthy of consideration, that the steps considered provide a good basis for further work, respectively to provide a good understanding for the various parties involved, Therefore, concepts are used as a basis for the sign-off, respectively for the implementation of software and hardware solutions.

 A concept provides information about the context, design, architecture, security, usage, purpose and/or objectives of the future solution together with the set of assumptions, constraints and implications. A concept is not necessarily a recipe because it attempts providing a solution for a given problem or situation that needs a solution. Even if it bears many similarities in content and structure a concept it also not a strategy, because the strategy offers an interpretation of the problem, and also not a business case, because the later focuses mainly on the financial aspects.

A concept proves thus to be a good basis for implementing the described solution, being often an important enabler. On the other side, a written concept is not always necessary, even if conceptualization must exist in implementers’ head.

The Bad: From these considerations projects often consider the elaboration of a concept before further work can be attempted. To write such a document is needed to understand the problem/situation and be capable of sketching a solution in which the various steps or components fit together as the pieces of a puzzle. The problem is that the more complex the problem to be solved, the fuzzier the view and understanding of the various pieces becomes, respectively, the more challenging it becomes to fit the pieces together. In certain situations, it becomes almost impossible for a single person to understand and handle all the pieces. Solving the puzzle becomes a collective approach where the complexity is broken in manageable parts in the detriment of other aspects.

Writing a concept is a time-consuming task. The more accuracy and details are needed, the longer it takes to write and review the document, time that’s usually stolen from other project phases, especially when the phases are considered as sequential. It takes about 20% from the total effort needed to write a ‘perfect’ concept for writing a concept that covers only 80% of the facts, while 80% from the effort to consider the remaining 20% of the facts as the later involve multiple iterations. In extremis, aiming for perfection will make one start the implementation late or not start at all. It’s a not understandable pedantry with an important impact on projects'
 timeline and quality in the hope of a quality increase, which is sometimes even illusory.

The Ugly: The concept-based approach is brought to extreme in ERP implementations where for each process or business area is needed to write a concept, which often carries fancy names – solution design document, technical design document, business process document, etc. Independently how it is called, the purpose is to describe how the solution is implemented. The problem is that the conceptualization phase tends to take much longer than planned given the dependencies between the various business area in terms of functionality and activities. The complexity can become overwhelming, with an important impact on project’s budget, time and quality.

🌡Performance Management: Mastery (Part I: The Need for Perfection vs. Excellence)

A recurring theme occurring in various contexts over the years seemed to be corroborated with the need for perfection, need going sometimes in extremis beyond common sense. The simplest theory attempting to explain at least some of these situations is that people tend to confuse excellence with perfection, from this confusion deriving false beliefs, false expectations and unhealthy behavior. 

Beyond the fact that each individual has an illusory image of what perfection is about, perfection is in certain situations a limiting force rooted in the idealistic way of looking at life. Primarily, perfection denotes that we will never be good enough to reach it as we are striving to something that doesn’t exist. From this appears the external and internal criticism, criticism that instead of helping us to build something it drains out our energy to the extent that it destroys all we have built over the years with a considerable effort. Secondarily, on the long run, perfection has the tendency to steal our inner peace and balance, letting fear take over – the fear of not making mistakes, of losing the acceptance and trust of the others. It focuses on our faults, errors and failures instead of driving us to our goals. In extremis it relieves the worst in people, actors and spectators altogether. 

In its proximate semantics though at diametral side through its implications, excellence focuses on our goals, on the aspiration of aiming higher without implying a limit to it. It’s a shift of attention from failure to possibilities, on what matters, on reaching our potential, on acknowledging the long way covered. It allows us building upon former successes and failures. Excellence is what we need to aim at in personal and professional life. Will Durant explaining Aristotle said that: “We are what we repeatedly do. Excellence, then, is not an act, but a habit.” 

People who attempt giving 100% of their best to achieve a (positive) goal are to admire, however the proximity of 100% is only occasionally achievable, hopefully when needed the most. 100% is another illusory limit we force upon ourselves as it’s correlated to the degree of achievement, completeness or quality an artefact or result can ideally have. We rightly define quality as the degree to which something is fit for purpose. Again, a moving target that needs to be made explicit before we attempt to reach it otherwise quality envisions perfection rather than excellence and effort is wasted. 

Considering the volume of effort needed to achieve a goal, Pareto’s principles (aka the 80/20 rule) seems to explain the best its underlying forces. The rule states that roughly 80% of the effects come from 20% of the causes. A corollary is that we can achieve 80% of a goal with 20% of the effort needed altogether to achieve it fully. This means that to achieve the remaining 20% toward the goal we need to put four times more of the effort already spent. This rule seems to govern the elaboration of concepts, designs and other types of documents, and I suppose it can be easily extended to other activities like writing code, cleaning data, improving performance, etc. 

Given the complexity, urgency and dependencies of the tasks or goals before us probably it's beneficial sometimes to focus first on the 80% of their extent, so we can make progress, and focus on the remaining 20% if needed, when needed. This concurrent approach can allow us making progress faster in incremental steps. Also, in time, through excellence, we can bridge the gap between the two numbers as is needed less time and effort in the process.

🌁Software Engineering: Proof-of-Concept [PoC] (Definitions)

"A software trial that allows a prospect to test the product before buying it, while at the same time delivering a realistic slice of functionality." (Jill Dyché & Evan Levy, "Customer Data Integration: Reaching a Single Version of the Truth", 2006)

"Originally a term referring to a prototype proving a technical solution’s feasibility. Recently, the term has become synonymous with early-stage revenue-proving market feasibility. For example, without proof of concept revenue for early adopters, the inventor was having difficulty proving anyone needed what he had built." (Brad Feld & Sean Wise, "Startup Opportunities" 2nd Ed., 2017)

"A minimal implementation or execution of a process that serves as a sample sufficient to prove the success of the whole implementation or process." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"Is a realization of a certain method or idea in order to demonstrate its feasibility or a demonstration in principle with the aim of verifying that some concept or theory has practical potential. PoC represents a stage during the development of a product when it is established that the product will function as intended." (Soraya Sedkaoui & SRY Consulting, "Big Data Applications in Business", 2019)

"A demonstration of the feasibility of a particular method. PoC is used to test theoretical calculations and hypotheses in practice." (Kaspersky)

"A proof of concept (POC) is a demonstration of a product, service or solution in a sales context. A POC should demonstrate that the product or concept will fulfill customer requirements while also providing a compelling business case for adoption." (Gartner)

"A proof of concept (POC) is a demonstration to verify that certain concepts or theories have the potential for real-world application. In a nutshell, a POC represents the evidence demonstrating that a project or product is feasible and worthy enough to justify the expenses needed to support and develop it." (Techopedia) [source]

"A proof of concept (POC) is an exercise in which work is focused on determining whether an idea can be turned into a reality. A proof of concept is meant to determine the feasibility of the idea or to verify that the idea will function as envisioned." (Search CIO) [source]

"A proof of concept (POC) or a proof of principle is a realization of a certain method or idea to demonstrate its feasibility, or a demonstration in principle, whose purpose is to verify that some concept or theory has the potential of being used. A proof of concept is usually small and may or may not be complete." (SensiML)