Business Intelligence: A Software Engineer's Perspective I (Houston, we have a Problem!)

Business Intelligence Series

One of the critics addressed to the BI/Data Analytics, Data Engineering and even Data Science fields is their resistance to applying Software Engineering (SE) methods in practice. SE can be regarded as the application of sound methods, methodologies, techniques, principles, and practices to obtain high quality economic software in a reproducible manner. At minimum, should be applied SE techniques and practices proven to work, for example the use of best practices, reference technologies, standardized processes for requirements gathering and management, etc. This doesn't mean that one should apply the full extent of SE but consider a minimum that makes sense to adopt.

Unfortunately, the creation of data artifacts (queries, reports, data models, data pipelines, data visualizations, etc.) as process seem to be done after the principle of least action, though least action means here the minimum interaction to push pieces on a board rather than getting the things done. At high level, the process is as follows: get the requirements, build something, present results, get more requirements, do changes, present the results, and the process is repeated ad infinitum.

Given that data artifact's creation finds itself at the intersection of two or more knowledge areas in which knowledge is exchanged in several iterations between the parties involved until a common ground is achieved, this process is totally inefficient from multiple perspectives. First of all, it takes considerably more time than planned to reach a solution, resources being wasted in the process, multiple forms of waste being involved. Secondly, the exchange and retention of knowledge resulting from the process is minimal, mainly on a need by basis. This might look as an efficient approach on the short term, but is inefficient overall.

BI reflects the general issues from SE - most of the issues can be traced back to requirements - if the requirements are incorrect and there's no magic involved in between, then one can't expect for the solution to be correct. The bigger the difference between the initial and final requirements elicited in the process, the more resources are wasted. The more time passes between the start of the development phase and the time a solution is presented to the customer, the longer it takes to build the final solution. Same impact have the time it takes to establish a common ground and other critical factors for success involved in the process.

One can address these issues through better requirements elicitation, rapid prototyping, the use of agile methodologies and similar approaches, though the general feeling is that even if they bring improvements, they don't address the root causes - lack of data literacy skills, lack of knowledge about the business, lack of maturity in planning and executing tasks, the inexistence of well-designed processes and procedures, respectively the lack of an engineering mindset.

These inefficiencies have low impact when building a report occasionally, though they accumulate and tend to create systemic issues in what concerns the overall BI effort. They are addressed locally by experts and in general through a strategic approach like the elaboration of a BI strategy, though organizations seldom pay attention to them. Some organizations consider that they are automatically addressed as part of the data culture, though data culture focuses in general on data literacy and not on the whole set of assumptions mentioned above.

An experienced data professional sees more likely the inefficiencies, tries to address them locally in his interactions with the various stakeholders, he/she can build a business case for addressing them, though it depends on organizations to recognize that they have a problem, respective address the inefficiencies in a strategic and systemic manner!

Previous Post <<||>> Next Post

Process Management: Workflow (Definitions)

"Similar to a business process, a description of the activities or tasks that have to be done to fulfill a certain business need." (Nicolai M Josuttis, "SOA in Practice", 2007)

"A series of granular steps that are put together in proper sequence to execute some bit of logic." (Tony Fisher, "The Data Asset", 2009)

"A set of components and relations between them, used to define a complex process from simple building blocks. Relations may be in the form of data links which allow the output of one component to be used as the input of another, or control links which state some conditions on the execution of a component." (Mark Olive, "SHARE: A European Healthgrid Roadmap", 2009)

"The sequence of steps needed to carry out a business process." (Judith Hurwitz et al, "Service Oriented Architecture For Dummies" 2nd Ed., 2009)

"An ordered series of steps that accomplish some defined purpose according to a set of rules." (Bruce Bukovics, "Pro WF: Windows Workflow in .NET 4", 2010)

"Similar to a business process; a description of the activities or tasks that have to be done to fulfill a certain business need. Some people differentiate between workflows and business processes by stating that business processes describe more generally what has to be done, whereas workflows describe how activities or tasks should be carried out." (David Lyle & John G Schmidt, "Lean Integration", 2010)

"System process to manage the routing and approval of documents and transactions across multiple people and/or departments within an organization. Also, automating a business approval process that will notify the appropriate resources when activities/approvals need to be performed." (Janice M Roehl-Anderson, "IT Best Practices for Financial Managers", 2010)

"A predefined sequence of activities that complete a process." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"Defines how people and tasks interact to create, update, manage, and deliver content. Workflow helps organizations perform tasks in an efficient and repeatable manner." (Charles Cooper & Ann Rockley, "Managing Enterprise Content: A Unified Content Strategy" 2nd Ed., 2012)

"This is a sequence of task-oriented steps needed to carry out a business process." (Marcia Kaufman et al, "Big Data For Dummies", 2013)

"The specification of actions, actors, sequencing of actions, and completion criteria that, taken together, accomplish a larger task." (O Sami Saydjari, "Engineering Trustworthy Systems: Get Cybersecurity Design Right the First Time", 2018)

Software Engineering: Requirements Specification (Just the Quotes)

"A clean design is more easily modified as requirements change or as more is learned about what parts of the code consume significant amounts of execution time. A 'clever' design that fails to work or to run fast enough can often be salvaged only at great cost. Efficiency does not have to be sacrificed in the interest of writing readable code - rather, writing readable code is often the only way to ensure efficient programs that are also easy to maintain and modify." (Brian W Kernighan & Phillip J Plauger, "The Elements of Programming Style", 1974)

"A good top-down design avoids bugs in several ways. First, the clarity of structure and representation makes the precise statement of requirements and functions of the modules easier. Second, the partitioning and independence of modules avoids system bugs. Third, the suppression of detail makes flaws in the structure more apparent. Fourth, the design can be tested at each of its refinement steps, so testing can start earlier and focus on the proper level of detail at each step." (Fred P Brooks, "The Mythical Man-Month: Essays", 1975)

"Far be it from me to suggest that all changes in customer objectives and requirements must, can, or should be incorporated in the design. Clearly a threshold has to be established, and it must get higher and higher as development proceeds, or no product ever appears." (Fred P Brooks, "The Mythical Man-Month: Essays", 1975)

"The beginning of wisdom for a programmer is to recognize the difference between getting his program to work and getting it right. A program which does not work is undoubtedly wrong; but a program which does work is not necessarily right. It may still be wrong because it is hard to understand; or because it is hard to maintain as the problem requirements change; or because its structure is different from the structure of the problem; or because we cannot be sure that it does indeed work." (Michael A Jackson, "Principles of Program Design", 1975)

"The hardest single part of building a software system is deciding precisely what to build." (Frederick P. Brooks, "The Mythical Man-Month", 1975) 

"[Enterprise Architecture is] the set of descriptive representations (i. e., models) that are relevant for describing an Enterprise such that it can be produced to management's requirements (quality) and maintained over the period of its useful life." (John Zachman, 1987)

"The hardest part of the software task is arriving at a complete and consistent specification, and much of the essence of building a program is in fact the debugging of the specification." (Frederick P Brooks, "No Silver Bullet", 1987)

"In programming, it’s often the buts in the specification that kill you." (Boris Beizer, "Software Testing Techniques", 1990)

"Object-oriented analysis is a method of analysis that examines requirements from the perspective of the classes and objects found in the vocabulary of the problem domain."(Grady Booch, "Object-oriented design: With Applications", 1991)

"The source code is often the only accurate description of the software. On many projects, the only documentation available to programmers is the code itself. Requirements specifications and design documents can go out of date, but the source code is always up to date. Consequently, it's imperative that the code be of the highest possible quality." (Steve C McConnell," Code Complete: A Practical Handbook of Software Construction", 1993) 

"Users can work with analysts and object designers to formulate and tune system requirements. People from business, analytical and object design disciplines can come together, learn from each other and generate meaningful descriptions of systems that are to be built. Each participant and each project has slightly different concerns and needs. Practical application of use cases can go a long way to improve our ability to deliver just what the customer ordered. (Rebecca Wirfs-Brock, "Designing scenarios: Making the case for a use case framework", 1993)

"Our experience with designing and analyzing large and complex software-intensive systems has led us to recognize the role of business and organization in the design of the system and in its ultimate success or failure. Systems are built to satisfy an organization's requirements (or assumed requirements in the case of shrink-wrapped products). These requirements dictate the system's performance, availability, security, compatibility with other systems, and the ability to accommodate change over its lifetime. The desire to satisfy these goals with software that has the requisite properties influences the design choices made by a software architect." (Len Bass et al, "Software Architecture in Practice", 1998)

"Agile development methodologies promise higher customer satisfaction, lower defect rates, faster development times and a solution to rapidly changing requirements. Plan-driven approaches promise predictability, stability, and high assurance. However, both approaches have shortcomings that, if left unaddressed, can lead to project failure. The challenge is to balance the two approaches to take advantage of their strengths and compensate for their weaknesses." (Barry Boehm & Richard Turner, "Observations on balancing discipline and agility", Agile Development Conference, 2003)

"The aim of architectural design is to prepare overall specifications, derived from the needs and desires of the user, for subsequent design and construction stages. The first task for the architect in each design project is thus to determine what the real needs and desires of the user are […]" (George J Klir & Doug Elias, "Architecture of Systems Problem Solving" 2nd Ed, 2003)

"The software architecture of a system or a family of systems has one of the most significant impacts on the quality of an organization's enterprise architecture. While the design of software systems concentrates on satisfying the functional requirements for a system, the design of the software architecture for systems concentrates on the nonfunctional or quality requirements for systems. These quality requirements are concerns at the enterprise level. The better an organization specifies and characterizes the software architecture for its systems, the better it can characterize and manage its enterprise architecture. By explicitly defining the systems software architectures, an organization will be better able to reflect the priorities and trade-offs that are important to the organization in the software that it builds." (James McGovern et al, "A Practical Guide to Enterprise Architecture", 2004)

"Enterprise architecture is the organizing logic for business processes and IT infrastructure reflecting the integration and standardization requirements of a company's operation model. […] The key to effective enterprise architecture is to identify the processes, data, technology, and customer interfaces that take the operating model from vision to reality." (Jeanne W Ross et al, "Enterprise architecture as strategy: creating a foundation for business", 2006)

"The role of conceptual modelling in information systems development during all these decades is seen as an approach for capturing fuzzy, ill-defined, informal 'real-world' descriptions and user requirements, and then transforming them to formal, in some sense complete, and consistent conceptual specifications." (Janis A Burbenko jr., "From Information Algebra to Enterprise Modelling and Ontologies", Conceptual Modelling in Information Systems Engineering, 2007)

"Writing the spec, a document that lays out copiously detailed instructions for the programmer, is a necessary step in any software building enterprise where the ultimate user of the product is not the same person as the programmer. The spec translates requirements - the set of goals or desires the software developer’s customers lay out - into detailed marching orders for the programmer to follow." (Scott Rosenberg, "Dreaming in Code", 2007)

"If the discipline of requirements specification has taught us anything, it is that well-specified requirements are as formal as code and can act as executable tests of that code!"  (Robert C Martin, "Clean Code: A Handbook of Agile Software Craftsmanship", 2008)

"Nothing has a more profound and long-term degrading effect upon a development project than bad code. Bad schedules can be redone, bad requirements can be redefined. Bad team dynamics can be repaired. But bad code rots and ferments, becoming an inexorable weight that drags the team down." (Robert C Martin, "Clean Code: A Handbook of Agile Software Craftsmanship", 2008)

"A good system design is based on a sound conceptual model (architecture). A system design that has no conceptual structure and little logic to its organization is ultimately going to be unsuccessful. Good architecture will address all the requirements of the system at the right level of abstraction." (Vasudeva Varma, "Software Architecture: A Case Based Approach", 2009)

"Design is the bridging activity between gathering and implementation of software requirements that satisfies the required needs. […] The fundamental goal of design is to reduce the number of dependencies between modules, thus reducing the complexity of the system. This is also known as coupling; lesser the coupling the better is the design. On the other hand, higher the binding between elements within a module (known as cohesion) the better is the design." (Vasudeva Varma, "Software Architecture: A Case Based Approach", 2009)

"Taking a systems approach means paying close attention to results, the reasons we build a system. Architecture must be grounded in the client’s/user’s/customer’s purpose. Architecture is not just about the structure of components. One of the essential distinguishing features of architectural design versus other sorts of engineering design is the degree to which architectural design embraces results from the perspective of the client/user/customer. The architect does not assume some particular problem formulation, as 'requirements' is fixed. The architect engages in joint exploration, ideally directly with the client/user/customer, of what system attributes will yield results worth paying for."  (Mark W Maier, "The Art Systems of Architecting" 3rd Ed., 2009)

"Agile approaches to software development consider design and implementation to be the central activities in the software process. They incorporate other activities, such as requirements elicitation and testing, into design and implementation. By contrast, a plan-driven approach to software engineering identifies separate stages in the software process with outputs associated with each stage." (Ian Sommerville, "Software Engineering" 9th Ed., 2011)

"Agile methods universally rely on an incremental approach to software specification, development, and delivery. They are best suited to application development where the system requirements usually change rapidly during the development process. They are intended to deliver working software quickly to customers, who can then propose new and changed requirements to be included in later iterations of the system. They aim to cut down on process bureaucracy by avoiding work that has dubious long-term value and eliminating documentation that will probably never be used." (Ian Sommerville, "Software Engineering" 9th Ed., 2011)

"Models are used during the requirements engineering process to help derive the requirements for a system, during the design process to describe the system to engineers implementing the system and after implementation to document the system’s structure and operation." (Ian Sommerville, "Software Engineering" 9th Ed., 2011)

"Software systems do not exist in isolation. They are used in a social and organizational context and software system requirements may be derived or constrained by that context. Satisfying these social and organizational requirements is often critical for the success of the system. One reason why many software systems are delivered but never used is that their requirements do not take proper account of how the social and organizational context affects the practical operation of the system."(Ian Sommerville, "Software Engineering" 9th Ed., 2011)

"Creating mockups to communicate is not intrinsically a bad idea. But, as we are subject to confirmation bias, there’s always a risk that we will stop at our first design attempt and become reluctant to ask if there are better ways to achieve the same goals. Making these first ideas very detailed; putting them into a document; and especially blessing that document with the label 'requirements' are all moves which make further revision less likely, and put us more at risk from confirmation bias." (Laurent Bossavit, "The Leprechauns of Software Engineering", 2015)

Software Quality Assurance: Black-Box Testing (Definitions)

"A specification-based test that looks at a system or unit exclusively from the outside, that is, over its public interface." (Johannes Link & Peter Fröhlich, "Unit Testing in Java", 2003)

"This test compares the externally observable behavior at the external software interfaces (without knowledge of their structure) with the desired behavior. Black-Box tests are frequently equated with »functional tests«, although they can of course also include non-functional tests." (Lars Dittmann et al, "Automotive SPICE in Practice", 2008)

"Repeatable procedure to derive and/or select test cases based on an analysis of the specification, either functional or nonfunctional, of a component or system without reference to its internal structure." (Tilo Linz et al, "Software Testing Foundations" 4th Ed., 2014)

"A software testing methodology that looks at available inputs for an application and the expected outputs from each input." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web" 2nd Ed., 2015)

[Data coverage (black-box) testing:] "Testing a program or subprogram based on the possible input values, treating the code as a black box" (Nell Dale & John Lewis, "Computer Science Illuminated" 6th Ed., 2015)

[black-box test design technique:] "Procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure." (Software Quality Assurance)

"Testing, either functional or non-functional, without reference to the internal structure of the component or system." (Software Quality Assurance)

Software Engineering: Specification (Definitions)

"A document used in procurement or development that describes the technical requirements for products, components, or services, usually including the procedures to be used to determine that the requirements have been met. Large systems may have multiple levels of specifications. Some specifications are common to multiple systems or products (e.g., interface definitions)." (Richard D Stutzke, "Estimating Software-Intensive Systems: Projects, Products, and Processes", 2005)

"A specific quantifiable set point that typically has a nominal or target value and a tolerance of acceptable values associated with it. This is what results when a team tries to use tools and best practices to fulfill a requirement. Requirements are hopefully completely fulfilled by a final design specification - but many times they are not." (Lynne Hambleton, "Treasure Chest of Six Sigma Growth Methods, Tools, and Best Practices", 2007)

"A detailed coherent statement of particulars. A requirements specification details requirements whereas a design specification details the design of a system or element." (Bruce P Douglass, "Real-Time Agility: The Harmony/ESW Method for Real-Time and Embedded Systems Development", 2009)

"A document that specifies, ideally in a complete, precise, concrete and verifiable form, the requirements or other characteristics of a component or system. It serves the developers as a basis for programming, and it serves the testers as a basis for developing test cases with black box test design methods. (Often, a specification includes procedures for determining whether these requirements have been satisfied.) special software Software developed for one or a group of customers." (Tilo Linz et al, "Software Testing Foundations" 4th Ed, 2014)

"A precise statement of the needs to be satisfied and the essential characteristics that are required." (Project Management Institute, "A Guide to the Project Management Body of Knowledge (PMBOK® Guide )", 2017)

"A statement of the intended functionality of a given system." (O Sami Saydjari, "Engineering Trustworthy Systems: Get Cybersecurity Design Right the First Time", 2018)

"A document that specifies, ideally in a complete, precise and verifiable manner, the requirements, design, behavior, or other characteristics of a system or component, and, often, the procedures for determining whether these provisions have been satisfied" (IEEE 610)

Software Engineering: Protocol (Definitions)

"The language or rules and conventions that two computers use to pass messages across a network medium. Networking software generally implements multiple levels of protocols layered one on top of another." (Owen Williams, "MCSE TestPrep: SQL Server 6.5 Design and Implementation", 1998)

"A set of rules or standards designed to enable computers to connect with one another and exchange information." (Microsoft Corporation, "SQL Server 7.0 System Administration Training Kit", 1999)

"The way in which two computers transfer data between each other." (Greg Perry, "Sams Teach Yourself Beginning Programming in 24 Hours 2nd Ed.", 2001)

"A list of methods that a class must implement to conform or adopt the protocol. Protocols provide a way to standardize an interface across classes." (Stephen G Kochan, "Programming in Objective-C", 2003)

"A set of rules that govern a transaction." (Marcus Green & Bill Brogden, "Java 2™ Programmer Exam Cram™ 2 (Exam CX-310-035)", 2003)

"A set of semantic and syntactic rules that determines the behavior of functions in achieving communication." (Sharon Allen & Evan Terry, "Beginning Relational Data Modeling" 2nd Ed., 2005)

"A language and a set of rules that allow computers to interact in a well-defined way. Examples are FTP, HTTP, and NNTP." (Craig F Smith & H Peter Alesso, "Thinking on the Web: Berners-Lee, Gödel and Turing", 2008)

"A specification - often a standard - that describes how computers communicate with each other, for example, the TCP/IP suite of communication protocols or the OAI-PMH." (J P Getty Trust, "Introduction to Metadata" 2nd Ed., 2008)

"To communicate effectively, client applications and database servers need a commonly agreed-upon approach. A protocol is a communication standard adhered to by both parties that makes these conversations possible." (Robert D Schneider and Darril Gibson, "Microsoft SQL Server 2008 All-In-One Desk Reference For Dummies", 2008)

"A set of rules that computers use to establish and maintain communication amongst themselves." (Judith Hurwitz et al, "Service Oriented Architecture For Dummies" 2nd Ed., 2009)

"the forms and ceremony used to manage the interaction of elements." (Bruce P Douglass, "Real-Time Agility: The Harmony/ESW Method for Real-Time and Embedded Systems Development", 2009)

"The rules governing the syntax, semantics, and synchronization of communication." (David Lyle & John G Schmidt, "Lean Integration", 2010)

"A list of methods that a class must implement to conform to or adopt the protocol. Protocols provide a way to standardize an interface across classes. See also formal protocol and informal protocol." (Stephen G Kochan, "Programming in Objective-C" 4th Ed., 2011)

"A set of conventions that govern the communications between processes. Protocol specifies the format and content of messages to be exchanged." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"The standard or set of rules that govern how devices on a network exchange and how they need to function in order to 'talk' to each other." (Linda Volonino & Efraim Turban, "Information Technology for Management" 8th Ed., 2011)

"A standard set of formats and procedures that enable computers to exchange information." (Microsoft, "SQL Server 2012 Glossary", 2012)

"In networking, an agreed-upon way of sending messages back and forth so that neither correspondent will get too confused." (Jon Orwant et al, "Programming Perl" 4th Ed., 2012)

"A set of guidelines defining network traffic formats for the easy communication of data between two hosts." (Mark Rhodes-Ousley, "Information Security: The Complete Reference, Second Edition, 2nd Ed.", 2013)

"A set of instructions, policies, or fully described procedures for accomplishing a service, operation, or task." (Jules H Berman, "Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information", 2013)

"A set of rules controlling the communication and transfer of data between two or more devices or systems in a communication network." (IBM, "Informix Servers 12.1", 2014)

"A rule or custom that governs how something is done. In a computer context, it refers to a standard for transferring data." (Faithe Wempen, "Computing Fundamentals: Introduction to Computers", 2015)

"A set of rules that defines how data is formatted and processed on a network" (Nell Dale & John Lewis, "Computer Science Illuminated" 6th Ed., 2015)

"Defined policies or standards that users adhere to. Protocols are well-defined and accepted procedures. In computer networking, the term refers to algorithms for exchanging various types of data and their interpretation at origination and destination." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web" 2nd Ed., 2015)