Business Intelligence: A One Man Show II (In the Cusps of Complexity)

Business Intelligence Series

I watched today on YouTube Power BI Tips' "One Person to Do Everything" episode I missed last week. The main topic is based on Christopher Laubenthal's article "Why one person can't do everything in the data space". Author's arguments are based on an analogy between the various data areas and a college's functional structure. Reading the article, I must say that it takes a poorly chosen analogy to mess messy things more!

One of the most confusing things is that there are so many data-related context-dependent roles with considerable overlapping, that it becomes more and more difficult to understand what they cover. The author considers the roles of Data Architect, Data Engineer, Database Administrator (DBA), Data Analyst, Information Designer and Data Scientist. However, to the every aspect of a data architecture there are also developers on the database (backend) and reporting side (front-end). Conversely, there are other data professionals on the management side for the various knowledge areas of Data Management: Data Governance, Data Strategy, Data Security, Data Operations, etc. There are also roles at the border between the business and the technical side like Data Stewards, Business Analysts, Data Citizen, etc. 

There are two main aspects here. According to the historical perspective, many of these roles appeared when a new set of requirements or a new layer appeared in the architecture. Firstly, it was maybe the DBA, who was supposed to primarily administer the database. Being a keeper of the data and having some knowledge of the data entities, it was easy for him/her to export data for the various reporting needs. In time such activities were taken over by a second category of data professionals. Then the data were moved to Decision Support Systems and later to Data Warehouses and Data Lakes/Lakehoses, this evolution requiring other professionals to address the challenges of each layer. Every activity performed on the data requires a certain type of knowledge that can result in the end in a new denomination. 

The second perspective results from the management of data and the knowledge areas associated with it. If in small organizations with one or two systems in place one doesn't need to talk about Data Operations, in big organizations, where a data center or something similar is maybe in place, Data Operations can easily become a topic on its own, a management structure needing to be in place for its "effective and efficient" management. And the same can happen in the other knowledge areas and their interaction with the business. It's an inherent tendency of answering to complexity with complexity, which on the long term can be in the detriment of any business. In extremis, organizations tend to have a whole team in each area, which can further increase the overall complexity by a small to not that small magnitude. 

Fortunately, one of the benefits of technological advancement is that much of the complexity can be moved somewhere else, and these are the areas where the cloud brings the most advantages. Parts or all architecture can be deployed into the cloud, being managed by cloud providers and third-parties on an on-demand basis at stable costs. Moreover, with the increasing maturity and integration of the various layers, the impact of the various roles in the overall picture is reduced considerably as areas like governance, security or operations are built-in as services, requiring thus less resources. 

With Microsoft Fabric, all the data needed for reporting becomes in theory easily available in the OneLake. Unfortunately, there is another type of complexity that is dumped on other professionals' shoulders and these aspects need to be furthered considered. 

Previous Post <<|||>> Next Post

Resources:
[1] Christopher Laubenthal (2024) "Why One Person Can’t Do Everything In Data" (link)
[2] Power BI tips (2024) Ep.292: One Person to Do Everything (link)

Business Intelligence: New Technologies, Old Challenges II (ETL vs. ELT)

 

Data lakes and similar cloud-based repositories drove the requirement of loading the raw data before performing any transformations on the data. At least that’s the approach the new wave of ELT (Extract, Load, Transform) technologies use to handle analytical and data integration workloads, which is probably recommendable for the mentioned cloud-based contexts. However, ELT technologies are especially relevant when is needed to handle data with high velocity, variance, validity or different value of truth (aka big data). This because they allow processing the workloads over architectures that can be scaled with workloads’ demands.

This is probably the most important aspect, even if there can be further advantages, like using built-in connectors to a wide range of sources or implementing complex data flow controls. The ETL (Extract, Transform, Load) tools have the same capabilities, maybe reduced to certain data sources, though their newer versions seem to bridge the gap.

One of the most stressed advantages of ELT is the possibility of having all the (business) data in the repository, though these are not technological advantages. The same can be obtained via ETL tools, even if this might involve upon case a bigger effort, effort depending on the functionality existing in each tool. It’s true that ETL solutions have a narrower scope by loading a subset of the available data, or that transformations are made before loading the data, though this depends on the scope considered while building the data warehouse or data mart, respectively the design of ETL packages, and both are a matter of choice, choices that can be traced back to business requirements or technical best practices.

Some of the advantages seen are context-dependent – the context in which the technologies are put, respectively the problems are solved. It is often imputed to ETL solutions that the available data are already prepared (aggregated, converted) and new requirements will drive additional effort. On the other side, in ELT-based solutions all the data are made available and eventually further transformed, but also here the level of transformations made depends on specific requirements. Independently of the approach used, the data are still available if needed, respectively involve certain effort for further processing.

Building usable and reliable data models is dependent on good design, and in the design process reside the most important challenges. In theory, some think that in ETL scenarios the design is done beforehand though that’s not necessarily true. One can pull the raw data from the source and build the data models in the target repositories.

Data conversion and cleaning is needed under both approaches. In some scenarios is ideal to do this upfront, minimizing the effect these processes have on data’s usage, while in other scenarios it’s helpful to address them later in the process, with the risk that each project will address them differently. This can become an issue and should be ideally addressed by design (e.g. by building an intermediate layer) or at least organizationally (e.g. enforcing best practices).

Advancing that ELT is better just because the data are true (being in raw form) can be taken only as a marketing slogan. The degree of truth data has depends on the way data reflects business’ processes and the way data are maintained, while their quality is judged entirely on their intended use. Even if raw data allow more flexibility in handling the various requests, the challenges involved in processing can be neglected only under the consequences that follow from this.

Looking at the analytics and data integration cloud-based technologies, they seem to allow both approaches, thus building optimal solutions relying on professionals’ wisdom of making appropriate choices.

Previous Post <<||>>Next Post

Microsoft Azure: Azure Data Factory (Notes)

Azure Data Factory (ADF)
• {definition} pay-per-use serverless cloud-based data integration service that orchestrates and automates the movement and transformation of both cloud-based and on-premises data sources [1]
  • ⇐ a hybrid and scalable data integration service for Big Data and advanced end-to-end analytics solutions [11]
  • ⇐ Microsoft Azure PaaS offering for ETL/ELT workloads found at its second generation [11]
  • allows creating data-driven flows to orchestrate movement of data between supported data stores and processing of data using compute services in other regions or in an on-premises environment
• {benefit} easy-to-use
  • {feature} allows creating code-free pipelines with drag-and-drop functionality [2]
  • {feature} uses JSON to describe each of its entities
• {benefit} cost-effective
  • pay-as-you-go model against the Azure subscription with no up-front costs
  • low price-to-performance ratio
    • ⇐ cost effective and performant at the same time
  • fully managed serverless cloud service that scales on demand [2]
    • ⇒requires zero hardware maintenance [1]
    • ⇒can easily scale beyond what was originally anticipated [1]
  • does not store any data [1]
  • provides additional cost-saving functionality [11]
    • {feature} it takes care of the provisioning and teardown of the cluster once the job has executed [11]
• {benefit} powerful
  • allows ingesting on-premise and cloud-based data sources
  • high-performance hybrid connectivity
    • over 90 built-in connectors make it easy to interact with all kinds of technologies [11]
  • orchestrate at scale
    • on-demand compute
    • Big Data workloads are scaled over multiple nodes to chunk data in parallel [11]
  • {feature} [ADFv2] monitoring
    • richer and natively integrating it with Azure Monitor and OMS [11]
      • includes feature-rich monitoring and management tools to visualize the current state of data pipelines, data lineage and pipeline dependencies [1]
  • {feature} [ADFv2] control flow functionality
    • lets define complex workflows using programmatic or UI mechanisms
      • allows defining parameters at pipeline level [11]
      • includes custom state passing and looping containers [11]
      • pipelines can be authored via additional tools
        • e.g. PowerShell, .NET, Python, REST APIs
        • ⇒ helps ISVs build SaaS-based analytics solutions on top of ADF app models
• {benefit} intelligent
  • autonomous ETL allows unlocking operational efficiencies and enable citizen integrators [2]
• {benefit} enterprise-grade security:
  • provides same security standards as any other Microsoft service [11]
• {benefit} monthly release cycle
  • {feature} via auto-update
  • improvements may include support for new connectors, bug fixes, security patches, and performance improvements [11]
• {benefit} backwards compatibility
  • {feature} [ADFv2] allows rehosting SSIS solutions [2]
    • ⇒ helpful for modernizing data warehouse solutions
• {prerequisite} an Azure subscription with the contributor role assigned to at least one resource group
• {limitation} availability
  • the service isn’t available in all regions
    • an instance can be made available in other region to trigger the job on customer’s computer environment [1]
      • ⇐ the time for executing the job on the compute environment doesn’t change [1]
• {concept} activity
  • the unit of orchestration in ADF [1]
  • defines the actions to perform on data [1]
  • takes zero or more datasets as inputs and produces one or more datasets as outputs [1]
  • activity types
    • data movement activities
    • data transformation activities
    • control activities
      • control how the pipeline works and interacts with the data [10]
      • allow executing pipelines [10]
      • allow running a foreach statement or Lookup activities [10]
• {concept] pipeline
  • logical grouping of activities that together perform a task [1]
    • the sequence can have a complex schedule and dependencies that need to be orchestrated and automated [1]
    • two activities can be chained by setting the output data set of one activity as the input dataset of the other activity
  • allows building ETL/ELT workloads
  • scheduled by scheduler triggers [10]
  • data in a pipeline is referred to by different names
    • ⇐ based on the amount of modification that has been performed
    • raw data
      • data with no processing applied [10]
        • ⇒does not yet have a schema applied
      • stored in the message encoding format used to send tracking events such as JSON.
      • can be organized into meaningful data stores and data lakes [10]
        • ⇐ further used in decision-making
      • it's common to send all tracking events as raw events
        • ⇐ because all events can be sent to a single endpoint and schemas can be applied later in the pipeline [10]
    • processed data
      • raw data that has been decoded in the event-specific formats with the schema applied
        • e.g. JSON tracking events that have been translated into a session start event with a fixed schema [10]
      • usually stored in different event tables and destination in a data pipeline [10]
    • cooked data
      • processed data that has been aggregated or summarized [10]
  • {concept} pipeline parameters
    • similar to SSIS package parameters
      • ⇐ need to be set from outside packages
    • can be passed from the parent pipeline
• {concept} dataset
  • named references/pointers to the data used as an input or an output of an activity [1]
  • identifies data structures within different (linked) data stores [1]
    • ⇐ before creating a dataset, a linked service must be created to link the data store to ADF [10]
    • once created, it can be used with activities in a pipeline [10]
      • e.g. a dataset can be an input or output dataset of a copy activity
• {concept} linked service
  • defines the information needed by ADF to connect to external resources at runtime
    • much like connection strings which define the connection information [10]
  • used to represent
    • {concept} data store
      • holds the input-output data to the ADF
      • e.g. tables, files, folders, and documents
    • {concept} compute resource
      • can host the execution of an activity [1]
• {concept} scheduler triggers
  • allow pipelines to be triggered on a wall-clock schedule [10]
    • pipelines and triggers have an n-m relationship
      • multiple triggers can kick off a single pipeline
      • the same trigger can kick off multiple pipelines
    • manual triggers trigger pipelines on demand [10]
  • once defined, it must be started to begin triggering the pipeline [10]
  • comes into effect only after publishing the solution to ADF [10]
    • ⇐ not when saving the trigger in the UI [10]
  • to run a pipeline, a pipeline reference must be included in trigger definition [10]
  • there is a cost associated with each pipeline run
    • {recommendation} when testing, make sure that the pipeline is triggered only a couple of times [10]
    • {recommendation} ensure that there is enough time for the pipeline to run between the published time and the end time [10]

Acronyms:

Azure Data Factory (ADF)

Continuous Integration/Continuous Deployment (CI/CD)

Extract Load Transform (ELT)

Extract Transform Load (ETL)

Independent Software Vendors (ISVs)

Operations Management Suite (OMS)

pay-as-you-go (PAYG)

SQL Server Integration Services (SSIS)

Resources:

[1] Microsoft (2020) "Microsoft Business Intelligence and Information Management: Design Guidance", by Rod College

[2] Microsoft (2021) Azure Data Factory [source]

[3] Microsoft (2018) Azure Data Factory: Data Integration in the Cloud [source]

[4] Microsoft (2021) Integrate data with Azure Data Factory or Azure Synapse Pipeline [source]

[10] Coursera (2021) Data Processing with Azure [source]

[11] Sudhir Rawat & Abhishek Narain (2019) "Understanding Azure Data Factory: Operationalizing Big Data and Advanced Analytics Solutions"

IT: Platform as a Service (PaaS)

"PaaS is defined as a computing platform delivered as a service." (Martin Oberhofer et al, "The Art of Enterprise Information Architecture", 2010)

"Delivery of an application development platform (hardware and software) from a third party via the Internet without having to buy and manage these resources." (Bill Holtsnider & Brian D Jaffe, "IT Manager's Handbook" 3rd Ed., 2012)

"A cloud service that abstracts the computing services, including the operating software and the development and deployment and management life cycle. It sits on top of Infrastructure as a Service." (Marcia Kaufman et al, "Big Data For Dummies", 2013)

"A cloud service that abstracts the computing services, including the operating software and the development, deployment, and management life cycle. It sits on top of Infrastructure as a Service (IaaS)." (Judith S Hurwitz, "Cognitive Computing and Big Data Analytics", 2015)

"Delivery of a computing platform as a service." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web" 2nd Ed., 2015)

"The capability provided to the customer to deploy onto the cloud infrastructure customer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including network, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations." (James R Kalyvas & Michael R Overly, "Big Data: A Businessand Legal Guide", 2015)

"A cloud-based service that typically provides a platform on which software can be developed and deployed." (H James Harrington & William S Ruggles, "Project Management for Performance Improvement Teams", 2018)

"A complete application platform for multitenant cloud environments that includes development tools, runtime, and administration and management tools and services, PaaS combines an application platform with managed cloud infrastructure services." (Forrester)

"A services providing all the necessary infrastructure for cloud computing solutions." (Analytics Insight)

IT: Cloud (Definitions)

"A set of computers, typically maintained in a data center, that can be allocated dynamically and accessed remotely. Unlike a cluster, cloud computers are typically managed by a third party and may host multiple applications from different, unrelated users." (Michael McCool et al, "Structured Parallel Programming", 2012)

"A network that delivers requested virtual resources as a service." (IBM, "Informix Servers 12.1", 2014)

"A secure computing environment accessed via the Internet." (Faithe Wempen, "Computing Fundamentals: Introduction to Computers", 2015)

"Products and services managed by a third-party company and made available through the Internet." (David K Pham, "From Business Strategy to Information Technology Roadmap", 2016)

"It has the ability to offer and to assist any kind of useful information without any limitations for users." (Shigeki Sugiyama. "Human Behavior and Another Kind in Consciousness: Emerging Research and Opportunities", 2019)

"Remote server and distributed computing environment used to store data and provision computing related services as and when needed on a pay-as-you-go basis." (Wissam Abbass et al, "Internet of Things Application for Intelligent Cities: Security Risk Assessment Challenges", 2021)

"The virtual world in which information technology tools and services are available for hire, use and storage via the internet, Wi-Fi and physical attributes ranging from IT components to data storage." (Sue Milton, "Data Privacy vs. Data Security", 2021)

"uses a network of remote servers hosted on the internet to store, manage, and process data, rather than requiring a local server or a personal computer." (Accenture)

IT: Cloud Computing (Definitions)

"The service delivery of any IT resource as a networked resource." (David G Hill, "Data Protection: Governance, Risk Management, and Compliance", 2009)

"A technology where the data and the application are stored remotely and made available to the user over the Internet on demand." (Janice M Roehl-Anderson, "IT Best Practices for Financial Managers", 2010)

"A business model where programs, data storage, collaboration services, and other key business tools are stored on a centralized server that users access remotely, often through a browser." (Rod Stephens, "Start Here! Fundamentals of Microsoft .NET Programming", 2011)

"Technology that is rented or leased on a regular, or as-needed basis." (Linda Volonino & Efraim Turban, "Information Technology for Management" 8th Ed, 2011)

"Using programs and data stored on servers connected to computers via the Internet rather than storing software and data on individual computers." (Gina Abudi & Brandon Toropov, "The Complete Idiot's Guide to Best Practices for Small Business", 2011)

"The delivery of computing as a service. Cloud computing applications rely on a network (typically the Internet) to provide users with shared resources, software, and data." (Craig S Mullins, "Database Administration", 2012)

"Using Internet-based resources (e.g., applications, servers, etc.) as opposed to buying and installing in-house." (Bill Holtsnider & Brian D Jaffe, "IT Manager's Handbook, 3rd Ed", 2012)

"A business strategy where part or all of an organization’s information processing and storage is done by online service providers." (Kenneth A Shaw, "Integrated Management of Processes and Information", 2013)

"A computing model that makes IT resources such as servers, middleware, and applications available as services to business organizations in a self-service manner." (Marcia Kaufman et al, "Big Data For Dummies", 2013)

"Computing resources provided over the Internet using a combination of virtual machines (VMs), virtual storage, and virtual networks." (Mark Rhodes-Ousley, "Information Security: The Complete Reference, Second Edition, 2nd Ed.", 2013)

"A model for network access in which large, scalable resources are provided via the Internet as a shared service to requesting users. Access, computing, and storage services can be obtained by users without the need to understand or control the location and configuration of the system. Users consume resources as a service, and pay only for the resources that are used." (Jim Davis & Aiman Zeid, "Business Transformation: A Roadmap for Maximizing Organizational Insights", 2014)

"The delivery of software and other computer resources as a service over the Internet, rather than as a stand-alone product." (Manish Agrawal, "Information Security and IT Risk Management", 2014)

"The provision of computational resources on demand via a network. Cloud computing can be compared to the supply of electricity and gas or the provision of telephone, television, and postal services. All of these services are presented to users in a simple way that is easy to understand without users' needing to know how the services are provided. This simplified view is called an abstraction. Similarly, cloud computing offers computer application developers and users an abstract view of services, which simplifies and ignores much of the details and inner workings. A provider's offering of abstracted Internet services is often called the cloud." (Robert F Smallwood, "Information Governance: Concepts, Strategies, and Best Practices", 2014)

"A computational paradigm that aims at supporting large-scale, high-performance computing in distributed environments via innovative metaphors such as resource virtualization and de-location." (Alfredo Cuzzocrea & Mohamed M Gaber, "Data Science and Distributed Intelligence", 2015)

"A computing model that makes IT resources such as servers, middleware, and applications available as services to business organizations in a self-service manner." (Judith S Hurwitz, "Cognitive Computing and Big Data Analytics", 2015)

"A delivery model for information technology resources and services that uses the Internet to provide immediately scalable and rapidly provisioned resources as services using a subscription or utility-based fee structure." (James R Kalyvas & Michael R Overly, "Big Data: A Businessand Legal Guide", 2015)

"A service that provides storage space and other resources on the Internet" (Nell Dale & John Lewis, "Computer Science Illuminated, 6th Ed.", 2015)

"Delivering hosted services over the Internet, which includes providing infrastructures, platforms, and software as services." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web 2nd Ed.", 2015)

"The delivery of computer processing capabilities as a service rather than as a product, whereby shared resources, software, and information are provided to end users as a utility. Offerings are usually bundled as an infrastructure, platform, or software." (Adam Gordon, "Official (ISC)2 Guide to the CISSP CBK" 4th Ed., 2015)

"A general term for anything that involves delivering hosted services over the Internet. These services are broadly divided into: Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS), and Analytics-as-a-Service (AaaS)."  (Suren Behari, "Data Science and Big Data Analytics in Financial Services: A Case Study", 2016)

"A type of Internet-based technology in which different services (such as servers, storage, and applications) are delivered to an organization’s or an individual’s computers and devices through the Internet." (Jonathan Ferrar et al, "The Power of People: Learn How Successful Organizations Use Workforce Analytics To Improve Business Performance", 2017)

"A form of distributed computing whereby many computers and applications share the same resources to work together, often across geographically separated areas, to provide a coherent service." (O Sami Saydjari, "Engineering Trustworthy Systems: Get Cybersecurity Design Right the First Time", 2018)

"Cloud computing is a general term for the delivery of hosted services over the Internet. Cloud computing enables companies to consume compute resources as a utility - just like electricity - rather than having to build and maintain computing infrastructures in-house." (Thomas Ochs & Ute A Riemann, "IT Strategy Follows Digitalization", 2018)

"Cloud computing refers to the provision of computational resources on demand via a network. Cloud computing can be compared to the supply of a utility like electricity, water, or gas, or the provision of telephone or television services. All of these services are presented to the users in a simple way that is easy to understand without the users’ needing to know how the services are provided. This simplified view is called an abstraction. Similarly, cloud computing offers computer application developers and users an abstract view of services, which simplifies and ignores many of the details and inner workings. A provider’s offering of abstracted Internet services is often called The Cloud." (Robert F Smallwood, "Information Governance for Healthcare Professionals", 2018)

"The delivery of computing services and resources such as the servers, storage, databases, networking, software, and analytic through the internet." (Babangida Zubairu, "Security Risks of Biomedical Data Processing in Cloud Computing Environment", 2018)

"The use of shared remote computing devices for the purpose of providing improved efficiencies, performance, reliability, scalability, and security." (Shon Harris & Fernando Maymi, "CISSP All-in-One Exam Guide" 8th Ed., 2018)

"A computing model that makes information technology resources such as servers, middleware, and applications available over the internet as services to business organizations in a self-service manner." (K Hariharanath, "BIG Data: An Enabler in Developing Business Models in Cloud Computing Environments", 2019)

"Cloud computing refers to the practice of using a network of remote servers, hosted on the Internet to manage, store and process data instead of using a local server or a personal computer." (Jurij Urbančič et al, "Expansion of Technology Utilization Through Tourism 4.0 in Slovenia", 2020)

"A standardized technology delivery capability (services, software, or infrastructure) delivered via internet-standard technologies in a pay-per-use, self-service way." (Forrester)

"Cloud computing is a style of computing in which scalable and elastic IT-enabled capabilities are delivered as a service using internet technologies." (Gartner)

Data Science: MapReduce (Definitions)

"A data processing and aggregation paradigm consisting of a 'map' phase that selects data and a 'reduce' phase that transforms the data. In MongoDB, you can run arbitrary aggregations over data using map-reduce." (MongoDb, "Glossary", 2008)

"A divide-and-conquer strategy for processing large data sets in parallel. In the 'map' phase, the data sets are subdivided. The desired computation is performed on each subset. The 'reduce' phase combines the results of the subset calculations into a final result. MapReduce frameworks handle the details of managing the operations and the nodes they run on, including restarting operations that fail for some reason. The user of the framework only has to write the algorithms for mapping and reducing the data sets and computing with the subsets." (Dean Wampler & Alex Payne, "Programming Scala", 2009)

"A method by which computationally intensive problems can be processed on multiple computers in parallel. The method can be divided into a mapping step and a reducing step. In the mapping step, a master computer divides a problem into smaller problems that are distributed to other computers. In the reducing step, the master computer collects the output from the other computers. Although MapReduce is intended for Big Data resources, holding petabytes of data, most Big Data problems do not require MapReduce." (Jules H Berman, "Principles of Big Data: Preparing, Sharing, and Analyzing Complex Information", 2013)

"An early Big Data (before this term became popular) programming solution originally developed by Google for parallel processing using very large data sets distributed across a number of computing and storage systems. A Hadoop implementation of MapReduce is now available." (Kenneth A Shaw, "Integrated Management of Processes and Information", 2013)

"Designed by Google as a way of efficiently executing a set of functions against a large amount of data in batch mode. The 'map' component distributes the programming problem or tasks across a large number of systems and handles the placement of the tasks in a way that balances the load and manages recovery from failures. After the distributed computation is completed, another function called 'reduce' aggregates all the elements back together to provide a result." (Marcia Kaufman et al, "Big Data For Dummies", 2013)

"A programming model consisting of two logical steps - Map and Reduce - for processing massively parallelizable problems across extremely large datasets using a large cluster of commodity computers." (Haoliang Wang et al, "Accessing Big Data in the Cloud Using Mobile Devices", Handbook of Research on Cloud Infrastructures for Big Data Analytics, 2014)

"Algorithm that is used to split massive data sets among many commodity hardware pieces in an effort to reduce computing time." (Billie Anderson & J Michael Hardin, "Harnessing the Power of Big Data Analytics", Encyclopedia of Business Analytics and Optimization, 2014)

"MapReduce is a parallel programming model proposed by Google and is used to distribute computing on clusters of computers for processing large data sets." (Jyotsna T Wassan, "Emergence of NoSQL Platforms for Big Data Needs", Encyclopedia of Business Analytics and Optimization, 2014)

"A concept which is an abstraction of the primitives ‘map’ and ‘reduce’. Most of the computations are carried by applying a ‘map’ operation to each global record in order to generate key/value pairs and then apply the reduce operation in order to combine the derived data appropriately." (P S Shivalkar & B K Tripathy, "Rough Set Based Green Cloud Computing in Emerging Markets", Encyclopedia of Information Science and Technology 3rd Ed., 2015) 

"A programming model that uses a divide and conquer method to speed-up processing large datasets, with a special focus on semi-structured data." (Alfredo Cuzzocrea & Mohamed M Gaber, "Data Science and Distributed Intelligence", Encyclopedia of Information Science and Technology 3rd Ed., 2015) 

"MapReduce is a programming model for general-purpose parallelization of data-intensive processing. MapReduce divides the processing into two phases: a mapping phase, in which data is broken up into chunks that can be processed by separate threads - potentially running on separate machines; and a reduce phase, which combines the output from the mappers into the final result." (Guy Harrison, "Next Generation Databases: NoSQL, NewSQL, and Big Data", 2015)

"MapReduce is a technological framework for processing parallelize-able problems across huge data sets using a large number of computers (nodes). […] MapReduce consists of two major steps: 'Map' and 'Reduce'. They are similar to the original Fork and Join operations in distributed systems, but they can consider a large number of computers that can be constructed based on the Internet cloud. In the Map-step, the master computer (a node) first divides the input into smaller sub-problems and then distributes them to worker computers (worker nodes). A worker node may also be a sub-master node to distribute the sub-problem into even smaller problems that will form a multi-level structure of a task tree. The worker node can solve the sub-problem and report the results back to its upper level master node. In the Reduce-step, the master node will collect the results from the worker nodes and then combine the answers in an output (solution) of the original problem." (Li M Chen et al, "Mathematical Problems in Data Science: Theoretical and Practical Methods", 2015)

"A programming model which process massive amounts of unstructured data in parallel and distributed cluster of processors." (Fatma Mohamed et al, "Data Streams Processing Techniques Data Streams Processing Techniques", Handbook of Research on Machine Learning Innovations and Trends, 2017)

"A data processing framework of Hadoop which provides data intensive computation of large data sets by dividing tasks across several machines and finally combining the result." (Rupali Ahuja, "Hadoop Framework for Handling Big Data Needs", Handbook of Research on Big Data Storage and Visualization Techniques, 2018)

"A high-level programming model, which uses the “map” and “reduce” functions, for processing high volumes of data." (Carson K.-S. Leung, "Big Data Analysis and Mining", Encyclopedia of Information Science and Technology 4th Ed., 2018)

"Is a computational paradigm for processing massive datasets in parallel if the computation fits a three-step pattern: map, shard and reduce. The map process is a parallel one. Each process executes on a different part of data and produces (key, value) pairs. The shard process collects the generated pairs, sorts and partitions them. Each partition is assigned to a different reduce process which produces a single result." (Venkat Gudivada et al, "Database Systems for Big Data Storage and Retrieval", Handbook of Research on Big Data Storage and Visualization Techniques, 2018)

"Is a programming model or algorithm for the processing of data using a parallel programming implementation and was originally used for academic purposes associated with parallel programming techniques. (Soraya Sedkaoui, "Understanding Data Analytics Is Good but Knowing How to Use It Is Better!", Big Data Analytics for Entrepreneurial Success, 2019)

"MapReduce is a style of programming based on functional programming that was the basis of Hadoop." (Alex Thomas, "Natural Language Processing with Spark NLP", 2020)

"Is a specific programming model, which as such represents a new approach to solving the problem of processing large amounts of differently structured data. It consists of two functions - Map (sorting and filtering data) and Reduce (summarizing intermediate results), and it is executed in parallel and distributed." (Savo Stupar et al, "Importance of Applying Big Data Concept in Marketing Decision Making", Handbook of Research on Applied AI for International Business and Marketing Applications, 2021)

"A software framework for processing vast amounts of data." (Analytics Insight)

Market Review: What’s New in Microsoft World II

Microsoft Office - Cloud Computing is the Word

    Two weeks ago, on 15^th of June 2010, Microsoft Office was shipped together with Visio and Project 2010, closing the cycle of releases started with SQL Server 2008 R2, Visual Studio 2010, Sharepoint 2010 (all 3 shipped in April 2010) and Windows Azure (available also in April). The words that describe/unite at best these software tools is cloud computing and collaboration, why that? First we have to consider Azure, the new product from Windows’ portfolio, a framework for cloud computing and SaaS (Software as a Service) architectures, and composed of 3 components, namely Windows Azure which allows running applications and accessing data in the cloud, SQL Azure Database  provides data services in the cloud, while Windows Azure platform AppFabric allows the communication between the applications residing in the cloud. Also MS Office 2010 is part of Microsoft’s strategy toward cloud computing, the weight falling on SharePoint 2010, a business collaboration platform that together with the other MS Office tools allow to manage information, automate and manage business processes, facilitate decision making process, etc. A cornerstone of the framework is the co-authoring tool that “allows multiple people to work on a single copy of a document at the same time or at different times, seamlessly, whether they are online or offline”. As it seems are provided also “community features that allows users to share data as they do on Twitter and Facebook”, a step toward social computing. Microsoft plans to offer an online version of Office 2010, called Office Web Apps (OWA), supposed to be also a competitor for Google Docs.

    There are also people who question the steps done by Microsoft toward cloud computing, but in the end is important to establish the software infrastructure in which cloud computing-based applications could be developed, futures that don’t exist currently could appear in future versions or could be provided by third-party vendors.

    Microsoft comes also with some unpleasant surprises, as it seems Microsoft’s SharePoint Server runs only on 64-bit hardware and requires also a 64 bit SQL Server edition, and this could be quite an important constraint for many customers. The most unpleasant surprise is that Microsoft renounces to the well-known upgrade schema, the reason for that, as mentioned in Ars Technica quoting a Microsoft spokesman, from the need to simplify the product lineup and pricing, based on “partner and customer feedback” (I’m sorry but I can’t really buy that!). The same source expects that upgrades will be available with promotions, after Office’s launch. The only promotion I heard of is Microsoft Office 2010 Technology Guarantee program but if refers only to the customers who “purchased, installed, and activated a qualifying Microsoft Office 2007 product between March 5, 2010, and September 30, 2010”, they being eligible to download Office 2010 at no additional cost. How about the ones who bought a Microsoft Office 2007 copy in 2010 but before 5^th of March (like I did)?!

Microsoft TechEd North America Sessions are Online

    The Microsoft TechEd North America sessions held in New Orleans were made available online (video and slides), an opportunity for technical professionals to get an overview on the new advancements in Microsoft technologies, being approached topics related to the various platforms of Windows, MS Office, Dynamics, Web, Cloud Computing & Online Services, etc. I really like the way Microsoft makes its technologies available to the public, especially the fact that it provides also Express versions of their software, allowing newbies and developers to get acquainted and use essential basic functionality. The MSDN, TechNet, webcasts, Channel9  and community and personal blogs bring the technical and non-technical closer to the company and its technologies.