Notes: Team Data Science Process (TDSP)

Team Data Science Process (TDSP)
• an agile, iterative data science methodology to deliver predictive analytics solutions and intelligent applications efficiently [1]
• {goal} help customers fully realize the benefits of their analytics program [1]
• {component} data science lifecycle definition
  • {description} a framework to structure the development of data science projects [1]
  • {goal} designed for data science projects that ship as part of intelligent applications that deploy ML & AI models for predictive analytics [1]
  • {benefit} can be used in the context of other DM methodologies as they have common ground [1]
    • e.g. CRISP-DM, KDD
  • {benefit} exploratory data science projects or improvised analytics projects can also benefit from using this process [1]
• {component} standardized project structure
  • {description} a directory structure that includes templates for project documents
    • ⇒makes it easy for team members to find information [1]
    • ⇐templates for the folder structure and required documents are provided in standard locations [1]
    • all code and documents are stored in an agile VCS tracking repository [1]
      • {recommendation} create a separate repository for each project on the VCS for versioning, information security, and collaboration [1]
  • {benefit} organizes the code for the various activities [1]
  • {benefit} allows tracking the progress [1]
  • {benefit} provides checklist with key questions for each project to guarantee process and deliverables’ quality [1]
  • {benefit} enables team collaboration [1]
  • {benefit} allows closer tracking of the code for individual features [1]
  • {benefit} enables teams to obtain better cost estimates [1]
  • {benefit} helps build institutional knowledge across the organization [1]
• {component} recommended infrastructure
  • {description} a set of recommendations for the infrastructure and resources needed for analytics and storage [1]
  • {benefit} addresses cloud and/or on-premises requirements [1]
  • {benefit} enables reproducible analysis [1]
  • {benefit} avoids infrastructure duplication [1]
    • ⇒minimizes inconsistencies and unnecessary infrastructure costs [1]
  • {tools} tools are provided to provision the shared resources, track them, and allow each team member to connect to those resources securely [1]
  • {good practice} create a consistent compute environment [1]
    • ⇐allows team members replicate and validate experiments [1]
• {component} recommended tools and utilities
  • {description} a set of recommendations for the tools and utilities needed for project’s execution [1]
  • {benefit} help lower the barriers and increase the consistency of their adoption [1]
  • {benefit} provides an initial set of tools and scripts to jump-start methodology’s adoption [1]
  • {benefit} helps automate some of the common tasks in the data science lifecycle [1]
    • e.g. data exploration and baseline modeling [1]
  • {benefit} well-defined structure provided for individuals to contribute shared tools and utilities into their team's shared code repository [1]
    • ⇐ resources can then be leveraged by other projects [1]
• {phase} 1: business understanding
  • {goal} define and document the business problem, its objectives, the needed attributes, and the metric(s) used to determine project’s success
  • {goal} identify and document the relevant data sources
  • {step} 1.1: define project’s objectives
    • elicit together with the stakeholders the requirements, define and document the problem and its objectives, respectively the metric(s) used to determine project’s success
      • requires a good understanding of the business processes, data and further characteristics
  • {step} 1.2: identify data sources
    • identify the attributes and the data sources relevant to the problem under study
  • {step} 1.3: define project plan and team*
    • develop a high-level milestone plan and identify the resources needed for executing it
  • {tool} project charter
    • standard template that documents the business problem, the scope of the project, the business objectives and metric(s) used to determine project’s success
• {phase} 2: data acquisition & understanding
  • {goal} prepare the base dataset(s) as needed by the modeling phase into the target repository
  • {goal} build the data ETL/ELT architecture and processes needed for provisioning the basis data
  • {step} 2.1: ingest data
    • make the required data available for the team in the repository where the analytics operations take place
  • {step} 2.2: explore data
    • understand data’s characteristics by leveraging specific tools (visualization, analysis)
    • prepare the data as needed for further processing
  • {step} 2.3: set up pipelines
    • build the pipelines needed for data actualization and qualitative assessment [3]
    • set up a process to score new data or refresh the data regularly [3]
  • {step} 2.4: feasibility analysis*
    • reevaluate the project to determine whether the value expected is sufficient to continue pursuing it
  • {tool} data quality report
    • report that includes data summaries, data mappings, variable ranking, data qualitative assessment(s) and further information [3]
  • {tool} solution architecture
    • diagram and/or textual-based description of the data pipeline(s), technical assumptions and further aspects
  • {tool} data reports
    • document the structure and statistics of the raw data
  • {tool} checkpoint decision
    • decision template document that
      • summarizes the findings of the feasibility analysis step
      • includes a set of choices and recommendations for the next steps
      • serves as basis for the decision on whether to continue or not the project, respectively what the next steps are
• {phase} 3: modeling
  • {goal} create a machine-learning model that addresses the prediction requirements and that's suitable for production
  • {step} 3.1: feature engineering
    • the inclusion, aggregation, and transformation of raw variables to create the features used in the analysis [4]
      • ⇐requires a good understanding of how the features relate to each other and how the ML algorithms use those features [4]
  • {step} 3.2: model selection*
    • choose one or more modeling algorithms that address problem’s characteristics the best
  • {step} 3.3: model training
    • involves the following steps:
      • split the input data into training and test datasets
      • build the models by using the training dataset
      • evaluate the training and the test data set
      • determine the optimal setup and methods
  • {step} 3.4: model evaluation
    • evaluate the performance of the model(s)
  • {step} 3.5: feasibility analysis*
    • evaluate the readiness of the models for use into production, respectively on whether they fulfill project’s objectives
  • {tool} feature sets
    • describe the features developed for the modeling and how they were generated
    • contains pointers to the code used to generate the features
  • {tool} model report
    • a standard, template-based report that provides details on each experiment’s outcomes
    • created for each model tried
  • {tool} checkpoint decision
  • {tool} model performance metrics
    • e.g. ROC curves or MSE
• {phase} 4: deployment
  • {goal} deploy the models and the data pipelines to the environment used for final user acceptance
  • {step} 4.1: operationalize architecture
    • prepare the models and data pipelines for use into production
    • {best practice} expose the models over an open API interface
      • enables models’ consumption from various applications
    • {best practice} build telemetry and monitoring into the models and the data pipelines [5]
      • helps in monitoring and troubleshooting [5]
  • {step} 4.2: deploy solution*
    • deploy the architecture into production
  • {tool} status dashboard
    • displays data on system’s health and key metrics
  • {tool} model report
    • the report in its final form with deployment information
  • {tool} solution architecture
    • the document in its final form
• {phase} 5: customer acceptance
  • {goal} confirm that project’s objectives were fulfilled and get customer’s acceptance
  • {step} 5.1: system validation
    • validate system’s performance and outcomes and confirm that it fulfills customer’s needs
  • {step} 5.2: project signoff*
    • finalize and review documentation
    • handover the solution and afferent documentation to customer
    • evaluate the project against the defined objectives and get customer’ signoff
  • {tool} exit report
  • {tool} technical report
    • contains all the details of the project that are useful for learning about how to operate the system [6]

Acronyms:

Artificial Intelligence (AI)

Cross-Industry Standard Process for Data Mining (CRISP-DM)

Data Mining (DM)

Knowledge Discovery in Databases (KDD)

Team Data Science Process (TDSP) 

Version Control System (VCS)

Visual Studio Team Services (VSTS)

Resources:

[1] Microsoft Azure (2020) What is the Team Data Science Process? [source]

[2] Microsoft Azure (2020) The business understanding stage of the Team Data Science Process lifecycle [source]

[3] Microsoft Azure (2020) Data acquisition and understanding stage of the Team Data Science Process [source]

[4] Microsoft Azure (2020) Modeling stage of the Team Data Science Process lifecycle [source] 

[5] Microsoft Azure (2020) Deployment stage of the Team Data Science Process lifecycle [source]

[6] Microsoft Azure (2020) Customer acceptance stage of the Team Data Science Process lifecycle [source]

Systems Engineering: Agent-Based Model/Modeling (Definition)

"Modeling refers to the process of designing a software representation of a real-world system or a small part of it with the purpose of replicating or simulating specific features of the modeled system. In an agent-based model, the model behavior results from behavior of many small software entities called agents. This technique is used to model real-world systems comprised of many decision-making entities with inhomogeneous preferences, knowledge, and decision-making processes. An advantage of this method is that no assumptions need to be made about an aggregate or mean behavior. Instead, this aggregation is made by the model." (E Ebenhoh, "Agent-Based Modelnig with Boundedly Rational Agents", 2007)

"A modeling and simulation approach applied to a complex system or complex adaptive system, in which the model is comprised of a large number of interacting elements (agents)." (Charles M Macal, "Agent Based Modeling and Artificial Life", 2009)

"A modeling technique with a collection of autonomous decision-making agents, each of which assesses its situation individually and makes decisions on the basis of a pre-set of rules. ABM is used to simulate land use land cover change, crowd behavior, transportation analysis and many other fine-scale geographic applications. (May Yuan, "Challenges and Critical Issues for Temporal GIS Research and Technologies", 2009)

"Agent-based models (ABM) are models where (i) there is a multitude of objects that interact with each other and with the environment; (ii) the objects are autonomous, i. e. there is no central, or top-down control over their behavior; and (iii) the outcome of their interaction is numerically computed." (Mauro Gallegati & Mateo G Richiardi, "Agent Based Models in Economics and Complexity", 2009)

"An agent-based model is a simulation made up of a set of agents and an agent interaction environment." (Michael J North & Charles M Macal, "Agent Based Modeling and Computer Languages", 2009)

"Systems composed of individuals who act purposely in making locational/spatial decisions." (Michael Batty, "Cities as Complex Systems: Scaling, Interaction, Networks, Dynamics and Urban Morphologies", 2009)

"A computational model for simulating the actions and interactions of autonomous individuals in a network, with a view to assessing their effects on the system as a whole. (Brian L. Heath & Raymond R Hill, "Agent-Based Modeling: A Historical Perspective and a Review of Validation and Verification Efforts", 2010)

"A model that involves numerous interacting autonomous agents, homogeneous or heterogeneous. The objective of agent-based modeling is to help us to understand effects and impacts of interactions of individuals." (Peter Mikulecký et al, "Possibilities of Ambient Intelligence and Smart Environments in Educational Institutions", 2011)

"a class of computational models for simulating interacting agents." (Enrico Franchi & Agostino Poggi, "Multi-Agent Systems and Social Networks", 2012)

Knowledge Management: Conceptual Model (Definitions)

"A conceptual model is a qualitative description of the system and includes the processes taking place in the system, the parameters chosen to describe the processes, and the spatial and temporal scales of the processes." (A Avogadro & R C Ragaini, "Technologies for Environmental Cleanup", 1993)

"A conceptual model is a model of the projected system that is independent of implementation details." (Michael Worboys, "GIS: A Computing Perspective", 1995)

"A conceptual model is what in the model theory is called a set of formulas making statements about the world." (Dickson Lukose [Eds.], "Conceptual Structures: Fulfilling Peirce's Dream" Vol 5, 1997)

"A conceptual model is a representation of the system expertise using this formalism. An internal model is derived from the conceptual model and from a specification of the system transactions and the performance constraints." (Zbigniew W. Ras & Andrzej Skowron [Eds.], Foundations of Intelligent Systems: 10th International Symposium Vol 10, 1997)

"A conceptual model is one which reflects reality by placing words which are concepts into the model in the same way that the model aeroplane builder puts wings, a fuselage, and a cockpit together." (Lynn Basford & ‎Oliver Slevin, "Theory and Practice of Nursing: An Integrated Approach to Caring Practice", 2003) 

"A conceptual model is simply a framework or schematic to understand the interaction of workforce education and development systems with other variables in a society." (Jay W Rojewski, "International Perspectives on Workforce Education and Development", 2004) 

"A conceptual model is a mental image of a system, its components, its interactions. It lays the foundation for more elaborate models, such as physical or numerical models. A conceptual model provides a framework in which to think about the workings of a system or about problem solving in general. An ensuing operational model can be no better than its underlying conceptualization." (Henry N Pollack, "Uncertain Science … Uncertain World", 2005)

"A particular kind of learning object design to be supplied to learners to support their mental modeling." (Daniel Churchill, "Mental Models" [in "Encyclopedia of Information Technology Curriculum Integration"], 2008)

"The concepts and constructs about real work things we have in our heads are called mental model." (Hassan Qudrat-Ullah, "System Dynamics Based Learning Environments" [in "Encyclopedia of Information Technology Curriculum Integration"], 2008)

"Representations of real or imaginary structure in the human mind enabling orientation as well as goal orientated actions and movements" (Ralf Wagner, "Customizing Multimedia with Multi-Trees" [in "Encyclopedia of Multimedia Technology and Networking" 2nd Ed.], 2009)

"A conceptual model is a qualitative description of 'some aspect of the behaviour of a natural system'. This description is usually verbal, but may also be accompanied by figures and graphs." (Howard S. Wheater et al., "Groundwater Modelling in Arid and Semi-Arid Areas, 2010) 

"[…] a conceptual model is a diagram connecting variables and constructs based on theory and logic that displays the hypotheses to be tested." (Mary Wolfinbarger Celsi et al, "Essentials of Business Research Methods", 2011) 

"A conceptual model of an interactive application is, in summary: the structure of the application - the objects and their operations, attributes, and relation-ships; an idealized view of the how the application works – the model designers hope users will internalize; the mechanism by which users accomplish the tasks the application is intended to support." (Jeff Johnson & Austin Henderson, "Conceptual Models", 2011)

"Simply put, a conceptual model is a simplified representation of reality, devised for a certain purpose and seen from a certain point of view."(David W Emble & Bernhard Thalheim, "Handbook of Conceptual Modeling", 2012) 

"Briefly, a conceptual model is the configuration of conceptual elements and the navigation between them. As such, a conceptual model is the foundation of the user interface of any interactive system." (Avi Parush, "Conceptual Design for Interactive Systems", 2015)

"A conceptual model is a framework that is initially used in research to outline the possible courses of action or to present an idea or thought. When a conceptual model is developed in a logical manner, it will provide a rigor to the research process." (N Elangovan & R Rajendran, "Conceptual Model: A Framework for Institutionalizing the Vigor in Business Research", 2015) 

"A model or conceptual model is a schematic or representation that describes how something works. We create and adapt models all the time without realizing it. Over time, as you gain more information about a problem domain, your model will improve to better match reality." (James Padolsey, "Clean Code in JavaScript", 2020)

Resources:
Quotes on "Conceptual Models" at the-web-of-knowledge.blogspot.com.

Dennis K Lieu - Collected Quotes

"Being a good team member takes work. Most people are used to working on their own - making decisions, prioritizing tasks, and being accountable for their own work. Working with others requires a different approach than working alone. To be a successful part of a team, you need to consider several issues. You should be prepared not to be in charge of everything. For some people, this requires a great deal of effort; for other people, it is less taxing. At times, you will be the supervisor; other times you will be supervised. You need to be flexible and understand that a team consisting only of leaders (or only of followers) is not likely to perform well." (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

"Charts are used to represent quantitative data in a graphic format. A chart visually illustrates relationships between numbers. When creating a chart, keep in mind that the goal is to represent the data in a simplified and appealing way so as not to muddle the message the chart is meant to convey." (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

"Design is a goal-oriented, problem-solving activity that typically takes many iterations - teams rarely come up with the 'optimal' design the first time around. [...] With each model, improvements were made to the original design such that the minivans of today are much improved compared to the initial product. The key activity in the design process is the development and testing of a descriptive model of the finished product before the product is finally manufactured or constructed." (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

"Designers are responsible for the project’s fit and finish, that is, specifying the geometry and sizes of components so they properly mate with each other and are ergonomically and aesthetically acceptable within the operating environment." (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

"Information graphics are an essential component of technical communication. Very few technical documents or presentations can be considered complete without graphical elements to present some essential data. Because engineers are visually oriented, graphic aids allow their thoughts and ideas to be better understood by other engineers. Information graphics are essential in presenting data because they simplify the content, offer a visually pleasing alternative to gray text in a proposal or an article, and thereby invite interest." (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

"Most importantly, prepare to learn how to be a team member. Share your strengths with the team and be willing to contribute. Remember, the combined efforts of all team members should yield a better outcome than the efforts of one individual. Learn new team skills and be adaptable." (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

"Reverse engineering is a systematic methodology for analyzing the design of an existing device or system, either as an approach to study the design or as a prerequisite for redesign. Reverse engineering essentially is a process used to gain information about the functionality and sizes of existing design components. [...] Reverse engineering is a technique within the practice of engineering design that can be useful in several ways. Reverse engineering can save time because there is no need to 'reinvent the wheel' when you can start from existing geometric data. The reverse engineering technique also can help an engineer develop a systematic approach to thinking about and improving the design of devices and systems." (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

"Tables work in a variety of situations because they convey large amounts of data in a condensed fashion. Use tables in the following situations: (1) to structure data so the reader can easily pick out the information desired, (2) to display in a chart when the data contains too many variables or values, and (3) to display exact values that are more important than a visual moment in time." (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

"The data [in tables] should not be so spaced out that it is difficult to follow or so cramped that it looks trapped. Keep columns close together; do not spread them out more than is necessary. If the columns must be spread out to fit a particular area, such as the width of a page, use a graphic device such as a line or screen to guide the reader’s eye across the row." (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

"Whereas charts generally focus on a trend or comparison, tables organize data for the reader to scan. Tables present data in an easy-read-format, or matrix. Tables arrange data in columns or rows so readers can make side-by-side comparisons. Tables work for many situations because they convey large amounts of data and have several variables for each item. Tables allow the reader to focus quickly on a specific item by scanning the matrix or to compare multiple items by scanning the rows or columns."  (Dennis K Lieu & Sheryl Sorby, "Visualization, Modeling, and Graphics for Engineering Design", 2009)

Randall E Schumacker - Collected Quotes

"Given the important role that correlation plays in structural equation modeling, we need to understand the factors that affect establishing relationships among multivariable data points. The key factors are the level of measurement, restriction of range in data values (variability, skewness, kurtosis), missing data, nonlinearity, outliers, correction for attenuation, and issues related to sampling variation, confidence intervals, effect size, significance, sample size, and power." (Randall E Schumacker & Richard G Lomax, "A Beginner’s Guide to Structural Equation Modeling" 3rd Ed., 2010)

"Need to consider outliers as they can affect statistics such as means, standard deviations, and correlations. They can either be explained, deleted, or accommodated (using either robust statistics or obtaining additional data to fill-in). Can be detected by methods such as box plots, scatterplots, histograms or frequency distributions." (Randall E Schumacker & Richard G Lomax, "A Beginner’s Guide to Structural Equation Modeling" 3rd Ed., 2010)

"Outliers or influential data points can be defined as data values that are extreme or atypical on either the independent (X variables) or dependent (Y variables) variables or both. Outliers can occur as a result of observation errors, data entry errors, instrument errors based on layout or instructions, or actual extreme values from self-report data. Because outliers affect the mean, the standard deviation, and correlation coefficient values, they must be explained, deleted, or accommodated by using robust statistics." (Randall E Schumacker & Richard G Lomax, "A Beginner’s Guide to Structural Equation Modeling" 3rd Ed., 2010)

"Structural equation modeling is a correlation research method; therefore, the measurement scale, restriction of range in the data values, missing data, outliers, nonlinearity, and nonnormality of data affect the variance–covariance among variables and thus can impact the SEM analysis." (Randall E Schumacker & Richard G Lomax, "A Beginner’s Guide to Structural Equation Modeling" 3rd Ed., 2010)

"Structural equation modeling (SEM) uses various types of models to depict relationships among observed variables, with the same basic goal of providing a quantitative test of a theoretical model hypothesized by the researcher. More specifically, various theoretical models can be tested in SEM that hypothesize how sets of variables define constructs and how these constructs are related to each other." (Randall E Schumacker & Richard G Lomax, "A Beginner’s Guide to Structural Equation Modeling" 3rd Ed., 2010)

"There are several key issues in the field of statistics that impact our analyses once data have been imported into a software program. These data issues are commonly referred to as the measurement scale of variables, restriction in the range of data, missing data values, outliers, linearity, and nonnormality." (Randall E Schumacker & Richard G Lomax, "A Beginner’s Guide to Structural Equation Modeling" 3rd Ed., 2010)