Software Quality Assurance: Black-Box Testing (Definitions)

"A specification-based test that looks at a system or unit exclusively from the outside, that is, over its public interface." (Johannes Link & Peter Fröhlich, "Unit Testing in Java", 2003)

"This test compares the externally observable behavior at the external software interfaces (without knowledge of their structure) with the desired behavior. Black-Box tests are frequently equated with »functional tests«, although they can of course also include non-functional tests." (Lars Dittmann et al, "Automotive SPICE in Practice", 2008)

"Repeatable procedure to derive and/or select test cases based on an analysis of the specification, either functional or nonfunctional, of a component or system without reference to its internal structure." (Tilo Linz et al, "Software Testing Foundations" 4th Ed., 2014)

"A software testing methodology that looks at available inputs for an application and the expected outputs from each input." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web" 2nd Ed., 2015)

[Data coverage (black-box) testing:] "Testing a program or subprogram based on the possible input values, treating the code as a black box" (Nell Dale & John Lewis, "Computer Science Illuminated" 6th Ed., 2015)

[black-box test design technique:] "Procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure." (Software Quality Assurance)

"Testing, either functional or non-functional, without reference to the internal structure of the component or system." (Software Quality Assurance)

Software Quality Assurance: White-Box Test/Testing (Definitions)

"An implementation-based test, in contrast to a specification-based test" (Johannes Link & Peter Fröhlich, "Unit Testing in Java", 2003)

"This test is derived knowing the inner structure of the software and based on the program code, design, interface descriptions, and so on. White-box tests are also called 'structure based tests'." (Lars Dittmann et al, "Automotive SPICE in Practice", 2008)

"Any technique used to derive and/or select test cases based on an analysis of the internal structure of the test object." (Tilo Linz et al, "Software Testing Foundations" 4th Ed., 2014)

"This kind of testing requires you to look at the code and see how it works, so you can test individual blocks and choices within the code." (Matt Telles, "Beginning Programming", 2014)

"White box test design technique in which the test cases are designed using the internal structure of the test object. Completeness of such a test is judged using coverage of structural elements (for example, branches, paths, data). General term for control- or data-flow-based test." (Tilo Linz et al, "Software Testing Foundations", 4th Ed., 2014)

"A software testing methodology that examines the code of an application. This contrasts with black box testing, which focuses only on inputs and outputs of an application." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web" 2nd Ed., 2015)

"A test designed by someone who knows how the code works internally. That person can guess where problems may lie and create tests specifically to look for those problems." (Rod Stephens, "Beginning Software Engineering", 2015)

"Procedure to derive and select test cases based on an analysis of the internal structure of a component or system." (Standard Glossary, "ISTQB", 2015)

"Testing based on an analysis of the internal structure of the component or system. " (Standard Glossary, "ISTQB", 2015)

Software Quality Assurance: Regression Testing (Defintiions)

"A test that exercises the entire application to verify that a new piece of code didn’t break anything." (Rod Stephens, "Beginning Software Engineering", 2015)

[regression test suite:] "A collection of tests that are run against a system on a regular basis to validate that it works according to the tests." (Pramod J Sadalage & Scott W Ambler, "Refactoring Databases: Evolutionary Database Design", 2006)

"Selective retesting of a modified system or component to verify that faults have not been introduced or exposed as a result of the changes, and that the modified system or component still meets its requirements." (Richard D Stutzke, "Estimating Software-Intensive Systems: Projects, Products, and Processes", 2005)

"Testing to verify that previously successfully tested features are still correct. It is necessary after modifications to eliminate undesired side effects." (Lars Dittmann et al, "Automotive SPICE in Practice", 2008)

"Testing a program to see if recent changes to the code have broken any existing features." (Rod Stephens, "Start Here!™ Fundamentals of Microsoft® .NET Programming", 2011)

"Testing a previously tested program or a partial functionality following modification to show that defects have not been introduced or uncovered in unchanged areas of the software as a result of the changes made. It is performed when the software or its environment is changed." (Tilo Linz et al, "Software Testing Foundations" 4th Ed., 2014)

"A software testing method that checks for additional errors in software that may have been introduced in the process of upgrading or patching to fix other problems." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web" 2nd Ed., 2015)

Software Quality Assurance: Stress Testing (Definitions)

"A test for the computer system to simulate real environment with real volume of data before moving it to production." (Timothy J  Kloppenborg et al, "Project Leadership", 2003)

"Testing that evaluates a system or component at or beyond its specified performance limits." (Richard D Stutzke, "Estimating Software-Intensive Systems: Projects, Products, and Processes", 2005)

"A form of simulation modeling that focuses specifically on identifying the response of a model under specific, often highly negative scenarios. Common examples include testing the profitability of a bank given catastrophic levels of mortgage defaults or modeling extreme macroeconomic conditions." (Evan Stubbs, "Delivering Business Analytics: Practical Guidelines for Best Practice", 2013)

"Test of system behavior with overload. For example, running it with too high data volumes, too many parallel users, or wrong usage." (Tilo Linz et al, "Software Testing Foundations, 4th Ed", 2014)

"Stress testing assesses the potential outcome of specific changes that are fundamental, material, and adverse." (Christopher Donohue et al, "Foundations of Financial Risk: An Overview of Financial Risk and Risk-based Financial Regulation" 2nd Ed., 2015)

"A type of performance testing conducted to evaluate a system or component at or beyond the limits of its anticipated or specified workloads, or with reduced availability of resources such as access to memory or servers." (IEEE 610)

Software Quality Assurance: Six Sigma (Definitions)

"A statistical term meaning six standard deviations from the norm. Used as the name for a quality improvement program that aims at reducing errors to one in a million." (Judith Hurwitz et al, "Service Oriented Architecture For Dummies" 2nd Ed., 2009)

"1.Generally, a rigorous and disciplined statistical analysis methodology to measure and improve a company’s operational performance, practices and systems. 2.In many organizations, simply a measure of quality near perfection. 3.In data quality, a level of quality in which six standard deviations of a population fall within the upper and lower control limits of quality, allowing no more than 3.4 defects per million parts or transactions." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"A methodology to manage process variations that cause defects, defined as unacceptable deviation from the mean or target, and to systematically work toward managing variation to prevent those defects." (Linda Volonino & Efraim Turban, "Information Technology for Management" 8th Ed., 2011)

"A systematic quality improvement process used on both the production and transactional sides of the business to design, manufacture, and market goods and services that customers may desire to purchase." (Joan C Dessinger, "Fundamentals of Performance Improvement" 3rd Ed., 2012)

"A highly structured approach for eliminating defects in any process, whether from manufacturing or transactional processes. It can be applied to a product or a service–oriented process in any organization. Further, Six Sigma is 'a statistical term that measures how far a given process deviates from perfection'. The goal of Six Sigma is to systematically measure and eliminate defects in a process, aiming for a level of less than 3.4 defects per million instances or 'opportunities'." (Robert F Smallwood, "Managing Electronic Records: Methods, Best Practices, and Technologies", 2013)

"A business management strategy originally developed by Motorola in the 1980s. It is essentially a business problem-solving methodology that supports process improvements through an understanding of customer needs, identification of causes of quality variations, and disciplined use of data and statistical analysis." (Sally-Anne Pitt, "Internal Audit Quality", 2014)

"An approach from the production environment for managing quality that targets a mere 3.4 errors per million instances as its performance goal." (Boris Otto & Hubert Österle, "Corporate Data Quality", 2015)

"A disciplined approach to enterprise-wide quality improvement and variation reduction. Technically, it is the denominator of the capability (Cp) index." (Clyde M Creveling, "Six Sigma for Technical Processes", 2006)

"A business management strategy focusing on quality control testing and optimizing processes through reducing process variance." (Evan Stubbs, "Big Data, Big Innovation", 2014)

Software Quality Assurance: Penetration Testing (Definition)

"A method for assessing information systems in an attempt to bypass controls and gain access." (Weiss, "Auditing IT Infrastructures for Compliance" 2nd Ed., 2015)

"An attempt to circumvent various layers of a system or application’s security controls for the purpose of seeing how far into the system the attacker can get." (Mike Harwood, "Internet Security: How to Defend Against Attackers on the Web" 2nd Ed., 2015)

"A method of evaluating the security of a computer system or network by simulating an attack that a malicious hacker would carry out. This is done so that vulnerabilities and weaknesses can be uncovered." (Shon Harris & Fernando Maymi, "CISSP All-in-One Exam Guide" 8th Ed., 2018)

"Security testing in which evaluators mimic real-world attacks in an attempt to identify ways to circumvent the security features of an application, a system, or a network." (William Stallings, "Effective Cybersecurity: A Guide to Using Best Practices and Standards", 2018)

"The portion of security testing in which evaluators attempt to circumvent the security features of a system. The evaluators may be assumed to use all system design and implementation documentation and may include listings of system source code, manuals, and circuit diagrams. The evaluators work under the same constraints applied to ordinary users." (Mark S Merkow & Lakshmikanth Raghavan, "Secure and Resilient Software Development", 2010)

"The specialized testing of a system to determine if it is possible to defeat its security controls." (O Sami Saydjari, "Engineering Trustworthy Systems: Get Cybersecurity Design Right the First Time", 2018)

Software Quality Assurance: Peer Review (Definitions)

"The review of work products performed by peers during development of the work products to identify defects for removal. The term peer review is used in the CMMI Product Suite instead of the term work product inspection. Essentially, these terms mean the same thing." (Sandy Shrum et al, "CMMI: Guidelines for Process Integration and Product Improvement", 2003)

"A formal review of a complete work product performed by a group to identify defects for removal, and to collect metrics. See also inspection." (Richard D Stutzke, "Estimating Software-Intensive Systems: Projects, Products, and Processes", 2005)

"A system using reviewers who are professional equals; a process used for checking the work performed by one’s equals (peers) to ensure that it meets specific criteria." (Mark S Merkow & Lakshmikanth Raghavan, "Secure and Resilient Software Development", 2010)

"A review of a software work product by colleagues of the producer of the product for the purpose of identifying defects and improvements. Examples are inspection, technical review and walkthrough." (IQBBA, "Standard glossary of terms used in Software Engineering", 2011)

"In relation to internal audit, a form of external assessment that involves a minimum of three organizations assessing each other's internal audit functions using a round-robin approach (A reviews B who reviews C who reviews A)." (Sally-Anne Pitt, "Internal Audit Quality", 2014)

Software Quality Assurance: Inspection (Definitions)

"Visual examination of work products to detect errors, violations of development standards, and other problems. See also peer review and static analysis." (Richard D Stutzke, "Estimating Software-Intensive Systems: Projects, Products, and Processes", 2005)

"The process of examining a component, subassembly, subsystem, or product for off-target performance, variability, and defects during either product development or manufacturing. The focus is typically on whether the item under inspection is within the allowable tolerances. As with all processes, inspection itself is subject to variability, and out-of-spec parts or functions might pass inspection inadvertently." (Clyde M Creveling, "Six Sigma for Technical Processes: An Overview for R Executives, Technical Leaders, and Engineering Managers", 2006)

"Examining or measuring to verify whether an activity, component, product, result, or service conforms to specified requirements. " (For Dummies, "PMP Certification All-in-One For Dummies" 2nd Ed., 2013)

"A verification method in which one member of a team reads the program or design aloud line by line and the others point out errors" (Nell Dale & John Lewis, "Computer Science Illuminated" 6th Ed., 2015)

"Examination of a work product to determine whether it conforms to documented standards." (Project Management Institute, "A Guide to the Project Management Body of Knowledge (PMBOK® Guide )", 2017)

"A type of peer review that relies on visual examination of documents to detect defects, e.g. violations of development standards and non-conformance to higher level documentation. The most formal review technique and therefore always based on a documented procedure." (IEEE 610, 1028)

Software Quality Assurance: Quality Control (Definitions)

"The operational techniques and activities that are used to fulfill requirements for quality." (Sandy Shrum et al, "CMMI: Guidelines for Process Integration and Product Improvement" 2nd Ed., 2006)

"Monitoring project performance for quality and identifying sources of unsatisfactory quality measures." (Bonnie Biafore, "Successful Project Management", 2011)

"Procedures and methods for measuring process quality, identifying unacceptable performance, variance and taking corrective action." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"A set of activities that measure, evaluate, and report on the quality of software project artifacts throughout the project life cycle." (Project Management Institute, "Software Extension to the PMBOK® Guide 5th Ed", 2013)

"Review of all elements of development and production, often reliant on inspection." (Sally-Anne Pitt, "Internal Audit Quality", 2014)

"The practice of testing or measuring and recording results at checkpoints to assess performance and ensure that the project performance meets the standard within appropriate parameters." (Bonnie Biafore & Teresa Stover, "Your Project Management Coach", 2012)

"The operational techniques and activities, part of quality management, that are focused on fulfilling quality requirements." (ISO 8402)

Software Quality Assurance: Testing (Definitions)

"The process of operating a system or component under controlled conditions to collect measurements needed to determine if the system or component meets its allocated requirements. See also dynamic analysis." (Richard D Stutzke, "Estimating Software-Intensive Systems: Projects, Products, and Processes", 2005)

"Activity to verify if an object conforms with its requirements and to detect deviations." (Lars Dittmann et al, "Automotive SPICE in Practice", 2008)

"The application of test cases against a build to ensure that a system performs correctly in those cases." (Bruce P Douglass, "Real-Time Agility: The Harmony/ESW Method for Real-Time and Embedded Systems Development", 2009)

"Generally, a validation process that compares in an organized fashion the functionality or content of a thing or process against pre-established requirements for that thing or process." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"A set of one or more test cases test automation 1. The use of software tools to design or program test cases with the goal to be able to execute them repeatedly using the computer. 2. To support any test activities by using software tools." (Tilo Linz et al, "Software Testing Foundations, 4th Ed", 2014)

"The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects." (Tilo Linz et al, "Software Testing Foundations, 4th Ed", 2014)

"Verifying that a program does what it is supposed to do - and doesn’t do what it is not supposed to do." (Matt Telles, "Beginning Programming", 2014)

"Software testing provides the mechanism for verifying that the requirements identified during the initial phases of the project were properly implemented and that the system performs as expected. The test scenarios developed through these competitions ensure that the requirements are met end-to-end." (Kamalendu Pal & Bill Karakostas, "Software Testing Under Agile, Scrum, and DevOps", 2021)

"A set or one of more test cases" (IEEE 829)

"Activity that verifies that a CI, service or process meets its specifications or agreed requirements" (ITIL)

Software Quality Assurance: Quality (Definitions)

"The totality of characteristics of an entity that bear on its ability to satisfy stated or implied needs." (Timothy J  Kloppenborg et al, "Project Leadership", 2003)

"The degree to which a system, component, or process meets specified requirements, user needs, or stakeholder expectations." (Richard D Stutzke, "Estimating Software-Intensive Systems: Projects, Products, and Processes", 2005)

"The degree or grade of excellence. In a product-development context, it is a product with superior features that performs on target with low variability throughout its intended life. In an economic context, it is the absence or minimization of costs associated with the purchase and use of a product or process." (Clyde M Creveling, "Six Sigma for Technical Processes: An Overview for R Executives, Technical Leaders, and Engineering Managers", 2006)

"A measure of the acceptability or 'goodness” of a system or element'." (Bruce P Douglass, "Real-Time Agility: The Harmony/ESW Method for Real-Time and Embedded Systems Development", 2009)

"1.The degree to which a set of inherent characteristics fulfills requirements. Quality is a multi-faceted concept. The dimensions of quality that are considered most important depend on user perspectives, needs and priorities, which vary across groups of users. 2.Adjective. In common use, of or having superior or high quality, or being perceived as superior, without specific qualification. 3.A peculiar and essential character, the degree of some characteristic meeting expectations. Quality is defined through four virtues - clarity, elegance, simplicity, and value." (DAMA International, "The DAMA Dictionary of Data Management", 2011)

"The degree to which a set of inherent characteristics fulfills requirements." (Cynthia Stackpole, "PMP® Certification All-in-One For Dummies®", 2011)

"The predetermined standard of excellence that may be applied to a product or service to measure how closely the product or service conforms to the standard and satisfies the customer." (Joan C Dessinger, "Fundamentals of Performance Improvement" 3rd Ed., 2012)

"A comparative concept that described the relative ability of two or more competing processes to drive outcomes. A higher-quality process drives a better outcome when considering a full variety of direct and indirect sources of value. Quality is subjective, and depending on organizational and personal objectives, the perception of which process is higher quality may vary." (Evan Stubbs, "Delivering Business Analytics: Practical Guidelines for Best Practice", 2013)

"A relative and unique concept that in effect refers to the standard of something as measured against other things of a similar kind; the degree of excellence of something." (Sally-Anne Pitt, "Internal Audit Quality", 2014)

"1. The totality of characteristics and their values relating to a product or service. They relate to the product’s ability to fulfill specified or implied needs. 2. The degree to which a component, system, or process meets user/customer needs and expectations. 3. The degree to which a set of inherent characteristics fulfills requirements." (Tilo Linz et al, "Software Testing Foundations, 4th Ed", 2014)

"A quality is an attribute or property of a resource. A quality is logically ascribable by a subject. (Ed.)" (Robert J Glushko, "The Discipline of Organizing: Professional Edition" 4th Ed., 2016)

"A subjective term for which each person or sector has its own definition. In technical usage, quality can have two meanings: 1) the characteristics of a product or service that bear on its ability to satisfy stated or implied needs; 2) a product or service free of deficiencies." (American Society for Quality)

"The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations." [IEEE 610]

"The degree to which a set of inherent characteristics fulfills requirements." (ISO 9000:2000) 

"The totality of features of a product or service that fulfill stated or implied needs." (ISO 8402)

"The degree to which a set of inherent characteristics fulfils requirements." (SDMX)