Skip to content

Software Quality

Comparing quality frameworks

Our dimensions mapped against ISO 25010, CISQ, Garvin, McCall, Boehm, and FURPS.

No way of cutting quality into dimensions is provably correct. The split this site uses — Functionality, Security, Reliability, Performance, Maintainability, with Usability and Aesthetics left to their own mature field — is one organizational choice among several that the field has converged on. The same underlying concerns, routed through our dimensions and through every major quality model, land in recognisably the same places; what differs is how each model lumps, splits, and drops them.

The split was modeled on ISO/IEC 25010 (ISO/IEC 2011)1. The older premise it rests on, that quality is several distinct dimensions rather than one, is usually credited to Garvin (Garvin 1987)2; treating each as needing its own verification is this project's lens.

The reshuffle, framework by framework

The widget maps a neutral set of concepts through two frameworks at once. The left column is our dimensions and stays fixed; the middle column is the neutral concepts; the right column is the characteristics of whichever framework the dropdown selects. A concept that fans to several nodes is one that framework (or we) treat as cross-cutting; a node that gathers many concepts is a lump; a concept that reaches the sink is one the framework has no home for.

Frameworks divide into a product-finish camp (Garvin covers features, aesthetics, and reputation but little of the engineering interior) and an engineering-interior camp (McCall, Boehm, and CISQ cover maintainability and portability but little of the finish); ISO/IEC 25010:2011 is the one model that spans both.

The concepts

The middle column is a neutral vocabulary, defined here so the mappings rest on definitions rather than shared words. Characteristics that an ISO model names are paraphrased from it; the rest are taken from the framework that introduced them.

Behavior and function

  • Functional correctness — the functions are present, return correct results, and fit the task (ISO functional suitability).
  • Features — breadth of capability, especially secondary functions beyond the core.
  • Conformance — the degree to which a product meets a declared specification or standard. A meta-property (adherence), not a behavior, so it cuts across the others.
  • Security — protection against malicious or unauthorized access: it assumes an adversary.
  • Safety / freedom from risk — the product avoids hazards to people, property, or the environment (ISO 25010:2023 Safety, a product property), and using it in context doesn't expose stakeholders to economic, health, or environmental risk (ISO 2011's in-use Freedom from risk). No adversary required — that is what separates it from Security.
  • Reliability — performs without failure over time: maturity, availability, fault tolerance, recoverability.
  • Performance — time behavior, resource use, and capacity under load (speed and resources, not Garvin's "performance").

Evolution and structure

  • Maintainability — the effort to change the software safely.
  • Testability — a sub-concern of maintainability in the modern models.
  • Modifiability — a sub-concern of maintainability in the modern models.
  • Reusability — a sub-concern of maintainability in the modern models.
  • Portability — the ease of moving to other hardware, software, or usage environments.
  • Compatibility — interoperability (exchanging data with another system) plus co-existence (running alongside others that share resources).

User-facing and market

  • Usability — specified users reach their goals with effectiveness, efficiency, and satisfaction.
  • Aesthetics — the look and feel of the interface.
  • Perceived quality — the user's subjective impression, shaped by reputation and indirect signals.

Product lifespan, and quality in use

  • Durability (lifespan) — for a physical product, the use it withstands before wearing out. Software does not wear, so this maps to our dimensions only weakly.
  • Effectiveness (in use) — an outcome of use in a context — ISO's separate quality-in-use model rather than a property of the product itself.
  • Efficiency (in use) — an outcome of use in a context — ISO's separate quality-in-use model rather than a property of the product itself.
  • Satisfaction (in use) — an outcome of use in a context — ISO's separate quality-in-use model rather than a property of the product itself.
  • Context coverage — an outcome of use in a context — ISO's separate quality-in-use model rather than a property of the product itself.

The frameworks

  • Garvin 1987 (Garvin 1987)2 — eight dimensions from a manufacturing and marketing perspective: performance, features, reliability, conformance, durability, serviceability, aesthetics, perceived quality.
  • McCall 1977 (McCall et al. 1977)3 — eleven factors in three perspectives (product operation, revision, transition).
  • Boehm 1978 (Boehm et al. 1978)4 — a tiered model under as-is utility, maintainability, and portability.
  • FURPS 1987 (Grady and Caswell 1987)5 — functionality, usability, reliability, performance, supportability, from Hewlett-Packard.
  • ISO/IEC 9126 (ISO/IEC 2001)6 — six product-quality characteristics; security and interoperability are sub-characteristics of functionality.
  • ISO/IEC 25010:2011 (ISO/IEC 2011)1 — eight product-quality characteristics plus a five-part quality-in-use model.
  • ISO/IEC 25010:2023 (ISO/IEC 2023)7 — nine product-quality characteristics: safety is added, usability becomes interaction capability, portability becomes flexibility, and the quality-in-use model moves to ISO/IEC 25019.
  • CISQ (Consortium for Information and Software Quality (CISQ) 2021)8 — four measures taken directly from source code (reliability, security, performance efficiency, maintainability), standardized as ISO/IEC 5055.

Naming collisions

The same word denotes different things across frameworks:

  • Performance — Garvin's is the product's main operating characteristics (closer to our Functionality); ours and ISO's is speed and resource use.
  • Flexibility — McCall's is ease of modification; ISO/IEC 25010:2023's is the renamed Portability.
  • Durability — Garvin's is physical lifespan and wear, which does not apply to software.

What the comparison shows

Only Reliability appears, by that name and meaning, in every framework. Everything else is renamed, split, merged, or absent somewhere. Our five developed dimensions are the engineering-interior cluster, which is almost exactly the four measures CISQ takes from source code (Consortium for Information and Software Quality (CISQ) 2021)8 plus Functionality.

Every concept from the lineage folds somewhere in our split: portability into Functionality and Maintainability; reusability, testability, and modifiability into Maintainability; compatibility (interoperability and co-existence) into Functionality; the quality-in-use outcomes into Functionality and Usability; safety and freedom-from-risk into Functionality, Reliability, and Usability; and the support cluster — Garvin's serviceability and FURPS's supportability — into Maintainability and Reliability. Several concerns get no dedicated dimension, deliberately: the methods the five already own exercise them.

The split therefore adds nothing the field has not seen, and that is the intended outcome.

Defining "quality"

Beneath any list of dimensions sits a prior question of what "quality" means, and the classic answers differ. Crosby defined it as conformance to requirements (Crosby 1979)9; Juran as fitness for use (Juran 1951)10; Deming framed it around the consumer's present and future needs and the reduction of variation (Deming 1986)11. Garvin later cataloged these as complementary stances, the manufacturing-based and user-based approaches among five (Garvin 1984)12. The dimensions on this site take the product's side of that divide, leaving the in-use and value-based views to usability research and product analytics.

Quality in use

Usability, Aesthetics, and the in-use measures (effectiveness, efficiency, satisfaction, freedom from risk, context coverage) belong to ISO's quality in use model, separate from product quality. This site treats their product-side drivers as part of Functionality and Usability. Measuring them needs real users in a real context, and belongs to usability research and experimentation.

Referenced by

References

  1. ISO/IEC. 2011. ISO/IEC 25010:2011: Systems and software engineering — SQuaRE — System and software quality models. International Organization for Standardization. https://www.iso.org/standard/35733.html. ↩↩

  2. Garvin, David A. 1987. "Competing on the Eight Dimensions of Quality." Harvard Business Review 65 (6): 101–9. https://hbr.org/1987/11/competing-on-the-eight-dimensions-of-quality. ↩↩

  3. McCall, Jim A., Paul K. Richards, and Gene F. Walters. 1977. Factors in Software Quality. Technical Report RADC-TR-77-369. Rome Air Development Center (RADC), Griffiss AFB. https://scispace.com/pdf/factors-in-software-quality-volume-iii-preliminary-handbook-idmaap9gk8.pdf. ↩

  4. Boehm, Barry W., J. R. Brown, H. Kaspar, M. Lipow, G. J. MacLeod, and M. J. Merritt. 1978. Characteristics of Software Quality. Vol. 1. TRW Series of Software Technology. North-Holland. ↩

  5. Grady, Robert B., and Deborah L. Caswell. 1987. Software Metrics: Establishing a Company-Wide Program. Prentice Hall. https://archive.org/details/softwaremetricse00grad. ↩

  6. ISO/IEC. 2001. ISO/IEC 9126: Software engineering — Product quality. International Organization for Standardization. https://www.iso.org/standard/22749.html. ↩

  7. ISO/IEC. 2023. ISO/IEC 25010:2023: Systems and software engineering — SQuaRE — Product quality model. International Organization for Standardization. https://www.iso.org/standard/78176.html. ↩

  8. Consortium for Information and Software Quality (CISQ). 2021. Automated Source Code Quality Measures: Reliability, Security, Performance Efficiency, Maintainability (ISO/IEC 5055:2021). https://www.it-cisq.org/standards/code-quality-standards/. ↩↩

  9. Crosby, Philip B. 1979. Quality Is Free: The Art of Making Quality Certain. McGraw-Hill. https://archive.org/details/qualityisfree00cros. ↩

  10. Juran, Joseph M. 1951. Quality Control Handbook. McGraw-Hill. https://archive.org/details/qualitycontrolha0000jmju. ↩

  11. Deming, W. Edwards. 1986. Out of the Crisis. MIT Center for Advanced Engineering Study. https://archive.org/details/outofcrisisquali00demi. ↩

  12. Garvin, David A. 1984. "What Does 'Product Quality' Really Mean?" Sloan Management Review 26 (1): 25–43. https://sloanreview.mit.edu/article/what-does-product-quality-really-mean/. ↩