Quantitative metrics & scoring models

Scope: every established way the field turns usabilityquality into numbers, with exact scoring math, a measurability tag, and a one-line note on how it feeds OUR composite 0–100 index. Tags: [survey] = needs human respondents · [behavioral-auto] = computable from instrumentationlogs · [composite] = combines other metrics.

1. SUS — System Usability Scale [survey]

The anchor of the whole field. 10 items, 5-point Likert (1 = Strongly disagree … 5 = Strongly agree), alternating positive/negative tone.

Items (odd = positive, even = negative):

1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.

Exact scoring algorithm:

1. Odd items (positive): contribution = position − 1 (so 1→0 … 5→4).
2. Even items (negative): contribution = 5 − position (so 1→4 … 5→0).
3. Sum the 10 contributions → range 0–40.
4. Multiply by 2.5 → 0–100.

Note: the 0–100 SUS score is not a percentage and not a percentile — it's a 0–100-scaled mean.

Benchmark: grand mean ≈ 68 (SD ≈ 12.5) across ~500 studies / 5,000+ responses (Sauro & Lewis). 68 = the 50th percentile. Above 68 = above average.

Sauro–Lewis curved grading scale (raw SUS → grade → percentile):

Feeds our index: SUS is the canonical 0–100 perceived-usability channel and a natural normalization target — express any survey channel on the SUS scale, then percentile-rank against the curve. Use 68 as our neutral midpoint anchor.

2. UMUX & UMUX-LITE [survey]

UMUX (Usability Metric for User Experience, Finstad 2010): 4 items, 7-point, designed to be a short proxy that recovers a SUS-like score and maps to ISO 9241-11 (effectivenessefficiencysatisfaction). Two positive, two negative tone (alternating). Raw item score normalized: for positive items score − 1; for negative 7 − score; sum the 4, divide by 24, ×100 → 0–100.

UMUX-LITE (Lewis, Utesch & Maher 2013) — the practical winner. Just 2 positive items, 7-point agreement:

• "This system's capabilities meet my requirements." (capability/usefulness)
• "This system is easy to use." (ease)

Raw UMUX-LITE score: [(item1 + item2 − 2) / 12] × 100 → 0–100.

Regression correction to SUS (the headline result):

SUS_estimate = 0.65 × ([(item1 + item2 − 2) × (100/12)]) + 22.9

i.e. apply slope 0.65 and intercept 22.9 to the raw UMUX-LITE. Validated to ~1% error (≈99% agreement) vs actual SUS across independent datasets (Lewis et al.; Borsci et al.). The correction exists because raw UMUX-LITE means run slightly below SUS.

Feeds our index: UMUX-LITE is our cheapest survey channel (2 questions, in-product micro-survey friendly) yet outputs a defensible SUS-equivalent 0–100. Best ROI per token of user attention. Use the regression form so it lands on the same scale as SUS.

3. SUPR-Q — Standardized UX Percentile Rank Questionnaire [survey] [composite]

Sauro/MeasuringU's website-UX instrument: 8 items (7 on a 5-point agreement scale + 1 NPS-style 0–10 likelihood-to-recommend), yielding an overall score + 4 sub-factor scores, all expressed as percentile ranks (0–100) against a rolling normative database of hundreds of sites.

Four sub-factors (2 items each, conceptually):

• Usability — easy to use / easy to navigate (α ≈ .88)
• Trust — credible / trustworthy (α ≈ .87)
• Appearance — attractive, clean, simple (α ≈ .80)
• Loyalty — likelihood to return / to recommend (α ≈ .73; carries the NPS item)

Overall reliability α ≈ .90. Scoring = raw mean per factor → look up percentile in the SUPR-Q database → a 0–100 percentile. A SUPR-Q of 50 = better than 50% of sites measured.

Feeds our index: This is the closest existing template to what we're building — a single instrument that decomposes UX into weighted sub-factors and norms each as a percentile. Our composite can mirror its structure (usability + trust + aesthetics + loyalty as sub-channels) and its percentile-norming approach directly.

4. SEQ — Single Ease Question [survey] (post-task)

One 7-point item asked immediately after each task: "Overall, how difficult or easy was this task?" (1 = Very difficult … 7 = Very easy).

Scoring: mean of the 7-point ratings. Benchmark ≈ 5.3–5.6 across 400+ tasks / 10,000+ users (use ~5.5 as neutral). Correlates with task-time and completion at r ≈ .5. Tip: scores ≤ ~4.8 flag tasks worth fixing.

Feeds our index: the per-task perceived-difficulty channel. Average SEQ across a representative task set → rescale (mean − 1)/6 × 100 to feed the 0–100 composite. Cheap, sensitive, task-granular — pairs naturally with behavioral task-success.

UMUX-Lite vs SUS correlation: strong — typically r ≈ .8–.9; with the 0.65/22.9 regression, UMUX-Lite reproduces SUS means at ~99%. SEQ vs SUS is weaker (different grain: task vs system) but both load on perceived usability.

5. Google HEART + Goals–Signals–Metrics (and PULSE) [behavioral-auto] mostly

Rodden, Hutchinson & Fu (Google, CHI 2010). Large-scale product-UX measurement framework. Not a single score — a category model for choosing metrics.

HEART dimensions:

• Happiness — attitudinal/satisfaction (survey: SUS, CSAT, perceived ease). [survey]
• Engagement — frequencyintensitydepth of interaction (e.g., sessionsuserweek). [behavioral-auto]
• Adoption — new users / uptake of a feature in a window. [behavioral-auto]
• Retention — % of users still active after N days/weeks (cohort survival). [behavioral-auto]
• Task success — completion rate, time-on-task, error rate. [behavioral-auto]

You don't use all five — pick the rows that matter per product.

Goals–Signals–Metrics (GSM) process (the operational core):

1. Goals — what success means for this product/feature (per HEART row).
2. Signals — observable user behaviors/attitudes indicating progress toward the goal.
3. Metrics — the precise, trackable numbers (often signal ÷ exposure, normalized as ratios) put on a dashboard.

PULSE (the older, operations-centric predecessor HEART reacted against): Page views, Uptime, Latency, Seven-day active users, Earnings. Critique: business/ops health, only indirectly tied to UX, hard to use as a design dependent variable.

Feeds our index: HEART supplies the behavioral half of our composite (EngagementAdoptionRetentionTask-success are all auto-computable) and GSM gives us the discipline to define each sub-metric as a normalized ratio before weighting. Happiness = where SUSUMUX-LITE/SUPR-Q plug in.

6. Behavioral metrics (ISO-grounded, mostly auto) [behavioral-auto]

Feeds our index: these are the fully automatable channels — no respondents needed. They map 1:1 to ISO 9241-11 effectiveness (success) + efficiency (time, lostness, errors). Each needs a target/spec limit to normalize (see §10).

7. ISO 9241-11:2018 — the usability definition [composite] framework

"Extent to which a system can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use."

• Effectiveness = accuracy & completeness of goal achievement → task success rate.
• Efficiency = resources (time/effort) relative to results → time-on-task, efficiency ratio.
• Satisfaction = physicalcognitiveemotional response meeting needs → SUSUMUXSEQ.

Feeds our index: the three-pillar skeleton. Our composite should guarantee at least one channel per pillar (success / efficiency / satisfaction) so the score isn't gameable by optimizing one dimension. This is the conceptual backbone behind SUM (§9).

8. ISO/IEC 25010 — Product Quality model (SQuaRE) [composite] framework

Software product quality (broader than usability), 2023 revision: 9 characteristics (Functional Suitability, Performance Efficiency, Compatibility, Interaction Capability [renamed from "Usability" in 2023], Reliability, Security, Maintainability, Portability, + Safety added 2023). Each scored by measures in the companion ISO/IEC 25023.

Usability / Interaction Capability sub-characteristics (the ones relevant to UI quality):

• Appropriateness recognizability — users recognize the product fits their needs.
• Learnability — can be learned within a specified time/effort.
• Operability — easy to operate and control.
• User error protection — system prevents/guards against operation errors.
• User interface aesthetics — UI is pleasing/satisfying to interact with.
• Accessibility — usable across the widest range of user characteristics/capabilities.
• (2023 adds self-descriptiveness / inclusivity considerations.)

Feeds our index: gives our sub-factor taxonomy a standards-backed vocabulary. Map each of our sub-channels onto a 25010 sub-characteristic (e.g. an automated a11y audit → Accessibility; aesthetic/visual score → UI aesthetics; error telemetry → User error protection). Lets us claim ISO traceability for the composite's decomposition.

9. SUM — Single Usability Metric [composite] (the canonical "combine into one score")

Sauro & Kindlund (CHI 2005) — the published method for collapsing usability metrics into one standardized score, built explicitly from the ISO 9241-11 definition.

Inputs (4, ≈ equal weight by PCA): task completion rate, errors, task time, post-task satisfaction.

Standardization → z-equivalents:

• Continuous (time, satisfaction): z = (spec_limit − mean) / SD — distance from a specification limit in SD units, converted to a 0–1 proportion via the normal CDF.
• Discrete (completion, errors): Six-Sigma style — defects / opportunities, converted to a proportion-meeting-spec (with small-sample Wilson/Laplace adjustments).

Combine: average the four standardized 0–100% values → task-level SUM (0–100%). PCA showed the four contribute roughly equally → equal weighting is defensible.

Feeds our index: SUM is the direct precedent for our formula — it proves the pattern (pick ISO-aligned metrics → standardize each against a spec limit into a common 0–1/0–100 scale → average with justified weights). We can generalize SUM beyond task-level to a product-level index.

10. NPS — Net Promoter Score [survey] (include, but weight low)

Single item: "How likely are you to recommend X to a friend or colleague?" 0–10.

Scoring: NPS = %Promoters (9–10) − %Detractors (0–6); Passives (7–8) ignored. Range −100 to +100.

Why it's weak for UX:

• Throws away information — bins 11 points into 3 groups and discards Passives; a shift from "2 (hated)" to "5 (disliked)" — a real UX gain — registers zero change (still a Detractor).
• Loyalty ≠ usability — measures brand/relationship intent, not interface quality; confounded by price, brand, support.
• Statistically inefficient — needs much larger samples for a stable estimate than a mean rating would.
• Contested predictive validity — no robust evidence it beats other loyalty items at predicting growth; Reichheld himself warned against over-reliance.

Feeds our index: keep as a small loyalty signal (it's already the Loyalty item inside SUPR-Q) but never as the headline. Prefer the raw 0–10 mean (more information) over the binned NPS. Cap its weight.

RECOMMENDATION — How to structure OUR composite 0–100 UI-Quality Index

Grounded in SUM (standardize-then-average), SUPR-Q (percentile-norm sub-factors), ISO 9241-11 (three pillars), 25010 (sub-characteristic taxonomy), and HEART/GSM (behavioral channels):

A. Three ISO-9241-11 pillars, each guaranteed ≥1 channel

Prevents gaming one dimension. Suggested channels & default weights (tune empirically via PCA like SUM did):

B. Normalization — two-stage, SUM + SUPR-Q hybrid

1. Standardize each channel to 0–1 against a specification limit (SUM's z-equivalent method): for continuous metrics z = (spec − mean)/SD then normal-CDF → 0–1; for rates, defects/opportunities → proportion-meeting-spec. This makes heterogeneous units (seconds, %, Likert) commensurable.
2. Percentile-norm against a rolling benchmark DB (SUPR-Q method) once we have ≥ a few dozen products measured — so "73" means "better than 73% of Koder UIs," which is far more interpretable than a raw weighted mean. Until the DB exists, fall back to fixed spec limits + the Sauro–Lewis curve for the SUS channel.

C. Aggregate

Index = 100 × Σ(wᵢ · normᵢ) with Σwᵢ = 1. Start with equal-ish weights (SUM's PCA found ≈equal contribution); re-derive weights from PCA/factor loadings once you have data. Report the sub-scores alongside the composite (never collapse them away — that's SUPR-Q's discipline and the fix for NPS's information loss).

D. Banding — adopt the Sauro–Lewis curve directly

Reuse the A+ → F bands and percentile mapping (§1) verbatim for the 0–100 composite, with 68 ≈ "C"/average as the neutral anchor and ≥ 80 = "A" (top decile). This gives stakeholders an instantly familiar grade without inventing a new scale.

E. Cadence (GSM)

Define each channel as a Goal→Signal→Metric triple; recompute behavioral channels continuously (auto), refresh survey channels per release/quarter. Behavioral channels are zero-friction; survey channels (UMUX-LITE = 2 questions) are the cheapest high-value attitudinal input.

Sources

• SUS scoring & benchmark (mean 68): MeasuringU – 5 Ways to Interpret a SUS Score · Item Benchmarks for the SUS (Lewis & Sauro)
• Sauro–Lewis curved grading scale: Curved Grading Scale table · MeasuringU interpret-sus-score
• UMUX / UMUX-LITE + regression to SUS: MeasuringU – From the SUS to the UMUX-Lite · Lewis, UMUX-LITE: When There's No Time for the SUS (PDF) · Correspondence Between UMUX-LITE and SUS (Springer)
• SUPR-Q (4 factors + percentile norming): SUPR-Q: A Comprehensive Measure of Website UX (UXPA Journal) · MeasuringU – Validating the SUPR-Q Model
• SEQ: MeasuringU – 10 Things to Know About the SEQ · NN/g – Measuring Perceived Usability (SUS, NASA-TLX, SEQ)
• HEART + GSM + PULSE: Kerry Rodden – The HEART framework · heartframework.com · MeasuringU – Should You Love the HEART Framework?
• Lostness: MeasuringU – Validating a Lostness Measure · Smith, Lost in Hyperspace: Metrics and Mental Models
• SUM (combine into one score): Sauro & Kindlund, A Method to Standardize Usability Metrics Into a Single Score (PDF) · MeasuringU – SUM
• ISO 9241-11:2018: ISO 9241-11:2018(en) official
• ISOIEC 25010 (2023): [iso25000.com – ISO 25010](https:iso25000.comindex.phpeniso-25000-standardsiso-25010) · [ISOIEC 25010:2023 guide](https:/log.pacificcert.comiso-25010-software-product-quality-model)
• NPS & UX critique: NN/g – NPS: What a Customer-Relations Metric Can Tell You About UX · MeasuringU – 10 Things to Know About NPS and UX