Read an inspection confusion matrix by deciding what counts as a defect and counting the four possible decisions on units with settled reference outcomes. Read those counts alongside the number of units sent to review, the tested conditions, and the consequences of a wrong decision.
Start with the four production outcomes
A defect flag is correct when the reference decision confirms a defect. A false alarm flags an acceptable unit. A miss clears a defective unit. A correct clear frees an acceptable unit. Put the reference result on one axis and the check result on the other. Label both axes, since software packages vary in how they arrange the cells.
| Fictional reference | Flagged | Cleared |
|---|---|---|
| Defect | 18 correct flags | 2 misses |
| Acceptable | 9 false alarms | 171 correct clears |
Give each rate its denominator
Precision is 18 / 27, or 66.7%. It answers how often an alert is a confirmed defect. Recall on decided cases is 18 / 20, or 90%. It answers how many of the reference defects in those cases the check caught. The false-alarm rate is 9 / 180, or 5%. Accuracy is 189 / 200, or 94.5%. Report the counts alongside percentages, and mark a zero denominator undefined.
The remaining 12 cases matter. If a person subsequently finds six defects in those cases, the operation found 24 of 26 known defects, while the automated flag caught 18 of 26. Those are two different measurements with different owners and staffing demands. Track the 12 as a separate decision state until you can report what happened.
Compare like with like
A study assembled with extra defective units can be useful for finding failure modes. Its precision describes the study mix. A line with rarer defects could send mostly acceptable units to review even with the same recall and false-alarm rate. Keep an operational sample with the usual product mix and use the deliberately enriched set to assess protected defect types. Compare performance on the same cases when choosing between threshold candidates.
Use the confusion-matrix calculator for a first pass and report errors by condition to see where the wrong decisions occurred. See scikit-learn’s metric definitions for the standard terminology.
Put the unit, reference, and decision stage on the report
A useful confusion matrix begins with a statement about what each entry counts. A line can have images, attempts, inspected features, product units, and final dispositions. An image may contain two labels and a unit might need several images. Decide whether the claim concerns a feature finding, an image decision, or the release of an item. Use a denominator whose member can be recovered from the evidence and appears once in the reported population. If both location and unit outcomes matter, report them in separate tables.
Next name the reference behind the matrix. A qualified inspector may follow an approved physical measurement, a verified image review, or another approved method suitable for the requirement. Record the rules and version used. In the matrix, “reference defect” means the underlying product met that definition. Two reviewers observing the same photograph can disagree and both records can be useful. Preserve unresolved reference outcomes outside the four cells until the applicable owner settles them. State how many cases remain unsettled and what the report can still describe.
Identify the moment represented by the output axis. An initial vision flag prompts an action. A final disposition after recapture and manual measurement tells how a combined process handled the unit. The four counts can change meaning entirely depending on that choice. Label the matrix “initial automatic decisions” when that is what it shows. If the decision system has a separate “review” output, report that group and its known reference outcomes beside the binary matrix or represent three output columns. Report final dispositions as a different stage with their own population accounting.
Finally state which conditions are included. List format, visibility, station and date window when relevant. If the report excludes unreadable images, tell the reader how records got that label and how many were affected. A matrix of usable images answers a real question about performance when evidence is usable. The fraction of units that actually reached that stage is a separate fact the owner needs before relying on an operational interpretation.
Derive the metrics from one fictional result
Suppose an approved sample contains 320 settled inspections. The chosen automatic result flagged 24 of 30 independently confirmed defects, and cleared the other 6. It also flagged 12 of 290 acceptable units and cleared 278. Label the four outcomes as true positive 24, false positive 12, false negative 6, and true negative 278. The defect reference total is 30, the acceptable reference total is 290, the flagged total is 36, and the cleared total is 284. Confirm these margins account for all 320 decisions before calculating percentages.
Think about who will use each metric. The quality manager asks how many true defects were identified and needs the recall denominator of 30. Recall is 24 divided by 30, or 80%. The operator asks how often the alert is a real problem and needs the precision denominator of 36. Precision is 24 divided by 36, or about 66.7%. The production lead asks what fraction of acceptable units were held or interrupted, and needs the false positive rate of 12 divided by 290, or roughly 4.1%.
A supervisor might ask how often a clear result was correct. The answer on these decided cases is 278 of 284, about 97.9%. This rate depends on the tested group’s underlying mix, the methods used to establish correct and incorrect finds, and the scope of the cases. It does not settle whether missing six reference defects is acceptable. Present counts by consequence and by relevant operating condition before making a product decision.
Accuracy in the fictional table is 302 correct of 320, about 94.4%. If every unit were released, 290 of 320 would be correct and 30 defects would be missed. That is about 90.6% accuracy, even with zero defect recall. The difference in total percentage hides much of the operational change. An owner judging a protected omission often needs the actual misses, the number of defect exposures, and how the workflow would handle those units.
Consider reporting a metric with no valid denominator. A slice with ten acceptable items and zero confirmed defects can show how often those acceptable items were flagged. Defect recall for that slice is undefined because there are no checked reference defects. A slice with no flags has undefined precision even if the tool did make clear decisions. Use a blank or “undefined” with the underlying counts and an evidence note so a consumer cannot mistake a display default for a measured zero or 100%.
Count review as a real third output when it exists
Add a review column to the previous fictional inspection before drawing conclusions. Imagine 42 additional units are evaluated and sent to a person. Their independent references include 10 defects, 28 acceptable items, and four unsettled cases. The binary decisions still account for 320 units. The evaluated group now totals 362, with 42 initially requiring review. The initial review share is 42 divided by 362, or about 11.6%. Of those reviewed, 38 have settled references. Do not distribute the four unsettled results among established classes without evidence.
Measure the routing effect on confirmed defects. The reference group now has 40 confirmed defects spread across 24 automatically flagged, six automatically cleared, and ten routed to review. On the automatic subset, recall remains 24 of 30. Across all settled reference defects, the automatic flag share is 24 of 40, or 60%. The fraction that automatically clears a known defect is six of 40, or 15%. Other counts or metrics can be appropriate, provided that the report names the exact workflow action and eligible population.
If the policy treats both flag and review as containment, you can count placed in either group. In this example 34 of 40 confirmed defects were initially withheld from automatic clear. Verify that a review status actually kept those units available and prevented them from moving without authorization. A UI state alone does not demonstrate containment. A review item that physically continued before its result arrived is a different operational failure, and the evaluation should account for it.
Next compare final outcomes after the routed units receive their approved inspections. Suppose manual review correctly holds nine of the ten reference defects and mistakenly accepts one. It correctly accepts 27 of the 28 acceptable units and holds one. Four remain unresolved. If you combine settled initial automatic results with those final reviewed decisions, you obtain 33 correct holds, 13 unneeded holds, seven cleared reference defects, and 305 correct clears. Preserve the four unsettled cases and the exact scope. The final table answers a broader workflow question than the initial one and contains the contribution of human review.
This calculation assumes the automatic decisions did not undergo another downstream review that modifies their final statuses. If they did, build the final outcomes from the event records. Reconcile repeated inspections and corrections by unit. Keep enough time context to explain cases still pending at cutoff. A matrix calculated before a shift is closed may contain a large unresolved proportion, and a matrix after the same units finish could cover a changed set of final outcomes.
Compare mixes before comparing results
Consider two stations whose automatic sensitivity and false-positive rate are similar under the same condition. One station reviews frequent setup checks with many intentionally staged defects. The other handles ordinary units where confirmed defects are uncommon. In a fictional example a classifier flags 90% of true defects and 5% of acceptable units on both. In a set of 1,000 units with 200 defects, that yields 180 true flags and 40 false flags. Precision is 180 of 220, about 81.8%. In a set of 1,000 units with 20 defects, it yields 18 true flags and 49 false flags. Precision is 18 of 67, about 26.9%.
The program behavior used in the example is deliberately held constant. The difference in precision comes from the change in prevalence in the compared groups. That is why a demo selected to include many challenging defects can answer whether those error modes are noticed while giving an unrealistic expectation for routine alert quality. Describe how the group entered the matrix and keep extrapolations explicit. A convenience set with unknown selection cannot establish an actual field rate for defects.
You can compare the counts by type and estimate what changes under a plausible mix with a separate hypothetical calculation. If the real prevalence of defects is uncertain, provide several what-if scenarios. Make the estimates conditional on performance and operating conditions. Report known confidence limits around the observed recall and false-alarm rates if appropriate. The numerical prediction is a projection based on the given assumptions. Keep the evidence and the projection labeled separately.
A related issue comes from a product scope change with limited support. If one format has more reviews and comes to represent a larger fraction of the line, overall review rate can grow while each format is stable. If one finish becomes harder to inspect, a lower share of the affected material can make the combined rate look flat. Break the counts down by the condition the owner can act on, then compare within each matched group. For a common mix comparison, state the weights and use consistent groups rather than averaging selected percentages without regard to their denominators.
Read a protected failure separately from an overall rate
Suppose a candidate has 30 reference defects with two misses and the team can identify their underlying cases. One is a minor cosmetic mark and the other is an omitted component that a product instruction treats as significant. A single recall figure groups both outcomes. Keep the protected case list and the operational consequence attached. Label the sample size for the protected group. If the independent set included one protected omission and it was cleared, that specific finding is important even though the overall denominator was larger.
A slightly different candidate can improve aggregate recall while losing a protected case. In a fictional comparison, the previous rule misses four mostly low-consequence faults and catches the rare approved challenge case. The new rule misses two overall, including the protected challenge. Look at the paired per-unit changes before treating the new result as improvement. A change may improve handling for several common examples and still fail a condition the owner must preserve before use. Document this boundary separately from the trend in counts.
Record what evidence could establish a broader claim. Observing no misses in a single small group does not establish that no miss could occur on the next lot or shift. As a simplified illustration, if three independent items were sampled from a stable population with a hypothetical miss chance of 20% each, the probability of seeing zero misses would be 0.8 cubed, or 51.2%. This example depends on independence and constant risk. It illustrates why small groups can show clean outcomes even at an operationally important failure rate. A repeated specimen may have correlated results and provides even less basis for that specific calculation.
Discuss consequence and coverage with the quality owner before widening a check. If rare cases are hard to collect, preserve what the trial actually demonstrates and keep suitable containment for unsupported conditions. Use approved staged specimens for protected behavior, independent representative samples where obtainable, and deliberate reviews after changes. In the report explain which examples demonstrated repeatability and which independently tested variation across units, setups or lots.
Verify the matrix with practical reconciliation
Collect four source records that stand in for the four cells and trace each from a known physical requirement to its observed and reference statuses. Do this before investigating a drift in aggregate performance. A report can silently flip “positive” and “negative” depending on whether the system refers to a positive quality status or presence of a defect. Use explicit labels in the table and a documented translation to the ground truth. Verify what the exporter means for a field like pass, pass count, violation, and review.
Trace the table totals back to the capture and disposition manifests. If 500 unique units entered evaluation and the four cells account for 420, reconcile the remaining 80. They may have missing output, unresolved reference, exclusions due to a capture gate, or distinct review states. Preserve such reasons if they affect how the reader can use the report. The pair of margins in a valid table should reconcile to the same population. Report the population start and exclusions so that someone can reproduce the funnel.
Audit repeated records and missing joins. A camera can retry a capture, and a reviewer can revise a decision after seeing the physical item. Joining an event table and all images only by the unit identifier might multiply rows. Decide which event belongs to the report stage and whether the evaluation counts product units or independent findings. Keep separate counts for attempts when evaluating workload. Ask an independent reviewer to look at a few repeated cases and verify the algorithm selects the right stage and applicability.
Choose an appropriate cutoff. If the reference review occurs on the next shift, avoid treating all cases whose references are still in progress as correct. Preserve the evaluation cohort and label the number pending. Compare metrics on the settled eligible subset and reopen that view when more references are resolved. State whether the completed subset differs by condition. Cases with poor images may take longer to settle, so the in-progress group can be unrepresentative of the remaining work.
Document any correction. A reference may be revised when the product requirement owner clarifies an edge case. Preserve the earlier distinct data version along with the current version and why the change was justified. When comparing candidates, recompute both against the same revised reference and separately describe what the revised data changes. It can be misleading to attribute a gain or loss to software adjustment when the definition of the underlying product outcome changed at the same time.
Use the matrix to start the right case review
For each false clear, begin by recovering the unit, applicable requirement, operating condition and evidence the inspection used. Check whether the relevant feature was fully visible, whether applicable inputs and configuration were correct, and whether the result followed the current scope. Verify the reference again when reasonable using the approved method. A physically valid defect and the software result should be connected before inferring a cause. Preserve the state of the original case even if further investigation changes the next action.
For each false flag, inspect the check’s reason. It might be associated with approved variation, poor crop, a borderline score, or the wrong configuration. Replaying both accepted and known defect examples under the same suspect condition can show whether a change helps. Use a small set that includes relevant controls. A simple plan can state which observation changed, what effect you expect on established accepted and failing references, and what group a new evaluation needs to cover before release.
For consistently reviewed cases, check the usefulness of the routing. A high review proportion can arise because a threshold band is protecting difficult calls, because images are unreadable, or because product identity is missing. Those mechanisms require different comparisons. Use the actual queue reason, final disposition, and handling times. Compare ordinary cases selected outside the queue too. A group identified only by review outcomes can describe that group without representing the entire production population.
Use case detail to understand the cost of errors instead of assigning a single expense without context. A held acceptable unit may require three minutes and a quick extra scan, or it may stop a process for longer if isolation space is limited. A cleared defect might be caught by a separate approved inspection step or reach a stage where correction is harder. Describe and quantify where evidence supports it. Decide whether more reliable evidence or better handling would change the practical effect of the result under review.
Practice interpreting two tables with the same headline
All numbers in this exercise are fictional. Two candidate workflows each have 190 correct decisions out of 200. Both groups contain 20 confirmed defects and 180 acceptable units. Candidate A flags 18 of the defects, misses two, falsely flags eight acceptable units, and correctly clears 172 acceptable units. Candidate B flags ten defects, misses ten, and correctly clears all 180 acceptable units. Both have 95% accuracy. A’s defect recall is 18 of 20 or 90%, and B’s is ten of 20 or 50%. A’s precision is 18 of 26 or about 69.2%, and B’s is ten of ten.
The figures describe different operating outcomes. B’s observed false positive rate is zero on the 180 eligible acceptable units and its alert precision is 100% on the ten flags. It also clears half the confirmed defects. A sends eight additional acceptable units into the flagged path but catches eight more defects. Ask which decisions affect quality and whether those eight interruptions can be handled. Examine the individual misses rather than deciding based entirely on which rate is larger.
Then introduce an uncertain path. Suppose A could send four of its current false flags and one of its current misses to manual review under a revised rule. While those five are unresolved, the automatic binary table changes to 18 correct flags, four false flags, one incorrect clear, and 172 correct clears. It covers 195 units and its precision and recall on automatically decided cases rise. A report must show the five routed cases, including the known defect, separately if the reader is supposed to understand all 200. Whether that policy works depends on final outcomes and the time needed to reach them.
Finally test how the candidate affects one protected case. Suppose B’s ten missed defects include two omissions that the approved instructions require it to detect. Suppose A’s remaining missed defect is an optical case the current station cannot judge on the required view. A suitable next test can compare the routing prerequisites and visibility with independent examples. Apply a comparable set of physical rules and state which unresolved risks require other evidence or an operating restriction.
Write a report that can be read on the shop floor
Include a title naming the exact question, supported scope, test dates, deployed evaluation configuration, and decision stage. Label the two axes in words. Give the counts and metric numerators and denominators. Beside them, describe the reference procedure and the sample composition, including staged, routine, reviewed or excluded units. Include the condition comparisons needed for the next decision and show any evidence gaps. Keep count units consistent even if the station sends multiple observations or scans for each item.
Ask for one explicit decision about the system or pilot. It might advance to a limited scope, stay under the current process, or extend an evaluation in a clearly named condition. Give the decision owner the specific misses, false flags and review facts relevant to the question. Assign a next comparison to each major unresolved group. A report that ends with a percentage alone leaves the next owner to guess which mechanism matters.
Use the confusion matrix calculator after reconciling the four cells. It provides repeatable arithmetic for settled cases. Keep the references and additional decision groups alongside its output. Where the work needs a review column and workflow-stage comparison, draw those extra states separately, then use the calculator on each valid binary subset. The result is useful when the owner can answer what was tested, what happened to the exceptions and where the matrix has sufficient evidence to guide the next action.
Calculate operational impact over a bounded window
A production lead can translate observed rates into conditional counts for an upcoming run. Start with the expected number of units that actually fall in the approved scope. Use observed or supplied estimates of the mix, and label any hypothetical values. Establish which outputs interrupt a station, require a second view, or need an approved manual inspection. Apply observed counts or rates within relevant conditions. Leave unresolved prevalence and support gaps visible rather than projecting through them silently.
Suppose a fictional line expects 2,000 of one shape under the measured setup. The team explores a hypothetical 2% true defect share, 90% recall, and 4% false alarm rate on settled automatic cases. If you apply those values without a separate review condition, 40 units are defective, 36 are flagged correctly, and four are missed. Among 1,960 acceptable units, about 78 would be flagged and 1,882 correctly cleared if the rates held. That creates approximately 114 flags. The underlying observed matrix or another reference needs to support each metric. Treat these counts as a scenario, with uncertainty around both the rate and expected groups.
Then consider staffing. If the approved process uses two minutes of attention per flag, 114 flags would imply about 228 minutes of work across the full run. If some alerts lead to a longer physical examination and others are resolved from the image, separate them using their expected times. This calculation does not tell you which part of the shift can absorb the work. Examine how often flags cluster after a material change or stop. The total count can fit within available staff-hours and still require an actionable overflow procedure during startup.
Consider what happens when one assumption changes. With the same performance rates and a 1% hypothetical defect fraction, the 2,000-unit example would contain 20 defects, about 18 true flags, and roughly 79 false flags. The total flagged count changes only moderately, but the valid alerts make up a smaller fraction. If a view change halves the false alarm rate while preserving defect outcomes, a separate validated scenario can estimate lower workload. Program owners should check that claim with actual data before using it as evidence to extend rollout.
Read uncertainty alongside exact numbers
Percentages computed from a sample are descriptions of the observed cases. A broader claim requires reasoning about how the sample was selected and what uncertainty remains. If units were selected independently and represent the target scope, more of them generally provide stronger evidence about frequent outcomes than a few convenience examples. Repeated captures of the same unit can test consistency under the allowed conditions but do not establish independent variation across units. State which evidence serves which purpose.
At a minimum, put the raw counts beside every important rate and make small groups apparent. A fictional result of 100% recall from two reference defects carries a very different evidence base from a trial with 200 independently confirmed defects and zero misses. Both computations are numerically correct on the selected sample. Add coverage of product format, material, station and time where relevant. Keep staged protected tests visible for severe but rare cases, and avoid confusing a deliberately challenging group with a prevalence estimate.
If the decision depends on formal uncertainty bounds, work with a qualified owner who knows how the sample was collected. Intervals based on independent observations may be misleading for images grouped by production run or physical specimen. Sampling only cases surfaced by the current model also conditions the analysis on its behavior. A clear statement about coverage can be more useful than a technically sophisticated number applied to an unsuitable sample. When evidence is thin, identify the independent comparisons the next evaluation should collect.
Run a hand calculation before trusting an export
Ask someone who has access to the raw case IDs to create a small example file. Include a true defect flag, an unnecessary flag, a missed defect, a correctly cleared item, an independently verified review outcome, an unresolved reference, and a duplicate image belonging to one of the same units. Record the intended stage and single identifier for each table contribution. Before running the report, have a second person produce the counts by hand. A deliberately engineered fixture exposes mistakes that may be difficult to see in thousands of routine rows.
Look for issues that visual formatting can hide. The labels on the predicted and reference axes may be swapped. That mistake leaves accuracy intact but can invert the interpretation of precision and recall. A dashboard may label the rate by columns while dividing by rows. An export may convert missing reference decisions to negative, increasing the apparent number of acceptable units. A row group can accidentally include a child image for each view of one unit. Validate the calculations using known IDs, not only a screenshot.
Repeat the small reconciliation after changing a data source, processing version, or scoring policy. If the decision statuses changed, update the report’s mapping explicitly and replay the earlier fixture to see whether historical interpretations remain valid. Give the person approving the release a direct readout of affected rows and cite representative examples. When the report uses cutoff time and includes later resolved items, write the time convention next to the data. Include exactly which exclusions and pending decisions are represented.
End the review with an actionable summary. Identify the protected findings, false alerts that affect staffing, missed cases that need follow-up, and the conditions with no usable denominator. Give the receiving team the underlying links and the right calculation for the next comparison. If the two table directions or workflow stages were unclear during the exercise, rename them and rerun the example. The matrix should tell someone where to go in the evidence, how to recover a specific error, and which conclusions remain conditional on further sampling.
