Evaluate a visual inspection pilot by comparing its decisions with independently reviewed ground truth, applying limits you agreed to before the run, and checking how errors cluster across actual conditions. Give missed defects, false alarms, uncertain frames, and unusable images their own counts. A pilot report should tell a quality owner what to do on the next shift.

The examples in this guide describe a fictional paper-carton line. The numbers demonstrate the decisions and calculations. They do not describe Shangil results or the performance of any particular software.

Write the decision the pilot must support

Begin with a narrow release question. A packaging team might ask whether a camera at its first labeling station can identify orange labels that sit outside an approved window on the front face of two carton sizes. That names a feature, view, place, and limited assortment. A request to evaluate “packaging quality” leaves room to accidentally treat scoring label location as proof of legibility, count, seal integrity, or the correctness of printed text.

Write three possible decisions before anyone sees the output. A go decision could allow the check to recommend routing in one supervised configuration. An extend decision could request additional samples for one finish while retaining the current process. A stop decision could keep manual review if the viewpoint routinely loses the relevant edge. Specify the people who can make each decision and what observations they can accept. A software operator can explain a capture failure. The production owner must confirm that a second look fits the operational handoff, and the quality owner decides whether the remaining miss risk is tolerable.

Separate the reading of the image from what happens to a physical carton. For example, a software result can identify a visible displacement. A supervisor may still have to decide whether to hold the unit, inspect the label on the line, or accept a documented deviation. Keep that action policy visible; otherwise the same evaluation can appear successful to the software team and unusable to the operator.

A complete decision statement usually names:

  • The station, camera, part, view, and permitted product versions.
  • The visible criterion and the exact label or rule version used to judge it.
  • The output states, who can resolve an uncertain image, and what happens when evidence is missing.
  • The intended role of automation in the next stage, including whether a person still makes the disposition decision.
  • Evidence required to advance, conditions that remain under the existing method, and a trigger for reopening the decision.

For the carton line, the initial decision could read: “Evaluate one supervised check for front-label position on matte cartons at station A. A camera result may suggest review. A quality reviewer still resolves defects and unreadable frames. Consider including the reflective finish after separate coverage.” This gives the supply and tooling teams a way to ask whether a new varnish changes the scope, rather than leaving the conclusion buried in a spreadsheet.

Decide how reference outcomes are assigned

The reference outcome is the decision your evaluation compares against. If the pilot checks physical label placement, the reference should be grounded in the physical requirement and the best available evidence of where the label actually sits. A prior operator action can be a starting point, but it may reflect a different rule or a judgment made from another side of the carton. Ask what evidence that person had at the time.

Create a reference rubric that distinguishes accepted, defect, uncertain, and unusable. Accepted examples show the necessary edges inside the tolerance. Defect examples show an edge beyond it. An uncertain sample shows visible evidence that deserves a second measurement or adjudication. An unusable capture omits a necessary edge or obscures it completely. For the pilot, retain the physical outcome when it is independently available, and record image usability separately. This allows an unreadable image of a known defect to count toward the capture and review problem instead of losing it from the record.

Have two reviewers make independent decisions on a small, mixed portion of the set before the main labeling pass. Include routine passes, visible defects, and genuine boundary cases. In this fictional review, a tilted image gives one reviewer the impression that the label is skewed. The other reviewer uses the carton’s left edge as the reference and accepts it. The disagreement identifies a gap in the instructions: the rubric must say whether position is measured relative to the image frame, fixture, or carton. With that resolved, revisit other frames captured at an angle using the same definition.

Keep a record for each contested outcome. Include both initial decisions, the visible evidence each reviewer used, the source or measurement that resolved the difference, and the final rubric version. If the dispute remains open, exclude the case from the binary reference denominator and retain it in a separate adjudication queue. Indicate how many cases are affected and whether the unresolved examples cluster by material, station, or requirement. A hard set can look easy after its difficult cases are silently removed.

Do not show model scores or candidate outcomes to reviewers assigning ground truth. If you need to reconcile a model error later, first preserve the original reference and then open a separate adjudication record. Correct a genuine label error through a documented change and rerun the affected comparison. An update to the reference is different from a change to the candidate system, so give the two their own timestamps and versions.

Build an evaluation set around actual conditions

Agree on the condition list before collecting examples. The key question is whether a material, station, camera setup, time window, package format, or handling pattern can change the visible evidence. In the carton case the matte surface, glossy film, slight package tilt, and partially masked edge all deserve explicit representation. Season and shift are meaningful only if they stand in for a real change such as a lighting fixture drawing daylight, operators using a different jig, or a different product mix.

Make two sample lists with separate purposes. A sample that reflects the ordinary volume mix helps estimate review workload and false alarms under everyday use. A focused set of uncommon defects tests whether the system detects examples that might be infrequent during the same observation period. Label deliberately added examples and describe how they entered the set. Counting a defect-heavy sample as if it were representative would overstate the number of cases a shift will see and can also mislead someone reading the share of correct flags.

Retain the unit of evaluation. Five near-identical frames of one flipped carton tell you something about repeated decisions and line behavior. They still represent one physical carton for a trial asking how many units are processed correctly. Preserve the repeated frames when the question is retry behavior or whether the second view brings genuinely new evidence. Store a linked unit identifier in both cases, so people analyzing the set can group it properly.

Choose useful evidence a shift can reproduce. For the fictional line, collect examples from two runs of each permitted size, including the leading and trailing ends of a run. Record whether the same light and fixture were in place and whether changeover occurred. Capture centered labels as well as full labels shifted in each direction, a small set that straddles the boundary, and obscured edges which force human review. Report which conditions did not occur during the pilot rather than implicitly assigning them to an “other” group.

At capture time, record:

  • Physical unit or batch grouping, and the relationship between alternate views.
  • Material and format in terms reviewers can recognize during a shift.
  • Station, fixture revision, camera view, and lighting configuration.
  • Reference decision status, and whether the decisive feature was visible.
  • Whether the case was observed in normal operation, deliberately introduced, or generated for a narrow visual demonstration.

A representative set does not require every imaginable bad image. It needs enough realistic coverage to answer the approved decision question and a mechanism to isolate what it omitted. Ask a line operator to name the top exceptions they expect to see at volume. Ask the quality lead which missing examples would prevent a release decision. If those answers differ, test both viewpoints and surface the remaining tradeoff.

Keep training and final evaluation independent

Group images by physical unit or tightly connected capture session before assigning them to training, tuning, and final evaluation. A video burst can produce dozens of nearly identical pictures. Assigning those pictures independently can put one view of a carton in training and a barely different view in final evaluation. The result then answers a question about familiar examples, even if the image files have different names or crops.

Use one round of data for development and threshold selection. Reserve a different round for the final decision. The final set should be locked before running the candidate, with an exportable manifest containing unit identifiers, conditions, and reference versions. If someone tunes a setting after seeing results for the final set, record that it was used for tuning and collect a fresh comparison for the decision. This follows the separation between development and independent testing described in Google’s dataset splitting guidance.

The grouping rule should reflect the risk that approved conditions will change. When the expected change is a new carton lot, reserve lots that the pilot has not used to adjust settings. When a second camera position is likely to be deployed, record it as its own condition and collect a separate set for that position before extending scope. If different sizes share graphics and shape, keep enough cases for each size to see a weakness near its alignment window.

Check the manifest before scoring anything. Confirm that there are no repeated unit identifiers spanning training and final evaluation, and spot-check visually similar cases for common parents. Compare image counts and unit counts by condition and record any excluded or replaced record. Use a new version for all corrections, including removing a mislabeled or duplicate file. A test may still be useful after the correction, but reviewers need to know which version produced earlier numbers.

Sometimes the team has only a handful of clear examples of a rare defect. A narrow pilot can still explore whether the feature is visible, establish review instructions, and identify what further evidence would be useful. Label the result as feasibility evidence. A small defect set cannot justify strong statements about rare failures, especially if it misses important finish and lighting conditions. In that case the decision owner can use targeted capture and preserve the existing response while the team learns more.

Read the counts before reading an average

Begin with raw counts and denominators. Use “defect” as the positive class for the following fictional result. Of 400 usable photographs with resolved reference decisions, 30 show a label outside the window and 370 meet the rule. The candidate flags 26 of the defects and clears four. It also flags 14 acceptable images and clears 356. Record additional pictures that went to human review or recapture separately, along with the decisions made after review.

Candidate decisionReference defectReference acceptedOutput total
Flag for defect261440
Clear4356360
Reference total30370400
Fictional 400-image test for carton label placement. Matrix totals exclude images pending review or recapture.

Recall is 26 divided by 30, about 86.7%. It represents the share of reference defects flagged in this binary evaluation. Precision is 26 divided by 40, or 65%. It reflects how many flags corresponded to a defect in this set. The false-alarm rate is 14 divided by 370, about 3.8% of reference-accepted examples. Overall accuracy is 382 divided by 400, or 95.5%. Read the definitions of classification metrics and work through the same example in the confusion-matrix calculator.

The same set supports multiple operational questions. The quality lead cares about the four errors that cleared a reference defect and wants to know if they look like each other. The station owner wants to know where the 14 false alarm cases go and whether they can be reviewed immediately. The engineering team needs to know which input conditions had a part in either error. The general number can be useful in a dashboard but it cannot settle any of these decisions alone.

Now suppose a separate 50 photographs were received during the same test. Thirty were ambiguous enough to require a human look. Twenty were unusable and needed recapture. The initial binary result covers 400 of 450 original samples. The other 50 have their own operational impact. If one goes through review and receives a correct final outcome, keep both the initial and final decisions. If you overwrite the intermediate result, a retrospective will miss the real effort and underestimate the load on the shift.

When some cells are empty, use exact labels. If a condition has no reference defects, recall in that condition is unavailable. If it has one and the candidate gets it right, say “one of one observed”; withholding broader claims preserves the value of the single observation. A report can pair small cells with kept examples and with a specific evidence request. That makes the next capture purposeful and the current limits legible.

Look for the condition hiding in the mistakes

Review the whole error set when the count is manageable. Put the original image beside the selected region, candidate decision, reference decision, and capture metadata. Avoid assigning causes from a filename or from the fact that a score was low. Use an example that lets an experienced line operator confirm what is visible.

In the fictional run, imagine three of the four cleared defects come from reflective film on the smaller carton. Two were captured on a shift when the light had been moved closer to reach a neighboring station. The remaining clear was captured on a matte sample at the normal setup, with a real label error close to the lower tolerance. These are different cases. Test the glossy visibility problem with a fixed light or a second camera view. Examine the matte case near the boundary against the actual tolerance and reference measurements.

Read both numerators and denominators by condition. Suppose the reflective film contributes six defects and the matte cartons contribute 24. Three misses among six reflective-film defects is a different result from one miss among 24 matte defects. If the team tested only six reflective-film defects, it has useful negative evidence about the current setup and a limited view of the full failure distribution. Report the six samples and attach examples; do not jump to a confidence claim about every reflective lot.

Review one pass from the same setting as each group of misses where possible. A clear reflective example may show that glare becomes decisive only when the edge faces the light at a specific angle. It helps a reviewer falsify a coarse diagnosis such as “all glossy material is too hard.” A visually similar accepted package can also reveal a reference issue. Compare the same label position under each setup before concluding that the tolerance or software needs a change.

Keep at least four kinds of outcomes in the post-test work list:

  1. A genuine error under a covered condition that needs a fix or scope decision.
  2. An example outside the documented release conditions that needs routing and perhaps a follow-up pilot.
  3. An incorrect or disputed reference that needs quality adjudication.
  4. A capture without the evidence required to make a call, which needs another view or a supported human path.

One person may own several categories, but each category needs an action. A log entry that simply says “model issue” leaves everyone guessing whether to collect images, adjust the setup, or correct a label. Assign the next check and a due point tied to the shift or pilot review. Save the before-and-after evidence so teams can confirm a change through the same case rather than through memory.

Review workload where it actually occurs

A queue that looks modest on a daily average can overwhelm a shift immediately after changeover. Convert rates into expected review minutes, then test the routing when the inspection receives a burst of similar images. Count forwarding and context switching as part of handling time when the reviewer has to open a record, pull a second view, or walk to the equipment.

Suppose the first supervised station sees 1,200 cartons per shift. The team estimates that about 4% will require a separate manual look in the ordinary product mix. That corresponds to 48 reviews. At 35 seconds of active handling each, the expected hands-on review work is 28 minutes. Those estimates are useful for staffing discussion, but they leave out waiting and the clustering of cases. If eight unclear frames appear in ten minutes during a changeover, a queue owner needs a way to identify, hold, and resolve those particular units without mixing them with later cartons.

Time reviews using the same information the operator will have after release. A quiet review room with prepared image pairs can underestimate turnaround on the line. Include ordinary interruptions in the trial when you can. Capture separately the time to open an example, time waiting for a second image, and time spent on the actual decision. That distinction identifies whether to simplify the reference screen, improve capture, or revisit the queue owner.

Write down what happens if review is unavailable. A lead might defer the recommendation and continue the existing check, or temporarily hold affected units according to the team’s existing process. The appropriate response depends on the consequence of the decision and the capacity of the station. Confirm the fallback in a walkthrough, including where a shift finds the latest instruction and who can clear the held item.

Estimate the workload from an ordinary sample mix. A curated challenge set overrepresents rare defects or hard images and is useful alongside that workload estimate. Use the actual station counts and a measured sample of handling times when available. In the report, state whether review rates in each condition come from production-like sampling or a targeted capture. Link the workbook or issue summary showing how the counts were calculated. The inspection workload calculator converts a supplied rate into estimated minutes and gives you a quick way to compare alternatives during scoping.

Test the whole path from image to disposition

Once the offline evaluation evidence is available, walk a small set of units through the intended handling process. Put the original station result, reference label, final disposition, and event times in a shared record. This is where a software result turns into a practical inspection outcome.

Test a normal clear, a real defect, an obscured view, and a repeated trigger. In a live process, an operator can move the wrong carton if the decision screen and station buffer disagree about which unit is current. Capture the identifier and sequence in your test record. Ask the reviewer to open the example without coaching and indicate how they know which item the decision concerns. It is much easier to find this problem in a rehearsal than while a station is processing a surge.

Test equipment changes that really occur. An operator might rearrange a fixture while clearing a jam, disconnect the camera for maintenance, or run a manual check during a short interruption. For each case record what the user sees, where the original image or failure appears, and whether a new image is retried or logged as another event. Reproduce the routine steps with supervisors from the shifts that will operate the check so the instructions match the way they work.

If the pilot includes a link to another system, trace one example to each interface the release will depend on. Confirm which field carries the unit key, required result code, capture time, review status, and rubric version. If an interface responds later or cannot receive the result, record what remains actionable to the person running the check. Keep any integration outside the stated pilot scope separate from claims about what has been validated.

Note the exact conditions in which the candidate is allowed to guide a decision. If no system evaluates image quality reliably during this pilot, keep the review instructions visible and verify that the operation has a practical way to handle unusable images. Set a follow-up action for conditions outside the scope before collecting more data. Some teams use a short supervised period to check interactions and capture reliability before changing any operating responsibility. The launch decision should describe what those people actually do.

Make a release, extension, or stop decision explicit

Return to the written decision statement. Read the evidence against the agreed limits and present the result by scope. A fictional team might accept a supervised matte-carton pilot at its first station while collecting more examples under reflective conditions. If the measured review workload exceeds the available staffing, it may instead retain the existing process and rerun a targeted test after changing capture. The reason belongs beside the decision.

For each possible next step, make the evidence and constraints visible.

  • Release within a narrow scope. Name the condition, candidate version, review rule, human owner, and date of the decision. Record the way an operator should respond when a sample falls outside the scope. Confirm that the previously accepted process can be used while an unexpected issue is investigated.
  • Extend the evaluation. Name the missing evidence, the target condition, and the question the next capture is meant to answer. A request to “improve recall” is too open-ended. Ask whether the same six reflective examples are visible with an adjusted light, then compare that setting with a separate held-out set before making a scope decision.
  • Stop or pause the check. Identify the observed blocker and the action owner. Record the applicable prior process. Keep the labeled cases so the team can reconsider the check if its capture conditions or decision requirements change.

Make the decision clear enough for another shift to reproduce. Add a copy of the rubric with marked edge examples and a short worksheet listing the evidence expected for release. Include conditions where the current operator must make a direct judgment, which examples need a second image, and the person to contact when the review is unresolved. Explain how a problem can reopen the decision; an increase in recaptures after a finish change is one plausible trigger in the fictional example.

Use a one-page decision record plus linked supporting evidence. Include the source image manifest, pre-agreed limits, result tables for each covered condition, open exceptions, candidate and reference versions, and approval names. Keep the raw counts with the reports rather than screenshots only. Record actual dates when measurements and approvals were made. Because the release decision rests on those inputs, anyone revisiting it should see where a limit came from and how a later update changed the evidence. The pilot acceptance worksheet provides a place to document required evidence before the run.

Schedule review and preserve the earlier baseline

A pilot creates a starting point for monitoring. Preserve a small set of representative reference pictures that the team can reproduce after a shift changes the lights, camera position, or packaging material. Keep notes on controlled conditions and on difficult images where the expected action is human review. Save the measured results tied to the date of the original run. Treat each condition as its own comparison when a change affects only part of the line.

Decide who sees the weekly or shift-level record and what they investigate. A jump in review share may reflect a new material, more units from a familiar difficult condition, a capture problem, a newly strict human reviewer, or a change to the candidate. Compare examples, denominators, and capture metadata before attributing the shift to performance. Check for changed throughput and outstanding queue entries at the same time. A rise in measured review counts with flat review share can have a very different explanation from a change in the fraction of images that need review.

Use a short list of observable triggers. A new packaging material goes back to the material owner before covered scope changes. A repeated missed defect escalates to the quality lead and pauses any affected decision until the owner evaluates the evidence. A camera movement leads to a baseline capture comparison before new settings are accepted. An unusually long review queue goes to the shift owner with the image and unit keys needed to resolve it.

Check which parts of the process a change actually touches. A new camera setting can require revisiting usable-image rates while keeping the same written decision rule. A changed tolerance requires revisiting reference labels as well as any candidate outputs or thresholds. An additional packaging format calls for representative images of the new format and a regression check for the forms already in use. Distinguishing these changes helps the team maintain the parts of the workflow it trusts and direct new collection where it matters.

Finish by naming the first action for the current pilot. Select one recurring check and one station, find the current written requirement, and ask the people who own the result to agree on the error limits and the handling of obscured images. Collect examples that let them resolve real disagreements. Use the next test to answer whether the check can make the specified decision under the stated conditions, and leave a record that the next reviewer can read without being in the room.

Rehearse the sign-off with a second reviewer

Use the same worksheets in a review meeting before involving the next shift. Ask someone who did not set up the pilot to reproduce one row of the findings from the image, the written reference, and the decision log. For example, give them a cleared matte carton near the lower tolerance and a reflective carton routed for review. Let them identify which section of the rubric applies and which field in the results captures the unresolved condition. If they cannot find the two decisions unaided, improve the record before making it the basis for changing a production step.

In the fictional carton evaluation, the second reviewer finds that three records share a shorthand note of “glare.” The first image has an unusable edge and no alternate view. The second has enough information after a reset, with the second capture attached. The third looks bright but still shows both edges and was scored by a person. One shorthand can hide three different actions. Split the notes into exposure obstruction, resolved by recapture, and readable under the approved rule. Ask the station owner to check whether that distinction will be meaningful in the next shift handoff.

Ask the reviewer to explain one limitation to the project sponsor. They might say that results represent only one station, the primary view, and the specific packages observed in the run. They should also be able to explain what a repeat run would collect before extending the scope. A useful limitation includes a linked observation and a way to test it. “More data needed” is less helpful than “collect disjoint groups of both reflective lots with the revised light and measure how often the lower edge remains visible.”

Close the meeting with a brief readback. The reviewer reads the authorized scope, which cases still need manual review, who can pause the check, and which signed record describes the previous process. The quality owner then reads one example of a reason to reopen the decision. Retain the result alongside the test evidence. A good pilot handoff leaves that information accessible when supervisors who were not in the meeting need to explain why a result was routed one way.

Use one follow-up rehearsal to verify the updated record. Ask a shift lead to pick a representative item, route it through the agreed process, and retrieve the resulting decision evidence. Complete the same exercise with an unreadable image or an out-of-scope package. Compare the actual actions and records with the pilot’s stated boundaries. The test is complete when the operating team can see why the chosen actions follow from the evidence and where an improvement would need another review.