Write inspection annotation rules around a visible feature, examples on both sides of the decision, and an explicit route for genuine uncertainty. Test the rule with independent reviewers before applying it to the whole collection.

Describe what belongs in the label

Name the unit, feature, and acceptable evidence. A garment-tag example might treat an off-center tag as a defect while accepting a small wrinkle at the opposite end of the package. An annotation that says only “poor quality” buries the distinction. State the written requirement, the visual clue reviewers can identify, and the lens or view under which it is visible.

Define the annotation type that matches the job. An image-level class can record whether a visible part is present. A localized region can retain the position of a particular mark. A more detailed outline can be necessary if downstream work relies on measuring its boundaries. Choose the simplest representation that delivers the evidence someone will use. Different labeling types and their purposes are described in LandingAI’s project types documentation.

Pair each failure with an accepted look-alike

During training for annotators, compare a clear defect, an accepted look-alike, and a case that cannot be resolved. Put the examples on the same page of the rulebook and use short notes pointing to the decisive evidence. If the accepted example contains the same apparent mark, explain which condition makes it acceptable. If the answer relies on the photograph’s context, point out where that context comes from.

Example labelFictional image evidenceReviewer action
AcceptedLabel edge sits inside the window, with a nearby creaseMark accepted and retain the crease as a look-alike
DefectLabel edge clearly crosses the placement windowMark defect and note the crossing edge
UnsureLabel lies at the tolerance limit in a tilted viewRequest an independent measurement
UnusableThe image omits the relevant carton edgeRecord the capture problem and request recapture
Fictional annotation rubric for a carton-label check.

Do not ask annotators to invent a precise class when evidence is missing. Include an unsure state and specify who resolves it. If a photo is unusable, track that reason separately. Where the model needs a binary class, preserve the original human state and resolution in the source metadata.

Calibrate independent reviewers on a small set

Give two reviewers a shared pack of examples with the same instructions. Ask them to label it separately. Start the reconciliation meeting with cases where they disagreed and compare the exact image regions each used. The point of the exercise is to find an instruction that fails under realistic working conditions, not to force agreement by majority vote.

Suppose two reviewers interpret “edge beyond the guide” differently when the carton rotates. Add a short rule naming the product edge, then select a fresh pair of tilted cases to check whether it resolves the disagreement. Keep the old and new rubric versions and record which existing labels may change. If you rewrite the meaning of the class, review affected cases rather than silently reusing earlier labels.

Keep the rule version with the evidence

When a reviewer fixes a label, retain the original decision, reason, and revised version. Give a contested example a resolution record or a named owner. In evaluation, use the frozen reference file without exposing candidate outcomes to the reviewer. For broader quality practices, see the reviewer disagreement guide and annotation brief builder.

Before labeling additional images, ask someone who was not present at calibration to apply the latest rules to each example. Make sure they can decide which examples require a human check. Update the brief with the tested rule, its limits, and a link to the source examples.

Begin with an observable reference decision

Write one annotation goal in plain language. Define the unit, the piece of evidence, the accepted range or classes, and the work that follows the decision. For example, the first check may ask whether a carton label extends below a defined boundary on the front face. Its rule names the visible label edge and a datum visible on the same face. A separate check can ask whether the label contains the intended print. Keeping those two outputs separate prevents missing print from being learned as evidence of poor placement.

Decide whether the reference is a physical product decision, an image decision, or both recorded separately. A physical measurement can show the defect even when the camera cannot see the edge. In that case, save the physical result and mark the image unavailable for this rule. The physical result can help improve capture and assess downstream handling. An annotation claiming the hidden edge has a specific location invents evidence the visual reviewer cannot recover.

Identify the authoritative product requirement. The annotation brief can make its interpretation repeatable, but the project should consult the person who can interpret the requirement. If different formats use different reference edges or tolerances, specify which rule applies to each format. Include a way to identify the format without relying on the predicted class. Agree who owns the decision when the format cannot be established. An “unknown format” label can protect the evaluation from a case that a person can read visually but cannot place inside an approved operating scope.

Choose the annotation grain deliberately. An image can have one caption, several instances of a part, a measured point, a polygon around a visible defect, or a row for a physical unit. Use the simplest representation that supplies the information the check needs. If the inspection is only whether a clip is present at a named location, the absence of an annotated box needs a clear meaning. It can represent confirmed absence, an unfinished image, or an occluded location unless the brief defines explicit states. The annotation brief builder gives you a place to record the target rule and its example set.

Separate the visual boundary from the action

Define the visible evidence first, then the result for each relevant condition. A scratch classification may depend on length, location and whether the surface is finished. The disposition can depend on whether the scratch sits under another part later or is visible at final inspection. If annotators draw around all scuffs while the quality decision rejects only scuffs in a defined zone, preserve location or a separate requirement label. A single “bad” class makes these distinctions hard to recover.

Write the exact treatment at the boundary. Suppose a fictional carton drawing specifies a placement limit of 3 mm and accepts equality. A physically measured displacement of 3 mm is accepted under that rule. A display with 1 mm rounding might show 3 mm for a case whose real position lies just outside the window. Ask which evidence reviewers should use and which cases they should escalate. Document a named method for measuring and interpreting uncertainty. The numeric acceptance limit, measurement resolution and reviewer route are three distinct fields.

Give each label a definition people can apply without a long chain of interpretation. For visual location, say which edge to trace where the label lifts or curls. For a multi-part assembly, define whether a partly visible fastener counts as present only when a specific portion is visible. For a surface mark, document whether the marked region includes the shadow of the scratch or only its visible boundary. Pair one expected annotation with one acceptable look-alike where those rules would yield different results.

Keep the required evaluation outcome visible beside the annotation. An example can show a displaced corner with a correct drawn box, yet the actual decision still needs a measurement against the datum. If you cannot derive the intended decision from the annotations, adjust the brief or add the missing reference fields. The labels should support the full question the check will be asked to evaluate. Use the label-placement guide for a worked position example.

Use reference pairs to define accepted variation

Start with the common cases that can be easy to overlook. An approved logo revision, a different printing lot or a standard texture can resemble a defect. A fold line inside the expected area can look like a scratch. If accepted examples all come from a clean demonstration set, the team may struggle to describe ordinary appearance. Ask the station and quality leads to nominate real acceptable variants and identify their applicable requirements.

Pair one failure with an accepted unit that shows the competing feature. For a fictional surface mark on brushed metal, select a machining streak and a scratch that looks similar at a first glance. Have a person compare how each mark behaves in the approved illumination and whether it crosses a visible reference area. Give annotators both images with a short description of the feature they can use. They can then explain why a mark was accepted from visible evidence, even if they have not seen the original two pieces.

Use variation within a class as well as variation across classes. A label that sits within its allowed region can move horizontally, vertically or around a small angle. A check prepared on only perfectly centered labels can give reviewers the impression that any deviation is a fault. Include accepted examples near more than one edge of the window and in the ordinary fixture pose range. Keep the intended reference disposition attached to each example. A known good demonstration unit is useful for calibration; it covers one point in the operating range.

When you cannot find a paired failure in production, ask the quality owner whether a prepared physical example is useful. Label how it was prepared and whether it resembles an actual known failure. The prepared case can support training and testing of one visual requirement. Keep its collection basis separate so someone later can estimate routine review or defect frequency from appropriate evidence. If the simulated defect contains a highly visible marker absent in normal production, move it out of the inspection view or record why the case only illustrates a training exercise.

Distinguish a difficult class from unavailable evidence

Define states for visible, borderline, unavailable, and unresolved inputs based on the procedure the team can use. An example may be borderline when the relevant feature is visible and close to a meaningful limit. An image is unavailable for a check when the required evidence cannot be observed. A reference can be unresolved when two appropriate evidence sources disagree or the relevant specification cannot be identified. Use status fields that distinguish these conditions from the accepted and defect classes.

Consider a fictional assembly guide with three pictures. In picture one, a clip is fully visible in its expected opening and is the correct shape. Picture two shows an open fastening point without a clip. Picture three shows a grasping tool covering that same location. The annotations may use two of those pictures as settled reference cases. The third is evidence for the uncertain path. If a second view of the same unit or a physical observation exists, save it separately and link to the original. Make it possible to reproduce what the inspection saw when the first review happened.

Test whether the image tool supports the distinction. A tagging interface may offer mandatory “accept” or “defect” options but allow a reviewer to save a separate exception reason. Verify that an exception cannot accidentally be exported as a labeled negative. If the system represents exceptions through exclusion from a training set, keep a second manifest listing which cases were excluded and why. Retain enough linkage to include these cases in the workflow assessment where they belong.

Check which reason reaches the operator and which is reserved for the annotation team. The station may need only “Get a second view of the fastener position.” A reviewer can record that the area is obscured by a bracket, and the analyst can investigate whether a new fixture caused it. Keep those levels related. The end-of-shift report can then show when the team routed cases successfully and which station improvements might reduce the rate. See the image usability guide for example reason categories.

Select shapes that convey the evidence

Give annotators a consistent mapping between target shape and use. A box that nearly fits a visible label may be adequate when the goal is localization for a later crop. A measurement of its lower edge may need well-defined points or a contour. A segmentation boundary for a light scratch may require a clear definition of where it begins and ends. Describe what the marked region represents and how to handle parts hidden by folds or another item.

Check the tool behavior around image transformations. Stored polygon coordinates may apply to an export that changed orientation, resolution or crop origin. Save the raw image reference and visible transformation with the annotation. Give one known sample to two reviewers and export it through the actual pipeline. Open the original and reapply the saved marks, confirming that they land on the same feature. A successful save dialog does not establish that the coordinate frame is correct.

Test partial visibility specifically. Suppose a label crosses a crease and a second face carries the continuation. An annotator who draws a smooth rectangle over the missing corner imposes an inferred location. If inference is allowed for a data purpose, state it and mark it clearly. If the measurement requires a visible corner, preserve the occlusion reason and use another image or reference. Write down whether a human estimates location, marks only the visible portion, or withholds a measurement. Compare accepted cases with minor naturally curled edges, then test combinations where the original rule needs a second view.

Keep the grain of annotations connected to the counts later reported. Picking an instance inside an image is different from judging the entire carton. If a unit has an upper and lower label, record which label each point applies to and how their outcomes combine in a unit result. An evaluation that divides by all labeled objects can hide a unit for which one missing object requires a separate route. Preserve image, object and unit identifiers in exports so downstream reviewers can calculate each denominator.

Run calibration before assigning a large batch

Create a small collection deliberately designed to expose disagreements. Include common clear cases, accepted look-alikes, examples right around the failure boundary, unusable views, and ambiguous product variants. Ask reviewers to work independently with the same reference photos and versioned rules. Collect their decisions, evidence region, reason and elapsed time. Note which tool view they used and whether they could access the original image.

A fictional ten-case calibration makes the separation concrete. Both reviewers agree on five obvious passes and two clear defects. On one boundary example, they agree on location and disagree about whether equality is allowed. For a second case, one reviewer uses the right corner of a folded label while the other uses the shadow. In the final case they both infer a defect in a fully blocked location. Their agreement is useful evidence in the first seven cases. It does not solve the three independent reasons for trouble. Involve the rule owner for equality, clarify the visual target where corners and shadow differ, and route the blocked case to the approved evidence path.

Do another round after revising the rule. Use new cases that require the same interpretation. A reviewer might remember the results of a short calibration set and repeat them without understanding the change. Ask them to identify the evidence and the rule they applied. Keep unresolved disagreement near accepted tolerance in the record for the owner. If the project uses repeated human labels for reference, define when the person with authority should interpret a physical sample.

Decide what drift looks like during long annotation work. A reviewer can get faster and miss rare application rules or see one lighting condition so often that they overinterpret it. Send a small reference batch across time and compare how the outcome and reason move. Record any new material or camera setting. Use the drift observation to focus the next calibration, especially if a new run causes disagreements a single dataset-wide rate could hide. The reviewer disagreement log keeps evidence, interpretation and correction together.

Work through the consequences of changing a definition

Treat a change to an annotation rule as a versioned change to evidence. Name which labels, views and operating conditions are affected. Determine whether the new rule clarifies a consistently applied boundary or changes which units should be accepted. Keep the original annotations and outcome summaries so an older report remains interpretable. A correction to a faulty reference is important even if the original model output never changes.

Suppose a fictional team originally regards small print bubbles as a surface defect everywhere on the package. The packaging owner clarifies that bubbles under an external band are acceptable if they do not affect a protected area. Now the visual appearance can stay the same while the outcome changes by location. Look back through examples of both areas. Decide whether the existing annotations include enough positional context to apply the new rule. The reference tool could retain a bubble label and compute a separate disposition from its zone, or update a final label with a separate reason and rule version.

Re-read changes that affect the frozen evaluation. When one case was demonstrably wrong because an annotator applied the wrong revision, correcting that case can be appropriate. Log the evidence and compare all candidate methods on the same corrected version. When a new product requirement is designed after examining a failed evaluation, the earlier report provides development observations. Evaluate the new requirement under an independent set. Keep the purpose of each record explicit so results are not merged across incompatible rules.

Give the operator and analyst distinct records if they need them. The operator may need the current disposition and instruction, while the analyst needs the original label, reviewed label and rule versions. Record reviewed changes as new events or approved revisions. Avoid instructions that require someone to retrieve an unpublished draft with no indication it superseded a prior requirement. The decision record can hold scope, approver and reopen trigger.

Audit a completed batch beyond annotation counts

Measure completeness by approved unit groups and label coverage. A high number of marked images can hide that a station, finish, or rejection category has never been reviewed. Inspect representative exports by condition and by annotator. Compare the shapes and reasons of similar cases. Confirm that partial views, review states, and approved formats survived export. Track whether fault preparation or station changes are visible to later analysts.

For a fictional batch of fifty independent cartons, assume forty have one clear applicable label, six require a second view and four cannot be decided from the existing evidence. That project can verify a settled annotation on forty-six if all six are recovered and independently checked. It should retain the remaining four as unresolved. If some recovered cases were reviewed with a later camera setup, preserve the date and assumption before treating their source state as equivalent. Confirm that the split logic groups both views from each carton.

Audit the written instruction using a retrieval test. Ask somebody who did not label the batch to recover one known acceptable look-alike, one marked failure, one uncertain example, and one specific corrected record. Have them identify the rule version and how the team established the label. Ask what action they would take if the original photo is clear but the missing part’s physical reference sheet says something different. If they have to talk to the original annotator for every example, tighten the references.

Close the annotation batch with an evidence summary, a rule version, and specific remaining exceptions. Give the next person a record of which items are ready for training or evaluation, which are preserved for workflow testing, and which need a decision. State the recall limits of the individual annotations. For example, a label describing “fastener visible” covers apparent presence within the accepted view. An approved placement judgment covers the fit of a measured location against a named tolerance. The dataset splitting guide explains how to carry those groups into evaluation.

Recognize differences between case selection and annotation quality

Identify which cases enter each batch and why. A reviewer can apply the rule faithfully to an unrepresentative sample. For instance, a group might consist of the first cartons from setup, when station staff check position especially carefully. Another group may be assembled only from flagged items. The labels describe actual evidence in those groups. Their proportions cannot stand in for the mix the inspection needs to handle on a normal run. Keep collection purpose in the record.

Review how examples are delivered to people. An annotator given a folder called rejects may use its name as an answer, even when file picks include glare and unrelated product complaints. Hide irrelevant group signals and any prediction if a reference needs independent judgement. Record exception handling when people have already seen the earlier disposition. Supply the evidence needed for the task, such as the correct specification and authorized measurement. Record where it came from and who interpreted it.

Look for classes associated with incidental collection details. A newly trained reviewer might annotate only glossy examples while an experienced reviewer handles matte. If they use slightly different rules, an apparent supply change can reflect reviewer interpretation. Include the same calibration cases in both groups and compare decisions on overlapping conditions. Where the rules are applied consistently but evidence differs by finish, look at the affected edges and document the actual visual difference. Changes in both result and method deserve note.

If an annotation has several usable levels of certainty, define what moves between them. An initial single-person label might be ready for review. A two-person consensus on visible evidence might be ready for evaluation. A physically measured verification might be required for a protected limit. The list of approved labels should indicate which categories each pilot analysis can use. A quick development label can still be helpful when the process knows it is provisional and it is kept away from final confirmation.

See how an export error changes the meaning of a class

Check the exported records against the interface people used to label them. Pick a few known cases, including a confirmed negative, an unresolved case, and an example with multiple features. Confirm that the field names, identities, coordinates and state meanings still match. A common failure mode in a structured export is using an empty list for several states. Empty could mean a person looked and found no defective feature. It could also mean nobody reviewed the evidence or the required field was hidden.

Take a fictional export with fifteen cases from a cap station. Eleven images visibly show the cap correctly seated, two show a cap above the required limit, one image is missing, and the far edge is hidden in another. If the project exports rectangles only around high caps, thirteen sets of rectangles are empty. A reader sees the same empty list for eleven confirmed clears, one missing frame and one unresolved view. Add a status to represent the underlying decision, or export the set of evaluable units and the reason ledger separately with stable IDs. Test the downstream import on exactly those cases.

Check the order of classes and categories in the imported tool. A project that relies on numerical class IDs needs a stable mapping. Adding a new class in alphabetical order may change indices if the exporter computes them on every run. Inspect the actual mapping on both sides and version it with the labels. Give the next reviewer human-readable names rather than requiring them to know which index was the positive class in a particular run. When conversions occur, preserve enough source detail to verify what changed.

Test a unit with two annotations that use different scopes. A carton can show one label in place and one with displaced print; a surface check can record a local scratch and an accepted texture nearby. Exporting one disposition per image without location or component ID can lose the fact that several independent observations occurred. Decide which notes the tool must keep for reproduction. Ask the downstream analysis owner to perform one reconstruction before accepting the export. The smallest useful manifest lets them reproduce one known answer without asking the sequence of tools that wrote it.

Review a difficult change in a short calibration meeting

Bring an original image, the disputed region, the current rule and an example that challenges the proposed change. Give the rule owner enough physical context to establish the correct requirement. The purpose of the meeting is to approve how a future reviewer can decide the case. Capturing the final annotation alone omits the explanation somebody will need for the next near-boundary example.

Suppose a fictional clip check labels a recessed head “present” whenever more than half of the head is visible. One reviewer counts the reflection from the head’s top. Another measures only the portion that shows the approved profile. For a highly reflective valid head the first convention can yield an acceptable result, although the second describes a hidden condition. The requirement owner consults the drawing and clarifies that the fastening point and identifiable head must be visible together, without demanding a percentage of reflected area. The calibration team records affected images and measures how the clarification changes recovery requests.

Before accepting the new rule, work through the consequence on known failure examples. A metal fleck lying near an empty recess could appear as the same small bright patch. Invite reviewers to use the surrounding structure to explain their decisions and classify unusable cases separately. Include at least one different finish and one product with a legitimately recessed fastener. If the camera can inspect only some variations, add the fallback the line already uses and record what could extend the approved scope later.

Record the change, owner and date. Tell annotators which reference version to apply to open work. Decide whether already exported sets need a reviewed correction and how to identify them. Validate the change in the tool and in a fresh sample group. When the change touches a report given to the decision owner, provide the old and revised outcomes so they can see the effect. The guide to adjudicating disagreements provides the workflow for a continuing dispute.

Use a small acceptance protocol for the annotation brief

Prepare a set that spans the actual operating conditions before calling the brief usable. The owner may choose a few independently collected samples from each significant format, including near-boundary disagreements and unavailable evidence. Have a person who did not write the instructions use the actual labeling interface. Include a pair of images whose correct outcomes depend on the identified rule version. Ask what the person would do when the governing product reference is unavailable.

Review the resulting record in three places. In the interface, verify that each case has the intended type of mark and can display the original evidence. In the export, check unit identity, class name, visibility, reference disposition and the rule version. In a regular workflow review, verify that the evidence fields and routing status support the next action. A usability failure in one place may be inexpensive to fix and severe in another. For instance, a correct reference label can still create confusion if the operator display hides which package position it belongs to.

Choose a next step from the evidence. If reviewers can apply the brief consistently, approve it for that scoped batch and retain the calibration set. If a visual boundary is unresolved, send the disputed evidence to the quality owner. If the problem is the missing view, capture it or preserve the approved review route. Document the rule or information that would change your choice. Add it to the annotation brief builder before distributing the next batch.