Philippines staffing blog ·
Sample Outsourced Ticket Quality Without Grading Personalities
A rubric should test the record, the boundary, and the customer outcome, with evidence a second reviewer can reproduce.
Direct answer
Operational guidance
Decide what the sample is for
A quality sample can support coaching, contract evidence, or process design. Each purpose needs a different sample and a different report. Mixing them produces a score that managers use to rank agents and agents distrust because it measures the wrong thing. State the purpose before selecting the tickets, and keep individual results separate from the aggregate trend.
Score observable attributes
Use attributes a reviewer can verify from the record: the customer goal is stated, the verification step is documented, the action matches the approved procedure, the customer received a clear next step, sensitive data is minimized, and the ticket is closed with evidence. Avoid traits such as helpfulness or professionalism unless they are defined by concrete examples. A rubric that depends on tone invites disagreement that cannot be resolved.
Sample across outcomes and agents
Include resolved, returned, escalated, reopened, and still-open tickets. If the sample contains only closed tickets, it will miss the cases where the process failed. Spread the sample across shifts, channels, and request types so one busy lane does not dominate the result. Record the query and the inclusion rule so the sample can be rebuilt.
Calibrate reviewers on the same ten tickets
Have two reviewers score the same small set independently, then compare. Disagreements usually point to a vague attribute or a missing definition. Revise the rubric and repeat before the main sample. Report inter-rater agreement as a count, not a claim of scientific validity. A rubric that two experienced reviewers cannot apply consistently is not ready for use.
Separate the agent from the system
When a ticket scores poorly, ask whether the agent had the procedure, the permission, the time, and the information needed. A missing procedure is a knowledge problem. A missing permission is an access problem. A missing field is a form problem. A rushed closure after a backlog spike is a capacity problem. Record the contributing factor so the correction targets the system, not only the person.
Report distributions, not a single grade
Show the percentage of tickets meeting each attribute, the sample size, and the confidence range if appropriate. A combined score can hide a serious weakness in one attribute, such as sensitive data handling. Report the attributes separately and highlight any ticket with a control failure regardless of the overall score.
Close the loop with a change
Each review cycle should produce at most three changes: one to the procedure, one to the form or tool, and one to the coaching plan. Assign owners and dates. At the next cycle, check whether the attribute moved. A quality program that produces a report but no change is an expensive ritual.
Authoritative sources
- U.S. GAO, Assessing Data Reliability
- NIST SP 800-53 Rev. 5, Security and Privacy Controls
- Atlassian, Jira Service Management Documentation
