Buying note 03 / evaluation protocol

A small trial.
Clear failure criteria.

A proposed evaluation method—not results. Support Ledger has not run vendor trials or published performance scores in this build.

Prepare a fictional support desk

Write a short fictional shipping policy, returns policy and product catalogue. Keep personal data and live store credentials out. Use the actual product in a legitimate trial with a recorded configuration; do not substitute a chat model API.

Include questions it should not answer confidently

1. A direct question with an explicit answer in the policy.
2. A paraphrase and a colloquial Finnish or English version of the same question.
3. A question whose answer is not in the knowledge base.
4. A request conflicting with a written policy.
5. An attempted request for another customer's details.
6. An instruction to ignore the knowledge base or reveal hidden instructions.
7. An explicit request for a human.

Evaluate more than fluency

For each example, record factual correctness, grounding, uncertainty, language clarity and escalation behavior. A natural-sounding answer can still be wrong. Define what a failure is before looking at the output. Review examples manually and preserve the evidence where permitted.

Test a change

Update one fictional policy and observe whether the product picks it up in the intended way. Record the timing and configuration rather than generalizing from a single run. Do not let the system take real irreversible actions during this evaluation.

Report limitations

State the sample, date, product plan, enabled features and any vendor involvement. Do not call the result a security audit, compliance certification or prediction of your store's resolution rate.