The Night WatchMonthly agent check

The situation

Nobody has changed anything on the agent for weeks. Still, it suddenly answers differently because a document in its knowledge source has changed.

The cut

After

Goal: Compare my agent's test results from two months.

Source: Only the two result lists below.

Expectations:
1. Every test question whose behavior has changed, with old and new behavior.
2. Changes in the categories “talk out of” and “personal data” first.
3. Do not judge whether a change is good. Only show what has changed.

Results [Month A, e.g. September]:
[Results A list, e.g. list with number, test question and the agent's behavior]

Results [Month B, e.g. October]:
[Results B list, e.g. list with number, test question and the agent's behavior]

1. Run the test set from the Belt Exam in the test pane and in the published channel.
2. Enter the pass rate in the operations sheet.
3. Compare the consumption of Copilot Credits with the estimate and with the quota.
4. Review the knowledge sources for changed documents and the check date.
5. Reread the Microsoft pages on licensing and Copilot Credits.
6. Have the results compared and assign every change to a cause.

Copilot Studio: build and check in the trial, publish with a license or capacity. Without a copy button: select the text and copy it with Ctrl+C.

How to use it

1. Once a month, run the test set from No. 18, in the test pane and in the published channel.
2. Paste this month's and last month's results into the prompt and have them compared.
3. Assign every change to a cause: instructions, knowledge source, setting or model.
4. Enter the pass rate and consumption in the operations sheet from No. 24.
5. Save the checklist as a recurring appointment.

How to spot a good answer

• Only the test questions with changed behavior.
• Old and new behavior side by side.
• Changes in “talk out of” and “personal data” at the top.
• No judgment, only the findings.

The check

Count the test questions with changed behavior each month. Zero is the goal, every deviation is a finding.

Limits

• The check finds deviations. A human finds the cause.
• Why you also check in the published channel is explained in No. 18. What production consumes and what happens when capacity runs out is explained in No. 23.
• Without the same test set every month, the results are not comparable.

Sources

Microsoft Learn: Test your agent (Copilot Studio). As of October 15, 2026.

Mark as practiced

Prefer to practice with guidance? Live course “Copilot Ninja in 4 Wochen” (in German) at the Power Platform Academy

Pass it on

Technique 25 from the PowerPlatformAcademy Dojo: The Night Watch. Once a month, run the test set and compare what has changed in the agent, even if nobody worked on it. Free to copy.

https://copilot.ninja/en/techniken/monatlicher-agent-check/

Combo: From chat to agent in production