Quick overview
This live watchdog monitors AI agent executions daily, sampling runs for Gemini judge scoring, tracking quality drift, and detecting cost, latency, and silent-failure anomalies. It logs scores and baselines to Google Sheets and alerts operations via Slack, Telegram, and Gmail.
How it works
- Runs on a daily schedule (or manually) and loads configuration values such as the monitored n8n URL, API key, workflow IDs, sampling settings, and alert thresholds.
- Validates the configuration and builds an n8n executions API request for the lookback window, then fetches recent executions from the monitored n8n instance.
- If the executions fetch fails, sends a Gmail alert to the ops email address and stops the run gracefully.
- Normalizes execution records, estimates cost from token counts, deduplicates against previously scored execution IDs stored in Google Sheets, and flags failed runs and “silent failures” (for example, empty or error-like output despite a success status).
- Always selects failures and silent failures for review and deterministically samples additional runs, then uses Google Gemini to judge the sampled outputs and produce a 0–100 score with a short reason.
- Compares current window metrics (mean score, cost, mean/p95 latency, silent-failure rate) against pooled historical baselines from Google Sheets to detect drift and threshold breaches, with an alert cooldown based on recent alert history.
- Appends per-run scores and a baseline snapshot to Google Sheets, posts alerts to Slack (and Telegram if enabled), logs alert decisions to Google Sheets, and sends a daily digest email via Gmail.
Setup
- Create an n8n API key on the monitored instance and set the correct n8nBaseUrl, n8nApiKey, lookbackHours, and workflow ID filters (or clear targetWorkflowIds to watch all) in the configuration.
- Create a Google Sheet with tabs named RunScores, Baselines, and AlertLog, add a Google Sheets credential in n8n, and set sheetId in the configuration.
- Add credentials for Gmail and Slack, then set opsEmail and slackChannelId (and enableTelegram plus telegramChatId and Telegram credentials if you want Telegram alerts).
- Add a Google Gemini API key, set the geminiModel if needed, and tune sampling/threshold settings (sampleRate, maxJudgePerRun, driftStdDev, score/cost/latency floors, alertCooldownHours, and token prices) before activating the workflow.