# Reporting

| Key | Value |
|-----|-------|
| Status | Active |
| Owner | QA Automation |
| Updated | 2026-03-26 |
| Scope | Slack alerts, run summaries, periodic reports, and recovery communication |

Reporting is where the system becomes readable. A good report answers three questions quickly:

1. what happened
2. how worried should we be
3. what should happen next

That sounds simple, but it took a lot of iteration to get there.

## Reporting Outputs

| Output | Audience | Typical Use |
|--------|----------|-------------|
| failure alerts | QA, on-call, engineers | immediate action |
| success summaries | operators and channel followers | reassurance and trend awareness |
| visual notifications | teams watching the visual suite | structural UI signal |
| weekly reports | managers and maintainers | broad operational summary |
| monthly reports | stakeholders | trend and stability review |
| recovery replies | anyone following a failure thread | closure after a fix |

## Slack Failure Alerts

Failure alerts are now designed to be more human and less robotic.

### Main Principles

| Principle | What It Looks Like |
|-----------|--------------------|
| calm headline | headline says what matters, not just “FAILED” |
| short first post | stats and failed tests, not a forensic dump |
| verdict-first thread | `Assessment`, `Why`, `Next` |
| clustering | repeated failures with one cause are grouped |
| confidence language | `Post-fix, watching`, `Likely flaky`, `Infra/CI suspicion`, and similar labels |

### Why This Matters

Operators need to know whether to escalate, rerun, ignore, or watch. Generic classifier text does not help much when a run is noisy.

## Success Summaries

Success messages are intentionally smaller. Their job is to confirm the run landed cleanly and leave useful links behind without taking over the channel.

## Weekly And Monthly Reports

Periodic reports are the long-view counterpart to failure alerts.

| Report Type | What It Tries To Show |
|-------------|-----------------------|
| weekly | what changed this week, where the risk is, and whether the trend is improving |
| monthly | slower moving stability and quality patterns |

Recent direction:

- plain-language executive summary
- root-cause rollups, not only symptom counts
- week-over-week context instead of isolated numbers
- better separation between recurring noise and material regressions

### What A Weekly Report Contains

| Section | What It Shows |
|---------|---------------|
| overall pass rate | percentage across all suites for the past 7 days |
| failure breakdown | per-site and per-suite failure counts |
| top failing tests | tests that failed most often during the week |
| top flaky tests | tests with mixed pass/fail outcomes |
| trend direction | whether pass rate improved, held steady, or declined versus the prior week |
| root-cause rollup | failures grouped by incident or category rather than listed one by one |

### What A Monthly Report Contains

| Section | What It Shows |
|---------|---------------|
| 30-day pass rate | overall quality trend over the month |
| stability patterns | which sites and suites are most stable or most volatile |
| recurring failures | incidents that appeared repeatedly over the month |
| recovery rate | how quickly failures were resolved after they first appeared |

### Schedule

| Report | When | Channel |
|--------|------|---------|
| weekly | Monday morning (GitLab schedule) | `#qa-reports` |
| monthly | 1st of each month (GitLab schedule) | `#qa-reports` |

### Triggering Reports Manually

| Command | What It Does |
|---------|-------------|
| `npm run report:weekly` | generates and posts a 7-day report |
| `npm run report:monthly` | generates and posts a 30-day report |
| `npm run data:weekly` | pulls weekly summary data without posting |

PDF output is saved to `test-results/reports/`.

### Reporting Data Flow

```mermaid
%%{init: {'theme':'base', 'themeVariables': {'primaryColor': '#4a90d9', 'primaryTextColor': '#fff', 'primaryBorderColor': '#2c6fad', 'lineColor': '#555', 'fontFamily': 'sans-serif'}}}%%
flowchart LR
    OS["OpenSearch\ncncqa_tests-*"] --> GEN["Report generator\nscripts/ci/generate-weekly-report.ts"]
    HIST["test-results/history/\nlocal run records"] --> GEN
    GEN --> PDF["PDF report\ntest-results/reports/"]
    GEN --> SLACK["Slack post\n#qa-reports"]
```

The generator pulls run records from OpenSearch when credentials are available, or falls back to local history files. It produces both a PDF artifact and a Slack post in the same run.

### Slack Channel Setup

| Variable | Purpose |
|----------|---------|
| `SLACK_REPORTS_CHANNEL` | target channel for weekly and monthly reports (default: `#qa-reports`) |
| `SLACK_BOT_TOKEN` | enables PDF attachment upload and thread replies |
| `SLACK_WEBHOOK_URL` | fallback delivery path if bot token is not set (message only, no PDF) |

## Recovery Replies

Recovery replies are one of the most useful newer reporting features because they stop failure threads from becoming dead ends.

| Recovery Reply Does | Why It Helps |
|---------------------|--------------|
| confirms a failure has stopped repeating | reduces uncertainty |
| links back to the original thread | keeps context together |
| records the recovery in history | improves future confidence |

## Visual Notifications

The visual suite needs separate reporting because its failure modes differ from text-first suites.

Important reporting rule:

- a missing baseline is a configuration problem
- a screenshot diff is a visual regression candidate
- those should not be reported as the same thing

## Report Consumers

| Consumer | What They Usually Need |
|----------|------------------------|
| QA operator | what failed and what to do next |
| engineer | enough evidence to reproduce or fix |
| team lead | whether the problem is broad or isolated |
| stakeholder | trend and operational stability, not every low-level detail |

## Good Reporting Style

- lead with the verdict
- prefer one clear action over five vague suggestions
- separate symptom from cause when confidence is low
- group shared incidents instead of repeating near-identical paragraphs
- avoid flooding the first message with low-level selectors or stack details

1. run completes
2. notifier reads merged results
3. history and incident services add context
4. Slack gets a short main alert
5. investigation thread carries the deeper reasoning
6. if the failure stops repeating, the thread can receive a recovery reply later

## Related Pages

| Need | Go To |
|------|-------|
| notifier commands | [CLI Reference](./cli-reference.md) |
| root-cause and symptom labels | [Failure Categories](./failure-categories.md) |
| Slack and service wiring | [Integrations](./integrations.md) |