PW-Tests

We moved from simple substring matching to a multi-layered classification engine. Failures are now matched against a structured incident store using weighted fingerprinting across seven dimensions. A historical confidence layer tracks how often each test has failed for each root cause, blending past patterns with current evidence to produce verdicts that get smarter over time.

• Incident store with six root cause domains and seventeen hierarchical tags
• Weighted incident matcher scoring fingerprint, site, error category, selector overlap, date window, URL pattern, and message content
• Historical prior service computing confidence bands from confirmed recovery events
• Cause assessor blending both layers into a final verdict with human-readable explanation

Nightly failure notifications were completely rewritten to communicate in human terms instead of dumping raw failure counts. Each failure now gets a verdict label — confirmed regression, post-fix watching, likely flaky, infra suspicion, or needs confirmation. Failures are clustered by site, and an investigation thread is posted automatically with per-failure breakdowns and next-step recommendations.

• Recovery detection posts confirmation to original failure thread when a test passes consecutively
• Thread state machine manages open, resolved, superseded, and stale failure threads
• Weekly report redesigned with executive summary, trend comparison, and root cause breakdown
• Escalation contact suggestion appended to alerts based on failure classification

A brand-new escalation system answers the question that classification alone could not: the test failed, QA confirmed it is real — now who do I contact? Three normalized JSON databases map contacts, sites, and routing rules across twelve escalation categories. A resolver module implements strict precedence matching and the portal page presents it all as an interactive workflow and lookup tool.

• Seventy-eight CNC sites and twenty-seven contacts seeded from the ownership spreadsheet
• Twelve escalation categories from content issues to video player failures
• Five-step visual workflow on the portal: Test Fails → QA Triages → Classify → Escalate → Recovery
• Build-time validation with eight error checks and five warning checks

The documentation portal launched with thirty-one pages, CNC brand design, content registry sidebar, and dark mode support. A second version is in progress with a modular build pipeline — separate collectors for git, OpenSearch, CI config, and test results feed into interactive templates that can show live data alongside static documentation.

• New pages: escalation matrix, content registry, per-suite operational runbooks
• Portal v2 architecture: collectors, derived data, template engine, cache layer

All three Grafana dashboards were redesigned. Status got a sites-by-suites matrix with per-cell drill-down links. Investigate replaced its table with a Dynamic Text panel showing formatted errors and stack traces. Trends got proper multi-select variables and fixed data links. The reporter was enhanced with ANSI stripping, normalization, and over five hundred step wrappers across twenty-seven test files.

• Resolved the UUID hyphen parsing bug that caused "No data" across Investigate queries
• Debunked the .keyword field mismatch — all indices have proper sub-fields
• Production ISM policies and index templates deployed for retention management

Several infrastructure issues were found and fixed. OpenSearch indices that had been accumulating indefinitely now rotate monthly with automatic thirty-day cleanup. A twelve-check verification script validates observability health. A Playwright-based Grafana monitor catches dashboard rendering failures that API queries cannot detect.

• Fixed disk-full incident caused by static index names without retention
• Unified telemetry: three loggers replaced by single EventLogger, removing twelve hundred lines of dead code
• Visual regression tests rewritten from fifty-four failing tests to eighteen passing in under thirty seconds

Wiki grew from fourteen to sixteen pages. Confluence standalone pages expanded from four to fifteen with per-suite operational runbooks. Eight mermaid diagrams were added or restored. A Slack message formatting guide documents verdict labels, clustered thread style, and wording rules.