feat(monitoring): probe public web endpoints (ENG-2324)

[lucas-koontz]

User story

As an engineer on call for MindsHub
I want the public Console routes, Cowork, and website checked through the edge
So that route, DNS, certificate, proxy, and website failures alert me before customers report them

Why this matters

Production and staging have returned 301 and 403 responses on routes that Kubernetes still considered healthy. The one-minute Console and Cowork checks bypass the public edge, and the Pages check does not cover the mindshub.ai custom-domain path. This workflow exercises the Console hosts, Cowork, and that custom domain every five minutes.

Acceptance criteria

• A workflow runs every five minutes and supports manual dispatch.
• It checks nine reviewed routes on production and staging, https://cowork.mindshub.ai/, and https://mindshub.ai/, for 20 exact public endpoints.
• Console and Cowork responses require exact status 200 and <div id="root"></div>; the website requires exact status 200 and its og:site_name marker.
• Redirects, 200 error pages, and a body carrying the wrong endpoint marker fail.
• The first attempt covers all 20 endpoints, then only failures retry after 15 seconds.
• A recovered retry remains green and silent; persistent failures make the run red.
• One notifier label names every persistent endpoint and observed result. The next green run uses the existing recovery path.
• The endpoint list, markers, cadence, activation gate, and excluded authenticated journey are documented and contract-tested.

How to test

1. Run uv run pytest, Ruff, and actionlint .github/workflows/console-route-probe.yml.
2. Run python3 scripts/probe_console_routes.py --retry-delay-seconds 0 only after ENG-2317 reaches production. All 20 endpoints must pass.
3. Dispatch the workflow after that gate and confirm a healthy run posts nothing.
4. In an isolated exercise, make one endpoint return 403 or 301 twice. Confirm the run fails and one message identifies the endpoint and result.
5. Restore the endpoint and confirm the next run posts recovery.
6. Return 200 with the wrong configured marker for a Console route, Cowork, and the website. Confirm all three fail.

Notes for the reviewer

• ENG-2317 is a hard activation gate. Production currently returns 301 for /cowork and 403 for /assets/. Do not merge, promote, or dispatch this workflow until ENG-2317 is deployed and all 20 local probes pass.
• No live workflow dispatch was run. A dispatch inherits the real notifier secrets and can send a failure or recovery message.
• Cowork and the website are standalone roots. Cowork uses the SPA marker, while the website uses its og:site_name marker. Retry objects retain the marker for the exact failed endpoint.
• The retry is a debounce. Only first-attempt failures repeat after 15 seconds, which keeps transient failures silent without delaying healthy endpoints.
• The authenticated 15-minute journey remains excluded. It needs its own interaction contract and test plan before it can provide an actionable signal.

Verified locally

Check	Result
uv run pytest	181 passed
uv run pytest tests/test_probe_console_routes.py	33 passed
Ruff check and format check	Passed
actionlint .github/workflows/console-route-probe.yml	Passed
Targeted endpoint mutation	The endpoint-count contract failed when a standalone target was removed, then passed after restoration
Live standalone public endpoint requests	Cowork and mindshub.ai returned 200 with their expected markers
git diff --check and pre-PR sweep	Passed

Ships with

Sibling: mindsdb/terraform#203

Merge order: Merge terraform#203 first and apply it after its destination and token gates pass. Merge this PR only after ENG-2317 is deployed and all 20 public endpoints pass locally.