Skip to content

docs: center scraping guidance on auto - #1255

Merged
gildesmarais merged 6 commits into
mainfrom
docs/gem-httpx-web-falcon-sync
Sep 6, 2026
Merged

docs: center scraping guidance on auto#1255
gildesmarais merged 6 commits into
mainfrom
docs/gem-httpx-web-falcon-sync

Conversation

@gildesmarais

@gildesmarais gildesmarais commented Sep 6, 2026

Copy link
Copy Markdown
Member

What changed

  • Make auto the recommended strategy across custom-feed, dynamic-content, CLI, MCP, and troubleshooting docs.
  • Document default as a fixed direct-request override without exposing HTTP adapter implementation details.
  • Keep explicit Botasaurus selection for browser-specific controls and targeted diagnosis.
  • Describe web workers and timeout contracts behaviorally, and keep reverse-proxy HTTPS as the supported deployment path.
  • Correct capture strategy persistence, MCP strategy discovery, and the UnknownStrategy troubleshooting entry against runtime behavior.

Why

End users should choose between the normal auto path and deliberate fixed-mode overrides. HTTP client and web server choices are implementation details; documenting them throughout the user journey made adapter replacements look like user-facing migrations and weakened the strategy interface.

Risk

  • Low — documentation-only changes.
  • The core runtime still publishes compatibility aliases through strategy discovery; the docs describe canonical choices without promoting those aliases.
  • html2rss-web/docker-compose.yml still sets the old timeout variable. These docs describe aligned runtime defaults and tell operators to set REQUEST_TIMEOUT_SECONDS explicitly rather than claiming Compose configures it.

Review map

Review map: whole PR is small — start at src/content/docs/ruby-gem/reference/strategy.mdx, then check src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx and src/content/docs/web-application/reference/env-variables.mdx.

Validation

  • make lint
  • make test — 7 files, 25 tests passed
  • make build — 44 pages built and 58 HTML files indexed
  • Fresh documentation findings review — readiness Yes

Deepen the public strategy interface around auto and default while keeping HTTP and server adapters behind their implementation seams. Remove migration narration and incomplete deployment side paths so end users can act on scrape outcomes instead of transport internals.
State capture persistence and MCP discovery behavior exactly as implemented while keeping compatibility aliases out of recommended user guidance. Use the runtime UnknownStrategy name in troubleshooting.
@gildesmarais gildesmarais changed the title docs: sync unreleased gem HTTPX default strategy and web Falcon stack docs: center scraping guidance on auto Sep 6, 2026
@gildesmarais
gildesmarais merged commit 4adfe88 into main Sep 6, 2026
5 checks passed
@gildesmarais
gildesmarais deleted the docs/gem-httpx-web-falcon-sync branch September 6, 2026 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant