Skip to content

Update fc2db scraper to handle Cloudflare and strict exceptions - #2826

Open
chocolate-croissant wants to merge 3 commits into
stashapp:masterfrom
chocolate-croissant:update-fc2db-flaresolverr
Open

Update fc2db scraper to handle Cloudflare and strict exceptions#2826
chocolate-croissant wants to merge 3 commits into
stashapp:masterfrom
chocolate-croissant:update-fc2db-flaresolverr

Conversation

@chocolate-croissant

Copy link
Copy Markdown
Contributor

Scraper type(s)

  • performerByName
  • performerByFragment
  • performerByURL
  • sceneByName
  • sceneByQueryFragment
  • sceneByFragment
  • sceneByURL
  • groupByURL
  • galleryByFragment
  • galleryByURL
  • imageByFragment
  • imageByURL

Examples to test

fc2db:

  • sceneByURL: https://fc2db.net/work/4925699/
  • performerByURL: https://fc2db.net/actress/2559/

Short description

This PR introduces critical updates to the new fc2db scraper to handle recent changes on the target site (fc2db.net), which recently enabled Cloudflare Anti-Bot protection and custom Age Verification checks.

Key Changes:

  1. FlareSolverr Integration:
    • fetch_page now detects Cloudflare challenges (403/503 or challenge text like "One moment, please...").
    • If a challenge is detected, the request is automatically routed through FlareSolverr to bypass the protection.
  2. Strict Error Handling:
    • Removed the silent HTML fallback for core metadata fields (Title, Code, etc.) if the JSON-LD payload is missing.
    • Now, if the VideoObject (for scenes) or Person (for performers) JSON-LD structure is not found in the HTML response, the scraper explicitly raises a hard Exception (Failed to find JSON-LD VideoObject/Person. Cloudflare challenge may have failed, or site layout changed.).
    • This ensures the scraper fails loudly and clearly instead of silently returning partial/empty scene data (unknown -).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant