add the bot detection feature to the crowdsec traefik bouncer

[sabban]

add the bot detection feature to the crowdsec traefik bouncer

This PR adds bot-detection/challenge-mode support to the Traefik bouncer, built on top of the crowdsec feature merged in master, and is ready for review.

The bouncer now parses structured AppSec JSON responses:

• action
• http_status
• user_body_content
• user_cookies
• user_headers

When AppSec returns an action other than allow, the plugin writes the provided status, headers, cookies, and body back to the client. This lets CrowdSec serve the challenge HTML, set the
__crowdsec_challenge cookie, and expose the browser-side challenge flow through Traefik.

Legacy behavior is preserved: an empty/non-structured AppSec 403 still falls back to the existing ban handling.

For challenge mode, the Traefik side does not need a new plugin option. It needs AppSec enabled, and the challenge protocol paths must be routed through the same CrowdSec middleware as the protected application route.

The challenge mode has to be enabled crowdsec side, though. Documentation is not published yet, but the WIP documentation is available at crowdsecurity/crowdsec-docs#1099 and crowdsecurity/crowdsec-docs#1100. Note that the hub items usable for this are still stored crowdsecurity/hub#1826, it's possible to use crowdsec with a cscli.hub_branch configuration item pinned to test-waf-challenge-mode-scenarios (see https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration/#configuration-directives)

My test configuration docker file.

name: crowdsec-bouncer-traefik-plugin
services:
  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    command:
      - --log.level=DEBUG
      - --accesslog
      - --accesslog.filepath=/var/log/traefik/access.log
      - --api.insecure=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entrypoints.web.address=:80
      - --experimental.localplugins.bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
    depends_on:
      crowdsec:
        condition: service_started
        required: true
    environment:
      DOCKER_API_VERSION: "1.40"
    ports:
      - 8000:80
      - 8080:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - logs-local:/var/log/traefik
      - ./ban.html:/ban.html:ro
      - ./captcha.html:/captcha.html:ro
      - ./:/plugins-local/src/github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
  crowdsec:
    image: crowdsecurity/crowdsec:dev
    container_name: crowdsec
    restart: unless-stopped
    environment:
      BOUNCER_KEY_TRAEFIK: 40796d93c2958f9e58345514e67740e5=
      COLLECTIONS: crowdsecurity/traefik crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules
      CUSTOM_HOSTNAME: crowdsec
    labels:
      - traefik.enable: "true"
      - traefik.http.routers.crowdsec-challenge.rule=PathPrefix(`/crowdsec-internal/challenge`)
      - traefik.http.routers.crowdsec-challenge.entrypoints=web
      - traefik.http.routers.crowdsec-challenge.service=crowdsec-challenge
      - traefik.http.routers.crowdsec-challenge.middlewares=crowdsec@docker
      - traefik.http.services.crowdsec-challenge.loadbalancer.server.port=7422
    volumes:
      - ./acquis.challenge.yaml:/etc/crowdsec/acquis.yaml:ro
      - logs-local:/var/log/traefik:ro
      - crowdsec-db:/var/lib/crowdsec/data
      - crowdsec-config:/etc/crowdsec
      - ./local-challenge.yaml:/etc/crowdsec/appsec-configs/local-challenge.yaml:ro

  bar:
    image: traefik/whoami
    container_name: simple-service-bar
    restart: unless-stopped
    labels:
       - traefik.enable: "true"
       - traefik.http.services.service-bar.loadbalancer.server.port: "80"
       - traefik.http.routers.bar.rule: PathPrefix(`/bar`)
       - traefik.http.routers.bar.middlewares: crowdsec@docker
       - traefik.http.routers.crowdsec-challenge.middlewares: crowdsec@docker
       - traefik.http.routers.crowdsec-challenge.service: service-bar
       - traefik.http.middlewares.crowdsec.plugin.bouncer.enabled: "true"
       - traefik.http.middlewares.crowdsec.plugin.bouncer.loglevel: DEBUG
       - traefik.http.middlewares.crowdsec.plugin.bouncer.metricsupdateintervalseconds: "15"
       - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdsecmode: stream
       - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapikey: 40796d93c2958f9e58345514e67740e5=
       - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapischeme: http
       - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdseclapihost: crowdsec:8080
       - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdsecappsecenabled: "true"
       - traefik.http.middlewares.crowdsec.plugin.bouncer.crowdsecappsechost: crowdsec:7422
       - traefik.http.middlewares.crowdsec.plugin.bouncer.remediationheaderscustomname: X-Remediation

  volumes:
    logs-local:
    crowdsec-db:
    crowdsec-config:

@maxlerebourg I'll happily discuss this PR further about the testing procedure or any other stuff.

[sabban]

hi @maxlerebourg,

An update on this. We released the bot detection feature as a release candidate (final release will come soon). This will be a bit easier to test, and you can still find the documentation (until merged) at https://pr-1099.d1to60jd2gb6y6.amplifyapp.com/docs/next/appsec/bot_detection/intro.

We are still available to answer any questions around this.

Regards,