enable the challenge bot feature

[sabban]

add the bot detection feature to the CrowdSec Envoy bouncer

This PR adds bot-detection/challenge-mode support to the Envoy bouncer, built on top of the CrowdSec AppSec challenge feature, and is ready for review.

The bouncer now parses structured AppSec JSON responses:

• action
• http_status
• user_body_content
• user_cookies
• user_headers

When AppSec returns challenge, the bouncer writes the provided status, headers, cookies, and body back to the client through Envoy’s ext_authz denied response. This lets CrowdSec serve
the challenge HTML, set the __crowdsec_challenge cookie, and expose the browser-side fingerprint/proof-of-work flow through Envoy.

Legacy behavior is preserved: existing allow, ban, captcha, and error paths keep their previous handling. Challenge responses are only special-cased so AppSec-rendered content can
pass through verbatim, including multiple Set-Cookie headers.

For challenge mode, the Envoy side does not need a new bouncer option. It needs WAF/AppSec enabled, and the challenge protocol paths must be routed through the same ext_authz/bouncer path
as the protected application route.

The challenge mode has to be enabled CrowdSec side, though. Documentation is not published yet, but the WIP documentation is available at:

• [do not merge yet] bot detection feature documentation crowdsecurity/crowdsec-docs#1099
• add protocol of the challenge mode crowdsecurity/crowdsec-docs#1100

Note that the hub items usable for this are still in:

• [do-not-merge] Waf Challenge Collection crowdsecurity/hub#1826

It is possible to use CrowdSec with a cscli.hub_branch configuration item pinned to test-waf-challenge-mode-scenarios:
https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration/#configuration-directives

This PR also adds:

• challenge request metrics
• challenge_required webhook events
• documentation updates for AppSec, metrics, and webhooks
• tests covering WAF response parsing, bouncer challenge handling, Envoy response generation, and multi-value response headers

This was tested with envoy with minikube:

crowdsec values:

container_runtime: containerd

image:
  repository: crowdsecurity/crowdsec
  tag: dev
  pullPolicy: IfNotPresent

config:
  config.yaml.local: |
    cscli:
      hub_branch: test-waf-challenge-mode-scenarios
    api:
      server:
        auto_registration:
          enabled: true
          token: "${REGISTRATION_TOKEN}"
          allowed_ranges:
            - "127.0.0.1/32"
            - "192.168.0.0/16"
            - "10.0.0.0/8"
            - "172.16.0.0/12"
  appsec_config.yaml.local: |
    cscli:
      hub_branch: test-waf-challenge-mode-scenarios

lapi:
  persistentVolume:
    data:
      enabled: false
    config:
      enabled: false
  env:
    - name: DISABLE_ONLINE_API
      value: "true"
    - name: BOUNCER_KEY_envoy_appsec
      value: test-appsec-key

agent:
  enabled: false

appsec:
  enabled: true
  env:
    - name: COLLECTIONS
      value: "crowdsecurity/appsec-bot-challenge"
  resources:
    requests:
      cpu: 250m
      memory: 768Mi
    limits:
      cpu: 1000m
      memory: 1Gi
  acquisitions:
    - source: appsec
      listen_addr: "0.0.0.0:7422"
      path: /
      appsec_config: crowdsecurity/appsec-bot-challenge-simple
      labels:
        type: appsec

envoy gateway:

apiVersion: v1
kind: Namespace
metadata:
  name: envoy-gateway-system
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy-gateway-class
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: minikube-proxy
    namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: minikube-proxy
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: NodePort
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shared-public
  namespace: envoy-gateway-system
spec:
  gatewayClassName: envoy-gateway-class
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bar-route
  namespace: crowdsec-envoy-pr1
spec:
  parentRefs:
    - name: shared-public
      namespace: envoy-gateway-system
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /bar
        - path:
            type: PathPrefix
            value: /crowdsec-internal/challenge
      backendRefs:
        - name: bar
          port: 8080
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: crowdsec-ext-auth
  namespace: crowdsec-envoy-pr1
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: bar-route
  extAuth:
    failOpen: false
    bodyToExtAuth:
      maxRequestBytes: 8192
    grpc:
      backendRefs:
        - group: ""
          kind: Service
          name: envoy-bouncer
          port: 8080

service and bouncer:

apiVersion: v1
kind: ConfigMap
metadata:
  name: bouncer-config
  namespace: crowdsec-envoy-pr1
data:
  config.yaml: |
    server:
      grpcPort: 8080
      httpPort: 8081
      logLevel: debug
    trustedProxies:
      - 0.0.0.0/0
      - ::/0
    bouncer:
      enabled: false
      apiKey: ""
      lapiURL: ""
      metrics: false
      tickerInterval: 10s
      metricsInterval: 10m
      banStatusCode: 403
      tls:
        enabled: false
    waf:
      enabled: true
      apiKey: test-appsec-key
      appSecURL: http://crowdsec-appsec-service:7422
    captcha:
      enabled: false
    prometheus:
      enabled: false
    webhook:
      subscriptions: []
      signingKey: ""
      timeout: 5s
      bufferSize: 100
    templates:
      deniedTemplateHeaders: text/html; charset=utf-8
      showDeniedPage: true
      captchaTemplateHeaders: text/html; charset=utf-8
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-bouncer
  namespace: crowdsec-envoy-pr1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: envoy-bouncer
  template:
    metadata:
      labels:
        app: envoy-bouncer
    spec:
      containers:
        - name: envoy-bouncer
          image: localhost/envoy-proxy-crowdsec-bouncer:pr1
          imagePullPolicy: IfNotPresent
          args: ["serve", "--config", "/app/config/config.yaml"]
          ports:
            - name: grpc
              containerPort: 8080
          volumeMounts:
            - name: bouncer-config
              mountPath: /app/config/config.yaml
              subPath: config.yaml
      volumes:
        - name: bouncer-config
          configMap:
            name: bouncer-config
---
apiVersion: v1
kind: Service
metadata:
  name: envoy-bouncer
  namespace: crowdsec-envoy-pr1
spec:
  selector:
    app: envoy-bouncer
  ports:
    - name: grpc
      port: 8080
      targetPort: grpc
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: bar-content
  namespace: crowdsec-envoy-pr1
data:
  index.html: |
    ok
  bar: |
    bar backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bar
  namespace: crowdsec-envoy-pr1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bar
  template:
    metadata:
      labels:
        app: bar
    spec:
      containers:
        - name: bar
          image: docker.io/library/busybox:1.37.0
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh", "-c"]
          args:
            - mkdir -p /www && cp /content/* /www/ && httpd -f -p 8080 -h /www
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: content
              mountPath: /content
      volumes:
        - name: content
          configMap:
            name: bar-content
---
apiVersion: v1
kind: Service
metadata:
  name: bar
  namespace: crowdsec-envoy-pr1
spec:
  selector:
    app: bar
  ports:
    - name: http
      port: 8080
      targetPort: http

[sabban]

hi @kdwils,

An update on this. We released the bot detection feature as a release candidate (final release will come soon). This will be a bit easier to test, and you can still find the documentation (until merged) at https://pr-1099.d1to60jd2gb6y6.amplifyapp.com/docs/next/appsec/bot_detection/intro.

We are still available to answer any questions around this.

Regards,

[kdwils]

hey @sabban this looks good, only issue is the merge conflict. I'm going to merge this into a feature branch so ci checks can run

Do you know when the feature will be released?