Skip to content

fix(http): preserve raw URL bytes via opt-in url_raw flag #1533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
juandiego-bmu wants to merge 3 commits into
base: master
Choose a base branch
from
+5 −0
Open

fix(http): preserve raw URL bytes via opt-in url_raw flag #1533

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6015314
fix(http): preserve raw URL bytes via opt-in url_raw flag
juandiego-bmu Apr 26, 2026
2cc2b2c
test(schema): allow url_raw key in HTTP_STEP_SCHEMA
juandiego-bmu Apr 26, 2026
95d7903
Merge branch 'master' into fix/url-raw-traversal
securestep9 Jul 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions nettacker/core/lib/http.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

import aiohttp
import uvloop
from yarl import URL

from nettacker.core.lib.base import BaseEngine
from nettacker.core.utils.common import (
Expand All @@ -34,6 +35,8 @@ async def perform_request_action(action, request_options):


async def send_request(request_options, method):
if request_options.pop("url_raw", False):
request_options["url"] = URL(request_options["url"], encoded=True)

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 25, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid leaving a yarl URL in the event dictionary

When url_raw is true, this assignment mutates the same sub_step dictionary that HttpEngine.run later passes to BaseEngine.process_conditions. That path serializes the event with json.dumps—and successful events are also serialized in submit_logs_to_db—but yarl.URL is not JSON-serializable, so raw requests end with a TypeError and cannot record a finding even when the Apache response matches. Convert the URL only in a copied request dictionary, or restore the string before processing, and cover the flag through HttpEngine.run.

AGENTS.md reference: AGENTS.md:L27-L31

Useful? React with 👍 / 👎.

async with aiohttp.ClientSession() as session:
Comment on lines +38 to 40

Copilot AI Apr 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

url_raw is popped from request_options before the URL(..., encoded=True) conversion runs. If the conversion raises (e.g., due to invalid percent-escapes or other characters that are only accepted in the non-encoded constructor), the caller’s retry loop will run again with url_raw already removed and will silently fall back to the default URL normalization path, defeating the opt-in behavior. Consider reading the flag with get() and only pop()-ing it after a successful conversion (or build a shallow copy of request_options for the aiohttp call and leave the original dict untouched).

Copilot uses AI. Check for mistakes.
action = getattr(session, method, None)
response = await asyncio.gather(
Expand Down
1 change: 1 addition & 0 deletions nettacker/modules/vuln/apache_cve_2021_41773.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ payloads:
headers:
User-Agent: "{user_agent}"
ssl: false
url_raw: true
url:
nettacker_fuzzer:
input_format: "{{schema}}://{target}:{{ports}}/{{path}}"
Expand Down
1 change: 1 addition & 0 deletions tests/test_yaml_schema_and_regex.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ def is_valid_regex(regex: str) -> bool:
Optional("timeout"): int,
Optional("allow_redirects"): bool,
Optional("ssl"): bool,
Optional("url_raw"): bool,
Optional("data"): object,
Optional("json"): object,
Optional("ports"): object,
Expand Down
Loading