Conversation
…atform fork vxcontrol#1) - new crtsh searcher: keyless CT-log lookup, %.domain wildcard, dedupe by CN+SAN+serial, newest-first, 80KB hard cap, retryable 429/5xx - web_search: new 'cert' mode + fallback chain (crtsh -> internal -> searxng) - DB migration adds 'crtsh' to searchengine_type enum - config: CRTSH_API_URL mirror override (default https://crt.sh) - unit tests: query normalization, dedupe, truncation, issuer shortening
Resource hardening pass after pentagi-vps evaluation (8G RAM host): - Add mem_limit to 13 managed containers based on active-state peaks. Prevents OOM killer from picking pentagi as victim under load. - ClickHouse TTL: 30d for observations/traces/scores (debug window), 14d for event_log/blob_storage_file_log (raw payloads recoverable from MinIO). ClickHouse 24 requires DateTime not DateTime64 in TTL, so each column wrapped with toDateTime(). - Mount ./langfuse-clickhouse-init:/docker-entrypoint-initdb.d:ro so fresh deploys get TTL automatically via 01-ttl.sql. - Fix minio image: minio/minio:RELEASE.2025-07-23T15-54-02Z no longer pulls from Docker Hub (anonymous denied); pin to quay.io/minio/minio:latest matching production override. Pairs with VPS-side changes in docker-compose.override.yml: - OLLAMA_KEEP_ALIVE=-1 (warm resident, no 3-8s reload per embedding) - mem_limit on ollama/searxng/flaresolverr And .env: MAX_GENERAL_AGENT_TOOL_CALLS=30 (was empty -> 100 default). Not pushed: origin only points to upstream vxcontrol/pentagi; intel-platform branch lives on local clone only. Awaiting fork URL to enable auto-push. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SearXNG was returning HTTP 400 to PentAGI's web_search requests because the default config required X-Forwarded-For headers from trusted proxy sources, but docker-network callers (PentAGI) connect directly with no proxy. Flow 7 (Puer.im Security Test) was stuck on web_search for this reason alongside the MiniMax API balance issue. Added minimal permissive limiter.toml: - botdetection.ip_limit = 0 (no per-IP rate limit) - botdetection.link_token = false (no link tokens for API callers) - real_ip_sources = RFC 1918 + loopback (trust docker network) - ratelimit.enable = false (PentAGI is sole internal caller) Also disabled DUCKDUCKGO_ENABLED in /opt/pentagi/.env because the VPS IP is captcha-blocked by DuckDuckGo. Searches still work via SearXNG's own engine pool (Brave etc). Verified: 'q=puer.im+test' now returns 19 results with 200 OK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second custom searcher of the intel-platform fork. FOFA complements crt.sh: where crt.sh yields subdomain names from passive CT logs, FOFA yields the *hosts* those names resolve to, with open ports, services, banners, and product fingerprints. Together they form a two-layer recon (names → addresses+services). Engine choice: - Position: 2nd in ModeLinks fallback (after Google, before DDG), so it leads over classic engines whenever configured. - IsAvailable(): requires BOTH FOFA_EMAIL and FOFA_API_KEY (FOFA rejects single-credential requests). - 401/403 are Fatal (bad credentials, won't fix on retry). 429/5xx are Retryable (rate limit / transient). 6 files + 1 migration: - backend/pkg/tools/searchers/fofa.go — Searcher implementation - backend/pkg/tools/searchers/fofa_test.go — 7 tests (gating, stub buildQuery, format, truncation, empty-state, response parsing) - backend/pkg/tools/web_search.go — EngineFofa + ModeLinks chain - backend/pkg/database/models.go — SearchengineTypeFofa constant - backend/pkg/config/config.go — FofaEmail/FofaAPIKey/FofaAPIURL - backend/migrations/sql/20260915_120000_add_fofa_search_type.sql — goose enum migration (adds 'fofa' to searchengine_type) - .env.example — FOFA_EMAIL/FOFA_API_KEY/FOFA_API_URL section 7 tests PASS (14 sub-tests), 26.4s full searchers suite green, no regression on crt.sh tests. Known TODO: buildQuery() is currently a pass-through stub (with scheme strip + lowercase). The user contribution point is the classifier that maps 'example.com' → domain="example.com" vs '1.2.3.4' → ip="1.2.3.4" vs already-DSL pass-through. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the pass-through stub with a full classifier that maps free-form
user queries to FOFA DSL:
1. Already DSL → whitelist-validate the field name (security: blocks
prompt-injected qbase64/exec operators).
2. IPv4 / IPv4 CIDR → ip="..." (checked BEFORE path/port strip so the
/24 suffix doesn't get eaten).
3. DNS hostname (any label count, valid public TLD incl. punycode) →
domain="..." (auto-wildcards all subdomains; LLM can use the
verbatim DSL path to get an exact host= match).
4. Fallback → title="..." (FOFA indexes page titles, so this always
returns SOMETHING — multi-word queries like 'Apache Struts' are
preserved verbatim).
TLD regex now accepts both plain alpha (com, org, cn) and IDN punycode
(xn--...) so Chinese IDN domains like xn--fiqs8s.xn--0zwm56d (.中国)
classify correctly.
Casing: DSL pass-through preserves the user's original case (FOFA field
names are case-sensitive; values are case-insensitive but cleaner to
preserve). Bare-word classification lowercases as before.
Test coverage:
- 38 sub-tests in TestFofaBuildQuery (every branch + adversarial cases)
- TestFofaAllowedFieldsWhitelist guards the public contract
- TestFofaIsDNSHostname covers the DNS validator directly
All FOFA tests + full searchers suite (28s) PASS.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Attempted to fix local Langfuse 3.225.7 + Next.js 16 production bind bug where the server ignores HOST env var and binds to its hostname IP only, making the docker port mapping 127.0.0.1:4000->3000 unreachable for inter-container traffic. Result: HOST=0.0.0.0 still gets ignored — Next.js binds to the resolved hostname IP regardless. Even setting hostname: 0.0.0.0 in compose doesn't reliably fix inter-container networking after multiple container recreates (iptables conntrack + Next.js event loop become inconsistent). DECISION (see pentagi-langfuse-cloud-decision.md memory): Use Langfuse Cloud instead. The local 6-container stack remains running as a backup but pentagi is configured to send traces to https://us.cloud.langfuse.com. Keeping this comment + line so a future Langfuse upgrade that fixes the bind bug has the documentation ready. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous limiter.toml (intel-platform fork hardening) used a TOML schema SearXNG 3.x rejects with "schema of /etc/searxng/limiter.toml is invalid!". SearXNG was caught in a crash-restart loop, breaking web_search for both PentAGI and manual testing. Without limiter.toml, SearXNG falls back to defaults (warning-only), which is sufficient for internal-only deployment where PentAGI is the sole caller and SearXNG is reachable via docker network only (no public exposure). If we need stricter rate limiting later, write a TOML that exactly matches SearXNG 3.x's pydantic schema (real_ip_sources must be a TOML array, not comma-separated string; [real_ip] is not a valid section; [ratelimit] needs per_second/burst, not enable). Verified post-fix: SearXNG Up, returns 200 for general queries, PentAGI container can reach it via docker DNS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the Change
Problem
Solution
Closes #
Type of Change
Areas Affected
Testing and Verification
Test Configuration
Test Steps
Test Results
Security Considerations
Performance Impact
Documentation Updates
Deployment Notes
Checklist
Code Quality
go fmtandgo vet(for Go code)pnpm run lint(for TypeScript/JavaScript code)Security
Compatibility
Documentation
Additional Notes