feat(search): add opt-in Parallel Search MCP fallback

[georgeatparallel]

Description of the Change

Problem

PentAGI users can already search without a key through DuckDuckGo. This adds another free option for public-web research: explicitly enabled Parallel Search MCP, with source excerpts and citations, through the existing search interface.

As of September 14, Artificial Analysis reports Parallel Fast at 80 DeepSearchQA F1, compared with 74 for Tavily basic and Firecrawl (research results). DeepSearchQA tests whether an agent can gather a complete, precise answer across many searches. Its time per task chart shows 18.8s for Fast, versus 45.0s for Tavily basic and 62.2s for Firecrawl, about 58% and 70% less time. That metric combines derived model time with measured search time. These are provider benchmark settings, not measurements of this PentAGI integration; PentAGI currently uses Tavily advanced. The free anonymous MCP path uses Fast.

Solution

Adds a Searcher adapter using the official MCP Go SDK and PentAGI's HTTP client settings. Set PARALLEL_SEARCH_ENABLED=true in .env or enable it in the installer's Search Engines form. It defaults to false and runs after existing engines in links, answer, and research modes. Existing priorities, DuckDuckGo's enabled default, exploit routing, and browser fetching remain in place.

The adapter discovers and calls only web_search, preserves source URLs, excerpts, dates and warnings, and enforces max_results locally. It handles HTTP, RPC and tool errors through the existing retry/fallback interface. The installer supports save, reset, status, configured count, and restart handling. A migration and regenerated database enum let search logs attribute the winning engine to parallel.

Related issue: none. This is a provider addition through the existing interface.

Type of Change

• 🐛 Bug fix
• 🚀 New feature
• 💥 Breaking change
• 📚 Documentation update
• 🔧 Configuration change
• 🧪 Test update
• 🛡️ Security update

Areas Affected

• Core Services (Frontend UI/Backend API)
• AI Agents (Researcher/Developer/Executor)
• Security Tools Integration
• Memory System (Vector Store/Knowledge Base)
• Monitoring Stack (Grafana/OpenTelemetry)
• Analytics Platform (Langfuse)
• External Integrations (LLM/Search APIs)
• Documentation
• Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: develop, based on ea665308baaff015b226f308438a68d929d0f29b
Docker Version: not used
Host OS: macOS arm64
Go Version: 1.26.5
PostgreSQL Version: 18.4, disposable instance
SQLC Version: 1.27.0
LLM Provider: not used
Enabled Features: Parallel Search MCP; no Parallel credentials

Test Steps

From backend/:

1. CGO_ENABLED=0 go test -timeout=4m ./pkg/tools/searchers ./pkg/tools ./pkg/config ./cmd/installer/wizard/models ./pkg/server/models github.com/golang-jwt/jwt/v5
2. CGO_ENABLED=0 go vet ./pkg/tools/searchers ./pkg/tools ./pkg/config ./cmd/installer/wizard/controller ./cmd/installer/wizard/models ./pkg/server/models ./pkg/server/auth ./pkg/server/services
3. CGO_ENABLED=0 go build ./cmd/pentagi ./cmd/installer
4. Apply the SQL migrations to a disposable PostgreSQL database, then run DATABASE_URL=<disposable connection> sqlc generate -f sqlc/sqlc.yml.
5. Enable Parallel, disable other search engines in a test configuration, and call the native tools.NewWebSearchTool with a public documentation query.

Test Results

Affected suites, vet, and both binary builds pass. Fixtures exercise the actual provider and SDK through the fixed endpoint using the existing proxy/CA path. They cover paginated discovery, repeated project attribution, no auth, structured and text results, errors, bounds, cancellation, redirects, and an SSE response held open until the call returns. Installer and orchestration tests cover persistence, reset, disabled defaults, preserved routing, and audit attribution.

A live call through tools.NewWebSearchTool, with no Parallel key and the other engines disabled, returned useful official documentation URLs and captured exactly one parallel search-log attribution. PostgreSQL checks verified upgrade and rollback behavior, including refusal to relabel historical Parallel logs. SQLC regenerated only the intended enum addition.

Lint was compared with the unmodified baseline using the CI version, 2.12.2, over the affected scopes with diagnostic limits removed. Both report the same 491 existing issues; this change introduces none. The full test suite was not run. Full auth/service tests cannot run locally with CGO disabled because their SQLite fixtures require it; enabling CGO reproduces the same pre-test Apple CPU initialization crash on the baseline. The auth key tests pass without CGO, and the upgraded JWT library's tests pass.

Security Considerations

Once enabled, agents may send their search query to https://search.parallel.ai/mcp as the objective and query. No Parallel authentication or other provider keys are forwarded. Requests carry pentagi/<binary-version> in the User-Agent so Parallel can measure aggregate free MCP usage. Optional conversation and model metadata are omitted because the host search interface does not expose them. Outbound project attribution is tested; warehouse analytics were not checked.

The adapter keeps PentAGI's proxy, CA, TLS and timeout settings. Redirects are rejected. Responses are capped at 2 MiB and formatted output at 32 KiB, with a truncation notice. Unsupported exploit controls are rejected before dispatch. Free access is rate limited and subject to Parallel's published terms and privacy policy.

Adds github.com/modelcontextprotocol/go-sdk v1.8.0 and its required dependencies. Its minimum versions raise JWT to v5.3.1 and golang.org/x/time to v0.15.0. The repository's license inventory script ran and the added modules' license files were inspected; the optional detailed license scanner was unavailable.

Performance Impact

There is no Parallel network traffic while disabled or when an earlier engine succeeds. A Parallel attempt establishes an MCP session, discovers the search tool, and invokes it. A positive HTTP_CLIENT_TIMEOUT bounds the whole operation; zero preserves unlimited timeout and caller cancellation. Cleanup is best effort. No end-to-end PentAGI latency benchmark was run.

Documentation Updates

• README.md updates
• API documentation updates
• Configuration documentation updates
• GraphQL schema updates
• Other: .env.example, installer help, and installation/configuration guide

Deployment Notes

Upgrade the backend so the search-engine migration runs before enabling the provider. PARALLEL_SEARCH_ENABLED=false is passed through Compose by default. Enabling or disabling it through the installer requires applying the restart change. Downgrade is refused while historical Parallel search logs exist, preserving their attribution.

Checklist

Code Quality

• My code follows the project's coding standards
• I have added/updated necessary documentation
• I have added tests to cover my changes
• All new and existing tests pass: affected suites pass; full-suite limits are described above
• I have run go fmt and go vet
• I have run pnpm run lint: no TypeScript/JavaScript changes

Security

• I have considered security implications
• Changes maintain or improve the security model
• Sensitive information has been properly handled

Compatibility

• Changes are backward compatible
• Breaking changes are clearly marked and documented: no default or routing break
• Dependencies are properly updated

Documentation

• Documentation is clear and complete
• Comments are added for non-obvious code
• API changes are documented: new setting and search-log engine

Additional Notes

I work at Parallel, which operates this service. This gives PentAGI a reviewable, opt-in research fallback while preserving existing operator choices.