feat: integrate OTLP support with native gRPC and HTTP flushers

[yyuuttaaoo]

Expands the data handling capabilities of the application, allowing for better integration with OTLP-compliant systems.

• Added OTLP gRPC and HTTP flushers to enhance data export capabilities.
• Implemented OTLPForwardService to handle logs, metrics, and traces via gRPC.
• Introduced FlusherOTLPNative and FlusherOTLPHttpNative for sending data to OTLP endpoints.
• Updated CMake configuration to include new protobuf dependencies and directories.
• Enhanced application startup and shutdown processes to initialize and stop gRPC sinks.
• Added new serializers for OTLP data formats and integrated them into the flushing process.
• Updated metric constants to include new runner types for gRPC sinks.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR adds first-class OTLP ingestion and export capabilities, including a new OTLP forward gRPC service and native OTLP flushers for both gRPC and HTTP, plus corresponding build/test infrastructure updates.

Changes:

• Introduces OTLP gRPC forwarding (logs/metrics/traces) and a new gRPC sink path for async exporting.
• Adds native OTLP flushers (gRPC + HTTP) and OTLP serializers, with unit tests and e2e docker-compose scenarios.
• Updates build configuration (CMake/protobuf) and documentation to reflect the new OTLP capabilities.

Reviewed changes

Copilot reviewed 90 out of 91 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/engine/verify/otlp_collector.go	Adds verification helper to assert OTLP Collector file-exporter received expected signals.
test/engine/trigger/otlp/otelgen.go	Adds otelgen-based trigger to generate OTLP test data inside compose networks.
test/engine/steps.go	Registers new OTLP trigger/verification steps and adjusts After hook behavior.
test/engine/setup/env.go	Skips PID setup for docker-compose until containers start.
test/engine/setup/docker_compose.go	Sets Agent PID after docker-compose container startup.
test/e2e/test_cases/flusher_otlp_native/otel-collector-config.yaml	Adds OTel Collector config for gRPC-only receiver and file exporters.
test/e2e/test_cases/flusher_otlp_native/docker-compose.yaml	Adds compose service for OTel Collector for gRPC native flusher tests.
test/e2e/test_cases/flusher_otlp_native/case.feature	Adds e2e scenarios for OTLP native gRPC flusher (logs/metrics/traces).
test/e2e/test_cases/flusher_otlp_http_native/otel-collector-config.yaml	Adds OTel Collector config for gRPC+HTTP receivers and file exporters.
test/e2e/test_cases/flusher_otlp_http_native/docker-compose.yaml	Adds compose service for OTel Collector for OTLP HTTP native flusher tests.
test/e2e/test_cases/flusher_otlp_http_native/case.feature	Adds e2e scenarios for OTLP HTTP native flusher (logs/metrics/traces).
docs/cn/plugins/input/native/input-forward.md	Documents OTLP as an input_forward protocol and its mapping/matching behavior.
docs/cn/plugins/flusher/native/flusher-otlp.md	Adds documentation for flusher_otlp_native (OTLP/gRPC).
docs/cn/plugins/flusher/native/flusher-otlp-http.md	Adds documentation for flusher_otlp_http_native (OTLP/HTTP).
docs/cn/plugins/flusher/flushers.md	Lists new OTLP flushers in the flusher catalog.
docker/Dockerfile_development_part	Updates development image layout and startup method (control script).
core/unittest/sender/FlusherRunnerGrpcUnittest.cpp	Adds unit tests for PushToGrpcSink -> GrpcSink path.
core/unittest/sender/CMakeLists.txt	Builds and registers new sender gRPC unit test target.
core/unittest/pipeline/GrpcSinkMock.h	Adds a mock GrpcSink implementation for unit tests.
core/unittest/forward/OTLPForwardServiceUnittest.cpp	Adds unit tests for OTLP forward gRPC service behavior and config matching.
core/unittest/forward/CMakeLists.txt	Builds and registers OTLP forward service unit tests.
core/unittest/flusher/FlusherOTLPHttpNativeUnittest.cpp	Adds unit tests for OTLP HTTP native flusher request building and send handling.
core/unittest/flusher/CMakeLists.txt	Builds new OTLP flusher unit test binaries.
core/unittest/CMakeLists.txt	Adds OTLP proto sources into unit test build inputs.
core/runner/sink/grpc/GrpcSink.h	Introduces GrpcSink interface for async OTLP gRPC exporting.
core/runner/sink/grpc/GrpcSink.cpp	Implements the gRPC sink worker loop and async dispatch.
core/runner/sink/SinkType.h	Adds GRPC sink type.
core/runner/ProcessorRunner.cpp	Adds additional processing log line (currently at INFO).
core/runner/FlusherRunner.h	Adds PushToGrpcSink entry point.
core/runner/FlusherRunner.cpp	Dispatches GRPC sink type and adds PushToGrpcSink implementation.
core/protobuf/opentelemetry/proto/resource/v1/resource.proto	Adds OTLP Resource proto.
core/protobuf/opentelemetry/proto/logs/v1/logs.proto	Adds OTLP Logs proto.
core/protobuf/opentelemetry/proto/common/v1/common.proto	Adds OTLP Common proto.
core/protobuf/opentelemetry/proto/collector/trace/v1/trace_service.proto	Adds OTLP TraceService proto.
core/protobuf/opentelemetry/proto/collector/metrics/v1/metrics_service.proto	Adds OTLP MetricsService proto.
core/protobuf/opentelemetry/proto/collector/logs/v1/logs_service.proto	Adds OTLP LogsService proto.
core/plugin/input/InputForward.cpp	Adds OTLP protocol support via OTLPForwardServiceImpl registration.
core/plugin/flusher/opentelemetry/FlusherOTLPNative.h	Adds OTLP native gRPC flusher interface and sender item/context types.
core/plugin/flusher/opentelemetry/FlusherOTLPHttpNative.h	Adds OTLP HTTP native flusher interface.
core/plugin/flusher/opentelemetry/FlusherOTLPHttpNative.cpp	Implements OTLP HTTP native flusher serialization and request building.
core/monitor/metric_constants/RunnerMetrics.cpp	Adds grpc_sink runner label constant.
core/monitor/metric_constants/MetricConstants.h	Declares grpc_sink runner label constant.
core/forward/otlp/OTLPForwardService.h	Adds OTLP forward service wrapper classes (logs/metrics/traces) and impl API.
core/forward/loongsuite/LoongSuiteForwardService.h	Adds GetGrpcServices override and minor header includes cleanup.
core/forward/GrpcInputManager.cpp	Registers all services returned by BaseService::GetGrpcServices().
core/forward/BaseService.h	Adds GetGrpcServices API to support multi-service servers.
core/dependencies.cmake	Adds OTLP proto compile steps to generate C++ sources.
core/collection_pipeline/serializer/OTLPHttpSerializer.h	Adds serializer that can output OTLP JSON and protobuf binary.
core/collection_pipeline/serializer/OTLPHttpSerializer.cpp	Implements OTLP request building for logs/metrics/traces and JSON/protobuf serialization.
core/collection_pipeline/plugin/PluginRegistry.cpp	Registers new OTLP native flushers.
core/application/Application.cpp	Initializes/stops GrpcSink alongside HttpSink.
core/CMakeLists.txt	Adds grpc sink + OTLP directories and includes proto-generated headers + sources.
.gitignore	Expands ignores for IDE configs and protobuf-generated headers.
.devcontainer/start-dind.sh	Adds helper script to start DinD with cgroup v1 nesting fix.
.devcontainer/devcontainer.json	Enables DinD data volume and auto-starts DinD after container start.
.devcontainer/Dockerfile	Installs DinD tooling and updates repo mirror settings.
.claude/skills/testing-standards/SKILL.md	Adds internal guidance for test standards.
.claude/skills/selfmonitor/SKILL.md	Adds internal guidance for metric/alarm conventions.
.claude/skills/security-check/scripts/security_reset.sh	Adds script to squash/reset commits containing sensitive info.
.claude/skills/security-check/scripts/security_check.sh	Adds script to scan staged changes/commits for sensitive tokens.
.claude/skills/security-check/SKILL.md	Documents sensitive info checks and cleanup flow.
.claude/skills/riper5-protocol/SKILL.md	Adds internal workflow protocol reference.
.claude/skills/review-standards/SKILL.md	Adds internal code review checklist and standards.
.claude/skills/project-knowledge/SKILL.md	Adds internal project architecture and conventions reference.
.claude/skills/omc-reference/SKILL.md	Adds internal multi-agent orchestration reference.
.claude/skills/mermaid/SKILL.md	Adds internal Mermaid formatting conventions.
.claude/skills/e2e-manual/SKILL.md	Adds internal e2e manual testing guidance.
.claude/skills/e2e-develop-guide/SKILL.md	Adds internal e2e framework development guidance.
.claude/skills/design-document/SKILL.md	Adds internal design doc conventions.
.claude/skills/compile/SKILL.md	Adds internal build instructions.
.claude/skills/commit/SKILL.md	Adds internal conventional commit guidance.
.claude/skills/code-review/scripts/update_comment_status.py	Adds tooling to track review comment status across rounds.
.claude/skills/code-review/scripts/init_review_workspace.py	Adds tooling to initialize code-review workspace files.
.claude/skills/code-review/scripts/generate_comment_status_report.py	Adds tooling to generate markdown status reports.
.claude/skills/code-review/scripts/fetch_review_comments.py	Adds tooling to fetch PR review comments with resolution state.
.claude/skills/code-review/scripts/build_snapshot.py	Adds tooling to snapshot changed files per review round.
.claude/skills/code-review/references/reviewed_commits.template.json	Adds template for reviewed commit tracking.
.claude/skills/code-review/references/review-plan.template.md	Adds template for review plan.
.claude/skills/code-review/references/meta.template.json	Adds template for review metadata.
.claude/skills/code-review/references/failure-playbook.md	Adds failure recovery playbook for review automation.
.claude/skills/code-review/references/comment-status.template.json	Adds template for comment status tracking.
.claude/settings.json	Enables an internal plugin configuration.
.claude/CLAUDE.md	Adds internal orchestration guidance for Claude tooling.

Comments suppressed due to low confidence (14)

test/engine/verify/otlp_collector.go:1

• context.WithTimeout(context.TODO(), ...) ignores the caller’s ctx (cancellation/deadlines won’t propagate). Also, resolveLocalPath’s fallback branch trims the same prefix again, so it never handles variants like "/tmp/otel-export" (no trailing slash). Use the passed-in ctx for the timeout base (e.g., ctx), and fix the fallback to handle both "/tmp/otel-export/" and "/tmp/otel-export" prefixes.
  test/engine/verify/otlp_collector.go:1
• context.WithTimeout(context.TODO(), ...) ignores the caller’s ctx (cancellation/deadlines won’t propagate). Also, resolveLocalPath’s fallback branch trims the same prefix again, so it never handles variants like "/tmp/otel-export" (no trailing slash). Use the passed-in ctx for the timeout base (e.g., ctx), and fix the fallback to handle both "/tmp/otel-export/" and "/tmp/otel-export" prefixes.
  test/engine/trigger/otlp/otelgen.go:1
• The docker argument list includes an extra "--" before the image name; docker run expects IMAGE directly after options, and "--" here can be interpreted as the image name and break the command. Also, for non-metrics paths, exec.Command ignores ctx cancellation; use exec.CommandContext(ctx, ...) so the test runner can reliably abort otelgen runs.
  test/engine/trigger/otlp/otelgen.go:1
• docker ps --format "{{.ID}}" can return multiple container IDs separated by newlines; trimming whitespace can leave multiple IDs, causing docker inspect ... <containerID> to fail. Split by lines and select the first non-empty ID (or iterate) to make network discovery deterministic.
  core/plugin/flusher/opentelemetry/FlusherOTLPHttpNative.cpp:1
• The implementation defaults mFormat to Protobuf (header), but the comment, warning text, and log line claim the default is JSON when Format is absent/invalid. This produces misleading logs and makes troubleshooting configuration behavior harder. Align the comment/log/warning with the actual default (or change the default to match the documented behavior), and ensure the “invalid Format” branch reports the real fallback.
  core/plugin/flusher/opentelemetry/FlusherOTLPHttpNative.cpp:1
• The implementation defaults mFormat to Protobuf (header), but the comment, warning text, and log line claim the default is JSON when Format is absent/invalid. This produces misleading logs and makes troubleshooting configuration behavior harder. Align the comment/log/warning with the actual default (or change the default to match the documented behavior), and ensure the “invalid Format” branch reports the real fallback.
  core/plugin/flusher/opentelemetry/FlusherOTLPHttpNative.cpp:1
• std::stoi will throw on malformed ports (e.g., http://host:abc/...) and can crash the process since the exception isn’t handled. Since Url is user configuration, catch conversion errors and return false with a clear errMsg (and set keepItem appropriately). Also consider validating/infering TLS from the URL scheme to avoid inconsistent configs like https://... with EnableTLS=false.
  core/plugin/flusher/opentelemetry/FlusherOTLPHttpNative.cpp:1
• std::stoi will throw on malformed ports (e.g., http://host:abc/...) and can crash the process since the exception isn’t handled. Since Url is user configuration, catch conversion errors and return false with a clear errMsg (and set keepItem appropriately). Also consider validating/infering TLS from the URL scheme to avoid inconsistent configs like https://... with EnableTLS=false.
  core/runner/ProcessorRunner.cpp:1
• This adds an unconditional LOG_INFO on the processor hot path and can spam logs / increase CPU under load. It should be downgraded to a lower verbosity (e.g., DEBUG/TRACE) and/or gated behind a flag to avoid production noise and performance regression.
  core/runner/sink/grpc/GrpcSink.cpp:1
• The “forced to stopped” path doesn’t actually stop the async worker; the std::future remains valid and may still block later (e.g., at destruction) depending on the implementation. Consider switching to a std::thread that you can deterministically join, or ensure Stop performs a definitive shutdown (cancel in-flight RPCs / clear pending queue / signal Run to exit) and then get() the future when ready to avoid hidden hangs.
  core/runner/sink/grpc/GrpcSink.cpp:1
• This busy-wait loop throttles concurrency via polling/sleep and will waste CPU under sustained pressure. Prefer a semaphore/condition-variable approach (signal on decrement) so dispatch blocks efficiently until capacity is available.
  docs/cn/plugins/flusher/flushers.md:1
• The documented plugin names don’t match the implemented/introduced native flusher names (flusher_otlp_native and flusher_otlp_http_native). This will mislead users and break copy/paste configs. Update the table entries to the correct plugin IDs.
  docker/Dockerfile_development_part:1
• The new CMD relies on loongcollector_control.sh being executable, but the Dockerfile doesn’t chmod +x it after copying. Add an explicit chmod (or ensure the repo file has the executable bit preserved in builds) to prevent container startup failures.
  docker/Dockerfile_development_part:1
• The new CMD relies on loongcollector_control.sh being executable, but the Dockerfile doesn’t chmod +x it after copying. Add an explicit chmod (or ensure the repo file has the executable bit preserved in builds) to prevent container startup failures.