Export aggregated OTLP client operation metrics

docs/guides/operations/logs-and-metrics.md

@@ -29,7 +29,8 @@ cortex-training tui JOB_ID
 Recipe-level metrics are also written under each recipe's `log_path`.
 
 PAT-authenticated Python clients also emit best-effort client-side operation
-metrics over Snowflake's OTLP endpoint. Failures of essential SDK methods are
-emitted automatically; successful outcomes require
+metrics over Snowflake's OTLP endpoint. Aggregate success and failure metrics
+for essential SDK methods are emitted automatically. Detailed failure logs are
+also automatic; detailed success logs require
 `CORTEX_TRAINING_ENABLE_SUCCESS_TELEMETRY=1`. See
 [Client metrics](../../reference/python-sdk.md#client-metrics).

docs/reference/cli.md

@@ -627,8 +627,9 @@ CORTEX_TRAINING_ENDPOINT
 
 `CORTEX_TRAINING_DISABLE_TELEMETRY` (truthy) skips OTLP client metrics on
 PAT-authenticated clients. `CORTEX_TRAINING_ENABLE_SUCCESS_TELEMETRY`
-(truthy) also emits successful outcomes for essential operations; failures
-are emitted by default. See the [Python SDK reference](python-sdk.md#client-metrics).
+(truthy) additionally emits detailed success logs; aggregate success and
+failure metrics are emitted by default. See the
+[Python SDK reference](python-sdk.md#client-metrics).
 
 ### Troubleshooting
 

docs/reference/python-sdk.md

@@ -26,9 +26,8 @@ Tuning knobs on the constructor: `endpoint`, `poll_interval` (0.5s),
 
 ## Client metrics
 
-Clients created with `CortexTrainingClient.from_pat` automatically emit one
-best-effort event when an essential operation fails. Set
-`CORTEX_TRAINING_ENABLE_SUCCESS_TELEMETRY=1` to also emit successful outcomes.
+Clients created with `CortexTrainingClient.from_pat` automatically aggregate
+best-effort success and failure metrics for essential operations.
 Local or mock clients constructed with an explicit `base_url` treat
 `emit_metric` as a no-op. Set `CORTEX_TRAINING_DISABLE_TELEMETRY=1` to skip
 constructing the emitter.
@@ -44,16 +43,29 @@ Tracked operations:
   `router_replay_discard`, `reset_prefix_cache`
 - Async requests: `poll_request`, `get_request_status`, `cancel_request`
 
-Each emitted event body contains `success`, `duration_ms`, `request_count`,
-`attempt_count`, and `retry_count`. Failures also include a bounded
-`error_message` with common credential patterns redacted. Queryable attributes
-include available `job_id`, sub-job identifiers, `request_id`,
-`checkpoint_id`, `error.type`, HTTP status, Snowflake request ID, and server
-error code. Records use OTLP resource attributes
-`service.name = cortex-training` and
-`snowflake.account_host = <normalized connection hostname>`.
-
-Applications can emit additional events through the same helper:
+The client periodically exports these delta metrics, grouped only by
+`operation` and `outcome` (`success` or `failure`):
+
+- `cortex.training.client.operation.count`
+- `cortex.training.client.operation.duration` (histogram in milliseconds)
+- `cortex.training.client.operation.retries`
+- `cortex.training.client.operation.requests`
+
+An operation is successful when the client method returns; asynchronous job
+completion is separate. Failures also emit a detailed log with a bounded,
+credential-redacted error message and available job, sub-job, request,
+checkpoint, HTTP status, Snowflake request ID, and server error-code fields.
+Set `CORTEX_TRAINING_ENABLE_SUCCESS_TELEMETRY=1` only for temporary diagnostics
+to emit detailed success logs too.
+
+Telemetry resource attributes include `service.name = cortex-training`, the
+installed package version in `service.version`, the package surface in
+`cortex.training.client.surface`, and the normalized connection hostname in
+`snowflake.account_host`. API requests use
+`User-Agent: cortex-training/<installed package version>`.
+
+Applications can emit additional diagnostic log events through the legacy
+helper:
 
 ```python
 client.emit_metric(

src/cortex_training/__init__.py

@@ -15,7 +15,7 @@
 
 """Public Python API for Cortex Training."""
 
-__version__ = "0.0.2"
+from ._version import __version__
 
 from . import wire
 from .client import ChunkGroupConflictError

src/cortex_training/_version.py

@@ -0,0 +1,15 @@
+# Copyright 2025 Snowflake Inc.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Installed package version and HTTP user agent."""
+
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version
+
+
+try:
+    __version__ = version("cortex-training")
+except PackageNotFoundError:  # Source tree imported without an installation.
+    __version__ = "0.0.3"
+
+USER_AGENT = f"cortex-training/{__version__}"

src/cortex_training/client.py

@@ -64,6 +64,8 @@
 from urllib3.exceptions import NewConnectionError
 
 from cortex_training import wire
+from cortex_training._version import USER_AGENT
+from cortex_training._version import __version__
 from cortex_training.telemetry import CachedSessionTokenProvider
 from cortex_training.telemetry import OtlpMetricEmitter
 
@@ -1098,6 +1100,7 @@ def __init__(
         self._metric_emitter: OtlpMetricEmitter | None = None
         self._operation_metric_state = threading.local()
         self._session = requests.Session()
+        self._session.headers["User-Agent"] = USER_AGENT
         adapter = HTTPAdapter(pool_connections=pool_maxsize, pool_maxsize=pool_maxsize)
         self._session.mount("https://", adapter)
         self._session.mount("http://", adapter)
@@ -1142,6 +1145,8 @@ def from_pat(
             client._metric_emitter = OtlpMetricEmitter(
                 client.base_url,
                 token_provider,
+                service_version=__version__,
+                service_surface="cortex-training",
                 verify_ssl=verify_ssl,
                 timeout=telemetry_timeout,
             )
@@ -1158,7 +1163,7 @@ def emit_metric(
         *,
         attributes: dict[str, Any] | None = None,
     ) -> None:
-        """Emit a best-effort OTLP log record used as a client metric.
+        """Emit a best-effort OTLP diagnostic log record.
 
         PAT clients lazily exchange the PAT for a cached session token on the
         first call. Local/mock clients, or clients constructed with
@@ -1196,11 +1201,9 @@ def _emit_operation_outcome(
         result: Any = None,
         error: BaseException | None = None,
     ) -> None:
-        """Emit one structured outcome without changing the operation result."""
+        """Record aggregate metrics and an optional detailed outcome log."""
         if self._metric_emitter is None:
             return
-        if error is None and not _success_telemetry_enabled():
-            return
         try:
             attempt_count = int(
                 getattr(self._operation_metric_state, "attempt_count", 0)
@@ -1215,6 +1218,15 @@ def _emit_operation_outcome(
                 "retry_count": max(attempt_count - request_count, 0),
                 "success": error is None,
             }
+            self._metric_emitter.record_operation(
+                operation,
+                outcome="success" if error is None else "failure",
+                duration_ms=value["duration_ms"],
+                retry_count=value["retry_count"],
+                request_count=value["request_count"],
+            )
+            if error is None and not _success_telemetry_enabled():
+                return
             attributes = _metric_identity(operation, arguments, result)
             if error is not None:
                 value["error_message"] = _safe_metric_error_message(error)