Memory issue after bumping to Node 22

[CPerry26]

Hey everyone,

I hope you're doing well!

After bumping to Node 22 (v22.22.0 and v8 12.4.254.21-node.33), we're running into memory issues in a Node API deployed into Kubernetes (not happening on another service running Node 18 in K8s). This API is predominantly GraphQL and uses Hapi under the hood, with the semi space size set to 16 and the pod limit at 1G. After some tuning and digging through heap snapshots, I can see the JSArrayBufferData growing by ~5MB every 10 minutes or so. When I look at the retainers, I can see some references to http-exporter-transport.js, http-transport-utils.js and AbstractAsyncHooksContextManager.js. I can also see the system context and MockHttpSockets growing. Some of this seems normal as the traces get exported to our configured exporter. I'm not a Node expert though so I could be interpreting incorrectly. I'm also not seeing any errors in the logs from the exports.

To be clear, I'm not saying there's a memory leak here. I know there was a bunch of changes under the hood in Node 22 which have caused us other issues (for examples how closure memory retainment works). But I'm running out of ideas and I'm wondering if anyone can help me understand some good next steps as I suspect it's a configuration issue on our end. Happy to provide a heap dump if that helps.

Below is our versions of dependencies and our OTEL setup code. Any help is appreciated!

Setup code (enabled instrumentations is controlled by environment variables - GraphQL, MongoDB, Dataloader, Hapi, HTTP, Winston):

const opentelemetry = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { diag, DiagConsoleLogger, DiagLogLevel } = require('@opentelemetry/api');
const { AsyncLocalStorageContextManager } = require('@opentelemetry/context-async-hooks');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.INFO);

const defaultInstrumentations = getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-graphql': {
        depth: 2,
        mergeItems: true,
        ignoreTrivialResolveSpans: true
    },
    '@opentelemetry/instrumentation-winston': {
        logHook: (span, record) => {
            const traceId = record.trace_id;
            const spanId = record.span_id;
            const traceFlags = record.trace_flags;

            record['traceId'] = traceId;
            record['spanId'] = spanId;
            record['traceFlags'] = traceFlags;

            delete record.trace_id;
            delete record.span_id;
            delete record.trace_flags;
        },
        disableLogSending: true
    },
    '@opentelemetry/instrumentation-http': {
        ignoreIncomingRequestHook: (request) => {
            const url = request.url || '';
            return url.includes('/_status') || url.includes('/metrics');
        }
    }
});

const spanProcessor = new BatchSpanProcessor(new OTLPTraceExporter(), {
    maxQueueSize: 150, // Reduced from default 2048
    maxExportBatchSize: 50, // Reduced from default 512
    scheduledDelayMillis: 1000, // Reduced from default 5000
    exportTimeoutMillis: 10000 // Reduced from default 30000
});

const sdk = new opentelemetry.NodeSDK({
    contextManager: new AsyncLocalStorageContextManager(),

    instrumentations: defaultInstrumentations,
    autoDetectResources: false,
    spanProcessor: spanProcessor,

    spanLimits: {
        attributeValueLengthLimit: 1024, // Max 1KB per attribute value
        attributeCountLimit: 32, // Max 32 attributes per span
        eventCountLimit: 32, // Max 32 events per span
        linkCountLimit: 32 // Max 32 links per span
    }
});

sdk.start();

process.on('SIGTERM', () => {
    sdk.shutdown()
        .then(
            () => console.log('OTEL shut down successfully.'),
            (err) => console.error('Error shutting down OTEL!', err)
        )
        .finally(() => process.exit(0));
});

The span processor and span limits were me attempting to try and tune. This was happening with the defaults as well.

Dependency versions:

├─┬ @opentelemetry/auto-instrumentations-node@0.69.0
│ ├── @opentelemetry/api@1.9.0
│ ├── @opentelemetry/core@2.5.0
│ ├── @opentelemetry/instrumentation-amqplib@0.58.0
│ ├── @opentelemetry/instrumentation-aws-lambda@0.63.0
│ ├── @opentelemetry/instrumentation-aws-sdk@0.66.0
│ ├── @opentelemetry/instrumentation-bunyan@0.56.0
│ ├── @opentelemetry/instrumentation-cassandra-driver@0.56.0
│ ├── @opentelemetry/instrumentation-connect@0.54.0
│ ├── @opentelemetry/instrumentation-cucumber@0.26.0
│ ├── @opentelemetry/instrumentation-dataloader@0.28.0
│ ├── @opentelemetry/instrumentation-dns@0.54.0
│ ├── @opentelemetry/instrumentation-express@0.59.0
│ ├── @opentelemetry/instrumentation-fastify@0.55.0
│ ├── @opentelemetry/instrumentation-fs@0.30.0
│ ├── @opentelemetry/instrumentation-generic-pool@0.54.0
│ ├── @opentelemetry/instrumentation-graphql@0.58.0
│ ├── @opentelemetry/instrumentation-grpc@0.211.0
│ ├── @opentelemetry/instrumentation-hapi@0.57.0
│ ├── @opentelemetry/instrumentation-http@0.211.0
│ ├── @opentelemetry/instrumentation-ioredis@0.59.0
│ ├── @opentelemetry/instrumentation-kafkajs@0.20.0
│ ├── @opentelemetry/instrumentation-knex@0.55.0
│ ├── @opentelemetry/instrumentation-koa@0.59.0
│ ├── @opentelemetry/instrumentation-lru-memoizer@0.55.0
│ ├── @opentelemetry/instrumentation-memcached@0.54.0
│ ├── @opentelemetry/instrumentation-mongodb@0.64.0
│ ├── @opentelemetry/instrumentation-mongoose@0.57.0
│ ├── @opentelemetry/instrumentation-mysql@0.57.0
│ ├── @opentelemetry/instrumentation-mysql2@0.57.0
│ ├── @opentelemetry/instrumentation-nestjs-core@0.57.0
│ ├── @opentelemetry/instrumentation-net@0.55.0
│ ├── @opentelemetry/instrumentation-openai@0.9.0
│ ├── @opentelemetry/instrumentation-oracledb@0.36.0
│ ├── @opentelemetry/instrumentation-pg@0.63.0
│ ├── @opentelemetry/instrumentation-pino@0.57.0
│ ├── @opentelemetry/instrumentation-redis@0.59.0
│ ├── @opentelemetry/instrumentation-restify@0.56.0
│ ├── @opentelemetry/instrumentation-router@0.55.0
│ ├── @opentelemetry/instrumentation-runtime-node@0.24.0
│ ├── @opentelemetry/instrumentation-socket.io@0.57.0
│ ├── @opentelemetry/instrumentation-tedious@0.30.0
│ ├── @opentelemetry/instrumentation-undici@0.21.0
│ ├── @opentelemetry/instrumentation-winston@0.55.0
│ ├── @opentelemetry/instrumentation@0.211.0
│ ├── @opentelemetry/resource-detector-alibaba-cloud@0.33.1
│ ├── @opentelemetry/resource-detector-aws@2.11.0
│ ├── @opentelemetry/resource-detector-azure@0.19.0
│ ├── @opentelemetry/resource-detector-container@0.8.2
│ ├── @opentelemetry/resource-detector-gcp@0.46.0
│ ├── @opentelemetry/resources@2.5.0
│ └── @opentelemetry/sdk-node@0.211.0 deduped
├─┬ @opentelemetry/sdk-node@0.211.0
│ ├── @opentelemetry/api-logs@0.211.0
│ ├── @opentelemetry/api@1.9.0 deduped
│ ├── @opentelemetry/configuration@0.211.0
│ ├── @opentelemetry/context-async-hooks@2.5.0
│ ├── @opentelemetry/core@2.5.0 deduped
│ ├── @opentelemetry/exporter-logs-otlp-grpc@0.211.0
│ ├── @opentelemetry/exporter-logs-otlp-http@0.211.0
│ ├── @opentelemetry/exporter-logs-otlp-proto@0.211.0
│ ├── @opentelemetry/exporter-metrics-otlp-grpc@0.211.0
│ ├── @opentelemetry/exporter-metrics-otlp-http@0.211.0
│ ├── @opentelemetry/exporter-metrics-otlp-proto@0.211.0
│ ├── @opentelemetry/exporter-prometheus@0.211.0
│ ├── @opentelemetry/exporter-trace-otlp-grpc@0.211.0
│ ├── @opentelemetry/exporter-trace-otlp-http@0.211.0
│ ├── @opentelemetry/exporter-trace-otlp-proto@0.211.0
│ ├── @opentelemetry/exporter-zipkin@2.5.0
│ ├── @opentelemetry/instrumentation@0.211.0 deduped
│ ├── @opentelemetry/propagator-b3@2.5.0
│ ├── @opentelemetry/propagator-jaeger@2.5.0
│ ├── @opentelemetry/resources@2.5.0 deduped
│ ├── @opentelemetry/sdk-logs@0.211.0
│ ├── @opentelemetry/sdk-metrics@2.5.0
│ ├── @opentelemetry/sdk-trace-base@2.5.0
│ ├── @opentelemetry/sdk-trace-node@2.5.0
│ └── @opentelemetry/semantic-conventions@1.39.0

[CPerry26]

I disabled the SDK and the memory looks much better. Sadly just disable GraphQL didn't help much.

[CPerry26]

I tried disabling some of the instrumentations (namely HTTP) to no avail. However, I switched the export protocol from http/json to gRPC. The pods have been running for about 17 hours and the memory looks much better. More fluctuation and not too high. I haven't pinpointed to something specific but it seems maybe something with the HTTP export on Node 22.