Skip to content

[BUG] multi-node gRPC tests are flaky #6431

Description

@rursprung

What is the bug?
see this description from @nibix:
#6310 (comment)

In the test framework, we have several nodes running in a single JVM. For an incoming GRPC request, the filter picks a backend registry instance from GuiceHolder. However, as this is a singleton, there is a 2:1 chance that it gets an instance from a wrong node. This will lead to the observed issue.

the problematic code is this here, which even includes a TODO:

// below is a hack because it seems not possible to access RepositoriesService from a non guice class
// the way of how deguice is organized is really a mess - hope this can be fixed in later versions
// TODO check if this could be removed
@Override
public Collection<Class<? extends LifecycleComponent>> getGuiceServiceClasses() {
if (client || disabled || SSLConfig.isSslOnlyMode()) {
return Collections.emptyList();
}
final List<Class<? extends LifecycleComponent>> services = new ArrayList<>(1);
services.add(GuiceHolder.class);
return services;
}

How can one reproduce the bug?
switch a gRPC test to DEFAULT rather than SINGLENODE cluster and run it several times. it'll be flaky.

What is the expected behavior?
no flaky tests

What is your host/environment?
tested on main with OpenSearch 3.9.0-SNAPSHOT

Do you have any screenshots?
n/a

Do you have any additional context?
n/a

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtriagedIssues labeled as 'Triaged' have been reviewed and are deemed actionable.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions