Ext authz caching

[toddmgreer]

Commit Message: Cache-enable Envoy External Authentication HTTP Filter
Additional Description: Give ext_authz a way to be overridden by cached authentication results and to inform a cache of the same.
Risk Level: Low. The new feature is only active when configured, and is limited in scope.
Testing: See new tests in PR.
Docs Changes: In ext_authz_filter.rst.
Release Notes: In current.yaml‎
Platform Specific Features: none
Fixes #44852

I used the Gemini generative AI for both code and tests. I have reviewed and understand all generated code, though I'm new to ext_authz, so I could have misunderstandings.

[repokitteh-read-only]

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @markdroth
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #44874 was opened by toddmgreer.

see: more, trace.

[toddmgreer]

Added docs and release notes.

[ggreenway]

Decoupling concerns is a nice goal, but this is very hard to use or configure. I think this needs a built-in cache implementation that does something sensible. This topic has come up before, and I think we should at least consider having the caching be an extension point on the ext_authz filter.

What is the expected flow for inserting something into the cache? How does the message get from the ext_authz response to the caching filter?

This requires an integration test.

/wait

[toddmgreer]

Thank you for the quick initial review. I clearly didn't adequately explain the motivation--sorry.

My team is working on a caching service for Envoy with the following goals:

1. cache ext_authz, ext_proc, and HTTP responses
2. minimize the burden on projects using Envoy proxies by integrating into Envoy using the ext_proc filter. (There is resistance to compiling and linking in new types of filters, both to minimize the code that could crash Envoy, and to reduce administrative burden.)

From here on, I'm just going to talk about caching for ext_authz filters, but caching for ext_proc filters will use the same flows. (It just gets confusing otherwise because the caching filter is itself an ext_proc filter.)

Users of this service will add an ext_proc filter downstream of their ext_authz filters. There are two new flows:

1. Cache hit:
   In decodeHeaders, the caching ext_proc filter will send a request to an external caching service. If it returns any cached responses, the caching ext_proc filter will put them in dynamic metadata. In ext_authz's decodeHeaders, it will find the cached response and apply it, as if it had received it as an RPC response.
2. Cache miss
   In decodeHeaders, the caching ext_proc filter will send a request to an external caching service, which will look for cached responses. If the external caching service doesn't return any cached responses, the caching ext_proc filter will let iteration continue normally. In ext_authz's decodeHeaders, it will store the response it got from the external authorization service. In its encodeHeaders, the caching ext_proc filter will send that (along with the request) to the external service for caching.

I'm leaving out a lot of details on the caching service, including mountains of configuration (various ttls, freshness and validation rules, cache key construction, variant handling, and so much more), because this PR is only about the ext_authz changes.

BTW, this feature could be used for purposes other than caching. I could imagine a filter that applies some sort of custom logic to force the ext_authz filter to accept/reject a request. Perhaps there are more use cases.

I'll add an integration test.

Thank you,
Todd

[ggreenway]

Why even use ext_authz? ext_proc is roughly a superset of what ext_authz can do. Why involve two filters and make this complicated relationship, when you could just have ext_proc send back an unauthorized response.

[toddmgreer]

Some of our proxies have multiple ext_authz filters. Their configurations have been debugged and work. They talk to mature authentication services owned by different groups. We want to put one caching ext_proc filter in front of them that makes one request to a caching service to look for cached responses for all of them. With this proposal, we add a single field to each of their configs (to say where in dynamic metadata the cached responses go), and we add one new ext_proc filter that is configured to talk to all of them.

If we just use ext_proc to do the caching and multiple auth checks, we'd have to write a custom filter, because ext_proc can't talk to multiple external services. Additionally, we'd have to combine all of the config from the cache and from all of the original ext_* filters. Our proxy owners would not accept this solution. (In fact, their dislike of compiling/linking/configuring custom filters is why we started on this path.) Alternatively, we could use a single ext_proc filter, and move all the logic about contacting external services into the caching service. This would be worse, because it would move proxy-specific behaviors and config into what is intended to be a proxy-agnostic caching service.

We're absolutely open to other ways to do this--this is the only good approach we've found, it's always possible that we may have overlooked something.

BTW, when I wrote the design doc and the PR, I was unsure how much to go into these questions. (The ratio of design doc length to code length started to get pretty high.) I may have favored brevity too much.

[ggreenway]

My concern is that you're adding code and complexity to ext_authz for a very specific use case that it's unlikely anyone else will use.

[toddmgreer]

It's much simpler than any other way to cache the results of ext_authz calls. If your concern is that we're the only ones who want to cache ext_authz, then my response is that there are several very different teams in Google with proxies that would like to use this, and Google isn't all that weird anymore, so something needed by several of our groups is probably also wanted by some non-Google users as well.

Added integration tests.

[ggreenway]

There's been plenty of interest in caching ext_authz response; I believe you on that. But this implementation is very specific and difficult for others to use.

I'd like to see an implementation that is in some way usable for others. Maybe make an additional filter that is a cache implementation (with no external calls; just in-memory).

I think using untyped dynamic metadata makes the interface harder to use; it would be better to fix ext_proc to support typed metadata.

[toddmgreer]

If we switch from untyped to typed metadata, do you think that would give this wide enough appeal? Seems like a reasonable approach.

[toddmgreer]

The advantage of using untyped metadata is that the cache doesn't have to know what type of thing ext_authz uses. ext_authz hands the cache an opaque string, and the cache knows to hand that same opaque string back later (when it sees a request with headers that match the headers that generated that opaque string).

IIUC, adding typed_metadata support to ext_proc would require adding a new field to ProcessingResponse:
envoy.service.auth.v3.CheckResponse authz_check_response = ;

I don't think that would be an improvement, but I'm not totally clear on how typed metadata works. I assume the code has to actually know what types are involved.

[toddmgreer]

Ok, now that I've read some typed metadata docs, it again sounds reasonable. Please ignore my previous message.

To use typed dynamic metadata:
We'd add a new field to ProcessingResponse.response:
google.protobuf.Any typed_dynamic_metadata = 13;

and another like it in ProcessingRequest.request,
and some new config to say where to put it during decode / where to load from during encode,
and ext_proc would just copy between that dynamic typed metadata location and the ProcessingRequest/Response, without having to know what it is.

It would be somewhat less efficient, due to sending the type info to the caching service, but that's probably not a big deal. I prefer the existing version, due to its slightly better efficiency, but I'll see if the rest of the team raise any big concerns with using typed metadata.

[toddmgreer]

Switched to typed dynamic metadata. You were right--this is better

[tyxia]

There's been plenty of interest in caching ext_authz response; I believe you on that. But this implementation is very specific and difficult for others to use.

I'd like to see an implementation that is in some way usable for others. Maybe make an additional filter that is a cache implementation (with no external calls; just in-memory).

I think using untyped dynamic metadata makes the interface harder to use; it would be better to fix ext_proc to support typed metadata.

Hi @ggreenway , I just wanted to clarify and give a e2e example flow here: This PR is to give ext_authz a way to use cached results from ext_proc callout. In other words, we leverage the ext_proc here for external callout to caching service and other filters can use it as well:
On cache hit: downstream ext_proc populates the metadata, the ext_authz checks metadata and then skips gRPC call. On cache miss: ext_proc sees the dynamic metadata set by the gRPC ext_authz call and fills it to the external cache.

Regarding the in-memory cache option: Yes, it is viable option and it is more performant. But it has its own limitation: in-memory cache also is tied with Envoy instance (depends on implementation: per filter, thread local, or process wide). This will lead to inaccurate cache miss. In order to have a global view across envoy/ext_authz instances, I think we will need to have some kinds of external service/storage. With that said, i think we can provide multiple options here : remote vs in-memory for customer to choose based on their use cases and performance requirement. We are currently first implement the remote option due to our deployment requirement and use case.

[kyessenov]

my 2c: RLQS is actually quite similar to what you'd want from a caching authz. The servers should specify a bucket and an expiration deadline for the decisions. The bucket is a matcher over the request so that you can group. I don't quite understand why you'd want to use ext_authz if the decision is already made, it might be easier to just wrap into some conditional filter.

[markdroth]

I added some comments in #44852. I'm not sure this is really the best approach, given that this is something we're going to also need to support in gRPC.

[toddmgreer]

@kyessenov The reason for this PR is that we have envoy deployments that currently use ext_authz, and we want to reduce their average latency and server load. We want to do this with minimal changes in the config of those envoys, and without requiring that they link in custom filters, and the many of the auth decisions are highly cacheable in nature.

The cache server will indeed be configured with a specification of how to match requests (like RLQS's buckets*) and with expiry information.

We considered something like a conditional feature, but that would only work well if the auth decisions were only accept/reject. However, they routinely include header modification, and we don't want to replicate that behavior into a different filter. Instead, we want to avoid the RPC to the auth server, but still let the ext_authz filter apply the decision as it does today.

• based on your description. I don't know enough about RLQS to say how similar they are or arent.

[markdroth]

I don't think this interface should use CheckRequest as the argument type, because we don't want the data plane to have to construct the ext_authz request proto if it doesn't actually have to send the ext_authz request.

I'm not sure what the right type is to use in Envoy's implementation, but I think what we want here is to pass in the request attributes.

[toddmgreer]

I'm guessing that you don't want me to pass the AttributeContext messages, because constructing them is most of the cost of constructing a RequestCheck. To minimize cost, I switched to passing the objects that ext_authz uses as inputs to construct the RequestCheck, or the decoder_callbacks for things that can be reached through that. This works well for our internal cache implementation and seems like a reasonable choice for the general case.

[toddmgreer]

All comments are addressed, and the PR should be ready for further review.

[markdroth]

This looks like the right approach! Just one small comment from my side.

[toddmgreer]

/publish

[toddmgreer]

/kick

[markdroth]

/lgtm api

[toddmgreer]

@ggreenway, @tyxia, and @yanjunxiang-google, the api review is done so you're up!

I think the failure to restore the docker cache in verify_distro must be a flake, but my attempts to trigger a retest didn't work. Maybe you have to be a maintainer? Should I push an empty commit?