Generic API for flag propagation

[gregestren]

Please note @aranguyen 's Generic API for Bazel Flag Propagation proposal.

Summary:

• Today, Bazel assumes all build flags propagate completely down the build graph
• That can waste resources (memory, speed) for flags only meant for small portions of the graph
• The only way to change that is with carefully crafted transitions or adjusting exec transition logic
• The doc proposes adding a "propagation level" enum to all flags. Which can be tuned in subsequent proposals for more intelligent guidance for which flags should propagate where
• One interesting use case would be "project-scoped" flags, i.e. --//foo:my_flag automatically resets on targets outside of //foo. This could be especially useful for the "set a flag and consume it in a select() in my project" pattern.

[gregestren]

@fmeum

[gregestren]

Proposal feedback summary:

What if we want to limit other flags like --copt?

The immediate proposal adds support for limiting Starlark flags to "project-scope". Where "project" is defined by aPROJECT.scl metadata source-of-truth we're adding into Bazel. It doesn't support other scopes or native flags.

But the core innovation is making Bazel able to strip flags from the build graph based on user-defined annotations.

With this, we could build additional proposals for other scopes and flags Examples:

1. dont_propagate_to_exec annotations on Starlark flags. See @bazel_tool//tools/cpp:link_extra_libs should not propagate to exec config #22457.
2. --copt that only applies to the immediate project being built
3. Entire sets of flags, i.e. my_project_config = ["--copt=a", "--//my:flag=b", --platforms=//c] only applies to the immediate project
4. rule-scoped settings (these settings apply to language X but not data dependencies)
5. scope logic determined by arbitrary Starlark

What if we project-scope a flag that leaves, then re-enters the project?

i.e.:

• limit a flag to //foo/...
• $ bazel build //foo:buildme, where //foo:buildme -> //bar:out_of_project -> //foo:back_in_project.

IMO, at least for now I think the flag should be removed when it leaves //foo:buildme. If people need it to pass to //foo:back_in_project they can refactor their project definitions to accommodate.

I'm open to contrary use cases. For example if //foo is some framework where users inject their own libraries and that frameworks calls out to the user-controlled libraries but also adds framework-defined deps to them.

This is also something we can incrementally model more based on need.

Why not just re-use visibility for Starlark flags?

The proposal suggests a new scope attribute.

Conceptually, visibility covers a similar purpose and we could just leverage that.

It's an interesting connection. Maybe one day they could merge? scope is intended to be more generic than what visibility is currently restricted to (i.e. visibility is a list of packages, whereas scope might be packages, might be target vs. exec, might be a Starlark function). I'm inclined to, at least for now, not try to think through the merging challenge. And just get out a V 1.0 that works.

That said, using visibility as syntactic sugar over an explicit scope might be nice.

[gregestren]

FYI we applied a first version of this to a Google project to great success. The case where this helps:

• //foo:target has a transition that sets --//foo:some_flag
• --//foo:some_flag only matters to targets in //foo, //foo_support, and //foo_tests (like maybe a select()). In other words: foo's project includes a few different packages, and those are the packages where the flag matters.
• //foo:target has a large, expensive subgraph outside those packages. We don't want to fork that subgraph.
• This feature automatically resets --//foo:some_flag on targets outside //foo, //foo_support, and //foo_tests. So the subgraph can't fork regardless of what the transition does.

This is especially noticeable for $ bazel build //foo:all type builds, where different targets in //foo might transition the flag differently. But it also improves remote caching on the non-foo deps, regardless of which targets are built when.