feat: per-module library dependency filter via ocamldep BFS#14516
Draft
robinbb wants to merge 3 commits into
Draft
feat: per-module library dependency filter via ocamldep BFS#14516robinbb wants to merge 3 commits into
robinbb wants to merge 3 commits into
Conversation
This was referenced May 13, 2026
robinbb
force-pushed
the
robinbb-14492-l3-includes-refactor
branch
from
84710e9 to
d197872
Compare
…h cctx Pure-additive scaffolding for #4572's per-module inter-library filter. Fields are populated but no consumer reads them yet; the per-module filter (which is the only caller) lands in a follow-up. [Compilation_context]: - [build_lib_index]: builds a [Lib_file_deps.Lib_index.t] from the cctx's direct + hidden libs. Each entry carries [Some Module.t] for unwrapped locals (tight-eligible) and [None] otherwise (wrapped locals / externals). Local libs whose source is preprocessed by a non-staged ppx are indexed on the post-pp module so the cross-lib walker reads ocamldep on the same source the dep lib's compile pipeline produces. - Three new fields on [t]: [lib_index] (Memo.Lazy.t computing the index), [has_virtual_impl] (Memo.Lazy.t flag — true iff any dep lib implements a virtual lib), and [pps_runtime_libs] (closure of ppx_runtime_libraries introduced by [pps] in this stanza, threaded through [create]). - Accessors for each. [for_module_generated_at_link_time] populates [lib_index] with a [Code_error.raise] sentinel — the per-module filter's [can_filter] guard prevents reaching it from synthesised link-time cctxs. [Lib_rules] / [Exe_rules]: compute [pps_runtime_libs] from the stanza's [compile_info] and thread it through [Compilation_context.create]. [Dep_graph]: expose [dir] and [mem]; the cross-lib walker's [can_filter] guard uses these to detect synthesised dummy graphs. [Modules]: add [as_singleton] returning [Some m] iff the module set is a single user-written module. Used by [build_lib_index] to detect the single-module-no-deps short-circuit case. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
force-pushed
the
robinbb-14492-l3-includes-refactor
branch
from
d197872 to
d3418da
Compare
Behavior-equivalent restructuring that opens the lib-deps computation to per-module filtering in a follow-up. No test promotions. [Compilation_context.Includes.make] previously emitted both [-I]/[-H] include flags AND [Hidden_deps] for the cctx's libs in a single [Command.Args.t]. The opaque-aware [Cmx] case duplicated the deps logic that already exists in [Lib_file_deps.deps_of_entries]. Simplified: [Includes.t] now carries only the include flags; the [~opaque] parameter (and the [for_module_generated_at_link_time] call site) drops out. [Module_compilation]: - [lib_deps_for_module]: scaffold form, returns [(cctx_includes, deps_of_entries libs)] where [libs = requires_compile @ requires_hidden]. The per-module tight filter activates in a follow-up; arguments [obj_dir], [for_], [dep_graph], [ml_kind], [mode] are threaded but ignored here. - [lib_cm_deps]: wraps [lib_deps_for_module] with [Action_builder.dyn_deps], yielding the include args and registering the lib file deps. - [build_cm]: gated route — [Alias _] (non-stdlib) and [Wrapped_compat] modules short-circuit to the cctx's now-flag-only [Includes] (no lib deps, matching prior behavior since [Includes.empty] was used for these); all other module kinds call [lib_cm_deps]. Replace the in-line [Includes] lookup at the [Command.run] site with [Command.Args.Dyn lib_cm_deps]. - [ocamlc_i]: same swap. Combined: every consumer that previously read [-I]/[-H] + [Hidden_deps] from [Includes] now reads [-I]/[-H] from [Includes] and the deps from [deps_of_entries] (via [lib_cm_deps]). Same flags, same deps. The [Alias]/[Wrapped_compat] short-circuit preserves the prior "no-lib-deps" behavior for those module kinds. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
Activates the tight branch in [lib_deps_for_module]: a per-module BFS over the cross-library dependency graph (built from each lib's [ocamldep -modules] output, normalised through [build_lib_index]'s post-pp module map) produces the set of dep-lib modules actually referenced by the consumer module. The compile rule sees only those [.cmi]/[.cmx] files; sibling-module recompilations on unreferenced dep-lib cmi changes drop out. Include flags are still the cctx-wide [-I]/[-H] in this layer; the filtered include flags ship separately. Wrapped-lib soundness recovery, virtual-impl gating on the deps side, ppx-runtime force-glob, and the new soundness test fixtures ship in a follow-up — this commit leaves five existing cram tests broken ([auto-wrapped-child-reexport.t], [ppx-runtime-libraries.t], [virtual-library.t], [wrapped-closure-precision.t], [wrapped-reexport-via-open-flag.t]) that the soundness recovery restores. [Module_compilation]: - [union_module_name_sets_mapped]: parallel fold over a list of [Module_name.Set.t] producers. - [module_kind_is_filterable]: predicate excluding kinds whose dep story is handled outside the BFS ([Root], [Wrapped_compat], [Impl_vmodule], [Virtual], [Parameter]). - [cross_lib_tight_set]: BFS expanding through the lib_index's [(lib, entry)] pairs, reading each entry's impl + intf [ocamldep] output. Non-tight-eligible libs terminate chains. - [lib_deps_for_module]: replaces the scaffold body. A [can_filter] guard (consumer-side virtual / parameter, dummy dep graph, module kind, [Module.has m ~ml_kind]) falls back to glob; otherwise runs the BFS, classifies libs via [Lib_file_deps.Lib_index.filter_libs_with_modules], and emits specific-file deps for tight libs + glob deps for non-tight / unreached-non-eligible libs. Returns the cctx-wide [Includes]; filtered include flags follow in a later layer. [Compilation_context.create]: peek [direct_requires] / [hidden_requires] and pass [has_library_deps] to [Dep_rules.rules]. Single-module stanzas with library deps now produce real dep graphs (the filter needs them). [Dep_rules.rules]: gate the singleton short-circuit on [(not has_library_deps) || for_ = Melange]. Other singletons fall through to the full dep-graph build. [Ocaml_flags]: [extract_open_module_names] surfaces [-open Foo] references that ocamldep doesn't see; they join [BFS]'s initial frontier. [Virtual_rules]: [is_virtual_or_parameter] — true for virtual impls and parameter cctxs; used by [can_filter] to suppress per-module filtering on consumer cctxs whose dep story [Dep_rules] handles specially. [Parameterised_rules]: pass [~has_library_deps:true] to the [Dep_rules.rules] call; conservative — the dep-rules path here serves external parameterised libs whose dep set is built from generated sources. Tests: rebuild-precision promotions for the existing modified-test set in #14492 — cram outputs reflect the tighter dep / rebuild behavior that L4 already produces. New soundness test fixtures (and the two tests gated on filtered include flags, [per-module-include-flags.t] / [add-unreferenced-sibling-lib.t]) are deferred to their respective follow-ups. Signed-off-by: Robin Bate Boerop <me@robinbb.com>
robinbb
force-pushed
the
robinbb-14492-l3-includes-refactor
branch
from
d3418da to
246a914
Compare
robinbb
force-pushed
the
robinbb-14492-l4-tight-branch
branch
from
da056f0 to
a0893d7
Compare
robinbb
force-pushed
the
robinbb-14492-l3-includes-refactor
branch
from
246a914 to
d88da28
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 4 of 9 of #14492. The core BFS.
Module_compilation.lib_deps_for_moduleactivates the tight branch: a per-module BFS over the cross-library dep graph (cross_lib_tight_set) computes the dep-lib modules the consumer actually references. The compile rule sees only those cmi/cmx files.can_filterfalls back to glob for Melange, dummy dep graphs, synthesised modules, non-filterable kinds, and consumer-side virtual / parameter cctxs.Dep_rules.rulesgates its singleton short-circuit on~has_library_deps || for_ = Melange.Compilation_context.createcomputes and passes the flag.Ocaml_flags.extract_open_module_namessurfaces-open Fooreferences for the BFS frontier.Virtual_rules.is_virtual_or_parameterpowers the consumer-sidecan_filtercheck.Parameterised_rulespasses~has_library_deps:trueto itsDep_rules.rulescall.Includes are still the cctx-wide
-I/-H; filtered include flags ship in layer 6. Soundness recovery for wrapped libs / ppx-runtime / virtual-impl deps ships in layer 5.Five existing cram tests fail on this PR; layer 5's soundness recovery restores them without test file changes:
test/blackbox-tests/test-cases/per-module-lib-deps/auto-wrapped-child-reexport.ttest/blackbox-tests/test-cases/per-module-lib-deps/ppx-runtime-libraries.ttest/blackbox-tests/test-cases/per-module-lib-deps/virtual-library.ttest/blackbox-tests/test-cases/per-module-lib-deps/wrapped-closure-precision.ttest/blackbox-tests/test-cases/per-module-lib-deps/wrapped-reexport-via-open-flag.tStack: rebases on #14515. Next: #14517.
Part of #14492. Related to #4572.