Skip to content

Add returnable: false field option#1108

Open
jwils wants to merge 2 commits intomainfrom
joshauw/fetchable-false-prototype
Open

Add returnable: false field option#1108
jwils wants to merge 2 commits intomainfrom
joshauw/fetchable-false-prototype

Conversation

@jwils
Copy link
Copy Markdown
Collaborator

@jwils jwils commented Apr 4, 2026

Summary

This PR adds a returnable: false option to field definitions in the schema definition DSL.

t.field "internal_code", "String", returnable: false

When a field is marked returnable: false, ElasticGraph:

  • excludes the field from GraphQL output types
  • keeps the field available for filtering, sorting, grouping, aggregations, and highlights
  • excludes the field from datastore _source via _source.excludes

This supports fields that are useful for query behavior but should not be returned in GraphQL responses, while also reducing stored _source size for those fields.

Implementation

The change is intentionally narrow:

  • Field now exposes a returnable? predicate with a default of true
  • output type SDL generation filters out non-returnable fields
  • generated datastore mappings emit _source.excludes for non-returnable field paths
  • generated filter fields explicitly reset returnable so they still appear in input types

Because the field remains part of the schema metadata used for non-output GraphQL features, existing generation for filter inputs, sort enums, grouped-by types, aggregated-values types, and highlights continues to work.

Tests

Updated unit specs cover:

  • excluding returnable: false fields from GraphQL output types
  • retaining those fields in filter, sort, grouped-by, aggregated-values, and highlights types
  • emitting _source.excludes in generated datastore mappings

I was not able to run the RSpec files in this local environment because Bundler is missing required gems/native extensions.

@jwils jwils force-pushed the joshauw/fetchable-false-prototype branch 4 times, most recently from 445fa89 to 6af4096 Compare April 5, 2026 02:21
Copy link
Copy Markdown
Collaborator

@myronmarston myronmarston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My initial thoughts (I haven't done a full review):

  • This seems useful and I like it in general!
  • I think I prefer it being named returnable rather than fetchable because fetchable is named more in terms of how EG interacts with the datastore (whether it "fetches" that field from the datastore or not) where as returnable, to me, is phrased in terms of the impact on the GraphQL schema: a field which is not returnable is not available to returned in the response. That's purely subjective, though--thoughts?
  • Elasticsearch/OpenSearch have a plethora of options we could leverage here--see the "retrieve selected fields from a search" docs for details.
    • Notably, it sounds like the fields parameter consults the index mappings rather than _source and might allow the same sort of storage savings your looking for while allowing the field to be returnable.
    • Doc value fields similarly may support returning a field while avoiding its storage costs in _source.
    • There's also stored_fields which I don't fully grok yet.

I suspect there's some deeper, more profound improvement we could make to EG leveraging these features, allowing for greater storage savings, query performance, or both, but I haven't dug deep enough to have an idea of what that would be yet.

If it interests you to dig into that and open a feature proposal I'd be happy to review. But in the meantime I think returnable as a feature is straightforward enough that we can move forward with this.

@jwils jwils changed the title Prototype: fetchable: false field option Prototype: returnable: false field option Apr 6, 2026
@github-advanced-security
Copy link
Copy Markdown
Contributor

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

@jwils jwils changed the title Prototype: returnable: false field option Add field option Apr 6, 2026
@jwils jwils changed the title Add field option Add returnable: false field option Apr 6, 2026
@jwils jwils marked this pull request as ready for review April 6, 2026 21:34
Adds a new `fetchable: false` option for field definitions that allows fields
to remain filterable, sortable, groupable, and aggregatable while being excluded
from the GraphQL output type and from `_source` in the datastore mapping.

Changes:
- Field struct: new `:fetchable` member with `fetchable?` predicate (defaults true)
- fields_sdl: filters out non-fetchable fields from output type SDL
- Index#mappings: emits `_source.excludes` for non-fetchable field paths
- TypeWithSubfields#non_fetchable_field_paths: collects paths using
  indexing_fields_by_name_in_index to avoid interface/union recursion cycles
- to_filter_field: resets fetchable to nil so filter fields still appear in
  filter input types

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Collaborator

@myronmarston myronmarston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other suggestion: for new features like this, it helps give much greater confidence that it works end-to-end if we (1) update the test schema to use it and (2) update the GraphQL acceptance tests to exercise it. However, I don't want to bloat this PR with that, so here's what I'd recommend:

  • Stack a PR on top of this one which sets returnable: false on at least one field of the widgets schema.
    • If you can, make that (+ factory updates, if needed) the only change in that PR. The diff on schema artifact updates is quite large and it's nice to not mix it with other meaningful changes.
  • Stack a PR on top of that one which runs some GraphQL queries demonstrating that the field can be filtered on, sorted on, grouped on, aggregated, and highlighted, but not returned.

Edit: also, it would be good to cover returnable: false on both a leaf and an object field in these end-to-end tests.

@jwils jwils force-pushed the joshauw/fetchable-false-prototype branch from 8e9b12f to bde522d Compare April 7, 2026 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants