Multi-language Generation Backend Refactor + Java Generation

[mbolt35]

Core Changes

• Introduction of a generator Backend that is capable of interfacing with a core generation approach
• Simplify variable name generation to just a simple v<digit>
• Build out Emit utility that can build an indention-aware code writer/accumulator

Go Generation Changes

• Refactor Go Template Generation into more of a dynamic code generation approach
• Changes to Read (primitive) approach to be a specific order for all generated code. This is nearly identical to the previous generation with the exception of reading non-string primitives. All reads are generically defined as:
  • Declare variable
  • Read Buffer Into Variable
  • Commit/Set Field to Variable
    What go non-string primitive reads looked like before:

target.Age = buff.ReadInt32()

will now appear as:

var v0 int32 
v0 = buff.ReadInt32()
target.Age = v0

Generation may not be as tidy, but this approach allows us to reduce the complexity in the core generation of reads. The go compiler will also likely optimize the extra steps away.

Java Generator

The java generation was a major challenge for many reasons, but differences in java and go being the main hurdle:

• Go has implicit interface implementation while Java requires explicit implements clauses.
• Go's time.Time encoding had to be reverse engineered into an OffsetDateTime
• Generation didn't have to export the actual model types (since they were read from Go code originally) -- Java has a different generation step which required exporting record types that mimic the Go types.
• Java doesn't have unsigned primitive types.
• Java package and directory requirements versus go.
• Java import rules
• Java List<T> and Map<K, V> translations and iteration.
• Java "streaming" equivalent to Go

Java and Go Test Parity

Lots of effort was put towards parity and roundtrip testing between Go and Java.

Architecture

Major overhaul to the underlying generation architecture to allow a generic set of generation steps and a custom backend to implement both language outputs. This is still not exactly where we want it, but it's getting there.

[Copilot]

Pull request overview

This PR refactors bingen’s generation pipeline around a language-neutral “core” backend (with a shared Emit writer + structural walkers) and adds a Java code generator/runtime with substantial test coverage to validate cross-language wire compatibility (including streaming and string-table behavior).

Changes:

• Introduces internal/generator/core primitives (Emit, streaming backend interfaces) and refactors Go generation to use the core backend.
• Adds a Java generator (internal/generator/java) with templates and runtime support classes, plus generated Java codecs for the tests/opencost fixture.
• Expands/updates tests to cover roundtrips, streaming, external types (time.Time), string-table behavior, and Java/Go parity.

Reviewed changes

Copilot reviewed 166 out of 168 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/var_test.go	Removes legacy alpha-based var-name tests (var naming scheme changed).
tests/timerec/timerec.go	Adds a small fixture to exercise time.Time external reference encoding.
tests/timerec/bingen.go	Adds @bingen:generate directive for the timerec fixture.
tests/sttable/sttable.go	Adds a fixture to exercise string-table encoding/dedup.
tests/sttable/bingen.go	Adds @bingen:generate[stringtable] directive for string-table fixture.
tests/streamfeed/streamfeed.go	Adds a fixture to exercise streaming parity-friendly fields.
tests/streamfeed/bingen.go	Adds @bingen:generate[streamable] directive for streaming fixture.
tests/shape/shape.go	Adds interface + concrete types fixture to exercise interface dispatch.
tests/shape/bingen.go	Adds generation directives for interface + concrete types.
tests/roundtrip_test.go	Adds Go-side runtime tests for roundtrip + streaming behavior.
tests/opencost/java/com/opencost/opencost/WindowEncoder.java	Generated Java encoder for Window.
tests/opencost/java/com/opencost/opencost/WindowDecoder.java	Generated Java decoder for Window.
tests/opencost/java/com/opencost/opencost/Window.java	Generated Java record + builder for Window.
tests/opencost/java/com/opencost/opencost/TypeRegistry.java	Generated interface wire-type registry for Java dispatch.
tests/opencost/java/com/opencost/opencost/SharedAssetEncoder.java	Generated Java encoder for SharedAsset.
tests/opencost/java/com/opencost/opencost/SharedAssetDecoder.java	Generated Java decoder for SharedAsset.
tests/opencost/java/com/opencost/opencost/SharedAsset.java	Generated Java record + builder for SharedAsset.
tests/opencost/java/com/opencost/opencost/RawAllocationOnlyDataEncoder.java	Generated Java encoder for RawAllocationOnlyData.
tests/opencost/java/com/opencost/opencost/RawAllocationOnlyDataDecoder.java	Generated Java decoder for RawAllocationOnlyData.
tests/opencost/java/com/opencost/opencost/RawAllocationOnlyData.java	Generated Java record + builder for RawAllocationOnlyData.
tests/opencost/java/com/opencost/opencost/PVKeyEncoder.java	Generated Java encoder for PVKey.
tests/opencost/java/com/opencost/opencost/PVKeyDecoder.java	Generated Java decoder for PVKey.
tests/opencost/java/com/opencost/opencost/PVKey.java	Generated Java record + builder for PVKey.
tests/opencost/java/com/opencost/opencost/PVAllocationEncoder.java	Generated Java encoder for PVAllocation.
tests/opencost/java/com/opencost/opencost/PVAllocationDecoder.java	Generated Java decoder for PVAllocation.
tests/opencost/java/com/opencost/opencost/PVAllocation.java	Generated Java record + builder for PVAllocation.
tests/opencost/java/com/opencost/opencost/package-info.java	Generated Java package documentation.
tests/opencost/java/com/opencost/opencost/NodeEncoder.java	Generated Java encoder for Node.
tests/opencost/java/com/opencost/opencost/Node.java	Generated Java record + builder for Node.
tests/opencost/java/com/opencost/opencost/NetworkEncoder.java	Generated Java encoder for Network.
tests/opencost/java/com/opencost/opencost/NetworkDecoder.java	Generated Java decoder for Network.
tests/opencost/java/com/opencost/opencost/Network.java	Generated Java record + builder for Network.
tests/opencost/java/com/opencost/opencost/LoadBalancerEncoder.java	Generated Java encoder for LoadBalancer.
tests/opencost/java/com/opencost/opencost/LoadBalancerDecoder.java	Generated Java decoder for LoadBalancer.
tests/opencost/java/com/opencost/opencost/LoadBalancer.java	Generated Java record + builder for LoadBalancer.
tests/opencost/java/com/opencost/opencost/DiskEncoder.java	Generated Java encoder for Disk.
tests/opencost/java/com/opencost/opencost/DiskDecoder.java	Generated Java decoder for Disk.
tests/opencost/java/com/opencost/opencost/Disk.java	Generated Java record + builder for Disk.
tests/opencost/java/com/opencost/opencost/ClusterManagementEncoder.java	Generated Java encoder for ClusterManagement.
tests/opencost/java/com/opencost/opencost/ClusterManagementDecoder.java	Generated Java decoder for ClusterManagement.
tests/opencost/java/com/opencost/opencost/ClusterManagement.java	Generated Java record + builder for ClusterManagement.
tests/opencost/java/com/opencost/opencost/CloudEncoder.java	Generated Java encoder for Cloud.
tests/opencost/java/com/opencost/opencost/CloudDecoder.java	Generated Java decoder for Cloud.
tests/opencost/java/com/opencost/opencost/Cloud.java	Generated Java record + builder for Cloud.
tests/opencost/java/com/opencost/opencost/BreakdownEncoder.java	Generated Java encoder for Breakdown.
tests/opencost/java/com/opencost/opencost/BreakdownDecoder.java	Generated Java decoder for Breakdown.
tests/opencost/java/com/opencost/opencost/Breakdown.java	Generated Java record + builder for Breakdown.
tests/opencost/java/com/opencost/opencost/AssetSetRangeEncoder.java	Generated Java encoder for AssetSetRange.
tests/opencost/java/com/opencost/opencost/AssetSetRangeDecoder.java	Generated Java decoder for AssetSetRange.
tests/opencost/java/com/opencost/opencost/AssetSetRange.java	Generated Java record + builder for AssetSetRange.
tests/opencost/java/com/opencost/opencost/AssetSetEncoder.java	Generated Java encoder for AssetSet (string-table enabled).
tests/opencost/java/com/opencost/opencost/AssetSet.java	Generated Java record + builder for AssetSet.
tests/opencost/java/com/opencost/opencost/AssetPropertiesEncoder.java	Generated Java encoder for AssetProperties.
tests/opencost/java/com/opencost/opencost/AssetPropertiesDecoder.java	Generated Java decoder for AssetProperties.
tests/opencost/java/com/opencost/opencost/AssetProperties.java	Generated Java record + builder for AssetProperties.
tests/opencost/java/com/opencost/opencost/Asset.java	Generated Java marker interface for Go interface Asset.
tests/opencost/java/com/opencost/opencost/AnyEncoder.java	Generated Java encoder for Any.
tests/opencost/java/com/opencost/opencost/AnyDecoder.java	Generated Java decoder for Any.
tests/opencost/java/com/opencost/opencost/Any.java	Generated Java record + builder for Any.
tests/opencost/java/com/opencost/opencost/AllocationSetRangeEncoder.java	Generated Java encoder for AllocationSetRange.
tests/opencost/java/com/opencost/opencost/AllocationSetRangeDecoder.java	Generated Java decoder for AllocationSetRange.
tests/opencost/java/com/opencost/opencost/AllocationSetRange.java	Generated Java record + builder for AllocationSetRange.
tests/opencost/java/com/opencost/opencost/AllocationSetEncoder.java	Generated Java encoder for AllocationSet (string-table enabled).
tests/opencost/java/com/opencost/opencost/AllocationSet.java	Generated Java record + builder for AllocationSet.
tests/opencost/java/com/opencost/opencost/AllocationProperties.java	Generated Java record + builder for AllocationProperties.
tests/opencost/java/com/opencost/opencost/AllocationEncoder.java	Generated Java encoder for Allocation.
tests/opencost/java/com/bingen/WriteBuffer.java	Adds Java runtime write buffer implementation.
tests/opencost/java/com/bingen/StringTableWriter.java	Adds Java string-table writer runtime.
tests/opencost/java/com/bingen/StringTableReader.java	Adds Java string-table reader runtime.
tests/opencost/java/com/bingen/EncodingContext.java	Adds Java encoding context runtime.
tests/opencost/java/com/bingen/DecodingContext.java	Adds Java decoding context runtime.
tests/opencost/java/com/bingen/BingenValue.java	Adds Java streaming value wrapper.
tests/opencost/java/com/bingen/BingenStreamFn.java	Adds Java functional interface for streamer entrypoints.
tests/opencost/java/com/bingen/BingenStreamException.java	Adds Java exception wrapper for pull-stream adaptation.
tests/opencost/java/com/bingen/BingenFieldInfo.java	Adds Java streamed field descriptor type.
tests/opencost/java/com/bingen/BingenElement.java	Adds Java pulled stream element type.
tests/opencost/java/com/bingen/BingenConsumer.java	Adds Java consumer interface for push streaming.
tests/opencost/java/com/bingen/BinEncoder.java	Adds Java encoder interface.
tests/opencost/java/com/bingen/BinDecoder.java	Adds Java decoder interface.
tests/generator_test.go	Updates generator tests to use new Go/Java generator packages and run Java gen for opencost.
tests/container/container_codecs.go	Updates generated Go codecs to new core backend var naming / structure.
tests/aliasnil/bingen.go	Adds generation directives for new alias-nilability test fixture.
tests/aliasnil/aliasnil.go	Adds alias-nilability fixture types.
internal/types/walk.go	Adds a shared structural type walker utility for generators.
internal/types/types.go	Adds helper predicates for primitive type-code classification.
internal/generator/java/validate.go	Adds Java package validation (identifier/reserved words).
internal/generator/java/templates/java/StringTableWriter.java.tmpl	Adds template for Java string-table writer runtime.
internal/generator/java/templates/java/StringTableReader.java.tmpl	Adds template for Java string-table reader runtime.
internal/generator/java/templates/java/streamer.java.tmpl	Adds template for Java push-streamer codegen.
internal/generator/java/templates/java/registry.java.tmpl	Adds template for Java wire-type registry codegen.
internal/generator/java/templates/java/record.java.tmpl	Adds template for Java records + builders.
internal/generator/java/templates/java/package-info.java.tmpl	Adds template for Java package-info emission.
internal/generator/java/templates/java/interface.java.tmpl	Adds template for Java marker interfaces (Go interfaces).
internal/generator/java/templates/java/EncodingContext.java.tmpl	Adds template for Java encoding context runtime.
internal/generator/java/templates/java/encoder.java.tmpl	Adds template for Java per-type encoders.
internal/generator/java/templates/java/DecodingContext.java.tmpl	Adds template for Java decoding context runtime.
internal/generator/java/templates/java/decoder.java.tmpl	Adds template for Java per-type decoders.
internal/generator/java/templates/java/BingenValue.java.tmpl	Adds template for Java stream value wrapper.
internal/generator/java/templates/java/BingenStreamFn.java.tmpl	Adds template for Java streamer function interface.
internal/generator/java/templates/java/BingenStreamException.java.tmpl	Adds template for Java stream exception wrapper.
internal/generator/java/templates/java/BingenFieldInfo.java.tmpl	Adds template for Java field descriptor.
internal/generator/java/templates/java/BingenElement.java.tmpl	Adds template for Java pulled stream element.
internal/generator/java/templates/java/BingenConsumer.java.tmpl	Adds template for Java stream consumer.
internal/generator/java/templates/java/BinEncoder.java.tmpl	Adds template for Java encoder interface.
internal/generator/java/templates/java/BinDecoder.java.tmpl	Adds template for Java decoder interface.
internal/generator/java/templates.go	Adds embedded template plumbing and runtime template registry for Java generator.
internal/generator/java/record_test.go	Adds Java codegen + compilation test for generated records.
internal/generator/java/options.go	Adds Java generator options parsing/validation and output dir resolution.
internal/generator/java/migration_test.go	Adds Java migration/versioning behavior test (Go v1 -> Java v2).
internal/generator/java/javatype_test.go	Adds Java type-mapping and field-name tests.
internal/generator/java/implements.go	Adds Go type-checking based interface-implementation graph + registry support.
internal/generator/java/hooks_test.go	Adds Java pre/post process hook tests.
internal/generator/java/generate_test.go	Adds basic Java generation tests (package-info, nil options).
internal/generator/java/externaltypes.go	Adds Java handling for external Go reference types (time.Time -> OffsetDateTime).
internal/generator/java/encoder_test.go	Adds Java->Go roundtrip test for encoding compatibility.
internal/generator/java/decoder_test.go	Adds Java-only roundtrip test for maps + nested pointer structs.
internal/generator/golang/templates/go/unmarshaller.go.tmpl	Updates Go unmarshaller template to accept core-generated decode body.
internal/generator/golang/templates/go/streamer.go.tmpl	Updates Go streamer template to accept core-generated stream body.
internal/generator/golang/templates/go/marshaller.go.tmpl	Updates Go marshaller template to accept core-generated encode body.
internal/generator/golang/templates.go	Refactors Go templates plumbing to accept pre-rendered bodies.
internal/generator/golang/generate.go	Refactors Go generator to use internal/generator/core backends.
internal/generator/golang/context.go	Adds a simplified Go generator context implementation (no error-handler plumbing).
internal/generator/core/stream.go	Adds core streaming codegen driver + stream backend interface.
internal/generator/core/scope.go	Adds debug scope emission helpers for core backends.
internal/generator/core/emit.go	Adds Emit indentation-aware writer + import tracking + var naming (v<n>).
cmd/bingen/bingen.go	Adds -lang and repeatable -opt support; routes to Go or Java generator.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mbolt35]

Is it more appropriate to use make() over create() - unsure of the java conventions here.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

[Copilot]

Pull request overview

Copilot reviewed 167 out of 169 changed files in this pull request and generated no new comments.

Suppressed comments (4)

internal/generator/core/scope.go:12

• Typo/grammar in comment: "delimitting" should be "delimiting" (and consider starting the sentence with a capitalized identifier name).

// just a default debug emission scope that writes a comment delimitting the start/end
// of an encode/decode scope

internal/generator/core/stream.go:45

• Grammar: "it's" should be "its" in this comment.

// StreamStruct renders the streaming-decode body for the provided StructType by yielding
// each of it's fields, in order.

internal/generator/golang/templates/go/marshaller.go.tmpl:6

• This template header comment says the unmarshaller retains read* defines because the streamer reuses them, but the streamer/unmarshaller bodies are now generated by the core backend and the old read/write sub-templates appear to be retired. Updating the comment avoids misleading future maintainers.

	Marshaller method skeleton. The field-encoding body ({{ .Body }}) is produced
	by the core Go backend (internal/generator/gobackend.go via
	internal/generator/core), so the write* sub-templates that previously lived
	here have been retired. The unmarshaller retains its read* defines because the
	streamer still reuses them.

internal/generator/core/scope.go:5

• Typo in comment: "lead" should be "leading".

This issue also appears on line 11 of the same file.

// DebugScope is used to output lead and trailing emissions during code generation