feat(go): support native fory row format - #3901
Conversation
wenshao
left a comment
wenshao
left a comment
There was a problem hiding this comment.
Reviewed. Suggestions are inline. Not reviewed: build-and-test — Go and Java toolchains not installed on this machine; CI passed all Go checks. Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/ci.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
— Qwen3-235B-A22B via Qwen Code /review (v0.21.2)
| } else if strings.HasPrefix(part, "ignore=") { | ||
| switch strings.TrimPrefix(part, "ignore=") { | ||
| case "true": |
There was a problem hiding this comment.
[Suggestion] hasIgnoreTag accepts only literal "true"/"false" while the core tag parser parseFieldTag delegates to parseBoolStrict, which accepts true/1/yes/false/0/no case-insensitively with trimming. The doc comment claims this function mirrors the core parser. — Failure scenario: a struct tagged fory:"ignore=1" serializes correctly through the core fory serializer, but InferSchema returns an error for the same tag.
| } else if strings.HasPrefix(part, "ignore=") { | |
| switch strings.TrimPrefix(part, "ignore=") { | |
| case "true": | |
| } else if strings.HasPrefix(part, "ignore=") { | |
| switch strings.ToLower(strings.TrimSpace(strings.TrimPrefix(part, "ignore="))) { | |
| case "true", "1", "yes": |
— Qwen3-235B-A22B via Qwen Code /review
| end := offset + size | ||
| if offset < 0 || size < 0 || end > len(data) { |
There was a problem hiding this comment.
[Suggestion] boundedSlice's bounds check is defeated by signed integer overflow: end := offset + size wraps negative when size is near MaxInt64 (reachable via NewMapData's attacker-controlled uint64→int cast). All three guards pass on the negative end, and the function reaches a generic runtime panic instead of the intended descriptive one. Go's runtime prevents memory-safety issues, but the defensive boundary is silently bypassed. — Failure scenario: a crafted keysSize = MaxInt64 makes end = 8 + MaxInt64 overflow to negative; data[8:negative] triggers an uninformative panic.
| end := offset + size | |
| if offset < 0 || size < 0 || end > len(data) { | |
| end := offset + size | |
| if offset < 0 || size < 0 || end < offset || end > len(data) { |
— Qwen3-235B-A22B via Qwen Code /review
| dataBytes := int64(numElements) * int64(w.elemSize) | ||
| if dataBytes > maxArrayDataBytes { | ||
| return fmt.Errorf("row: array of %d elements exceeds maximum size", numElements) | ||
| } |
There was a problem hiding this comment.
[Suggestion] The int64 multiplication int64(numElements) * int64(w.elemSize) overflows for extreme numElements, wrapping dataBytes negative and bypassing the > maxArrayDataBytes guard. The trigger is physically unallocatable, but the explicit guard is provably defeated. — Failure scenario: numElements > MaxInt64/8 wraps dataBytes negative, producing an obscure runtime panic instead of the intended clean error.
| dataBytes := int64(numElements) * int64(w.elemSize) | |
| if dataBytes > maxArrayDataBytes { | |
| return fmt.Errorf("row: array of %d elements exceeds maximum size", numElements) | |
| } | |
| if int64(numElements) > maxArrayDataBytes/int64(w.elemSize) { | |
| return fmt.Errorf("row: array of %d elements exceeds maximum size", numElements) | |
| } | |
| dataBytes := int64(numElements) * int64(w.elemSize) |
— Qwen3-235B-A22B via Qwen Code /review
| elem, err := inferField(listItemName, t.Elem(), path) | ||
| if err != nil { | ||
| return Field{}, err | ||
| } | ||
| return Field{Name: name, Type: &ListType{Elem: elem}, Nullable: true}, nil |
There was a problem hiding this comment.
[Suggestion] inferField leaves the list-element Nullable flag at its zero value false for value-type elements (e.g. []int32), diverging from Java's TypeInference (which produces Nullable: true for List<Integer>) and Go's own List()/Map() factories (which always set Nullable: true). The nullable flag is encoded as bit 6 of the field header in SchemaToBytes, so schema bytes differ. The cross-language test fixture uses only pointer elements ([]*string), masking this divergence. — Failure scenario: adding a []int32 field to the xlang fixture would fail both parsed.Equal(encoder.Schema()) and bytes.Equal(reencoded, schemaBytes).
| elem, err := inferField(listItemName, t.Elem(), path) | |
| if err != nil { | |
| return Field{}, err | |
| } | |
| return Field{Name: name, Type: &ListType{Elem: elem}, Nullable: true}, nil | |
| elem, err := inferField(listItemName, t.Elem(), path) | |
| if err != nil { | |
| return Field{}, err | |
| } | |
| elem.Nullable = true | |
| return Field{Name: name, Type: &ListType{Elem: elem}, Nullable: true}, nil |
— Qwen3-235B-A22B via Qwen Code /review
| key := reflect.New(goType.Key()).Elem() | ||
| if err := keyCodec.read(keys, j, key); err != nil { | ||
| return err | ||
| } | ||
| value := reflect.New(goType.Elem()).Elem() |
There was a problem hiding this comment.
[Suggestion] newMapCodec's read loop allocates two heap objects per map entry via reflect.New(...).Elem() inside the loop. Since SetMapIndex copies the key and value, and the read functions fully overwrite their targets (per the valueCodec doc contract), both allocations can be hoisted before the loop. — Concrete cost: deserializing a map with N entries performs 2N short-lived heap allocations that immediately become garbage, adding GC pressure proportional to map size.
| key := reflect.New(goType.Key()).Elem() | |
| if err := keyCodec.read(keys, j, key); err != nil { | |
| return err | |
| } | |
| value := reflect.New(goType.Elem()).Elem() | |
| key := reflect.New(goType.Key()).Elem() | |
| value := reflect.New(goType.Elem()).Elem() | |
| for j := 0; j < n; j++ { | |
| if err := keyCodec.read(keys, j, key); err != nil { | |
| return err | |
| } |
— Qwen3-235B-A22B via Qwen Code /review
|
@chaokunyang ready for review |
| // schema is inferred per InferSchema. | ||
| // | ||
| // An Encoder owns a reusable write buffer and is NOT goroutine-safe; | ||
| // create one Encoder per goroutine. Decoding is safe on corrupt or |
There was a problem hiding this comment.
Keep Encoder decoding on the trusted-data boundary
The repository security model classifies Row Format as a trusted in-memory format unless a runtime explicitly exposes an untrusted API. This sentence promotes Decode and FromRow into that stronger contract, but the implementation does not provide the allocation and work limits required for hostile bytes. Since this package is intended for trusted data, remove the corrupt/untrusted safety promise and document that callers must provide trusted, schema-matched row bytes.
| } | ||
|
|
||
| // SchemaFromBytes deserializes a schema from the cross-language wire | ||
| // format. It is safe on untrusted input: declared sizes are checked |
There was a problem hiding this comment.
Do not expose schema parsing as untrusted-safe
This comment explicitly makes SchemaFromBytes an untrusted deserialization API, which is broader than the trusted-only Row Format boundary. It is also stronger than the implementation: nested struct counts can allocate from the same unread tail at every level. Remove the untrusted-input claim and document trusted schema bytes as the precondition.
| // Fixed-width getters return the zero value for null fields; use | ||
| // IsNullAt to distinguish. Variable-width getters return nil (or "") | ||
| // for null fields. An out-of-range field index panics; offsets in | ||
| // corrupt or untrusted data may panic on slice bounds, so wrap |
There was a problem hiding this comment.
Keep raw Row views trusted-only
Suggesting that callers can make arbitrary untrusted rows safe by wrapping getters with recover is misleading: recover handles ordinary panics, not resource exhaustion, and these zero-copy views do not fully validate the row graph. State that Row and its nested views require trusted, schema-matched bytes instead.
| fields: make([]fieldCodec, 0, layout.schema.NumFields()), | ||
| } | ||
| for ordinal, goIndex := range layout.indexes { | ||
| codec, err := newValueCodec(t.Field(goIndex).Type, layout.schema.Field(ordinal).Type, buf) |
There was a problem hiding this comment.
Reject null when the Go carrier cannot represent it
The codec is built from Field.Type only, so nullability and the target carrier capability are lost. A frame produced for {nil} has the same type hash as []int32 and decodes successfully as {0}; nullable strings and value structs similarly become empty values. Pass the Field/nullability into codec construction and return an error when a non-pointer carrier receives null.
| return Field{}, err | ||
| } | ||
| key.Nullable = false | ||
| value, err := inferField(mapValueName, t.Elem(), path) |
There was a problem hiding this comment.
Canonicalize map value nullability during inference
For map[string]int32 this leaves the value field non-nullable, while both Java MapType and the Go schema parser canonicalize map values as nullable. As a result, SchemaFromBytes(SchemaToBytes(inferred)) is not equal to the inferred schema and Java emits different schema bytes. Force the canonical map child fields here and add a primitive-valued map round-trip/interoperability case.
| case reflect.String: | ||
| return Field{Name: name, Type: StringType{}, Nullable: true}, nil | ||
| case reflect.Slice: | ||
| if t.Elem().Kind() == reflect.Uint8 { |
There was a problem hiding this comment.
Align []byte with the Java row carrier
Go infers []byte as BINARY/raw bytes, but the current Java row TypeInference treats ordinary byte[] as LIST; only BinaryArray maps to BINARY. Common Go and Java models therefore get different hashes and incompatible raw layouts. Align Java row inference with the shared byte[] mapping, or define and test the exact Java carrier required for this Go type.
| if version != schemaVersion { | ||
| return nil, fmt.Errorf("row: unsupported schema version %d, expected %d", version, schemaVersion) | ||
| } | ||
| numFields := int(r.buf.ReadVarUint32Small7(&r.err)) |
There was a problem hiding this comment.
Validate wire integers before narrowing to int
On 32-bit Go, converting the uint32 count first can produce a negative int that bypasses the remaining-byte check and reaches make with a negative capacity. Array counts and map key sizes have the same truncation problem, and INT64 decoded into a Go int can silently narrow. Validate against MaxInt before conversion, check reflect.Value.OverflowInt, and make the new tests cross-compile on 386 instead of using untyped 1<<40/1<<33 int arguments.
| TestUtils.executeCommand( | ||
| buildCommand, 120, Collections.emptyMap(), new File("../../go/fory")); | ||
| if (!buildSuccess || !new File("../../go/fory/tests/" + GO_BINARY).exists()) { | ||
| throw new SkipException("Skipping GoCrossLanguageTest: failed to build " + GO_BINARY); |
There was a problem hiding this comment.
Fail the enabled cross-language test when the Go peer cannot build
FORY_GO_JAVA_CI=1 explicitly opts into this suite, but a missing Go toolchain, failed go build, or missing binary is still converted to SkipException. The normal Go task does not compile tests/row_xlang, so a peer-only compile regression can leave CI green. Once enabled, every setup failure should fail the test; only the non-opted-in case should skip.
| if t.Elem().Kind() == reflect.Uint8 { | ||
| return Field{Name: name, Type: BinaryType{}, Nullable: true}, nil | ||
| } | ||
| elem, err := inferField(listItemName, t.Elem(), path) |
There was a problem hiding this comment.
Detect recursive slice and map aliases
The cycle check is only entered for structs. Valid declarations such as type L []L and type M map[string]M recurse indefinitely through inferField and eventually overflow the stack. Track every composite type in the active path and return the same cycle error used for structs.
| } | ||
|
|
||
| func (r *Row) Duration(i int) time.Duration { | ||
| return time.Duration(r.Int64(i)) * time.Microsecond |
There was a problem hiding this comment.
Check the duration carrier range before multiplying
The wire stores an int64 number of microseconds, but time.Duration stores int64 nanoseconds. Multiplication silently wraps for otherwise valid wire values; MaxInt64 microseconds becomes negative. Reject values outside MinInt64/1000 through MaxInt64/1000 before conversion, including the ArrayData path.
| return nil | ||
| } | ||
|
|
||
| func (w *RowWriter) WriteTimestamp(i int, t time.Time) { |
There was a problem hiding this comment.
Use a checked timestamp-to-microseconds conversion
time.Time.UnixMicro has undefined results when the timestamp cannot be represented as int64 microseconds. This writer currently succeeds with a corrupted value for sufficiently large years. Make timestamp writes return an error and use checked seconds/nanoseconds arithmetic in both RowWriter and ArrayWriter.
| } | ||
| switch t { | ||
| case goDateType: | ||
| return Field{Name: name, Type: Date32Type{}}, nil |
There was a problem hiding this comment.
Align temporal field nullability with Java
Go infers Date and Timestamp value carriers as non-nullable, whereas Java infers LocalDate, Timestamp, and Instant fields as nullable. Their schema bytes therefore differ, while the type-only hash does not detect the difference; a Java null then becomes a zero Date or year-1 Time in Go. Align the inferred field metadata and require pointer carriers when null must be preserved.
| if numElements < 0 { | ||
| return fmt.Errorf("row: negative array length %d", numElements) | ||
| } | ||
| dataBytes := int64(numElements) * int64(w.elemSize) |
There was a problem hiding this comment.
Check the array limit before multiplying
With an 8-byte element, Reset(math.MaxInt) overflows dataBytes to a negative value, bypasses the maximum-size check, and later panics while computing or reserving the region. Compare numElements against maxArrayDataBytes/elemSize before multiplying and test that extreme input returns an error.
| return writeSchemaType(buf, f.Type) | ||
| } | ||
|
|
||
| func writeSchemaType(buf *fory.ByteBuffer, dataType DataType) error { |
There was a problem hiding this comment.
Validate schema types, depth, and cycles before writing
The writer can emit a 65-level schema that its own reader rejects, and a self-referential ListType or StructType recurses until a fatal stack overflow. Dynamic type handling is also incomplete: *DecimalType writes only the type ID, out-of-range precision/scale is silently truncated to byte, and an external composite DataType can omit child metadata. Perform an exhaustive validation pass before writing any bytes.
| } | ||
|
|
||
| func inferField(name string, t reflect.Type, path []reflect.Type) (Field, error) { | ||
| if t.Kind() == reflect.Ptr { |
There was a problem hiding this comment.
Reject pointers to already-nilable carriers
For *[]T, *map[K]V, and *[]byte, a nil outer pointer and a non-nil pointer to a nil container both produce the same row null bit. Decode always returns a nil outer pointer, so valid Go values cannot round-trip. Reject these double-nilable shapes unless the format gains a separate representation for both states.
| } | ||
| return Field{Name: name, Type: &ListType{Elem: elem}, Nullable: true}, nil | ||
| case reflect.Map: | ||
| if t.Key().Kind() == reflect.Ptr { |
There was a problem hiding this comment.
Require map keys to preserve Go equality
Rejecting only a direct pointer key is insufficient. A comparable struct key can contain ignored or unexported equality state, causing distinct keys to encode identically and one entry to overwrite the other on decode; pointer identity nested inside a struct is also not preserved. Recursively restrict keys to shapes whose encoded fields fully determine Go ==.
| byName[f.Name] = i | ||
| } | ||
| } | ||
| return &Schema{fields: fields, byName: byName} |
There was a problem hiding this comment.
Copy schema fields before caching their indexes
NewSchema stores the caller's slice after building byName. Mutating the original fields slice can therefore make Field(i) report a new name while FieldIndex still resolves the old one. Defensively copy the input, including nested schema state that participates in cached encoder/hash behavior.
| mapWriter.Reset() | ||
| // Go map iteration order changes between iterations, so | ||
| // snapshot entries once to keep keys and values aligned. | ||
| keys := make([]reflect.Value, 0, n) |
There was a problem hiding this comment.
Remove per-entry reflection allocations from map codecs
The write path retains two []reflect.Value snapshots and MapIter.Key/Value can allocate per entry; the read path separately performs two reflect.New calls per entry. Write keys while taking one typed value snapshot with SetIterValue, and hoist reusable key/value scratch values outside the decode loop. Add an allocation benchmark so this hot path stays linear without 2N temporary objects.
| // little-endian schema hash followed by the row bytes, matching the | ||
| // Java and Python row encoders. | ||
| func (e *Encoder[T]) Encode(v *T) ([]byte, error) { | ||
| rowBytes, err := e.ToRow(v) |
There was a problem hiding this comment.
Avoid copying the complete row twice in Encode
Encode calls ToRow, which already allocates and copies the reusable buffer, then allocates the framed result and copies the row again. Reserve or prepend the hash in the encoder buffer and make only the final owned-output copy; add a benchmark for framed encoding.
| part = strings.TrimSpace(part) | ||
| if part == "ignore" { | ||
| ignore = true | ||
| } else if strings.HasPrefix(part, "ignore=") { |
There was a problem hiding this comment.
Use the core fory tag grammar
This parser does not actually mirror parseFieldTag: the core accepts case-insensitive true/1/yes and false/0/no, trims both sides of '=', and rejects duplicate keys. Here ignore=yes is rejected, ignore = true is ignored, and duplicates silently overwrite. Reuse the core parser or implement the same grammar and update the test that currently treats ignore=yes as invalid.
| } | ||
|
|
||
| func (w *RowWriter) WriteString(i int, s string) error { | ||
| start := appendStringRegion(w.buf, s) |
There was a problem hiding this comment.
Reject strings that are not valid UTF-8
A Go string may contain arbitrary bytes, but the Row Format string type is UTF-8. Writing string([]byte{0xff}) succeeds here; Java decodes it with a replacement character, so the value changes on cross-language round-trip. Validate UTF-8 in both row and array writers and cover nested/list/map strings.
| func (e *Encoder[T]) SchemaHash() int64 { return e.hash } | ||
|
|
||
| // ToRow serializes v to row bytes (no framing, just the row). | ||
| func (e *Encoder[T]) ToRow(v *T) ([]byte, error) { |
There was a problem hiding this comment.
Return argument errors for nil public inputs
ToRow(nil) and Encode(nil) panic at reflect.Value.Elem instead of using their error result. FromRowInto with a nil output is mislabeled as corrupt data and can even succeed for an empty struct; InferSchema(nil) and SchemaToBytes(nil) have analogous behavior. Validate nil at each public entry point and return a direct argument error.
|
|
||
| func (e *Encoder[T]) Schema() *Schema { return e.codec.schema } | ||
|
|
||
| // SchemaHash returns the cross-language schema hash used by Encode and |
There was a problem hiding this comment.
Describe SchemaHash as a type-shape fingerprint
The compatible hash folds recursive type IDs only; it does not cover field names, nullability, decimal parameters, or reordering of same-typed fields. It therefore cannot generally detect out-of-sync struct definitions. Keep the wire-compatible algorithm, but narrow this documentation and the mismatch error to the actual type-shape check.
| buf *fory.ByteBuffer | ||
| } | ||
|
|
||
| func NewEncoder[T any]() (*Encoder[T], error) { |
There was a problem hiding this comment.
Add the Go Row Format public documentation surface
This adds a complete public package, but the specification, xlang guide, README, and Go guide still list only Java/C++/Python and do not explain NewEncoder, ToRow versus Encode, trusted-data scope, nullable carriers, supported types, thread safety, or zero-copy lifetimes. Update the support matrix and add a runnable Go example before exposing the API.
| func NewSchema(fields []Field) *Schema { | ||
| byName := make(map[string]int, len(fields)) | ||
| for i, f := range fields { | ||
| if _, ok := byName[f.Name]; !ok { |
There was a problem hiding this comment.
Make duplicate field-name handling cross-language deterministic
Go intentionally keeps the first duplicate name, while Java Schema and StructType keep the last. Identical schema bytes can therefore resolve a name to different ordinals across runtimes. Reject duplicate names during schema construction and parsing, or align the lookup rule and test it cross-language.
3 participants
Why?
What does this PR do?
Related issues
AI Contribution Checklist
yes/noyes, I included a completed AI Contribution Checklist in this PR description and the requiredAI Usage Disclosure.yes, my PR description includes the requiredai_reviewsummary and screenshot evidence or equivalent persisted links of the final clean AI review results from both fresh reviewers described inAI_POLICY.md, the Fory-guided reviewer and the independent general reviewer, on the current PR diff or current HEAD after the latest code changes.Does this PR introduce any user-facing change?
Benchmark