diff --git a/docs/benchmarks/index.md b/docs/benchmarks/index.md index 46af48783df..f9a50cc9877 100644 --- a/docs/benchmarks/index.md +++ b/docs/benchmarks/index.md @@ -1,5 +1,5 @@ --- -title: Benchmarks +title: Results sidebar_position: 1 id: index license: | diff --git a/docs/benchmarks/object-serialization/xlang/swift/README.md b/docs/benchmarks/object-serialization/xlang/swift/README.md index f3322f902b5..91d4568221f 100644 --- a/docs/benchmarks/object-serialization/xlang/swift/README.md +++ b/docs/benchmarks/object-serialization/xlang/swift/README.md @@ -2,11 +2,11 @@ This benchmark compares serialization and deserialization throughput for Apache Fory, Protocol Buffers, and JSON in Swift. -## Benchmark Products +## Benchmark Executables The ordinary/xlang cases are built by `swift-benchmark`. External-type and carrier comparisons are built by the separate `swift-external-benchmark` -product, so building the ordinary product does not compile those models or +executable, so building the ordinary executable does not compile those models or serializer specializations. ## Throughput Plot diff --git a/docs/community/how_to_release.md b/docs/community/how_to_release.md index ddefad8b1aa..6b34c3e13b9 100644 --- a/docs/community/how_to_release.md +++ b/docs/community/how_to_release.md @@ -446,7 +446,7 @@ To learn more about Fory, please see https://fory.apache.org/ [ ] No compiled archives bundled in source archive. [ ] Can compile from source. -How to Build and Test, please refer to: https://github.com/apache/fory/blob/main/docs/development/building.md +How to Build and Test, please refer to: https://github.com/apache/fory/blob/main/docs/development/index.md Thanks, diff --git a/docs/compiler/generated-code/index.md b/docs/compiler/generated-code/index.md index bf76e92ae35..ac9908fa7b1 100644 --- a/docs/compiler/generated-code/index.md +++ b/docs/compiler/generated-code/index.md @@ -1,5 +1,5 @@ --- -title: Generated Code +title: Overview sidebar_position: 1 id: index license: | diff --git a/docs/development/building.md b/docs/development/building.md deleted file mode 100644 index 4e6bb138468..00000000000 --- a/docs/development/building.md +++ /dev/null @@ -1,135 +0,0 @@ ---- -title: Development -sidebar_position: 20 -id: building -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -## How to build Apache Fory™ - -Clone the source tree from https://github.com/apache/fory. - -### Build Apache Fory™ Java - -```bash -cd java -mvn -T16 package -``` - -#### Environment Requirements - -- JDK 17+ -- Maven 3.6.3+ - -### Build Apache Fory™ Python - -```bash -cd python -pip install -v -e . - -# Optional: build Cython extension (replace X.Y with your Python version) -bazel build //:cp_fory_so --@rules_python//python/config_settings:python_version=X.Y -``` - -#### Environment Requirements - -- CPython 3.8+ -- Bazel 8+ (required when building Cython extensions) - -### Build Apache Fory™ C++ - -```bash -cd cpp -bazel build //cpp/... -``` - -#### Environment Requirements - -- C++17 compiler -- Bazel 8+ - -### Build Apache Fory™ Go - -```bash -cd go/fory -go test -v ./... -``` - -Run Go xlang tests from Java test module: - -```bash -cd java -mvn -T16 install -DskipTests -cd fory-core -FORY_GO_JAVA_CI=1 ENABLE_FORY_DEBUG_OUTPUT=1 mvn test -Dtest=org.apache.fory.xlang.GoXlangTest -``` - -#### Environment Requirements - -- Go 1.24+ - -### Build Apache Fory™ Rust - -```bash -cd rust -cargo build -cargo test --features tests - -# Debug a specific test -RUST_BACKTRACE=1 FORY_PANIC_ON_ERROR=1 ENABLE_FORY_DEBUG_OUTPUT=1 \ - cargo test --test mod $dir$::$test_file::$test_method -- --nocapture -``` - -#### Environment Requirements - -- Rust toolchain via rustup -- `cargo-expand` (optional, for macro expansion debugging) - -### Build Apache Fory™ JavaScript - -```bash -cd javascript -npm install - -npm run build -node ./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit -``` - -#### Environment Requirements - -- Node.js (LTS) -- npm - -### Lint Markdown Docs - -```bash -cd docs -npx prettier --write "**/*.md" -``` - -#### Environment Requirements - -- Node.js (LTS) -- npm - -## Contributing - -For contribution details, see [How to contribute to Apache Fory™](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). -For AI-assisted contributions, follow the -[AI Contribution Policy](https://github.com/apache/fory/blob/main/AI_POLICY.md), including the -required self-review, two-reviewer AI review loop, disclosure, and verification evidence for -substantial AI assistance. diff --git a/docs/development/cpp-debugging.md b/docs/development/cpp-debugging.md index 360eef894bb..761a972e42c 100644 --- a/docs/development/cpp-debugging.md +++ b/docs/development/cpp-debugging.md @@ -18,9 +18,9 @@ license: | limitations under the License. --- -import JumpGeneratedFile from "@site/docs/images/jmp_generate_file.png"; -import VscodeDebugFory from "@site/docs/images/vscode_debug_fory.jpg"; -import VscodeSelectDebugRun from "@site/docs/images/vscode_select_debug_run.png"; +import JumpGeneratedFile from "@site/docs/development/jmp_generate_file.png"; +import VscodeDebugFory from "@site/docs/development/vscode_debug_fory.jpg"; +import VscodeSelectDebugRun from "@site/docs/development/vscode_select_debug_run.png"; ## Debugging C++ diff --git a/docs/development/index.md b/docs/development/index.md index 7044b921dea..1584659bb0c 100644 --- a/docs/development/index.md +++ b/docs/development/index.md @@ -1,5 +1,5 @@ --- -title: Development +title: Build and Test sidebar_position: 1 id: index license: | @@ -22,11 +22,121 @@ license: | Contributor documentation covers repository setup, builds, tests, debugging, and release-oriented workflows. It is separate from the user guides for released artifacts. -## Start here +Clone the source tree from https://github.com/apache/fory. -- [Build and test the repository](building.md) -- [Debug the C++ implementation](cpp-debugging.md) -- [Contributing guide](https://github.com/apache/fory/blob/main/CONTRIBUTING.md) +## Java + +```bash +cd java +mvn -T16 package +``` + +Requirements: + +- JDK 17+ +- Maven 3.6.3+ + +## Python + +```bash +cd python +pip install -v -e . + +# Optional: build Cython extension (replace X.Y with your Python version) +bazel build //:cp_fory_so --@rules_python//python/config_settings:python_version=X.Y +``` + +Requirements: + +- CPython 3.8+ +- Bazel 8+ (required when building Cython extensions) + +## C++ + +```bash +cd cpp +bazel build //cpp/... +``` + +Requirements: + +- C++17 compiler +- Bazel 8+ + +## Go + +```bash +cd go/fory +go test -v ./... +``` + +Run Go xlang tests from the Java test module: + +```bash +cd java +mvn -T16 install -DskipTests +cd fory-core +FORY_GO_JAVA_CI=1 ENABLE_FORY_DEBUG_OUTPUT=1 mvn test -Dtest=org.apache.fory.xlang.GoXlangTest +``` + +Requirement: Go 1.24+. + +## Rust + +```bash +cd rust +cargo build +cargo test --features tests + +# Debug a specific test +RUST_BACKTRACE=1 FORY_PANIC_ON_ERROR=1 ENABLE_FORY_DEBUG_OUTPUT=1 \ + cargo test --test mod $dir$::$test_file::$test_method -- --nocapture +``` + +Requirements: + +- Rust toolchain via rustup +- `cargo-expand` (optional, for macro expansion debugging) + +## JavaScript + +```bash +cd javascript +npm install + +npm run build +node ./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit +``` + +Requirements: + +- Node.js (LTS) +- npm + +## Markdown + +```bash +cd docs +npx prettier --write "**/*.md" +``` + +Requirements: + +- Node.js (LTS) +- npm + +## Debugging + +See [Debugging C++](cpp-debugging.md) for the repository's VS Code, Bazel, LLDB, and GDB setup. Runtime-specific contributor instructions remain in each runtime's source tree and contributor README when they are not part of the shared repository build. + +## Contributing + +For contribution details, see +[How to contribute to Apache Fory™](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). +For AI-assisted contributions, follow the +[AI Contribution Policy](https://github.com/apache/fory/blob/main/AI_POLICY.md), including the +required self-review, two-reviewer AI review loop, disclosure, and verification evidence for +substantial AI assistance. diff --git a/docs/images/jmp_generate_file.png b/docs/development/jmp_generate_file.png similarity index 100% rename from docs/images/jmp_generate_file.png rename to docs/development/jmp_generate_file.png diff --git a/docs/images/vscode_debug_fory.jpg b/docs/development/vscode_debug_fory.jpg similarity index 100% rename from docs/images/vscode_debug_fory.jpg rename to docs/development/vscode_debug_fory.jpg diff --git a/docs/images/vscode_select_debug_run.png b/docs/development/vscode_select_debug_run.png similarity index 100% rename from docs/images/vscode_select_debug_run.png rename to docs/development/vscode_select_debug_run.png diff --git a/docs/grpc/architecture.md b/docs/grpc/architecture.md deleted file mode 100644 index ffe4c30162a..00000000000 --- a/docs/grpc/architecture.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Architecture -sidebar_position: 2 -id: architecture -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Generated service companions use normal gRPC servers, channels, method descriptors, deadlines, -status codes, interceptors, and streaming APIs. Fory-generated marshallers encode and decode the -generated request and response models. - -## Ownership boundary - -Fory can generate service companions for application-provided gRPC runtimes. -Those companions provide Fory serialization for request and response objects; -the application and gRPC stack still own listeners, channels, credentials, -authentication, authorization, deadlines, retries, and transport lifecycle. - -Fory packages do not add a gRPC implementation as a hard dependency. The application selects and -configures the runtime's gRPC libraries. - -## Generated service surface - -The compiler emits runtime-idiomatic service bases, clients or stubs, method metadata, and Fory -marshallers. Model generation is documented under -[Generated Code](../compiler/generated-code/index.md); the runtime pages document server and client -integration. diff --git a/docs/grpc/index.md b/docs/grpc/index.md index 5e019a79ffc..a0c4712e0a4 100644 --- a/docs/grpc/index.md +++ b/docs/grpc/index.md @@ -1,5 +1,5 @@ --- -title: Fory gRPC +title: Overview sidebar_position: 1 id: index license: | @@ -54,8 +54,52 @@ generation with `--grpc-python-mode=sync`. JavaScript uses `@grpc/grpc-js` for Node.js; browser clients are generated separately with `--grpc-web` and use `grpc-web`. +## Architecture + +Generated service companions use normal gRPC servers, channels, method descriptors, deadlines, +status codes, interceptors, and streaming APIs. Fory-generated marshallers encode and decode the +generated request and response models. + +### Ownership Boundary + +Fory can generate service companions for application-provided gRPC runtimes. Those companions +provide Fory serialization for request and response objects; the application and gRPC stack still +own listeners, channels, credentials, authentication, authorization, deadlines, retries, and +transport lifecycle. + +Fory packages do not add a gRPC implementation as a hard dependency. The application selects and +configures the runtime's gRPC libraries. + +### Generated Service Surface + +The compiler emits runtime-idiomatic service bases, clients or stubs, method metadata, and Fory +marshallers. Model generation is documented under +[Generated Code](../compiler/generated-code/index.md); the runtime pages document server and client +integration. + +## Interoperability + +Fory gRPC peers interoperate only when they use the same generated service contract, matching Fory +type identities, and compatible generated model schemas. + +### Protocol Boundary + +The transport is gRPC, but the message bytes are Fory payloads. Generic protobuf clients and server +reflection tools cannot decode those payloads as protobuf messages. Generate every peer through a +supported Fory compiler frontend. + +### Verification + +Test at least one unary call and every streaming shape used by the service. A protobuf +`UNIMPLEMENTED` or decode failure usually means the peer used an ordinary protobuf stub or a +different generated service contract. + ## Runtime Guides +Java, Python, C++, Go, Rust, JavaScript/TypeScript, C#, Dart, Scala, and Kotlin have documented gRPC +companions. Use the [support matrix](../introduction/support-matrix.md) and the selected runtime page +for current dependencies and streaming support. + | Runtime | Guide | | --------------------- | ------------------------------------- | | Java | [Java](java.md) | @@ -68,6 +112,3 @@ Node.js; browser clients are generated separately with `--grpc-web` and use | Dart | [Dart](dart.md) | | Scala | [Scala](scala.md) | | Kotlin | [Kotlin](kotlin.md) | - -See [Architecture](architecture.md) for payload and transport ownership and -[Interoperability](interoperability.md) for peer compatibility. diff --git a/docs/grpc/interoperability.md b/docs/grpc/interoperability.md deleted file mode 100644 index 1de36c48855..00000000000 --- a/docs/grpc/interoperability.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Interoperability -sidebar_position: 3 -id: interoperability -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Fory gRPC peers interoperate only when they use the same generated service contract, matching Fory -type identities, and compatible generated model schemas. - -## Protocol boundary - -The transport is gRPC, but the message bytes are Fory payloads. Generic protobuf clients and server -reflection tools cannot decode those payloads as protobuf messages. Generate every peer through a -supported Fory compiler frontend. - -## Supported generated companions - -Java, Python, C++, Go, Rust, JavaScript/TypeScript, C#, Dart, Scala, and Kotlin have documented gRPC -companions. Use the [support matrix](../introduction/support-matrix.md) and the selected runtime page -for current dependencies and streaming support. - -## Verification - -Test at least one unary call and every streaming shape used by the service. A protobuf -`UNIMPLEMENTED` or decode failure usually means the peer used an ordinary protobuf stub or a -different generated service contract. diff --git a/docs/images/logo/fory-horizontal.png b/docs/images/fory-logo.png similarity index 100% rename from docs/images/logo/fory-horizontal.png rename to docs/images/fory-logo.png diff --git a/docs/images/idea_jdk11.png b/docs/images/idea-jdk11.png similarity index 100% rename from docs/images/idea_jdk11.png rename to docs/images/idea-jdk11.png diff --git a/docs/images/logo/fory-horizontal-black.png b/docs/images/logo/fory-horizontal-black.png deleted file mode 100644 index 2fff2442c4d..00000000000 Binary files a/docs/images/logo/fory-horizontal-black.png and /dev/null differ diff --git a/docs/images/logo/fory-horizontal-black1.png b/docs/images/logo/fory-horizontal-black1.png deleted file mode 100644 index d2495dc8d61..00000000000 Binary files a/docs/images/logo/fory-horizontal-black1.png and /dev/null differ diff --git a/docs/images/logo/fory-horizontal-white.png b/docs/images/logo/fory-horizontal-white.png deleted file mode 100644 index 759074c2229..00000000000 Binary files a/docs/images/logo/fory-horizontal-white.png and /dev/null differ diff --git a/docs/images/logo/fory-horizontal-white1.png b/docs/images/logo/fory-horizontal-white1.png deleted file mode 100644 index a310816cf9c..00000000000 Binary files a/docs/images/logo/fory-horizontal-white1.png and /dev/null differ diff --git a/docs/images/logo/fory-horizontal1.png b/docs/images/logo/fory-horizontal1.png deleted file mode 100644 index eb7039f122a..00000000000 Binary files a/docs/images/logo/fory-horizontal1.png and /dev/null differ diff --git a/docs/images/logo/fory-icon-black.png b/docs/images/logo/fory-icon-black.png deleted file mode 100644 index 5056720de4a..00000000000 Binary files a/docs/images/logo/fory-icon-black.png and /dev/null differ diff --git a/docs/images/logo/fory-icon-white.png b/docs/images/logo/fory-icon-white.png deleted file mode 100644 index 6f75304c23b..00000000000 Binary files a/docs/images/logo/fory-icon-white.png and /dev/null differ diff --git a/docs/images/logo/fory-icon.png b/docs/images/logo/fory-icon.png deleted file mode 100644 index fb3c9749f39..00000000000 Binary files a/docs/images/logo/fory-icon.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical-black.png b/docs/images/logo/fory-vertical-black.png deleted file mode 100644 index 4a6cba6723c..00000000000 Binary files a/docs/images/logo/fory-vertical-black.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical-black1.png b/docs/images/logo/fory-vertical-black1.png deleted file mode 100644 index 2bef2eda275..00000000000 Binary files a/docs/images/logo/fory-vertical-black1.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical-white.png b/docs/images/logo/fory-vertical-white.png deleted file mode 100644 index d2daf413e8e..00000000000 Binary files a/docs/images/logo/fory-vertical-white.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical-white1.png b/docs/images/logo/fory-vertical-white1.png deleted file mode 100644 index c56e4a66bfd..00000000000 Binary files a/docs/images/logo/fory-vertical-white1.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical.png b/docs/images/logo/fory-vertical.png deleted file mode 100644 index 8962eba810c..00000000000 Binary files a/docs/images/logo/fory-vertical.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical1.png b/docs/images/logo/fory-vertical1.png deleted file mode 100644 index 1fa11e55c4a..00000000000 Binary files a/docs/images/logo/fory-vertical1.png and /dev/null differ diff --git a/docs/introduction/choose-a-format.md b/docs/introduction/choose-a-format.md index 0d7225c6a48..240590d1c57 100644 --- a/docs/introduction/choose-a-format.md +++ b/docs/introduction/choose-a-format.md @@ -19,12 +19,16 @@ license: | limitations under the License. --- -| Format | Use it when | Start here | -| ------------- | ------------------------------------------------------------- | -------------------------------------------------------------- | -| Xlang binary | Data crosses language boundaries | [Cross-language guide](../object-serialization/xlang/index.md) | -| Native binary | Producer and consumer are in the same language | Language guide | -| Row format | You need random field access or analytics-style partial reads | [Row format spec](../row-format/index.md) | -| Fory JSON | Java applications need high-performance standard JSON | [Fory JSON guide](../json/index.md) | +| Format or mode | Data model | Use it when | Interoperability | Start here | +| -------------- | ------------------------------------ | ------------------------------------------------------------- | --------------------------------------------------------------------------- | -------------------------------------------------------- | +| Xlang binary | Portable object graphs | Data crosses language boundaries | Shared wire format across supported runtimes | [Cross-language guide](../object-serialization/xlang.md) | +| Native binary | Runtime-native object graphs | Producer and consumer use the same language | Same runtime family only | [Object Serialization](../object-serialization/index.md) | +| Row Format | Random-access binary rows | You need random field access or analytics-style partial reads | Standard Row is shared by Java, Python, C++, and Rust; Compact is Java-only | [Row Format guide](../row-format/index.md) | +| Fory JSON | Standard JSON mapped to Java objects | Java applications need high-performance standard JSON | Standard JSON text | [Fory JSON guide](../json/index.md) | + +Xlang and native are sibling modes of Object Serialization. Use them when the receiver needs to +reconstruct an object graph. Row Format and Fory JSON are separate formats, not additional Object +Serialization modes. For Java, Scala, Kotlin, Python, C++, Go, and Rust, use native mode for same-language traffic. It avoids xlang's cross-language type mapping and @@ -59,3 +63,12 @@ Choose xlang or native mode when you need to reconstruct object graphs. Choose R trusted analytical data that benefits from random field access. Choose Fory JSON for standard JSON in Java applications. Use Fory IDL and the compiler when multiple teams need one schema-first contract; it generates models that use the relevant Fory capability. + +## Related capabilities + +[Fory IDL and the compiler](../compiler/index.md) generate native models for supported runtimes. +Service definitions can also generate [Fory gRPC](../grpc/index.md) code. Neither capability defines +another serialization format. + +Protocol implementers should follow the normative format documentation under +[Specifications](../specification/xlang_serialization_spec.md). diff --git a/docs/introduction/index.md b/docs/introduction/index.md index 3f0bd9eed06..2245ef3700a 100644 --- a/docs/introduction/index.md +++ b/docs/introduction/index.md @@ -1,5 +1,5 @@ --- -title: Introduction +title: Overview sidebar_position: 1 id: index license: | diff --git a/docs/introduction/product-model.md b/docs/introduction/product-model.md deleted file mode 100644 index 3379dd6b529..00000000000 --- a/docs/introduction/product-model.md +++ /dev/null @@ -1,47 +0,0 @@ ---- -title: Product Model -sidebar_position: 3 -id: product-model -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Apache Fory exposes three serialization products and one schema toolchain. Start from the data -contract you need, then choose a runtime API. - -## Serialization products - -| Product | Data model | Interoperability boundary | -| ---------------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------- | -| Binary Object Serialization: xlang mode | Portable object graphs | Shared wire format across supported runtimes | -| Binary Object Serialization: native mode | Runtime-native object graphs | Same runtime family only | -| Row Format | Random-access binary rows | Standard Row is shared by Java, Python, C++, and Rust; Compact Row is Java-only | -| Fory JSON | Standard JSON mapped to Java objects | Java API with text interoperability | - -Xlang and native are sibling modes of Binary Object Serialization. Row Format and Fory JSON are -separate products; neither is a third object-serialization mode. - -## Schema and services - -[Fory IDL and the compiler](../compiler/index.md) generate native models for supported runtimes. -Service definitions can also generate [Fory gRPC](../grpc/index.md) companions. The compiler and -gRPC integration do not define additional serialization formats. - -## Normative formats - -Protocol implementers should use the unchanged [Specification](../specification/xlang_serialization_spec.md) -surface. User guides explain tasks and link to the exact specification instead of duplicating its -wire-level rules. diff --git a/docs/introduction/support-matrix.md b/docs/introduction/support-matrix.md index 5127bd48fbe..7fee55a3953 100644 --- a/docs/introduction/support-matrix.md +++ b/docs/introduction/support-matrix.md @@ -20,7 +20,7 @@ license: | --- Use this matrix to confirm the documented API surface before choosing a capability. A runtime page -does not imply support for every Fory product. +does not imply support for every Fory capability. | Capability | Documented runtimes | Interoperability | | --------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------- | @@ -33,5 +33,5 @@ does not imply support for every Fory product. | Fory gRPC | Java, Python, C++, Go, Rust, JavaScript/TypeScript, C#, Dart, Scala, Kotlin | Peers must use matching generated Fory service contracts | Platform constraints such as [Android](../object-serialization/java/android.md) and -[GraalVM Native Image](../object-serialization/java/graalvm.md) are documented by the owning Java -object-serialization and Fory JSON products. +[GraalVM Native Image](../object-serialization/java/graalvm.md) are documented in the Java Object +Serialization and Fory JSON guides. diff --git a/docs/json/index.md b/docs/json/index.md index 6ab963dd7a5..47a2e3683fd 100644 --- a/docs/json/index.md +++ b/docs/json/index.md @@ -1,5 +1,5 @@ --- -title: Fory JSON +title: Overview sidebar_position: 1 id: index license: | @@ -63,6 +63,6 @@ native or xlang protocol when reference identity or cycles are required. ## Related Java guides For binary serialization, start with [Java Object Serialization](../object-serialization/java/index.md) -and choose [xlang](../object-serialization/java/xlang.md) or +and choose [xlang](../object-serialization/java/basic-serialization.md#cross-language-interoperability) or [native](../object-serialization/java/native.md). Binary builder options are documented separately in [Java Configuration](../object-serialization/java/configuration.md). diff --git a/docs/object-serialization/core-concepts.md b/docs/object-serialization/core-concepts.md new file mode 100644 index 00000000000..c122e6d450d --- /dev/null +++ b/docs/object-serialization/core-concepts.md @@ -0,0 +1,122 @@ +--- +title: Core Concepts +sidebar_position: 1 +id: core-concepts +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Fory object serialization turns an object graph into bytes and reconstructs that graph later. The +same concepts apply to the default [xlang mode](xlang.md) and to supported +[native modes](native.md); the selected mode determines which types and wire rules are available. + +## Object graphs + +A root value may contain scalar fields, collections, maps, nested objects, repeated references, and +cycles. Serialization walks that graph from the root. Deserialization creates a new graph from the +encoded type and field data. + +This is different from serializing a row or a JSON document. Object serialization can preserve +runtime types and object identity so the reader can reconstruct application objects rather than +only values. Use [Row Format](../row-format/index.md) for trusted analytical rows and +[Fory JSON](../json/index.md) for JSON interchange. + +## Runtime instances and registration + +A Fory instance owns its mode, schema behavior, reference settings, registered types, custom +serializers, and read limits. Configure and register the instance before its first root +serialization or deserialization operation, then reuse it. Registration is frozen after the first +root operation so the same instance always resolves a type in the same way. + +Thread-safety differs by runtime. Some runtimes provide a thread-safe wrapper or pool; others use +one instance per thread or task. Follow the selected language guide instead of sharing an ordinary +instance without checking its concurrency contract. + +## Types and type identity + +Built-in types have identities owned by Fory. Application structs, classes, enums, unions, and +extension types use a registered numeric ID or name. Type identity answers _which serializer and +model should read this value_; a field schema describes _what data that model contains_. + +A statically known field can use its declared type directly. A dynamic field also carries the +concrete runtime type needed for interfaces, abstract classes, trait objects, broad object types, +or heterogeneous values. Dynamic typing is more flexible but requires every possible concrete type +to be registered and supported by the selected mode. + +In xlang mode, peers must coordinate the same portable type identity and mapping. Native mode may +use runtime-specific identities and types. See [Xlang Serialization](xlang.md) for the portable +rules and each language's Type Registration page for its exact API. + +## Schemas and evolution + +A schema describes the fields and nested types of a structured value. Compatible mode carries +metadata that lets a reader handle supported additions, removals, reordering, and type adaptations. +Use it when readers and writers may deploy independently. + +Same-schema mode assumes both sides use the same type identity, fields, nested types, nullability, +and reference metadata. It reduces metadata and payload size, but a schema mismatch is an error. +Use it only when one release process keeps every reader and writer aligned. + +Field IDs or names should remain stable after a contract is published. Renaming or reusing an +identity can turn an intended evolution into a different field or type. + +## Nullability + +Nullability determines whether a value position may contain no value. Languages express it through +nullable references, option types, pointers, annotations, or schema metadata. A nullable field is +not the same as a field whose value happens to use a default. + +Keep nullability consistent across readers and writers. Compatible mode can handle documented +nullable and missing-field cases, but it cannot place a remote null into a local carrier that has +no valid null or missing-value representation. + +## Reference tracking + +Reference tracking preserves object identity. Enable it when a graph contains the same object more +than once or contains a cycle. Without reference tracking, repeated values may become separate +objects and cycles may recurse until the operation fails. + +Leave reference tracking disabled for value-shaped, acyclic data when identity does not matter; it +adds per-object metadata and lookup work. Some runtimes combine a global setting with field-level +metadata, so use the language-specific References or Basic Serialization page for exact behavior. + +## Polymorphism + +Polymorphism stores the concrete type of a value whose declared position is broader. The reader +must know and accept that concrete type, and the type must be representable in the selected mode. + +Host-language inheritance alone does not create a portable contract. For cross-language data, +model only alternatives that have xlang mappings on every peer. For same-runtime data, native mode +may support additional runtime-specific class, trait, or hook behavior. + +## Custom serializers + +Use a custom serializer when a type needs a representation that built-in schema inference cannot +provide. Registration connects the custom serializer to the application type. A custom serializer +must follow the selected mode's rules: xlang serializers need a portable representation, while +native serializers may use runtime-specific data and hooks. + +Prefer built-in serializers and generated models when they already describe the type. They keep +schema evolution, reference handling, and cross-language behavior easier to reason about. + +## Continue with a mode + +- [Xlang Serialization](xlang.md) is the default and is required when different language runtimes + exchange bytes. +- [Native Serialization](native.md) is for supported same-runtime use cases that need native types + or behavior. +- Choose a language section after selecting a mode to find installation, API, configuration, + registration, platform, security, and troubleshooting guidance. diff --git a/docs/object-serialization/cpp/core-api.md b/docs/object-serialization/cpp/basic-serialization.md similarity index 51% rename from docs/object-serialization/cpp/core-api.md rename to docs/object-serialization/cpp/basic-serialization.md index c9ca5cbc3d6..275ad53ed48 100644 --- a/docs/object-serialization/cpp/core-api.md +++ b/docs/object-serialization/cpp/basic-serialization.md @@ -1,7 +1,7 @@ --- title: Basic Serialization -sidebar_position: 3 -id: core-api +sidebar_position: 1 +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,7 +19,7 @@ license: | limitations under the License. --- -This page covers basic object graph serialization and the core serialization APIs. +This page covers basic object graph serialization and the core serialization APIs in the default xlang mode. ## Object Graph Serialization @@ -355,6 +355,285 @@ fory.register_struct(2); - **Disable Tracking**: Use `track_ref(false)` when references aren't needed - **Compact Encoding**: Variable-length encoding for space efficiency +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + +This page explains how to use Fory xlang serialization between C++ and other languages. + +### Overview + +Apache Fory™ enables seamless data exchange between C++, Java, Python, Go, Rust, +JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. Xlang mode ensures +binary compatibility across all supported languages. + +### Xlang Configuration + +C++ defaults to xlang mode. Compatible schema evolution is also the xlang default. Set the mode explicitly in xlang examples: + +```cpp +#include "fory/serialization/fory.h" + +using namespace fory::serialization; + +auto fory = Fory::builder().xlang(true).build(); +``` + +### Xlang Example + +#### C++ Producer + +```cpp +#include "fory/serialization/fory.h" +#include + +using namespace fory::serialization; + +struct Message { + std::string topic; + int64_t timestamp; + std::map headers; + std::vector payload; + + bool operator==(const Message &other) const { + return topic == other.topic && timestamp == other.timestamp && + headers == other.headers && payload == other.payload; + } +}; +FORY_STRUCT(Message, topic, timestamp, headers, payload); + +int main() { + auto fory = Fory::builder().xlang(true).build(); + fory.register_struct(100); + + Message msg{ + "events.user", + 1699999999000, + {{"content-type", "application/json"}}, + {'h', 'e', 'l', 'l', 'o'} + }; + + auto result = fory.serialize(msg); + if (result.ok()) { + auto bytes = std::move(result).value(); + // write to file, send over network, etc. + std::ofstream file("message.bin", std::ios::binary); + file.write(reinterpret_cast(bytes.data()), bytes.size()); + } + return 0; +} +``` + +#### Java Consumer + +```java +import org.apache.fory.Fory; + +public class Message { + public String topic; + public long timestamp; + public Map headers; + public byte[] payload; +} + +public class Consumer { + public static void main(String[] args) throws Exception { + Fory fory = Fory.builder() + .withXlang(true) + .build(); + fory.register(Message.class, 100); // Same ID as C++ + + byte[] bytes = Files.readAllBytes(Path.of("message.bin")); + Message msg = (Message) fory.deserialize(bytes); + + System.out.println("Topic: " + msg.topic); + System.out.println("Timestamp: " + msg.timestamp); + } +} +``` + +#### Python Consumer + +```python +import pyfory + +class Message: + topic: str + timestamp: int + headers: dict[str, str] + payload: bytes + +fory = pyfory.Fory(xlang=True) +fory.register(Message, type_id=100) # Same ID as C++ + +with open("message.bin", "rb") as f: + data = f.read() + +msg = fory.deserialize(data) +print(f"Topic: {msg.topic}") +print(f"Timestamp: {msg.timestamp}") +``` + +### Type Mapping + +#### Primitive Types + +| C++ Type | Java Type | Python Type | Go Type | Rust Type | +| ------------------ | ---------- | ----------------- | ------------------- | ---------- | +| `bool` | `boolean` | `bool` | `bool` | `bool` | +| `int8_t` | `byte` | `int` | `int8` | `i8` | +| `int16_t` | `short` | `int` | `int16` | `i16` | +| `int32_t` | `int` | `int` | `int32` | `i32` | +| `int64_t` | `long` | `int` | `int64` | `i64` | +| `float` | `float` | `float` | `float32` | `f32` | +| `double` | `double` | `float` | `float64` | `f64` | +| `fory::float16_t` | `Float16` | `pyfory.Float16` | `float16.Float16` | `Float16` | +| `fory::bfloat16_t` | `BFloat16` | `pyfory.BFloat16` | `bfloat16.BFloat16` | `BFloat16` | + +#### String Types + +| C++ Type | Java Type | Python Type | Go Type | Rust Type | +| ------------- | --------- | ----------- | -------- | --------- | +| `std::string` | `String` | `str` | `string` | `String` | + +#### Collection Types + +| C++ Type | Java Type | Python Type | Go Type | Rust Type | +| ------------------------------------------- | -------------- | --------------- | --------------------- | --------------- | +| `std::vector` | `List` | `list` | `[]T` | `Vec` | +| `std::vector` | `Float16List` | `Float16Array` | `[]float16.Float16` | `Vec` | +| `std::vector` | `BFloat16List` | `BFloat16Array` | `[]bfloat16.BFloat16` | `Vec` | +| `std::set` | `Set` | `set` | `map[T]struct{}` | `HashSet` | +| `std::map` / `std::unordered_map` | `Map` | `dict` | `map[K]V` | `HashMap` | + +#### Lists and Dense Arrays + +`std::vector` maps to Fory `list` by default in handwritten C++ structs. +Use the field metadata DSL's array node when the schema is dense `array`. + +| Fory schema | C++ metadata sketch | +| ----------------- | ---------------------------------------- | +| `list` | `fory::F(id).list(fory::T::int32())` | +| `array` | `fory::F(id).array(fory::T::bool_())` | +| `array` | `fory::F(id).array(fory::T::int8())` | +| `array` | `fory::F(id).array(fory::T::int16())` | +| `array` | `fory::F(id).array(fory::T::int32())` | +| `array` | `fory::F(id).array(fory::T::int64())` | +| `array` | `fory::F(id).array(fory::T::uint8())` | +| `array` | `fory::F(id).array(fory::T::uint16())` | +| `array` | `fory::F(id).array(fory::T::uint32())` | +| `array` | `fory::F(id).array(fory::T::uint64())` | +| `array` | `fory::F(id).array(fory::T::float16())` | +| `array` | `fory::F(id).array(fory::T::bfloat16())` | +| `array` | `fory::F(id).array(fory::T::float32())` | +| `array` | `fory::F(id).array(fory::T::float64())` | + +#### Temporal Types + +| C++ Type | Java Type | Python Type | Go Type | +| ----------------- | ----------- | --------------- | --------------- | +| `fory::Timestamp` | `Instant` | `datetime` | `time.Time` | +| `fory::Duration` | `Duration` | `timedelta` | `time.Duration` | +| `fory::Date` | `LocalDate` | `datetime.date` | `time.Time` | + +### Field Order Requirements + +**Critical:** Fields are sorted by snake_case field name. The converted names must match across languages. + +#### C++ + +```cpp +struct Person { + std::string name; // Field 0 + int32_t age; // Field 1 + std::string email; // Field 2 +}; +FORY_STRUCT(Person, name, age, email); // Order matters! +``` + +#### Java + +```java +public class Person { + public String name; // Field 0 + public int age; // Field 1 + public String email; // Field 2 +} +``` + +#### Python + +```python +class Person: + name: str # Field 0 + age: int # Field 1 + email: str # Field 2 +``` + +### Type ID Consistency + +All languages must use the same type IDs: + +```cpp +// C++ +fory.register_struct(100); +fory.register_struct
(101); +fory.register_struct(102); +``` + +```java +// Java +fory.register(Person.class, 100); +fory.register(Address.class, 101); +fory.register(Order.class, 102); +``` + +```python +# Python +fory.register(Person, type_id=100) +fory.register(Address, type_id=101) +fory.register(Order, type_id=102) +``` + +### Compatible Mode + +Xlang mode already uses compatible schema evolution by default. Keep that default for schemas that +may evolve independently: + +```cpp +auto fory = Fory::builder().xlang(true).build(); +``` + +Compatible mode allows: + +- Adding new fields (with defaults) +- Removing unused fields +- Reordering fields + +### Interoperability Troubleshooting + +#### Type Mismatch Errors + +``` +Error: Type mismatch: expected 100, got 101 +``` + +**Solution:** Ensure type IDs match across all languages. + +#### Encoding Errors + +``` +Error: Invalid UTF-8 sequence +``` + +**Solution:** Ensure strings are valid UTF-8 in all languages. + +### Related Guides + +- [Configuration](configuration.md) - Builder options +- [Type Registration](type-registration.md) - Registering types +- [Supported Types](supported-types.md) - Type compatibility + ## Related Topics - [Configuration](configuration.md) - Builder options diff --git a/docs/object-serialization/cpp/configuration.md b/docs/object-serialization/cpp/configuration.md index 389636732fe..78c77939d2e 100644 --- a/docs/object-serialization/cpp/configuration.md +++ b/docs/object-serialization/cpp/configuration.md @@ -252,20 +252,10 @@ auto fory = Fory::builder().build_thread_safe(); // Returns ThreadSafeFory ## Security -Security-related configuration: - -- Register all structs and polymorphic implementations before deserializing untrusted payloads. -- Use `check_struct_version(true)` with `compatible(false)` for intentional same-schema payloads. -- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a - positive value for a trusted workload that needs a different collection/map/struct gate. -- Keep `max_dyn_depth(...)` as low as your model permits to reject unexpectedly deep polymorphic - graphs. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer concrete fields over broad polymorphic fields for untrusted input. +See [C++ Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) - Using configured Fory -- [Xlang Serialization](xlang.md) - xlang mode details +- [Basic Serialization](basic-serialization.md) - Using configured Fory +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - xlang mode details - [Type Registration](type-registration.md) - Registering types diff --git a/docs/object-serialization/cpp/custom-serializers.md b/docs/object-serialization/cpp/custom-serializers.md index bc7106e79ee..089e10df087 100644 --- a/docs/object-serialization/cpp/custom-serializers.md +++ b/docs/object-serialization/cpp/custom-serializers.md @@ -366,6 +366,6 @@ static MyType read_data(ReadContext &ctx) { ## Related Topics - [Type Registration](type-registration.md) - Registering serializers -- [Basic Serialization](core-api.md) - Using FORY_STRUCT macro +- [Basic Serialization](basic-serialization.md) - Using FORY_STRUCT macro - [Schema Evolution](schema-evolution.md) - Compatible mode -- [Xlang Serialization](xlang.md) - Cross-language serialization +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language serialization diff --git a/docs/object-serialization/cpp/index.md b/docs/object-serialization/cpp/index.md index c69bcd5215e..a18ee4f393b 100644 --- a/docs/object-serialization/cpp/index.md +++ b/docs/object-serialization/cpp/index.md @@ -222,7 +222,7 @@ Use xlang mode for cross-language payloads and schemas shared with other Fory im Use native mode for C++-only traffic. Native mode is selected with `.xlang(false)` and keeps C++ object serialization in C++-native form. It is optimized for C++ types and avoids portable xlang type-mapping constraints when the payload never leaves C++. Compatible mode is enabled by default. Set `.compatible(false)` only when every reader and writer uses the same C++ schema and you want faster serialization and smaller size. -See [Xlang Serialization](xlang.md) for C++ xlang registration and interoperability rules, and [Native Serialization](native.md) for C++-only payloads. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for C++ xlang registration and interoperability rules, and [Native Serialization](native.md) for C++-only payloads. ## Thread Safety @@ -264,8 +264,7 @@ std::thread t2([&]() { ## Next Steps - [Configuration](configuration.md) - Builder options and modes -- [Basic Serialization](core-api.md) - Object graph serialization -- [Xlang Serialization](xlang.md) - xlang mode and interoperability +- [Basic Serialization](basic-serialization.md) - Default xlang object graphs and interoperability - [Native Serialization](native.md) - C++-only serialization - [Schema Metadata](schema-metadata.md) - Field-level metadata (nullable, ref tracking) - [Schema Evolution](schema-evolution.md) - Compatible mode and schema changes @@ -274,3 +273,6 @@ std::thread t2([&]() { - [Custom Serializers](custom-serializers.md) - Extend serialization behavior - [Row Format](../../row-format/cpp.md) - Zero-copy row-based format - [gRPC Support](../../grpc/cpp.md) - Fory payloads over gRPC C++ + +Before decoding bytes from outside the application trust boundary, read +[C++ Security](security.md). diff --git a/docs/object-serialization/cpp/native.md b/docs/object-serialization/cpp/native.md index a93c9ed06d9..70df2a91ba0 100644 --- a/docs/object-serialization/cpp/native.md +++ b/docs/object-serialization/cpp/native.md @@ -23,7 +23,7 @@ C++ native serialization is the C++-only wire mode selected with `.xlang(false)` writer and reader is C++ and the payload should follow C++ type behavior instead of the portable xlang type system. -Use [Xlang Serialization](xlang.md), the default C++ mode, when +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default C++ mode, when bytes must be read by Java, Python, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another non-C++ Fory implementation. @@ -206,9 +206,9 @@ Enable `.track_ref(true)` and verify the graph uses supported pointer patterns. ## Related Topics -- [Xlang Serialization](xlang.md) - Cross-language C++ payloads +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language C++ payloads - [Configuration](configuration.md) - Builder options -- [Basic Serialization](core-api.md) - Object graph serialization +- [Basic Serialization](basic-serialization.md) - Object graph serialization - [Supported Types](supported-types.md) - C++ type support - [Polymorphic Serialization](polymorphism.md) - Polymorphic object models - [Schema Evolution](schema-evolution.md) - Compatible mode diff --git a/docs/object-serialization/cpp/polymorphism.md b/docs/object-serialization/cpp/polymorphism.md index 3f609ad2802..383ba7c36d8 100644 --- a/docs/object-serialization/cpp/polymorphism.md +++ b/docs/object-serialization/cpp/polymorphism.md @@ -487,4 +487,4 @@ if (!decoded_result.ok()) { - [Schema Metadata](schema-metadata.md) - Field-level metadata and options - [Supported Types](supported-types.md) - Smart pointers and collections - [Configuration](configuration.md) - `max_dyn_depth` and other settings -- [Basic Serialization](core-api.md) - Core serialization concepts +- [Basic Serialization](basic-serialization.md) - Core serialization concepts diff --git a/docs/object-serialization/cpp/schema-evolution.md b/docs/object-serialization/cpp/schema-evolution.md index 8ca540479d4..4b7ab928f82 100644 --- a/docs/object-serialization/cpp/schema-evolution.md +++ b/docs/object-serialization/cpp/schema-evolution.md @@ -421,4 +421,4 @@ Both instances can exchange data even with different schema versions. - [Configuration](configuration.md) - Enabling compatible mode - [Type Registration](type-registration.md) - Type ID management -- [Xlang Serialization](xlang.md) - Cross-language considerations +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language considerations diff --git a/docs/object-serialization/cpp/security.md b/docs/object-serialization/cpp/security.md new file mode 100644 index 00000000000..fe86e37d2db --- /dev/null +++ b/docs/object-serialization/cpp/security.md @@ -0,0 +1,57 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a C++ reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register all structs and polymorphic implementations before deserializing untrusted payloads. +- Use `check_struct_version(true)` with `compatible(false)` for intentional same-schema payloads. +- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a + positive value for a trusted workload that needs a different collection/map/struct gate. +- Keep `max_dyn_depth(...)` as low as your model permits to reject unexpectedly deep polymorphic + graphs. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer concrete fields over broad polymorphic fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/cpp/supported-types.md b/docs/object-serialization/cpp/supported-types.md index 11f21a9e332..4c95f5307e7 100644 --- a/docs/object-serialization/cpp/supported-types.md +++ b/docs/object-serialization/cpp/supported-types.md @@ -295,6 +295,6 @@ Currently not supported: ## Related Topics -- [Basic Serialization](core-api.md) - Using these types +- [Basic Serialization](basic-serialization.md) - Using these types - [Type Registration](type-registration.md) - Registering types -- [Xlang Serialization](xlang.md) - Cross-language compatibility +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language compatibility diff --git a/docs/object-serialization/cpp/type-registration.md b/docs/object-serialization/cpp/type-registration.md index 6e2a3e51892..62f525ef945 100644 --- a/docs/object-serialization/cpp/type-registration.md +++ b/docs/object-serialization/cpp/type-registration.md @@ -249,6 +249,6 @@ if (!result.ok()) { ## Related Topics -- [Basic Serialization](core-api.md) - Using registered types -- [Xlang Serialization](xlang.md) - Cross-language considerations +- [Basic Serialization](basic-serialization.md) - Using registered types +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language considerations - [Supported Types](supported-types.md) - All supported types diff --git a/docs/object-serialization/cpp/xlang.md b/docs/object-serialization/cpp/xlang.md deleted file mode 100644 index 726f63bda58..00000000000 --- a/docs/object-serialization/cpp/xlang.md +++ /dev/null @@ -1,295 +0,0 @@ ---- -title: Xlang Serialization -sidebar_position: 1 -id: xlang -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page explains how to use Fory xlang serialization between C++ and other languages. - -## Overview - -Apache Fory™ enables seamless data exchange between C++, Java, Python, Go, Rust, -JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. Xlang mode ensures -binary compatibility across all supported languages. - -## Create an Xlang Fory Instance - -C++ defaults to xlang mode. Compatible schema evolution is also the xlang default. Set the mode explicitly in xlang examples: - -```cpp -#include "fory/serialization/fory.h" - -using namespace fory::serialization; - -auto fory = Fory::builder().xlang(true).build(); -``` - -## Xlang Example - -### C++ Producer - -```cpp -#include "fory/serialization/fory.h" -#include - -using namespace fory::serialization; - -struct Message { - std::string topic; - int64_t timestamp; - std::map headers; - std::vector payload; - - bool operator==(const Message &other) const { - return topic == other.topic && timestamp == other.timestamp && - headers == other.headers && payload == other.payload; - } -}; -FORY_STRUCT(Message, topic, timestamp, headers, payload); - -int main() { - auto fory = Fory::builder().xlang(true).build(); - fory.register_struct(100); - - Message msg{ - "events.user", - 1699999999000, - {{"content-type", "application/json"}}, - {'h', 'e', 'l', 'l', 'o'} - }; - - auto result = fory.serialize(msg); - if (result.ok()) { - auto bytes = std::move(result).value(); - // write to file, send over network, etc. - std::ofstream file("message.bin", std::ios::binary); - file.write(reinterpret_cast(bytes.data()), bytes.size()); - } - return 0; -} -``` - -### Java Consumer - -```java -import org.apache.fory.Fory; - -public class Message { - public String topic; - public long timestamp; - public Map headers; - public byte[] payload; -} - -public class Consumer { - public static void main(String[] args) throws Exception { - Fory fory = Fory.builder() - .withXlang(true) - .build(); - fory.register(Message.class, 100); // Same ID as C++ - - byte[] bytes = Files.readAllBytes(Path.of("message.bin")); - Message msg = (Message) fory.deserialize(bytes); - - System.out.println("Topic: " + msg.topic); - System.out.println("Timestamp: " + msg.timestamp); - } -} -``` - -### Python Consumer - -```python -import pyfory - -class Message: - topic: str - timestamp: int - headers: dict[str, str] - payload: bytes - -fory = pyfory.Fory(xlang=True) -fory.register(Message, type_id=100) # Same ID as C++ - -with open("message.bin", "rb") as f: - data = f.read() - -msg = fory.deserialize(data) -print(f"Topic: {msg.topic}") -print(f"Timestamp: {msg.timestamp}") -``` - -## Type Mapping - -### Primitive Types - -| C++ Type | Java Type | Python Type | Go Type | Rust Type | -| ------------------ | ---------- | ----------------- | ------------------- | ---------- | -| `bool` | `boolean` | `bool` | `bool` | `bool` | -| `int8_t` | `byte` | `int` | `int8` | `i8` | -| `int16_t` | `short` | `int` | `int16` | `i16` | -| `int32_t` | `int` | `int` | `int32` | `i32` | -| `int64_t` | `long` | `int` | `int64` | `i64` | -| `float` | `float` | `float` | `float32` | `f32` | -| `double` | `double` | `float` | `float64` | `f64` | -| `fory::float16_t` | `Float16` | `pyfory.Float16` | `float16.Float16` | `Float16` | -| `fory::bfloat16_t` | `BFloat16` | `pyfory.BFloat16` | `bfloat16.BFloat16` | `BFloat16` | - -### String Types - -| C++ Type | Java Type | Python Type | Go Type | Rust Type | -| ------------- | --------- | ----------- | -------- | --------- | -| `std::string` | `String` | `str` | `string` | `String` | - -### Collection Types - -| C++ Type | Java Type | Python Type | Go Type | Rust Type | -| ------------------------------------------- | -------------- | --------------- | --------------------- | --------------- | -| `std::vector` | `List` | `list` | `[]T` | `Vec` | -| `std::vector` | `Float16List` | `Float16Array` | `[]float16.Float16` | `Vec` | -| `std::vector` | `BFloat16List` | `BFloat16Array` | `[]bfloat16.BFloat16` | `Vec` | -| `std::set` | `Set` | `set` | `map[T]struct{}` | `HashSet` | -| `std::map` / `std::unordered_map` | `Map` | `dict` | `map[K]V` | `HashMap` | - -### Lists and Dense Arrays - -`std::vector` maps to Fory `list` by default in handwritten C++ structs. -Use the field metadata DSL's array node when the schema is dense `array`. - -| Fory schema | C++ metadata sketch | -| ----------------- | ---------------------------------------- | -| `list` | `fory::F(id).list(fory::T::int32())` | -| `array` | `fory::F(id).array(fory::T::bool_())` | -| `array` | `fory::F(id).array(fory::T::int8())` | -| `array` | `fory::F(id).array(fory::T::int16())` | -| `array` | `fory::F(id).array(fory::T::int32())` | -| `array` | `fory::F(id).array(fory::T::int64())` | -| `array` | `fory::F(id).array(fory::T::uint8())` | -| `array` | `fory::F(id).array(fory::T::uint16())` | -| `array` | `fory::F(id).array(fory::T::uint32())` | -| `array` | `fory::F(id).array(fory::T::uint64())` | -| `array` | `fory::F(id).array(fory::T::float16())` | -| `array` | `fory::F(id).array(fory::T::bfloat16())` | -| `array` | `fory::F(id).array(fory::T::float32())` | -| `array` | `fory::F(id).array(fory::T::float64())` | - -### Temporal Types - -| C++ Type | Java Type | Python Type | Go Type | -| ----------------- | ----------- | --------------- | --------------- | -| `fory::Timestamp` | `Instant` | `datetime` | `time.Time` | -| `fory::Duration` | `Duration` | `timedelta` | `time.Duration` | -| `fory::Date` | `LocalDate` | `datetime.date` | `time.Time` | - -## Field Order Requirements - -**Critical:** Fields are sorted by snake_case field name. The converted names must match across languages. - -### C++ - -```cpp -struct Person { - std::string name; // Field 0 - int32_t age; // Field 1 - std::string email; // Field 2 -}; -FORY_STRUCT(Person, name, age, email); // Order matters! -``` - -### Java - -```java -public class Person { - public String name; // Field 0 - public int age; // Field 1 - public String email; // Field 2 -} -``` - -### Python - -```python -class Person: - name: str # Field 0 - age: int # Field 1 - email: str # Field 2 -``` - -## Type ID Consistency - -All languages must use the same type IDs: - -```cpp -// C++ -fory.register_struct(100); -fory.register_struct
(101); -fory.register_struct(102); -``` - -```java -// Java -fory.register(Person.class, 100); -fory.register(Address.class, 101); -fory.register(Order.class, 102); -``` - -```python -# Python -fory.register(Person, type_id=100) -fory.register(Address, type_id=101) -fory.register(Order, type_id=102) -``` - -## Compatible Mode - -Xlang mode already uses compatible schema evolution by default. Keep that default for schemas that -may evolve independently: - -```cpp -auto fory = Fory::builder().xlang(true).build(); -``` - -Compatible mode allows: - -- Adding new fields (with defaults) -- Removing unused fields -- Reordering fields - -## Troubleshooting - -### Type Mismatch Errors - -``` -Error: Type mismatch: expected 100, got 101 -``` - -**Solution:** Ensure type IDs match across all languages. - -### Encoding Errors - -``` -Error: Invalid UTF-8 sequence -``` - -**Solution:** Ensure strings are valid UTF-8 in all languages. - -## Related Topics - -- [Configuration](configuration.md) - Builder options -- [Type Registration](type-registration.md) - Registering types -- [Supported Types](supported-types.md) - Type compatibility diff --git a/docs/object-serialization/csharp/core-api.md b/docs/object-serialization/csharp/basic-serialization.md similarity index 55% rename from docs/object-serialization/csharp/core-api.md rename to docs/object-serialization/csharp/basic-serialization.md index 43020f81300..7c77560a7d3 100644 --- a/docs/object-serialization/csharp/core-api.md +++ b/docs/object-serialization/csharp/basic-serialization.md @@ -1,7 +1,7 @@ --- title: Basic Serialization -sidebar_position: 3 -id: core-api +sidebar_position: 1 +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,7 +19,7 @@ license: | limitations under the License. --- -This page covers typed serialization APIs in Apache Fory™ C#. +This page covers typed serialization APIs in the default xlang mode for Apache Fory™ C#. ## Object Graph Serialization @@ -159,6 +159,125 @@ fory.Serialize(dynamicWriter, value); - Register user types handled by `[ForyStruct]`, `[ForyEnum]`, `[ForyUnion]`, external structural serializers, or custom serializers explicitly. +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + +Apache Fory™ C# supports xlang serialization with other Fory implementations. + +### Xlang Configuration + +C# always writes and reads the xlang frame header. There is no mode switch, so interoperability code +only needs to configure the remaining settings such as compatibility mode and reference +tracking. + +```csharp +Fory fory = Fory.Builder() + .Build(); +``` + +### Register with Stable IDs + +```csharp +[ForyStruct] +public sealed class Person +{ + public string Name { get; set; } = string.Empty; + public int Age { get; set; } +} + +Fory fory = Fory.Builder() + .Build(); + +fory.Register(100); +``` + +Use the same ID mapping on all languages. + +Third-party classes, structs, and enums can use +[external-type serialization](external-types.md). Register the target type, +not its local serializer declaration, with the same ID or name used by the +other language peers. + +### Register by Name + +```csharp +fory.Register("com.example.Person"); +``` + +### Xlang Example + +#### C# (Serializer) + +```csharp +Person person = new() { Name = "Alice", Age = 30 }; +byte[] payload = fory.Serialize(person); +``` + +#### Java (Deserializer) + +```java +Fory fory = Fory.builder() + .withXlang(true) + .withRefTracking(true) + .build(); + +fory.register(Person.class, 100); +Person value = (Person) fory.deserialize(payloadFromCSharp); +``` + +#### Python (Deserializer) + +```python +import pyfory + +fory = pyfory.Fory(xlang=True, ref=True) +fory.register_type(Person, type_id=100) +value = fory.deserialize(payload_from_csharp) +``` + +### Type Mapping Reference + +See [cross-language interoperability guide](../xlang.md) for complete mapping. + +For reduced-precision numeric payloads, use `Half` / `Half[]` or `List` for xlang `float16`, and `BFloat16` / `BFloat16[]` or `List` for xlang `bfloat16`. + +### Lists and Dense Arrays + +C# `List` maps to Fory `list`. Use the schema marker +`Apache.Fory.Schema.Types.Array` when a field is dense `array`. + +| Fory schema | C# schema marker sketch | +| ----------------- | ----------------------- | +| `list` | `S.List` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | + +### Interoperability Best Practices + +1. Keep type IDs stable and documented. +2. Keep compatible mode enabled for rolling upgrades. +3. Register all user types on both read/write peers. +4. Validate integration with real payload round trips. + +### Related Guides + +- [Type Registration](type-registration.md) +- [External Types](external-types.md) +- [Schema Evolution](schema-evolution.md) +- [Supported Types](supported-types.md) + ## Related Topics - [Type Registration](type-registration.md) diff --git a/docs/object-serialization/csharp/configuration.md b/docs/object-serialization/csharp/configuration.md index 2d45d409d97..5bac8df215d 100644 --- a/docs/object-serialization/csharp/configuration.md +++ b/docs/object-serialization/csharp/configuration.md @@ -200,19 +200,10 @@ ThreadSafeFory fory = Fory.Builder() ## Security -Security-related configuration: - -- Register only the expected types before deserializing untrusted payloads. -- Use `CheckStructVersion(true)` with `Compatible(false)` for intentional same-schema payloads. -- Set `MaxDepth(...)` to reject unexpectedly deep dynamic object graphs. -- Set `MaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input bytes. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer generated or registered concrete models over broad dynamic fields for untrusted input. +See [C# Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Evolution](schema-evolution.md) - [Thread Safety](thread-safety.md) diff --git a/docs/object-serialization/csharp/external-types.md b/docs/object-serialization/csharp/external-types.md index b1802f07bef..0b1e18fd1a5 100644 --- a/docs/object-serialization/csharp/external-types.md +++ b/docs/object-serialization/csharp/external-types.md @@ -220,7 +220,7 @@ long as each concrete annotated child has a legal parameterless construction path. First-party bases use direct `[ForyStruct]` annotations instead. See -[Class Inheritance](core-api.md#class-inheritance). +[Class Inheritance](basic-serialization.md#class-inheritance). ## Declaration and Target Requirements diff --git a/docs/object-serialization/csharp/index.md b/docs/object-serialization/csharp/index.md index 66e3d2b5ffc..0b98a6df9fe 100644 --- a/docs/object-serialization/csharp/index.md +++ b/docs/object-serialization/csharp/index.md @@ -86,24 +86,26 @@ User decoded = fory.Deserialize(payload); ## Documentation -| Topic | Description | -| ------------------------------------------- | --------------------------------------------- | -| [Configuration](configuration.md) | Builder options and mode settings | -| [Basic Serialization](core-api.md) | Typed and dynamic serialization APIs | -| [Xlang Serialization](xlang.md) | Interoperability guidance | -| [Schema Metadata](schema-metadata.md) | `[ForyField]` ids and schema type descriptors | -| [Type Registration](type-registration.md) | Registering user types and custom serializers | -| [External Types](external-types.md) | Serializers for third-party types | -| [Custom Serializers](custom-serializers.md) | Implementing `Serializer` | -| [References](references.md) | Shared/circular reference handling | -| [Schema Evolution](schema-evolution.md) | Compatible mode behavior | -| [Supported Types](supported-types.md) | Built-in and generated type support | -| [Thread Safety](thread-safety.md) | `Fory` vs `ThreadSafeFory` usage | -| [gRPC Support](../../grpc/csharp.md) | Generated Fory-backed gRPC service companions | -| [Troubleshooting](troubleshooting.md) | Common errors and debugging steps | +| Topic | Description | +| --------------------------------------------- | --------------------------------------------- | +| [Configuration](configuration.md) | Builder options and mode settings | +| [Basic Serialization](basic-serialization.md) | Default xlang APIs and interoperability | +| [Schema Metadata](schema-metadata.md) | `[ForyField]` ids and schema type descriptors | +| [Type Registration](type-registration.md) | Registering user types and custom serializers | +| [External Types](external-types.md) | Serializers for third-party types | +| [Custom Serializers](custom-serializers.md) | Implementing `Serializer` | +| [References](references.md) | Shared/circular reference handling | +| [Schema Evolution](schema-evolution.md) | Compatible mode behavior | +| [Supported Types](supported-types.md) | Built-in and generated type support | +| [Thread Safety](thread-safety.md) | `Fory` vs `ThreadSafeFory` usage | +| [gRPC Support](../../grpc/csharp.md) | Generated Fory-backed gRPC service companions | +| [Troubleshooting](troubleshooting.md) | Common errors and debugging steps | ## Related Resources - [Xlang serialization specification](../../specification/xlang_serialization_spec.md) -- [Xlang guide](../xlang/index.md) +- [Cross-language guide](../xlang.md) - [C# source directory](https://github.com/apache/fory/tree/main/csharp) + +Before decoding bytes from outside the application trust boundary, read +[C# Security](security.md). diff --git a/docs/object-serialization/csharp/references.md b/docs/object-serialization/csharp/references.md index 00d044ba07a..723503136bc 100644 --- a/docs/object-serialization/csharp/references.md +++ b/docs/object-serialization/csharp/references.md @@ -76,6 +76,6 @@ supported; Fory rejects unresolved refs instead of returning a partial union. ## Related Topics - [Configuration](configuration.md) -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [External Types](external-types.md) - [Thread Safety](thread-safety.md) diff --git a/docs/object-serialization/csharp/security.md b/docs/object-serialization/csharp/security.md new file mode 100644 index 00000000000..06416a73e0b --- /dev/null +++ b/docs/object-serialization/csharp/security.md @@ -0,0 +1,56 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a C# reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected types before deserializing untrusted payloads. +- Use `CheckStructVersion(true)` with `Compatible(false)` for intentional same-schema payloads. +- Set `MaxDepth(...)` to reject unexpectedly deep dynamic object graphs. +- Set `MaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input bytes. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer generated or registered concrete models over broad dynamic fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/csharp/supported-types.md b/docs/object-serialization/csharp/supported-types.md index 7f4a060456a..bf4f0ad08dd 100644 --- a/docs/object-serialization/csharp/supported-types.md +++ b/docs/object-serialization/csharp/supported-types.md @@ -107,11 +107,11 @@ Dynamic object payloads via `Serialize` / `Deserialize` suppor ## Notes - User-defined types should be registered explicitly. -- For cross-language usage, follow the [xlang guide](../xlang/index.md). +- For cross-language usage, follow the [cross-language interoperability guide](../xlang.md). ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [External Types](external-types.md) - [Type Registration](type-registration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/csharp/type-registration.md b/docs/object-serialization/csharp/type-registration.md index 2d64ee2c494..0c6b629310a 100644 --- a/docs/object-serialization/csharp/type-registration.md +++ b/docs/object-serialization/csharp/type-registration.md @@ -94,7 +94,7 @@ fory.Register(101); ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [External Types](external-types.md) - [Custom Serializers](custom-serializers.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/csharp/xlang.md b/docs/object-serialization/csharp/xlang.md deleted file mode 100644 index fe17a459c54..00000000000 --- a/docs/object-serialization/csharp/xlang.md +++ /dev/null @@ -1,135 +0,0 @@ ---- -title: Xlang Serialization -sidebar_position: 1 -id: xlang -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Apache Fory™ C# supports xlang serialization with other Fory implementations. - -## Xlang Fory Instance - -C# always writes and reads the xlang frame header. There is no mode switch, so interoperability code -only needs to configure the remaining settings such as compatibility mode and reference -tracking. - -```csharp -Fory fory = Fory.Builder() - .Build(); -``` - -## Register with Stable IDs - -```csharp -[ForyStruct] -public sealed class Person -{ - public string Name { get; set; } = string.Empty; - public int Age { get; set; } -} - -Fory fory = Fory.Builder() - .Build(); - -fory.Register(100); -``` - -Use the same ID mapping on all languages. - -Third-party classes, structs, and enums can use -[external-type serialization](external-types.md). Register the target type, -not its local serializer declaration, with the same ID or name used by the -other language peers. - -## Register by Name - -```csharp -fory.Register("com.example.Person"); -``` - -## Xlang Example - -### C# (Serializer) - -```csharp -Person person = new() { Name = "Alice", Age = 30 }; -byte[] payload = fory.Serialize(person); -``` - -### Java (Deserializer) - -```java -Fory fory = Fory.builder() - .withXlang(true) - .withRefTracking(true) - .build(); - -fory.register(Person.class, 100); -Person value = (Person) fory.deserialize(payloadFromCSharp); -``` - -### Python (Deserializer) - -```python -import pyfory - -fory = pyfory.Fory(xlang=True, ref=True) -fory.register_type(Person, type_id=100) -value = fory.deserialize(payload_from_csharp) -``` - -## Type Mapping Reference - -See [xlang guide](../xlang/index.md) for complete mapping. - -For reduced-precision numeric payloads, use `Half` / `Half[]` or `List` for xlang `float16`, and `BFloat16` / `BFloat16[]` or `List` for xlang `bfloat16`. - -## Lists and Dense Arrays - -C# `List` maps to Fory `list`. Use the schema marker -`Apache.Fory.Schema.Types.Array` when a field is dense `array`. - -| Fory schema | C# schema marker sketch | -| ----------------- | ----------------------- | -| `list` | `S.List` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | - -## Best Practices - -1. Keep type IDs stable and documented. -2. Keep compatible mode enabled for rolling upgrades. -3. Register all user types on both read/write peers. -4. Validate integration with real payload round trips. - -## Related Topics - -- [Type Registration](type-registration.md) -- [External Types](external-types.md) -- [Schema Evolution](schema-evolution.md) -- [Supported Types](supported-types.md) diff --git a/docs/object-serialization/dart/xlang.md b/docs/object-serialization/dart/basic-serialization.md similarity index 68% rename from docs/object-serialization/dart/xlang.md rename to docs/object-serialization/dart/basic-serialization.md index e65c9210f96..0b28adb02b3 100644 --- a/docs/object-serialization/dart/xlang.md +++ b/docs/object-serialization/dart/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,9 +19,132 @@ license: | limitations under the License. --- +This page shows how to serialize and deserialize values in the default xlang mode for Apache Fory™ Dart. + +## Create a `Fory` Instance + +Create one instance and reuse it — creating a new `Fory` for every call wastes resources. + +```dart +import 'package:fory/fory.dart'; + +final fory = Fory(); +``` + +## Serialize and Deserialize Annotated Types + +```dart +import 'package:fory/fory.dart'; + +part 'person.fory.dart'; + +@ForyStruct() +class Person { + Person(); + + String name = ''; + + @ForyField(type: Int32Type()) + int age = 0; +} + +void main() { + final fory = Fory(); + PersonForyModule.register( + fory, + Person, + name: 'example.Person', + ); + + final person = Person() + ..name = 'Ada' + ..age = 36; + + final bytes = fory.serialize(person); + final roundTrip = fory.deserialize(bytes); + print(roundTrip.name); +} +``` + +`deserialize` returns the decoded value cast to `T`. If the payload describes a different type than `T`, it throws. + +## Null Values + +Serializing `null` is supported directly: + +```dart +final fory = Fory(); +final bytes = fory.serialize(null); +final value = fory.deserialize(bytes); +``` + +## Serialize Collections and Dynamic Payloads + +You can serialize collection values directly: + +```dart +final fory = Fory(); +final bytes = fory.serialize([ + 'hello', + 42, + true, +]); +final value = fory.deserialize>(bytes); +``` + +For heterogeneous collections, deserialize to `Object?`, `List`, or `Map`. + +## Reference Tracking + +By default, Fory does not track object identity — if the same object appears twice in a list, it is serialized twice. Enable reference tracking when your data contains shared references or circular structures. + +For a top-level collection: + +```dart +final fory = Fory(); +final shared = String.fromCharCodes('shared'.codeUnits); +final bytes = fory.serialize([shared, shared], trackRef: true); +final roundTrip = fory.deserialize>(bytes); +print(identical(roundTrip[0], roundTrip[1])); // true +``` + +For fields inside a generated struct, use `@ForyField(ref: true)` on that field instead. + +## Reusing a Buffer + +If you want to avoid allocating a new `Uint8List` on every call, use `serializeTo` and `deserializeFrom` with an explicit `Buffer`: + +```dart +final fory = Fory(); +final buffer = Buffer(); + +fory.serializeTo('Ada', buffer); +final value = fory.deserializeFrom(buffer); +``` + +This is an optimization. For most applications the default `serialize`/`deserialize` pair is fine. + +## Register Your Types Before Serializing + +Before you can serialize a custom class or enum, register it with `Fory`. The generated code makes this easy: + +```dart +PersonForyModule.register( + fory, + Person, + id: 100, +); +``` + +If you skip registration, deserialization fails with `Type ... is not registered`. See [Type Registration](type-registration.md) and [Code Generation](code-generation.md). + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Apache Fory™ Dart serializes to the same binary format as the Java, Go, C#, Python, Rust, and Swift Fory implementations. You can write a message in Dart and read it in Java — or any other direction — without any conversion layer. -## Setup +### Xlang Configuration Create a `Fory` instance as normal. There is no separate xlang option to enable in Dart: @@ -31,11 +154,11 @@ final fory = Fory(); // xlang payloads with compatible schema evolution The key requirement is that both sides register the same type using the same identity. -## Registration Identity +### Registration Identity The most important rule: **use the same type identity on every side**. You have two options: -### Numeric ID +#### Numeric ID Simpler for small, tightly-coordinated teams: @@ -44,7 +167,7 @@ Simpler for small, tightly-coordinated teams: ModelsForyModule.register(fory, Person, id: 100); ``` -### Namespace + Type Name +#### Namespace + Type Name Better when multiple teams define types independently: @@ -59,7 +182,7 @@ ModelsForyModule.register( Do not mix the two strategies for the same type across implementations. -## External Types +### External Types For a struct class owned by another Dart package, define an [external structural serializer](external-types.md) and register the target @@ -87,9 +210,9 @@ define the Dart-side xlang schema. An external declaration may explicitly list an accessible inherited target property, but Fory does not automatically scan the external target hierarchy. -## Dart to Java Example +### Dart to Java Example -### Dart +#### Dart ```dart import 'package:fory/fory.dart'; @@ -113,7 +236,7 @@ final bytes = fory.serialize(Person() ..age = 30); ``` -### Java +#### Java ```java Fory fory = Fory.builder() @@ -124,9 +247,9 @@ fory.register(Person.class, 100); Person value = (Person) fory.deserialize(bytesFromDart); ``` -## Dart to C# Example +### Dart to C# Example -### Dart +#### Dart ```dart final fory = Fory(); @@ -136,7 +259,7 @@ final bytes = fory.serialize(Person() ..age = 30); ``` -### CSharp +#### CSharp ```csharp [ForyStruct] @@ -153,9 +276,9 @@ fory.Register(100); Person person = fory.Deserialize(payloadFromDart); ``` -## Dart to Go Example +### Dart to Go Example -### Dart +#### Dart ```dart final fory = Fory(); @@ -165,7 +288,7 @@ final bytes = fory.serialize(Person() ..age = 30); ``` -### Go +#### Go ```go type Person struct { @@ -180,7 +303,7 @@ var person Person _ = f.Deserialize(bytesFromDart, &person) ``` -## Field Matching Rules +### Field Matching Rules Fory matches fields by name or by stable field ID. For robust cross-language interop: @@ -204,7 +327,7 @@ metadata use the same reference behavior as fields declared directly on the child. Inheritance does not change xlang reference framing or add parent-level reference state. -## Type Mapping Notes for Dart +### Type Mapping Notes for Dart Because Dart `int` is not itself a promise about the exact xlang wire width, prefer explicit field metadata when exact cross-language interpretation matters: @@ -217,7 +340,7 @@ Because Dart `int` is not itself a promise about the exact xlang wire width, pre - `Float16List` and `Bfloat16List` for 16-bit floating-point array payloads - `Timestamp`, `LocalDate`, and `Duration` for explicit temporal semantics -### Lists and Dense Arrays +#### Lists and Dense Arrays `List` always represents Fory `list` unless a field has explicit array metadata. Use `array` only for dense one-dimensional bool or numeric data. @@ -241,7 +364,7 @@ metadata. Use `array` only for dense one-dimensional bool or numeric data. See [Supported Types](supported-types.md) and [xlang type mapping](../../specification/xlang_type_mapping.md). -## Validation +### Validation Before relying on a cross-language contract in production, test a payload end-to-end through every implementation you support. @@ -253,10 +376,16 @@ dart analyze dart test ``` -## Related Topics +### Related Guides - [Struct Inheritance](inheritance.md) - [Type Registration](type-registration.md) - [External-Type Serialization](external-types.md) - [Schema Evolution](schema-evolution.md) -- [Xlang guide](../xlang/index.md) +- [Cross-language guide](../xlang.md) + +## Related Topics + +- [Configuration](configuration.md) +- [Type Registration](type-registration.md) +- [Schema Metadata](schema-metadata.md) diff --git a/docs/object-serialization/dart/configuration.md b/docs/object-serialization/dart/configuration.md index cbacbb5f654..7094e9b4a90 100644 --- a/docs/object-serialization/dart/configuration.md +++ b/docs/object-serialization/dart/configuration.md @@ -162,19 +162,10 @@ When Fory is used to communicate between services written in different languages ## Security -Security-related configuration: - -- Register only the expected generated models before deserializing untrusted payloads. -- Use `checkStructVersion: true` with `compatible: false` for intentional same-schema payloads. -- Set `maxDepth` to reject unexpectedly deep payload shapes. -- Keep `maxGraphMemoryBytes` at the default for most inputs, or set an explicit positive byte gate - for known trusted collection/map/struct-heavy payloads. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer generated schemas and explicit field metadata over broad dynamic fields for untrusted input. +See [Dart Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Evolution](schema-evolution.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/dart/core-api.md b/docs/object-serialization/dart/core-api.md deleted file mode 100644 index 08d586971a1..00000000000 --- a/docs/object-serialization/dart/core-api.md +++ /dev/null @@ -1,145 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 2 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page shows how to serialize and deserialize values with Apache Fory™ Dart. - -## Create a `Fory` Instance - -Create one instance and reuse it — creating a new `Fory` for every call wastes resources. - -```dart -import 'package:fory/fory.dart'; - -final fory = Fory(); -``` - -## Serialize and Deserialize Annotated Types - -```dart -import 'package:fory/fory.dart'; - -part 'person.fory.dart'; - -@ForyStruct() -class Person { - Person(); - - String name = ''; - - @ForyField(type: Int32Type()) - int age = 0; -} - -void main() { - final fory = Fory(); - PersonForyModule.register( - fory, - Person, - name: 'example.Person', - ); - - final person = Person() - ..name = 'Ada' - ..age = 36; - - final bytes = fory.serialize(person); - final roundTrip = fory.deserialize(bytes); - print(roundTrip.name); -} -``` - -`deserialize` returns the decoded value cast to `T`. If the payload describes a different type than `T`, it throws. - -## Null Values - -Serializing `null` is supported directly: - -```dart -final fory = Fory(); -final bytes = fory.serialize(null); -final value = fory.deserialize(bytes); -``` - -## Serialize Collections and Dynamic Payloads - -You can serialize collection values directly: - -```dart -final fory = Fory(); -final bytes = fory.serialize([ - 'hello', - 42, - true, -]); -final value = fory.deserialize>(bytes); -``` - -For heterogeneous collections, deserialize to `Object?`, `List`, or `Map`. - -## Reference Tracking - -By default, Fory does not track object identity — if the same object appears twice in a list, it is serialized twice. Enable reference tracking when your data contains shared references or circular structures. - -For a top-level collection: - -```dart -final fory = Fory(); -final shared = String.fromCharCodes('shared'.codeUnits); -final bytes = fory.serialize([shared, shared], trackRef: true); -final roundTrip = fory.deserialize>(bytes); -print(identical(roundTrip[0], roundTrip[1])); // true -``` - -For fields inside a generated struct, use `@ForyField(ref: true)` on that field instead. - -## Reusing a Buffer - -If you want to avoid allocating a new `Uint8List` on every call, use `serializeTo` and `deserializeFrom` with an explicit `Buffer`: - -```dart -final fory = Fory(); -final buffer = Buffer(); - -fory.serializeTo('Ada', buffer); -final value = fory.deserializeFrom(buffer); -``` - -This is an optimization. For most applications the default `serialize`/`deserialize` pair is fine. - -## Register Your Types Before Serializing - -Before you can serialize a custom class or enum, register it with `Fory`. The generated code makes this easy: - -```dart -PersonForyModule.register( - fory, - Person, - id: 100, -); -``` - -If you skip registration, deserialization fails with `Type ... is not registered`. See [Type Registration](type-registration.md) and [Code Generation](code-generation.md). - -## Related Topics - -- [Configuration](configuration.md) -- [Type Registration](type-registration.md) -- [Schema Metadata](schema-metadata.md) diff --git a/docs/object-serialization/dart/custom-serializers.md b/docs/object-serialization/dart/custom-serializers.md index 7284d148519..55ae78a846f 100644 --- a/docs/object-serialization/dart/custom-serializers.md +++ b/docs/object-serialization/dart/custom-serializers.md @@ -140,5 +140,5 @@ Skipping this step causes back-references to that object to resolve to `null`. - [Type Registration](type-registration.md) - [External-Type Serialization](external-types.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Troubleshooting](troubleshooting.md) diff --git a/docs/object-serialization/dart/index.md b/docs/object-serialization/dart/index.md index abe3b3eae13..e43822d7607 100644 --- a/docs/object-serialization/dart/index.md +++ b/docs/object-serialization/dart/index.md @@ -137,26 +137,28 @@ constructors, mixins, and field inclusion options. ## Documentation -| Topic | Description | -| ------------------------------------------------ | --------------------------------------------------------------- | -| [Configuration](configuration.md) | Fory options, compatible mode, and safety limits | -| [Basic Serialization](core-api.md) | `serialize`, `deserialize`, generated registration, root graphs | -| [Code Generation](code-generation.md) | `@ForyStruct`, build runner, and generated modules | -| [Struct Inheritance](inheritance.md) | Superclasses, mixins, private fields, and constructors | -| [External-Type Serialization](external-types.md) | Generated serializers for classes owned by another package | -| [Xlang Serialization](xlang.md) | Interoperability rules and field alignment | -| [Schema Metadata](schema-metadata.md) | `@ForyField`, field IDs, nullability, references, polymorphism | -| [Type Registration](type-registration.md) | ID-based vs name-based registration and registration rules | -| [Custom Serializers](custom-serializers.md) | Custom `Serializer` implementations and unions | -| [Supported Types](supported-types.md) | Built-in xlang values, wrappers, collections, and structs | -| [Schema Evolution](schema-evolution.md) | Compatible structs and evolving schemas | -| [Web Platform Support](web-platform-support.md) | Dart VM/AOT, Flutter, and web support, limits, and validation | -| [gRPC Support](../../grpc/dart.md) | Generated Fory-backed gRPC service companions | -| [Troubleshooting](troubleshooting.md) | Common errors, diagnostics, and validation steps | +| Topic | Description | +| ------------------------------------------------ | -------------------------------------------------------------- | +| [Configuration](configuration.md) | Fory options, compatible mode, and safety limits | +| [Basic Serialization](basic-serialization.md) | Default xlang APIs, registration, and interoperability | +| [Code Generation](code-generation.md) | `@ForyStruct`, build runner, and generated modules | +| [Struct Inheritance](inheritance.md) | Superclasses, mixins, private fields, and constructors | +| [External-Type Serialization](external-types.md) | Generated serializers for classes owned by another package | +| [Schema Metadata](schema-metadata.md) | `@ForyField`, field IDs, nullability, references, polymorphism | +| [Type Registration](type-registration.md) | ID-based vs name-based registration and registration rules | +| [Custom Serializers](custom-serializers.md) | Custom `Serializer` implementations and unions | +| [Supported Types](supported-types.md) | Built-in xlang values, wrappers, collections, and structs | +| [Schema Evolution](schema-evolution.md) | Compatible structs and evolving schemas | +| [Web Platform Support](web-platform-support.md) | Dart VM/AOT, Flutter, and web support, limits, and validation | +| [gRPC Support](../../grpc/dart.md) | Generated Fory-backed gRPC service companions | +| [Troubleshooting](troubleshooting.md) | Common errors, diagnostics, and validation steps | ## Related Resources - [Xlang serialization specification](../../specification/xlang_serialization_spec.md) - [Xlang implementation guide](../../specification/xlang_implementation_guide.md) -- [Xlang guide](../xlang/index.md) +- [Cross-language guide](../xlang.md) - [Dart implementation source directory](https://github.com/apache/fory/tree/main/dart) + +Before decoding bytes from outside the application trust boundary, read +[Dart Security](security.md). diff --git a/docs/object-serialization/dart/schema-evolution.md b/docs/object-serialization/dart/schema-evolution.md index 45f5c328e15..6f290413a15 100644 --- a/docs/object-serialization/dart/schema-evolution.md +++ b/docs/object-serialization/dart/schema-evolution.md @@ -132,4 +132,4 @@ runtime reference protocol or add a compatibility reader. - [Configuration](configuration.md) - [External-Type Serialization](external-types.md) - [Schema Metadata](schema-metadata.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/dart/schema-metadata.md b/docs/object-serialization/dart/schema-metadata.md index bd2edffe7dc..3073cf6bb83 100644 --- a/docs/object-serialization/dart/schema-metadata.md +++ b/docs/object-serialization/dart/schema-metadata.md @@ -169,4 +169,4 @@ When the same model is defined in multiple languages: - [Code Generation](code-generation.md) - [External-Type Serialization](external-types.md) - [Schema Evolution](schema-evolution.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/dart/security.md b/docs/object-serialization/dart/security.md new file mode 100644 index 00000000000..43671482911 --- /dev/null +++ b/docs/object-serialization/dart/security.md @@ -0,0 +1,56 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Dart reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected generated models before deserializing untrusted payloads. +- Use `checkStructVersion: true` with `compatible: false` for intentional same-schema payloads. +- Set `maxDepth` to reject unexpectedly deep payload shapes. +- Keep `maxGraphMemoryBytes` at the default for most inputs, or set an explicit positive byte gate + for known trusted collection/map/struct-heavy payloads. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer generated schemas and explicit field metadata over broad dynamic fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/dart/supported-types.md b/docs/object-serialization/dart/supported-types.md index 2136bf78dea..52c17b9b7ca 100644 --- a/docs/object-serialization/dart/supported-types.md +++ b/docs/object-serialization/dart/supported-types.md @@ -173,5 +173,5 @@ width is one of the most common cross-language bugs. - [Struct Inheritance](inheritance.md) - [Schema Metadata](schema-metadata.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Schema Evolution](schema-evolution.md) diff --git a/docs/object-serialization/dart/troubleshooting.md b/docs/object-serialization/dart/troubleshooting.md index 0a73e138747..038ee4c9d62 100644 --- a/docs/object-serialization/dart/troubleshooting.md +++ b/docs/object-serialization/dart/troubleshooting.md @@ -217,7 +217,7 @@ separate protobuf service endpoint for generic protobuf clients. ## Related Topics - [Struct Inheritance](inheritance.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Code Generation](code-generation.md) - [Custom Serializers](custom-serializers.md) - [Web Platform Support](web-platform-support.md) diff --git a/docs/object-serialization/dart/type-registration.md b/docs/object-serialization/dart/type-registration.md index ff57fdbd1c9..2364c81bd65 100644 --- a/docs/object-serialization/dart/type-registration.md +++ b/docs/object-serialization/dart/type-registration.md @@ -119,12 +119,12 @@ See [Custom Serializers](custom-serializers.md) for how to implement a serialize ## Xlang Requirements -The same numeric ID or name must be used in every peer that reads or writes the type. See [Xlang Serialization](xlang.md) for examples. +The same numeric ID or name must be used in every peer that reads or writes the type. See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for examples. ## Related Topics - [Struct Inheritance](inheritance.md) - [Code Generation](code-generation.md) - [External-Type Serialization](external-types.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Custom Serializers](custom-serializers.md) diff --git a/docs/object-serialization/deserialization-security-model.md b/docs/object-serialization/deserialization-security-model.md deleted file mode 100644 index 11ba637b177..00000000000 --- a/docs/object-serialization/deserialization-security-model.md +++ /dev/null @@ -1,712 +0,0 @@ ---- -title: Deserialization Security Model -sidebar_position: 99 -id: deserialization-security-model -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This document defines the security model for Apache Fory deserialization. It is -a public security reference for classifying deserialization behavior and -deciding where validation is required. It is not a vulnerability disclosure, -does not describe exploit techniques, and does not document implementation -history. - -The model is intentionally narrow. Fory should prevent resource and policy -failures caused by untrusted input, but it should not add hot-path validation -that only enforces byte-form strictness when doing so does not protect a Fory -security boundary. - -## Scope - -This model applies only to deserializing Fory binary object-serialization data -from untrusted or partially trusted sources. Fory JSON has a separate -[security guide](../json/security.md) because it uses different readers, -policies, codecs, and resource-accounting rules. - -It does not treat the semantic content of a successfully deserialized value as a -Fory security boundary. A sender can always construct protocol-valid data whose -value is chosen by that sender. Application authorization, object-level business -rules, and domain-specific validation remain application responsibilities. -The selected business invariant remains an application policy rather than a -Fory protocol security boundary. - -This model does not govern memory-format paths unless a runtime explicitly -exposes such a path through an untrusted deserialization API. - -## Trust Boundaries - -Fory deserialization should treat the encoded input as untrusted at API -boundaries that accept external bytes or streams. - -Fory security boundaries include: - -- Resource ownership, such as memory, CPU progress, stream buffering, file - handles, native allocations, callbacks, and retained read-side tables. -- Runtime safety, such as avoiding crashes, panics, undefined behavior, and - out-of-bounds reads or writes. -- Explicit Fory policy checks, such as type, function, method, class, or - registration policies that are intended to restrict what may be materialized. -- Cleanup boundaries, where state created during a failed read must be released - or reset before the next root operation. - -Fory security boundaries do not include: - -- The business meaning of a protocol-valid value. -- Which protocol-allowed byte form was used for a value. -- Whether a map, set, object, or metadata value uses one specific encoding - shape, unless rejecting other shapes is an explicit owner policy or protects - one of the boundaries above. - -## Type And Class Policy - -Type, class, function, method, registration, and deserialization policies are -security boundaries when they are intended to restrict what untrusted bytes may -materialize. - -For untrusted data, a bypass is security-relevant when encoded bytes can -materialize a type, function, method, class, or dynamic object that the active -Fory policy should reject. This includes bypasses of class or type -registration, allow-list checkers, strict-mode checks, or language-specific -deserialization policies. - -An application explicitly trusts a class when it registers that class or -registers a serializer for that class. Both operations are configuration-time -trust decisions under the class-registration policy. Explicitly selecting a -static root serializer or static root target at the deserialization call is -also an application authorization decision for that root path. Authorization -of that statically selected root does not depend on a separate registration -lookup; any registration needed to access registered identity or -registration-backed metadata remains access-driven. - -Explicitly declaring or selecting a static field codec is itself an application -authorization decision for that field; the codec does not need to be registered -separately for authorization. Registering an enclosing class or schema also -authorizes the statically declared field codecs and serializers that belong to -that registered owner. This applies equally to declared Array, Set, Map, Struct, -and other statically composed field paths. Those declared field paths do not -require independent registration merely because their bodies are decoded -without another type lookup. Likewise, an encoded declared-type marker does not -create a registration bypass when it can only invoke the codec already selected -by the authorized root or enclosing schema. - -These static authorization paths do not authorize an arbitrary alternative -chosen by encoded type metadata. A dynamic or polymorphic type selected by -input must still pass the active registration and deserialization-policy checks -for that type. A serializer that Fory merely discovers or generates, and that -is not reached through an explicitly selected static root or a registered -enclosing owner, is serialization mechanics only and does not by itself -authorize a dynamically selected class. - -Disabling registration or dynamic-type checks for trusted data is a caller -configuration choice. That choice only removes the arbitrary-type materialization -claim provided by that policy; it does not remove Fory's runtime-safety, -resource, cleanup, retained-state, or no-progress-loop requirements for -untrusted deserialization paths. - -Fory is not a sandbox for application-owned types. If a registered type or -serializer is allowed by the active policy, the application owns whether that -type's construction, hooks, setters, finalizers, or other logic is safe for the -application's trust boundary. - -When policy-approved construction or callable execution is allowed, resource -accounting should not claim to bound arbitrary code outside Fory's ownership. -Fory-owned accounting can cover only objects and storage that Fory itself -clearly creates or copies and that remain reachable from the materialized graph. -Temporary helper allocations and user-code internals remain outside that -accounting boundary. - -## Depth And Progress - -Deserialization paths that recurse through objects, metadata, containers, or -references should enforce the runtime's configured depth limit before crafted -nesting can exhaust the call stack or bypass cleanup. A malformed input that -exceeds the configured depth should fail the root operation instead of -continuing unbounded recursion. - -Loops that consume encoded data should guarantee byte progress, logical -progress, or a terminal error. Inputs that can keep a reader in a no-progress -loop are security-relevant even when they do not allocate memory. - -## Security Invariants - -Deserialization code must prevent the following outcomes for untrusted input: - -- Crash, panic, undefined behavior, or out-of-bounds memory access. -- OOM or disproportionate allocation compared with bytes that are already - supplied or proven readable. -- No-progress loops, including loops where neither logical progress nor byte - progress is guaranteed after malformed input. -- Stream-buffer growth to an attacker-declared size before the corresponding - bytes have been read or skipped exactly. -- Resource leaks, including native allocations, handles, callbacks, or - registered cleanup work that cannot run. -- Retained attacker-controlled state after failure when that state can affect a - later root operation or grow across operations. -- Successful bypass of an explicit Fory policy boundary. - -When a path cannot produce one of these outcomes, earlier rejection of malformed -bytes is normally a correctness or interoperability choice, not a security -requirement. - -## Robustness Scope Gate - -Before reporting or fixing a deserialization robustness finding, establish a -concrete consequence in the current implementation: - -- Crash, panic, undefined behavior, or out-of-bounds access. -- Disproportionate allocation, CPU work, or stream growth. -- A no-progress loop. -- Persistent state, reference-table, or cache pollution. -- Later-root corruption or a failed-root cleanup leak. -- A concrete type, registration, callable, or deserialization-policy violation. - -Protocol strictness alone is outside this gate. Do not change code merely -because a malformed or noncanonical flag, enum value, marker, length form, or -reserved value is accepted, rejected late, decoded differently, or produces a -less precise error. Such validation is actionable only when it prevents one of -the concrete consequences above or implements an explicit public contract. - -## Controlled Deserialization Errors - -When a decoder determines that input is invalid for the active owner path, the -root operation must return an error and run its normal failure cleanup. This is -an outcome requirement, not an error-taxonomy requirement. - -Unless a public API or specification explicitly promises otherwise, Fory does -not require a particular exception type, error code, message, detection layer, -input offset, or earliest possible detection point. An existing bounded -downstream buffer-underflow, type, reference, depth, or serializer error is a -valid rejection. A decoder does not need a new local check merely to replace -that controlled failure with a more specific or more uniform error. - -Tests for malformed input should prove that the root operation fails, cleanup -remains correct, and any relevant security invariant is preserved. They should -not pin an exact error type or message when doing so would require additional -successful-path validation that protects no security boundary. - -## Non-Security Semantics - -The following patterns are not vulnerabilities by default: - -- Protocol-allowed collection chunking, map chunking, and field ordering. -- Duplicate keys, set elements, or compatible fields that collapse according to - the target data structure or owning serializer semantics. -- Malformed ref, null, or type flags that eventually produce a read error. -- Malformed scalar bytes that are consumed linearly and eventually produce a - read error. -- Reading an encoded body before later shape validation when the operation - ultimately returns an error and does not create a security-invariant failure. -- Materializing an array whose component is an interface already allowed as a - class token. Allocating the reference array does not instantiate or execute - the interface, and every non-null element must still pass the active policy - for its concrete type. Treat this as security-relevant only if the array path - bypasses that concrete element check, invokes a policy-forbidden callback, or - violates a runtime-safety or resource invariant owned by Fory. - -Fory may still reject malformed forms for specification strictness or -interoperability. That validation should be added only when it is required by -the protocol owner, is effectively free on the relevant path, or protects a -security invariant listed above. Do not add protocol-layer validation solely to -reject scalar byte forms whose only effect is extra decode cost. - -### Value-bearing ref flags - -Some read paths intentionally share handling for multiple value-bearing flags. -For example, when both `NotNullValue` and `RefValue` mean that an encoded value -follows, a reader may merge their hot-path handling. This is not a malformed -flag bug by itself. Treat it as a bug only if the merged handling loses required -reference semantics, returns success across an explicit owner policy, or creates -a resource or runtime-safety failure. - -## Allocation And Byte Availability - -Fory should not make large allocations from attacker-declared lengths before -the required bytes are available or have been read exactly. - -For buffer-backed input: - -- Fixed-size binary values and primitive dense arrays should call the byte - owner's readability check for the required encoded byte size before allocating - the destination. For buffer-backed input this is normally a remaining-byte - comparison. -- Multi-byte element arrays should compute the required byte size with overflow - checks before allocation. -- Container readers that allocate backing storage or size-hint from a declared - logical element count should call the byte owner's readability check for that - count before that backing allocation or capacity reservation. This is not a - full container-body validation; it is the allocation proof that the sender has - supplied at least proportional input bytes before the reader preallocates from - the count. Estimated memory-budget accounting may reserve budget before this - byte check because it does not allocate backing storage. -- Readers should not add count-based readability checks merely because a loop - will read that many values when the destination grows incrementally and each - item read still uses the normal byte-owner checks. The security boundary is - direct preallocation from an untrusted count, not the existence of a counted - loop. - -For stream-backed input: - -- Reading or skipping a large byte region is the proof that the bytes exist. -- Byte-counted variable-length result allocation should use the byte owner's - readability check before allocation. Skip paths may use bounded skip without - materializing the skipped value. -- A stream-backed buffer may hold the full requested encoded body after that - body has been read from the stream. It must not reserve the attacker-declared - length before input bytes prove that length exists. -- Stream-backed fill buffers should grow geometrically from the current proven - buffer size, such as by doubling current capacity. Growth must not be capped - to the immediate fill target: for small fills the target is barely above the - current capacity, so cap-to-target degenerates into constant-size growth - steps that copy the whole buffer on every small read and make stream - deserialization O(n^2) overall. A byte owner may use an owner-local - availability signal as a one-shot growth hint when the stream implementation - itself is caller-owned trusted code, and may then reserve the full immediate - target at once while keeping at least the geometric growth step; if that hint - is absent or insufficient, the reader must fall back to bounded geometric - growth from already buffered bytes. Serializers should not add their own - availability branches. -- A truncated stream should fail before allocating the final deserialized value - and should allocate only for bytes actually read plus bounded spare capacity. - -The byte owner should stay byte-oriented. Buffer, reader, or read-context APIs -may expose byte read and byte skip operations, but string decoding, decimal -parsing, primitive-array encoding, compression modes, and collection capacity -policy belong to the owning serializers. - -## Collection And Map Capacity - -Large valid collection inputs are allowed. If the input contains many encoded -elements, proportional deserialization is expected. - -The security requirement is to avoid disproportionate preallocation from a -declared logical count before enough input bytes justify that capacity. When -the repeated element or entry body is proven to consume at least one byte, a -reader that allocates or reserves from the declared count should call -`checkReadableBytes(logicalCount)` or the runtime equivalent before that -allocation. When the body may consume no bytes, the readable-byte requirement -may exclude the root operation's remaining unbacked-container allowance. The -reader must still account for actual input progress while reading the -container. The byte check does not decode the whole container, validate element -semantics, or replace chunk validation. Readers that do not preallocate from -the logical count may still grow proportionally as elements are actually read. - -Map or collection chunk validation is security-relevant only when missing -validation can cause a no-progress loop, unbounded resource growth, retained -state, or success across a Fory policy boundary. Protocol-allowed chunk -segmentation is normal input and is not a security issue by itself. - -## Unbacked Container Work Budget - -Runtimes enforce a root-scoped limit on count-driven collection elements and -map entries whose repeated read bodies are not backed by input progress. The -public option is named `maxUnbackedContainerItems` or the language-equivalent -spelling. Its default is `8192`; values must be non-negative, and zero is a -strict limit rather than an unlimited sentinel. - -The allowance is shared by all nested collections, maps, and compatible field -skip operations in one root read. Collection readers account for completed -items every 1024 elements and at the final partial window. Map readers account -at existing protocol chunk boundaries. Bytes actually consumed by the repeated -item bodies offset the completed item count in the same window. The budget does -not add framing, reject values on write, change reference publication, or -replace graph-memory accounting. - -Readers whose exact repeated operation is known to consume at least one byte -retain their direct loop and proportional readable-byte check. Generated and -compiled serializers should remove budget access and periodic branches from -those proven-positive paths. - -## Graph Memory Budget - -Runtimes should enforce a per-operation approximate gate for estimated memory created by one -materialized graph. This is cumulative accounting for graph owners created by one top-level -deserialization operation; it is not exact heap measurement and it is not a raw element-slot limit. -Actual process memory can be higher than the configured gate. - -The public configuration is `maxGraphMemoryBytes`. The default is a fixed `128 MiB` for all input -forms; positive user configuration overrides the default. Explicit non-positive configuration is -invalid and should be rejected when the runtime is created. The budget is not derived from input -size, and stream budgeting should not depend on dynamic bytes-read accounting. - -Graph budget accounting should: - -- be initialized in top-level read state, with cleanup owned by the top-level deserialization - `finally`; -- account only for Fory-created objects or storage that are retained by the - returned value graph; temporary helper objects used only during construction - are outside the graph budget; -- not claim to budget arbitrary constructor, callable, descriptor, finalizer, - or state-restoration internals that run after an explicit policy allows that - code; -- keep read context/read state limited to raw byte reservation; counted arithmetic and collection, - map, array, struct, and object storage formulas belong in the concrete serializer or generated - serializer owner; -- reject arithmetic overflow before comparing budget or allocating; -- estimate lower-bound shallow owner storage: reference-backed or heap-materialized collections, - maps, sets, and reference arrays reserve nonzero shallow self cost plus - backing/reference/inline storage, and reference-backed or heap-materialized struct, record, - POJO, tuple/product, compatible, generated, and dynamic object owners reserve a nonzero shallow - self cost plus shallow field storage; -- use a 4-byte reference slot when the actual reference slot size is not cheap or reliable to query, - and use primitive/value field widths for inline storage; -- preserve existing byte-availability checks before backing allocation or capacity reservation; -- skip enum/union as separate owners and skip dedicated string, binary, primitive scalar, primitive - array, and primitive dense-array leaf owners unless a runtime-specific owner section explicitly - includes them. - -Skipped leaf owners must still be gated by remaining input bytes. If the unread input does not -contain enough bytes for a string, binary value, primitive scalar, primitive array, or primitive -dense array, the runtime must not read or create that leaf value. - -Each runtime must inspect the concrete owner path before choosing formulas. Reserve self storage -exactly once at the owner that stores, boxes, or allocates the value. Deserialization facades may -reset the budget for each operation, but must not pre-reserve the top-level result type, self bytes, -or value storage. -Reference-backed paths reserve parent owner self cost plus reference storage, while each referenced -heap owner reserves its own shallow self cost when materialized. Inline/value paths reserve inline -element, field, or boxed storage in the holder/allocation owner; top-level value serializers and -generated struct/product read paths must not charge their own self storage. -For inline/value collection or map runtimes, the top-level value container itself is not charged by -the deserialization facade or by the container serializer only because it is the returned value. -Nested value containers are charged as inline slots of the parent holder or as backing storage -elements of the outer collection that actually owns those slots. Pointer, box, smart-pointer, or -type-erased materialization paths reserve the shallow storage for the heap value they allocate. -Parents must not recursively include child object, collection, map, string, binary, or primitive -dense-array contents; the child owner reserves its own shallow memory when it is materialized. - -### Java Fory Core - -Java Fory core primitive-array serializers reserve the portable array header plus the logical -length multiplied by the primitive storage width. Primitive-list serializers reserve the returned -list's shallow owner, the backing-array header, and the same primitive storage. These known-length -paths reserve once after their existing proportional readable-byte check and before allocation; -they do not use incremental batches. Compressed inputs use the decompressed logical length, while -temporary compressed arrays remain construction scratch outside the retained graph budget. -Float16 and BFloat16 dense-array carriers also include their wrapper's shallow owner. When a boxed -list conversion first decodes a primitive array, the array's reservation remains as credit toward -the final list estimate, and the conversion reserves only a positive remaining difference. - -### Generated Structural Targets - -Wire members and physical storage are separate inputs. Properties, accessors, interfaces, and -logical schema aliases are not physical fields and must not be charged as storage. A field that is -both serialized and stored is counted once. A storage-only declaration contributes its field width -but must not enter wire metadata or generated reads and writes. - -For C# ordinary classes, each directly annotated class owns the physical instance fields declared -by that class. An inheritable class provider publishes the cumulative parent-provider value plus -those direct fields. A sealed concrete serializer uses the same cumulative expression privately. -A concrete descendant uses the immediate accessible provider value and its own direct fields; it -must not enumerate referenced private metadata or reconstruct parent storage. The concrete object -serializer reserves one shallow object owner plus this cumulative field storage. - -A C# external class declaration owns the exact third-party physical fields it lists. An exact -field mapping contributes storage. A visible property mapping does not, so its backing field must -be listed separately. An ignored mapping must identify one exact class field and is storage-only. -External struct declarations support visible member mappings only. -Discoverable unmapped public instance fields may be added once. A `BaseOnly` declaration can own -the complete target and target-ancestor prefix used by an ordinary child. It must list every -non-public physical field in that prefix; the generator does not scan the referenced assembly for -private layout. - -Exact external private identities are version-pinned package ABI assertions. Runtime wire access -uses exact accessors and must not fall back to reflection, layout probing, or a different member. -Storage-only private declarations have no runtime accessor, so the application must validate them -against the pinned package version. - -Dart generators may additionally include public instance fields visible on the target at -generation time. Swift macros cannot inspect another type's stored layout and therefore use only -the external declaration. In every runtime, these formulas are resolved during generation and -must not add reflection, layout probing, allocation, or field enumeration to deserialization hot -paths. The normal owner rules still apply: a reference target reserves its shallow owner and field -storage, while an inline value target is charged by the holder that owns its storage. - -### Runtime-Specific Owner Notes - -#### C++ - -C++ plain structs, products, and standard-library containers are value storage unless a pointer, -smart pointer, or type-erased owner allocates them on the heap. Top-level deserialization initializes -the remaining graph budget but does not reserve `sizeof(T)` for the returned value. Plain value -serializers must not reserve their own `sizeof(T)` only because they are reading a value. - -Generic collection and map serializers reserve the lower-bound element, key, and value storage -owned by the container path. Nested value container headers are charged when they are inline slots -of a parent object or elements in an outer container backing store. Smart-pointer and type-erased -materialization paths reserve the shallow storage for the heap value they allocate before publishing -or returning it. Generic C++ paths must not invent standard-library header, node, bucket, allocator, -or debug-layout overheads. - -#### Rust - -Rust structs, tuples, enums, and collection values are inline value storage unless a `Box`, `Rc`, -`Arc`, or type-erased owner allocates them. Top-level and derived value read paths initialize or -consume the budget but do not reserve `size_of::()` for the value being read. `Vec`, `HashMap`, -`BTreeMap`, and similar serializers reserve backing or entry value storage that they allocate from -counts; nested value container headers are charged as parent inline fields or outer backing elements. - -Boxed, reference-counted, and type-erased materialization paths reserve `size_of::()` for the heap -payload they create. Compile-time `size_of::()` formulas are acceptable in those allocation -owners, but value serializers should not add a parallel self-reserve for the same `T`. - -Before count-derived allocation, Rust owners whose exact repeated operation is proven to consume at -least one byte retain the full readable-byte gate. Uncertain owners require readable bytes only for -the portion of the count not covered by the remaining unbacked-item allowance. Apply the selected -gate exactly once at the allocation owner; do not repeat it after reading shared metadata. Writers -continue to encode legal compact or empty bodies and do not enforce this reader-side allowance. - -Fixed arrays do not allocate from their validated wire count and omit the allocation gate. `Vec`, -`VecDeque`, and `BinaryHeap` also omit it for zero-sized elements because they create no -count-derived backing allocation in that case. Node, bucket, and entry owners retain the gate where -the declared count drives allocation. Implementations must not substitute guessed allocation costs, -padding bytes, a global compact-body bypass, or a second collection or map codec. - -#### Swift - -Swift structs, enums, tuples, and collection values are value storage. Top-level value reads and -nested value serializers should not reserve their own self storage. The holder that owns the value, -such as a struct field, array backing store, dictionary entry storage, or boxed/dynamic -materialization path, owns the corresponding graph-budget reservation. - -Array, dictionary, and set serializers may reserve lower-bound backing storage using stable Swift -type-size information, such as `MemoryLayout.stride`, when they allocate or reserve that storage. -Class, existential, or boxed materialization paths reserve owner storage when Fory creates the -retained object or box. Runtime object-layout probing should not be added to hot read paths. - -#### Go - -Go structs and slice or map headers are value storage unless a pointer, interface materialization, or -other heap owner allocates them. Top-level deserialization and struct value serializers should not -reserve the returned struct or a nested inline struct by themselves. Pointer serializers reserve the -concrete struct storage when they allocate a retained `*T`. - -Slice, array, map, and set serializers reserve the backing or entry storage they allocate from -declared counts. Element and entry widths should come from stable type information captured by the -serializer or resolver when possible; read loops should not recompute reflective size information -when the owner already knows the concrete type. Interface or dynamic paths reserve only storage that -Fory clearly materializes and retains. - -#### C\# - -C# combines reference owners and inline value types. Classes, arrays, lists, dictionaries, hash sets, -and other heap containers reserve a nonzero shallow owner cost plus direct backing, reference-slot, -or inline element storage. A dictionary is a reference-type container even when its key or value type -is a struct, so the dictionary owner is still charged separately from its entry storage. - -Value structs do not reserve their own self storage when read inline; the holder that stores the -struct, such as an object field, array element, list backing store, dictionary entry, box, or dynamic -materialization path, owns that reservation. Boxing, `object`, and dynamic materialization paths -reserve a boxed owner when Fory creates the retained box. Owner constants should be real portable -lower bounds for the relevant C# object or container shape, not placeholder markers. - -Runtimes should not guess object headers, array headers, allocator headers, debug-mode fields, hash -buckets, tree links, hash-chain links, node headers, map-entry objects, spare blocks, or runtime -table layouts unless the owner path has a cheap, stable, explicit lower-bound storage signal and -documents the formula. Owner constants should be real lower bounds for the owner shape, not -placeholder markers. - -## Skip Semantics - -Skipping unknown or incompatible data is classified by concrete impact, not by -whether the runtime materializes a temporary value. - -Directly consuming encoded contents is useful when it is simple and owned by the -current runtime path. It is not a security requirement for complex fields such -as lists, sets, and maps. A runtime may materialize a value and discard it when -that preserves the existing serializer ownership model. - -For extension, dynamic, or user-owned types, the owning runtime may not always -have enough information to skip without invoking a registered serializer. In -that case, classify the behavior by concrete impact: - -- Resource leak, retained state, no-progress loop, or policy bypass is - security-relevant. -- Bounded materialization followed by an error or discard is allowed unless it - creates meaningful memory or CPU pressure. -- Pure strictness about whether a skipped value used one specific encoding shape - is not a security issue. - -## Metadata And Type Resolution - -Metadata parsing is security-sensitive when it affects retained read-side state, -type dispatch, or policy decisions. - -Metadata readers should: - -- Avoid unbounded recursion in nested metadata structures. -- Avoid unbounded table growth from attacker-controlled metadata streams. -- Validate metadata bodies before using them to bypass or replace existing - policy decisions. -- For Java metadata paths, keep name-level checks such as `TypeChecker` and the - disallowed-class list before `Class.forName` by routing remote class-name - loading through the existing `TypeResolver.loadClass` owner. Do not bypass - that owner with direct class loading from TypeDef or TypeMeta names. A rejected - input name must not cause class loading. Preserve registration, dynamic-loading, - and unknown-type semantics while moving this decision before loading. Checks - that require a materialized `Class` remain after loading; do not replace - them with string-only approximations. -- Pass a complete input array descriptor to `TypeChecker`. Input may derive up - to six array dimensions from an accepted component class. Higher-dimensional - arrays require an exact trusted full-array registration or checked name-cache - entry so input cannot make the JVM derive an unbounded family of array classes. -- Reset or release metadata state at the correct root-operation boundary. - -A class-resolution cache reachable from untrusted deserialization may publish -an entry only from explicit trusted configuration or after the active class -policy has accepted the resolved class. A cache hit therefore represents an -already trusted and validated `Class` and should use that cached class -without repeating class loading or name-level `TypeChecker` work. Only a cache -miss performs those name-level checks and publishes the accepted result. -Checks that require the materialized `Class` remain owned by their existing -caller. A cache entry that stores a data-only unknown-class placeholder may -return that same placeholder on an exact hit, but must not authorize loading the -original missed wire name. -Exact registered-name-table hits are trusted for both ID and name registrations, -and exact checked name-cache hits are trusted. After both exact lookups miss, a -reader must not infer another accepted name from inverse registration, -class-keyed state, or `Class.getName()`. A custom-name registration does not by -itself publish the Java class name as an additional alias; ID registration does -publish the Java class name. - -Remote metadata that can create persistent read state must be bounded before -that state is retained. The check is resource control only: it must not change -wire compatibility, type registration, dynamic class loading, unknown-type -handling, deserialization policy, or schema-evolution semantics. Failed or -incompatible metadata must not consume schema-version limits, and metadata -cache hits or generated field readers must not add validation, hashing, -allocation, or policy work for these limits. The concrete sequence for metadata -parsing, cache publishing, exact-local matching, and counting belongs to the -[xlang implementation guide](../specification/xlang_implementation_guide.md). - -The checked metadata cache is the only owner of whether a received TypeDef or -TypeMeta header has already been validated. A metadata cache hit means the -header was previously parsed, body/hash-validated, policy-checked, and -published by the owning cache, so the reader must skip the remaining metadata -body and use the cached metadata without repeating body validation, hash -validation, limit checks, exact-local checks, or policy work. A metadata cache -miss is the only path that parses the metadata body, validates its hash and -shape, enforces metadata limits, performs exact-local byte comparison, and -publishes to the cache. Do not add separate nullable flags, sentinel headers, -per-TypeInfo acceptance markers, or parallel state to represent this decision. - -Only metadata that is actually carried as a TypeDef or TypeMeta body is subject -to metadata body and schema-version limits. Compatible named enum, ext, and -union metadata normally has one version, but still counts against remote -metadata total limits when it is sent as shared metadata. Pure id-based enum, -ext, and typed-union values use type id plus user type id and must not be moved -onto this metadata body path. - -Remote metadata bodies and struct field lists must also be bounded on the cold -metadata parse path. `maxTypeMetaBytes` limits the encoded metadata body bytes -for one received TypeDef or TypeMeta body, excluding the 8-byte header and any -extended-size varint. `maxTypeFields` limits the number of fields declared by -one received struct metadata body. For Java native TypeDef class layers, the -field limit applies to the total field count across the class layers in that -one TypeDef. These limits are checked before copying, decompressing, reserving, -or allocating from attacker-declared metadata sizes or field counts. - -The default limits are `maxTypeFields = 512` and `maxTypeMetaBytes = 4096`. -Runtimes should report limit failures as possible malicious data and tell users -to increase the exact option only when the data is not malicious. These limits -must not introduce validation on metadata cache-hit, generated serializer, or -already-resolved type-id hot paths. - -Metadata byte-form strictness alone is not a security requirement. Rejecting a -metadata shape is useful only when the owner wants that strictness or when the -shape changes type identity, retained state, resource use, or policy behavior. - -## Reference Tracking - -Reference tracking is part of the wire protocol and is performance-sensitive. -Readers may use sentinel values and shared value-bearing branches to keep hot -paths compact. - -Reference tracking validation is security-relevant when malformed input can: - -- Access an out-of-range reference without reporting an error. -- Leave retained reference state after a failed root operation. -- Register unbounded callbacks or resolver state before the referenced value is - available. -- Cause a no-progress loop or crash. - -Reference tracking validation is not required merely because a malformed flag is -not rejected at the earliest possible byte. Lazy rejection is acceptable when -the root operation still returns an error and no security invariant is violated. -The downstream error does not need to be a dedicated reference-protocol error. - -## Error Propagation And Cleanup - -Fory runtimes may intentionally use lazy error propagation. After a read records -an error, later read steps may continue until the outer operation observes and -returns the error. - -This is acceptable when the continued work cannot: - -- Crash or panic. -- Allocate or retain attacker-controlled state. -- Leak resources. -- Bypass required cleanup. -- Return success across an explicit validation or policy boundary. - -Nested `try`/`finally` or equivalent cleanup should be added only when the -outer root-operation cleanup cannot cover the state or resource owned by the -nested path. - -## Performance Requirements - -Security validation must preserve Fory hot-path performance. Do not add -validation solely for strictness when it introduces: - -- Per-element object allocation. -- Dynamic dispatch or callbacks in hot loops. -- Wrapper objects or result carriers on success paths. -- Extra copying for buffer-backed string, binary, or primitive-array reads. -- Branches that do not protect a security invariant. -- Helper calls or generated-code expansion whose only purpose is to normalize - an eventual error's type, message, location, or timing. - -Prefer owner-local checks that can be inlined and that already use information -available in the current serializer. Do not move serializer-owned semantics into -generic read-context helpers. - -## Classification Guide - -Use the following questions when reviewing deserialization behavior: - -1. Can this input crash, panic, or access memory out of bounds? -2. Can a small or unproven input length cause disproportionate allocation? -3. Can a stream-backed reader grow a buffer before exact read or skip proves the - bytes exist? -4. Can a loop continue without byte progress or logical progress? -5. Can the path retain attacker-controlled state after the root operation fails? -6. Can the path leak resources or skip required cleanup? -7. Can the path return success across an explicit Fory policy boundary? -8. Is the proposed validation effectively free in the relevant hot path? - -If the answer to the first seven questions is no, the issue is normally not a -security finding. If the validation is not effectively free, avoid adding it -unless the protocol owner explicitly requires it. - -## Documentation Boundaries - -Security model documents must not include exploit samples, CVE narratives, -line-level vulnerability candidates, branch history, migration timelines, or -cleanup plans. Keep those details in private reports, issues, or pull requests -as appropriate. - -Public security documentation should describe durable boundaries and invariants, -not the history of how the implementation reached them. diff --git a/docs/object-serialization/go/xlang.md b/docs/object-serialization/go/basic-serialization.md similarity index 54% rename from docs/object-serialization/go/xlang.md rename to docs/object-serialization/go/basic-serialization.md index d2f94726968..e5b19b3fc57 100644 --- a/docs/object-serialization/go/xlang.md +++ b/docs/object-serialization/go/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,11 +19,397 @@ license: | limitations under the License. --- +This guide covers the core serialization APIs in the default xlang mode for Fory Go. + +## Creating a Fory Instance + +Create a Fory instance and register your types before serialization: + +```go +import "github.com/apache/fory/go/fory" + +f := fory.New(fory.WithXlang(true)) + +// Register struct with a type ID +f.RegisterStruct(User{}, 1) +f.RegisterStruct(Order{}, 2) + +// Or register with a name (more flexible, less prone to ID conflicts, but higher serialization cost) +f.RegisterStructByName(User{}, "example.User") + +// Register enum types +f.RegisterEnum(Color(0), 3) +``` + +`fory.New()` uses xlang mode with compatible schema evolution. The example sets +`fory.WithXlang(true)` explicitly so the mode choice is visible. For Go-only +payloads that need native mode, configure `fory.WithXlang(false)` explicitly in +the native-mode examples. + +**Important**: The Fory instance should be reused across serialization calls. Creating a new instance involves allocating internal buffers, type caches, and resolvers, which is expensive. The default Fory instance is not thread-safe; for concurrent usage, use the thread-safe wrapper (see [Thread Safety](thread-safety.md)). + +See [Type Registration](type-registration.md) for more details. + +## Core API + +### Serialize and Deserialize + +The primary API for serialization: + +```go +// Serialize any value +data, err := f.Serialize(value) +if err != nil { + // Handle error +} + +// Deserialize into target +var result MyType +err = f.Deserialize(data, &result) +if err != nil { + // Handle error +} +``` + +### Marshal and Unmarshal + +Aliases for `Serialize` and `Deserialize` (familiar to Go developers): + +```go +data, err := f.Marshal(value) +err = f.Unmarshal(data, &result) +``` + +## Serializing Primitives + +```go +// Integers +data, _ := f.Serialize(int64(42)) +var i int64 +f.Deserialize(data, &i) // i = 42 + +// Floats +data, _ = f.Serialize(float64(3.14)) +var fl float64 +f.Deserialize(data, &fl) // fl = 3.14 + +// Strings +data, _ = f.Serialize("hello") +var s string +f.Deserialize(data, &s) // s = "hello" + +// Booleans +data, _ = f.Serialize(true) +var b bool +f.Deserialize(data, &b) // b = true +``` + +## Serializing Collections + +### Slices + +```go +// String slice +strs := []string{"a", "b", "c"} +data, _ := f.Serialize(strs) + +var result []string +f.Deserialize(data, &result) +// result = ["a", "b", "c"] + +// Integer slice +nums := []int64{1, 2, 3} +data, _ = f.Serialize(nums) + +var intResult []int64 +f.Deserialize(data, &intResult) +// intResult = [1, 2, 3] +``` + +### Maps + +```go +// String to string map +m := map[string]string{"key": "value"} +data, _ := f.Serialize(m) + +var result map[string]string +f.Deserialize(data, &result) +// result = {"key": "value"} + +// String to int map +m2 := map[string]int64{"count": 42} +data, _ = f.Serialize(m2) + +var result2 map[string]int64 +f.Deserialize(data, &result2) +// result2 = {"count": 42} +``` + +## Serializing Structs + +### Basic Struct Serialization + +Only **exported fields** (starting with uppercase) are serialized: + +```go +type User struct { + ID int64 // Serialized + Name string // Serialized + password string // NOT serialized (unexported) +} + +f.RegisterStruct(User{}, 1) + +user := &User{ID: 1, Name: "Alice", password: "secret"} +data, _ := f.Serialize(user) + +var result User +f.Deserialize(data, &result) +// result.ID = 1, result.Name = "Alice", result.password = "" +``` + +### Nested Structs + +```go +type Address struct { + City string + Country string +} + +type Person struct { + Name string + Address Address +} + +f.RegisterStruct(Address{}, 1) +f.RegisterStruct(Person{}, 2) + +person := &Person{ + Name: "Alice", + Address: Address{City: "NYC", Country: "USA"}, +} + +data, _ := f.Serialize(person) + +var result Person +f.Deserialize(data, &result) +// result.Address.City = "NYC" +``` + +### Pointer Fields + +```go +type Node struct { + Value int32 + Child *Node +} + +// Use WithTrackRef for pointer fields +f := fory.New(fory.WithXlang(true), fory.WithTrackRef(true)) +f.RegisterStruct(Node{}, 1) + +root := &Node{ + Value: 1, + Child: &Node{Value: 2, Child: nil}, +} + +data, _ := f.Serialize(root) + +var result Node +f.Deserialize(data, &result) +// result.Child.Value = 2 +``` + +## Streaming API + +For scenarios where you want to control the buffer: + +### SerializeTo + +Serialize to an existing buffer: + +```go +buf := fory.NewByteBuffer(nil) + +// Serialize multiple values to same buffer +f.SerializeTo(buf, value1) +f.SerializeTo(buf, value2) + +// Get all serialized data +data := buf.GetByteSlice(0, buf.WriterIndex()) +``` + +### DeserializeFrom + +Deserialize from an existing buffer: + +```go +buf := fory.NewByteBuffer(data) + +var result1, result2 MyType +f.DeserializeFrom(buf, &result1) +f.DeserializeFrom(buf, &result2) +``` + +## Generic API (Type-Safe) + +Fory Go provides generic functions for type-safe serialization: + +```go +import "github.com/apache/fory/go/fory" + +type User struct { + ID int64 + Name string +} + +// Type-safe serialization +user := &User{ID: 1, Name: "Alice"} +data, err := fory.Serialize(f, user) + +// Type-safe deserialization +var result User +err = fory.Deserialize(f, data, &result) +``` + +The generic API: + +- Infers type at compile time +- Provides better type safety +- May offer performance benefits + +## Error Handling + +Always check errors from serialization operations: + +```go +data, err := f.Serialize(value) +if err != nil { + switch e := err.(type) { + case fory.Error: + fmt.Printf("Fory error: %s (kind: %d)\n", e.Error(), e.Kind()) + default: + fmt.Printf("Unknown error: %v\n", err) + } + return +} + +err = f.Deserialize(data, &result) +if err != nil { + // Handle deserialization error +} +``` + +Common error kinds: + +- `ErrKindBufferOutOfBound`: Read/write beyond buffer bounds +- `ErrKindTypeMismatch`: Type ID mismatch during deserialization +- `ErrKindUnknownType`: Unknown type encountered +- `ErrKindMaxDepthExceeded`: Recursion depth limit exceeded +- `ErrKindHashMismatch`: Struct hash mismatch (schema changed) + +See [Troubleshooting](troubleshooting.md) for error resolution. + +## Nil Handling + +### Nil Pointers + +```go +var ptr *User = nil +data, _ := f.Serialize(ptr) + +var result *User +f.Deserialize(data, &result) +// result = nil +``` + +### Empty Collections + +```go +// Nil slice +var slice []string = nil +data, _ := f.Serialize(slice) + +var result []string +f.Deserialize(data, &result) +// result = nil + +// Empty slice (different from nil) +empty := []string{} +data, _ = f.Serialize(empty) + +f.Deserialize(data, &result) +// result = [] (empty, not nil) +``` + +## Complete Example + +```go +package main + +import ( + "fmt" + "github.com/apache/fory/go/fory" +) + +type Order struct { + ID int64 + Customer string + Items []Item + Total float64 +} + +type Item struct { + Name string + Quantity int32 + Price float64 +} + +func main() { + f := fory.New(fory.WithXlang(true)) + f.RegisterStruct(Order{}, 1) + f.RegisterStruct(Item{}, 2) + + order := &Order{ + ID: 12345, + Customer: "Alice", + Items: []Item{ + {Name: "Widget", Quantity: 2, Price: 9.99}, + {Name: "Gadget", Quantity: 1, Price: 24.99}, + }, + Total: 44.97, + } + + // Serialize + data, err := f.Serialize(order) + if err != nil { + panic(err) + } + fmt.Printf("Serialized %d bytes\n", len(data)) + + // Deserialize + var result Order + if err := f.Deserialize(data, &result); err != nil { + panic(err) + } + + fmt.Printf("Order ID: %d\n", result.ID) + fmt.Printf("Customer: %s\n", result.Customer) + fmt.Printf("Items: %d\n", len(result.Items)) + fmt.Printf("Total: %.2f\n", result.Total) +} +``` + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Fory Go enables seamless data exchange with Java, Python, C++, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. This guide covers xlang compatibility and type mapping. -## Create an Xlang Fory Instance +### Xlang Configuration Go defaults to xlang mode with compatible schema evolution. Set the mode explicitly in xlang examples: @@ -31,11 +417,11 @@ Go defaults to xlang mode with compatible schema evolution. Set the mode explici f := fory.New(fory.WithXlang(true)) ``` -## Type Registration for Xlang +### Type Registration for Xlang Use consistent type IDs across all languages: -### Go +#### Go ```go type User struct { @@ -48,7 +434,7 @@ f.RegisterStruct(User{}, 1) data, _ := f.Serialize(&User{ID: 1, Name: "Alice"}) ``` -### Java +#### Java ```java public class User { @@ -60,7 +446,7 @@ fory.register(User.class, 1); User user = fory.deserialize(data, User.class); ``` -### Python +#### Python ```python from dataclasses import dataclass @@ -76,11 +462,11 @@ fory.register(User, type_id=1) user = fory.deserialize(data) ``` -## Type Mapping +### Type Mapping See [Type Mapping Specification](../../specification/xlang_type_mapping.md) for detailed type mappings across all languages. -## Field Ordering +### Field Ordering Cross-language serialization requires consistent field ordering. Fory sorts fields by their snake_case names alphabetically. @@ -106,9 +492,9 @@ type Example struct { } ``` -## Examples +### Examples -### Go to Java +#### Go to Java **Go (Serializer)**: @@ -149,7 +535,7 @@ fory.register(Order.class, 1); Order order = fory.deserialize(data, Order.class); ``` -### Python to Go +#### Python to Go **Python (Serializer)**: @@ -187,11 +573,11 @@ f.Deserialize(data, &msg) fmt.Println(msg.Content) // "Hello from Python" ``` -### Nested Structures +#### Nested Structures Cross-language nested structures require all types to be registered: -## Lists and Dense Arrays +### Lists and Dense Arrays Go slices are ordinary `list` carriers unless a field tag explicitly requests the dense `array` schema. Use `array` only for one-dimensional bool or @@ -251,9 +637,9 @@ fory.register(Address.class, 1); fory.register(Company.class, 2); ``` -## Common Issues +### Common Issues -### Field Name Mismatch +#### Field Name Mismatch Go uses PascalCase, other languages may use camelCase or snake_case. Fields are matched by their snake_case conversion: @@ -269,7 +655,7 @@ public class User { } ``` -### Type Interpretation +#### Type Interpretation Go unsigned types map to Java signed types with the same bit pattern: @@ -279,7 +665,7 @@ var value uint64 = 18446744073709551615 // Max uint64 Java's `long` holds the same bits but interprets as -1. Use `Long.toUnsignedString()` in Java if unsigned interpretation is needed. -### Nil vs Null +#### Nil vs Null Go nil slices/maps serialize differently based on configuration: @@ -290,7 +676,7 @@ var slice []string = nil Ensure other languages handle null appropriately. -## Best Practices +### Interoperability Best Practices 1. **Use consistent type IDs**: Same numeric ID for the same type across all languages 2. **Register all types**: Including nested struct types @@ -298,7 +684,7 @@ Ensure other languages handle null appropriately. 4. **Test cross-language**: Run integration tests early and often 5. **Handle type differences**: Be aware of signed/unsigned interpretation differences -## Related Topics +### Related Guides - [Type Registration](type-registration.md) - [Supported Types](supported-types.md) @@ -306,7 +692,7 @@ Ensure other languages handle null appropriately. - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Type Mapping Specification](../../specification/xlang_type_mapping.md) -## Built-in values +### Built-in values ```go package main @@ -344,7 +730,7 @@ func main() { } ``` -## Custom values +### Custom values ```go package main @@ -407,7 +793,7 @@ func main() { } ``` -## Shared and circular references +### Shared and circular references ```go package main @@ -440,3 +826,10 @@ func main() { fmt.Println(newValue) } ``` + +## Related Topics + +- [Configuration](configuration.md) +- [Type Registration](type-registration.md) +- [Supported Types](supported-types.md) +- [References](references.md) diff --git a/docs/object-serialization/go/configuration.md b/docs/object-serialization/go/configuration.md index 1e94af89428..e3de7416ed5 100644 --- a/docs/object-serialization/go/configuration.md +++ b/docs/object-serialization/go/configuration.md @@ -417,17 +417,11 @@ for req := range requests { ## Security -Security-related configuration: - -- Register only the expected structs before deserializing untrusted data. -- Use `WithMaxDepth(...)` to reject unexpectedly deep payloads. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer concrete struct fields over broad `any` or interface-typed fields for untrusted input. +See [Go Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [References](references.md) - [Schema Evolution](schema-evolution.md) - [Thread Safety](thread-safety.md) diff --git a/docs/object-serialization/go/core-api.md b/docs/object-serialization/go/core-api.md deleted file mode 100644 index 0a92aa70cf0..00000000000 --- a/docs/object-serialization/go/core-api.md +++ /dev/null @@ -1,409 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This guide covers the core serialization APIs in Fory Go. - -## Creating a Fory Instance - -Create a Fory instance and register your types before serialization: - -```go -import "github.com/apache/fory/go/fory" - -f := fory.New(fory.WithXlang(true)) - -// Register struct with a type ID -f.RegisterStruct(User{}, 1) -f.RegisterStruct(Order{}, 2) - -// Or register with a name (more flexible, less prone to ID conflicts, but higher serialization cost) -f.RegisterStructByName(User{}, "example.User") - -// Register enum types -f.RegisterEnum(Color(0), 3) -``` - -`fory.New()` uses xlang mode with compatible schema evolution. The example sets -`fory.WithXlang(true)` explicitly so the mode choice is visible. For Go-only -payloads that need native mode, configure `fory.WithXlang(false)` explicitly in -the native-mode examples. - -**Important**: The Fory instance should be reused across serialization calls. Creating a new instance involves allocating internal buffers, type caches, and resolvers, which is expensive. The default Fory instance is not thread-safe; for concurrent usage, use the thread-safe wrapper (see [Thread Safety](thread-safety.md)). - -See [Type Registration](type-registration.md) for more details. - -## Core API - -### Serialize and Deserialize - -The primary API for serialization: - -```go -// Serialize any value -data, err := f.Serialize(value) -if err != nil { - // Handle error -} - -// Deserialize into target -var result MyType -err = f.Deserialize(data, &result) -if err != nil { - // Handle error -} -``` - -### Marshal and Unmarshal - -Aliases for `Serialize` and `Deserialize` (familiar to Go developers): - -```go -data, err := f.Marshal(value) -err = f.Unmarshal(data, &result) -``` - -## Serializing Primitives - -```go -// Integers -data, _ := f.Serialize(int64(42)) -var i int64 -f.Deserialize(data, &i) // i = 42 - -// Floats -data, _ = f.Serialize(float64(3.14)) -var fl float64 -f.Deserialize(data, &fl) // fl = 3.14 - -// Strings -data, _ = f.Serialize("hello") -var s string -f.Deserialize(data, &s) // s = "hello" - -// Booleans -data, _ = f.Serialize(true) -var b bool -f.Deserialize(data, &b) // b = true -``` - -## Serializing Collections - -### Slices - -```go -// String slice -strs := []string{"a", "b", "c"} -data, _ := f.Serialize(strs) - -var result []string -f.Deserialize(data, &result) -// result = ["a", "b", "c"] - -// Integer slice -nums := []int64{1, 2, 3} -data, _ = f.Serialize(nums) - -var intResult []int64 -f.Deserialize(data, &intResult) -// intResult = [1, 2, 3] -``` - -### Maps - -```go -// String to string map -m := map[string]string{"key": "value"} -data, _ := f.Serialize(m) - -var result map[string]string -f.Deserialize(data, &result) -// result = {"key": "value"} - -// String to int map -m2 := map[string]int64{"count": 42} -data, _ = f.Serialize(m2) - -var result2 map[string]int64 -f.Deserialize(data, &result2) -// result2 = {"count": 42} -``` - -## Serializing Structs - -### Basic Struct Serialization - -Only **exported fields** (starting with uppercase) are serialized: - -```go -type User struct { - ID int64 // Serialized - Name string // Serialized - password string // NOT serialized (unexported) -} - -f.RegisterStruct(User{}, 1) - -user := &User{ID: 1, Name: "Alice", password: "secret"} -data, _ := f.Serialize(user) - -var result User -f.Deserialize(data, &result) -// result.ID = 1, result.Name = "Alice", result.password = "" -``` - -### Nested Structs - -```go -type Address struct { - City string - Country string -} - -type Person struct { - Name string - Address Address -} - -f.RegisterStruct(Address{}, 1) -f.RegisterStruct(Person{}, 2) - -person := &Person{ - Name: "Alice", - Address: Address{City: "NYC", Country: "USA"}, -} - -data, _ := f.Serialize(person) - -var result Person -f.Deserialize(data, &result) -// result.Address.City = "NYC" -``` - -### Pointer Fields - -```go -type Node struct { - Value int32 - Child *Node -} - -// Use WithTrackRef for pointer fields -f := fory.New(fory.WithXlang(true), fory.WithTrackRef(true)) -f.RegisterStruct(Node{}, 1) - -root := &Node{ - Value: 1, - Child: &Node{Value: 2, Child: nil}, -} - -data, _ := f.Serialize(root) - -var result Node -f.Deserialize(data, &result) -// result.Child.Value = 2 -``` - -## Streaming API - -For scenarios where you want to control the buffer: - -### SerializeTo - -Serialize to an existing buffer: - -```go -buf := fory.NewByteBuffer(nil) - -// Serialize multiple values to same buffer -f.SerializeTo(buf, value1) -f.SerializeTo(buf, value2) - -// Get all serialized data -data := buf.GetByteSlice(0, buf.WriterIndex()) -``` - -### DeserializeFrom - -Deserialize from an existing buffer: - -```go -buf := fory.NewByteBuffer(data) - -var result1, result2 MyType -f.DeserializeFrom(buf, &result1) -f.DeserializeFrom(buf, &result2) -``` - -## Generic API (Type-Safe) - -Fory Go provides generic functions for type-safe serialization: - -```go -import "github.com/apache/fory/go/fory" - -type User struct { - ID int64 - Name string -} - -// Type-safe serialization -user := &User{ID: 1, Name: "Alice"} -data, err := fory.Serialize(f, user) - -// Type-safe deserialization -var result User -err = fory.Deserialize(f, data, &result) -``` - -The generic API: - -- Infers type at compile time -- Provides better type safety -- May offer performance benefits - -## Error Handling - -Always check errors from serialization operations: - -```go -data, err := f.Serialize(value) -if err != nil { - switch e := err.(type) { - case fory.Error: - fmt.Printf("Fory error: %s (kind: %d)\n", e.Error(), e.Kind()) - default: - fmt.Printf("Unknown error: %v\n", err) - } - return -} - -err = f.Deserialize(data, &result) -if err != nil { - // Handle deserialization error -} -``` - -Common error kinds: - -- `ErrKindBufferOutOfBound`: Read/write beyond buffer bounds -- `ErrKindTypeMismatch`: Type ID mismatch during deserialization -- `ErrKindUnknownType`: Unknown type encountered -- `ErrKindMaxDepthExceeded`: Recursion depth limit exceeded -- `ErrKindHashMismatch`: Struct hash mismatch (schema changed) - -See [Troubleshooting](troubleshooting.md) for error resolution. - -## Nil Handling - -### Nil Pointers - -```go -var ptr *User = nil -data, _ := f.Serialize(ptr) - -var result *User -f.Deserialize(data, &result) -// result = nil -``` - -### Empty Collections - -```go -// Nil slice -var slice []string = nil -data, _ := f.Serialize(slice) - -var result []string -f.Deserialize(data, &result) -// result = nil - -// Empty slice (different from nil) -empty := []string{} -data, _ = f.Serialize(empty) - -f.Deserialize(data, &result) -// result = [] (empty, not nil) -``` - -## Complete Example - -```go -package main - -import ( - "fmt" - "github.com/apache/fory/go/fory" -) - -type Order struct { - ID int64 - Customer string - Items []Item - Total float64 -} - -type Item struct { - Name string - Quantity int32 - Price float64 -} - -func main() { - f := fory.New(fory.WithXlang(true)) - f.RegisterStruct(Order{}, 1) - f.RegisterStruct(Item{}, 2) - - order := &Order{ - ID: 12345, - Customer: "Alice", - Items: []Item{ - {Name: "Widget", Quantity: 2, Price: 9.99}, - {Name: "Gadget", Quantity: 1, Price: 24.99}, - }, - Total: 44.97, - } - - // Serialize - data, err := f.Serialize(order) - if err != nil { - panic(err) - } - fmt.Printf("Serialized %d bytes\n", len(data)) - - // Deserialize - var result Order - if err := f.Deserialize(data, &result); err != nil { - panic(err) - } - - fmt.Printf("Order ID: %d\n", result.ID) - fmt.Printf("Customer: %s\n", result.Customer) - fmt.Printf("Items: %d\n", len(result.Items)) - fmt.Printf("Total: %.2f\n", result.Total) -} -``` - -## Related Topics - -- [Configuration](configuration.md) -- [Type Registration](type-registration.md) -- [Supported Types](supported-types.md) -- [References](references.md) diff --git a/docs/object-serialization/go/custom-serializers.md b/docs/object-serialization/go/custom-serializers.md index 5ab02224d22..3f809fd32bc 100644 --- a/docs/object-serialization/go/custom-serializers.md +++ b/docs/object-serialization/go/custom-serializers.md @@ -282,4 +282,4 @@ func TestMySerializer(t *testing.T) { - [Type Registration](type-registration.md) - [Supported Types](supported-types.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/go/index.md b/docs/object-serialization/go/index.md index b053d1e0b7c..a4e11d7e189 100644 --- a/docs/object-serialization/go/index.md +++ b/docs/object-serialization/go/index.md @@ -91,7 +91,7 @@ Use xlang mode for cross-language payloads and schemas shared with other Fory im Use native mode for Go-only traffic. Native mode is selected with `fory.WithXlang(false)` and keeps Go object serialization in Go-native form. It is optimized for Go structs, pointers, interfaces, and Go-specific type behavior that does not need a portable xlang mapping. Compatible mode is enabled by default. Set `fory.WithCompatible(false)` only when every reader and writer uses the same Go struct schema and you want faster serialization and smaller size. -See [Xlang Serialization](xlang.md) for Go xlang registration and interoperability rules, and [Native Serialization](native.md) for Go-only payloads. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for Go xlang registration and interoperability rules, and [Native Serialization](native.md) for Go-only payloads. ## Configuration @@ -133,28 +133,30 @@ data, _ := f.Serialize(&User{ID: 1, Name: "Alice"}) // 'data' can be deserialized by Java, Python, etc. ``` -See [Xlang Serialization](xlang.md) for type mapping and compatibility details. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for type mapping and compatibility details. ## Documentation -| Topic | Description | -| ------------------------------------------- | -------------------------------------- | -| [Basic Serialization](core-api.md) | Core APIs and usage patterns | -| [Xlang Serialization](xlang.md) | Multi-language serialization | -| [Native Serialization](native.md) | Go-only serialization | -| [Configuration](configuration.md) | Options and settings | -| [Schema Metadata](schema-metadata.md) | Field-level configuration | -| [Type Registration](type-registration.md) | Registering types for serialization | -| [Supported Types](supported-types.md) | Complete type support reference | -| [References](references.md) | Circular references and shared objects | -| [Schema Evolution](schema-evolution.md) | Forward/backward compatibility | -| [Custom Serializers](custom-serializers.md) | Extend serialization behavior | -| [Thread Safety](thread-safety.md) | Concurrent usage patterns | -| [gRPC Support](../../grpc/go.md) | Fory payloads over grpc-go | -| [Troubleshooting](troubleshooting.md) | Common issues and solutions | +| Topic | Description | +| --------------------------------------------- | --------------------------------------- | +| [Basic Serialization](basic-serialization.md) | Default xlang APIs and interoperability | +| [Native Serialization](native.md) | Go-only serialization | +| [Configuration](configuration.md) | Options and settings | +| [Schema Metadata](schema-metadata.md) | Field-level configuration | +| [Type Registration](type-registration.md) | Registering types for serialization | +| [Supported Types](supported-types.md) | Complete type support reference | +| [References](references.md) | Circular references and shared objects | +| [Schema Evolution](schema-evolution.md) | Forward/backward compatibility | +| [Custom Serializers](custom-serializers.md) | Extend serialization behavior | +| [Thread Safety](thread-safety.md) | Concurrent usage patterns | +| [gRPC Support](../../grpc/go.md) | Fory payloads over grpc-go | +| [Troubleshooting](troubleshooting.md) | Common issues and solutions | ## Related Resources - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Xlang Type Mapping](../../specification/xlang_type_mapping.md) - [GitHub Repository](https://github.com/apache/fory) + +Before decoding bytes from outside the application trust boundary, read +[Go Security](security.md). diff --git a/docs/object-serialization/go/native.md b/docs/object-serialization/go/native.md index ac1b831f721..cf72cb866c1 100644 --- a/docs/object-serialization/go/native.md +++ b/docs/object-serialization/go/native.md @@ -23,7 +23,7 @@ Go native serialization is the Go-only wire mode selected with `fory.WithXlang(f when every writer and reader is a Go service and the payload should follow Go's type system instead of the portable xlang type system. -Use [Xlang Serialization](xlang.md), the default Go mode, when bytes must be read by +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default Go mode, when bytes must be read by Java, Python, C++, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another non-Go Fory implementation. @@ -210,7 +210,7 @@ The default `Fory` instance reuses its buffer. Copy the byte slice or use `threa ## Related Topics -- [Xlang Serialization](xlang.md) - Cross-language Go payloads +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language Go payloads - [Configuration](configuration.md) - Go options - [Type Registration](type-registration.md) - Struct and enum registration - [References](references.md) - Shared and circular references diff --git a/docs/object-serialization/go/references.md b/docs/object-serialization/go/references.md index 92470b51933..0469e33eb86 100644 --- a/docs/object-serialization/go/references.md +++ b/docs/object-serialization/go/references.md @@ -353,4 +353,4 @@ func main() { - [Configuration](configuration.md) - [Struct Tags](schema-metadata.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/go/schema-evolution.md b/docs/object-serialization/go/schema-evolution.md index b46aecf2097..6a8afb47ba8 100644 --- a/docs/object-serialization/go/schema-evolution.md +++ b/docs/object-serialization/go/schema-evolution.md @@ -389,5 +389,5 @@ func main() { ## Related Topics - [Configuration](configuration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Troubleshooting](troubleshooting.md) diff --git a/docs/object-serialization/go/schema-metadata.md b/docs/object-serialization/go/schema-metadata.md index 02ac8f5d095..878661dabbf 100644 --- a/docs/object-serialization/go/schema-metadata.md +++ b/docs/object-serialization/go/schema-metadata.md @@ -389,5 +389,5 @@ type Session struct { ## Related Topics - [References](references.md) -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Evolution](schema-evolution.md) diff --git a/docs/object-serialization/go/security.md b/docs/object-serialization/go/security.md new file mode 100644 index 00000000000..916643e3e34 --- /dev/null +++ b/docs/object-serialization/go/security.md @@ -0,0 +1,53 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Go reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected structs before deserializing untrusted data. +- Use `WithMaxDepth(...)` to reject unexpectedly deep payloads. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer concrete struct fields over broad `any` or interface-typed fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/go/supported-types.md b/docs/object-serialization/go/supported-types.md index e21e0c48de9..c0098d6649f 100644 --- a/docs/object-serialization/go/supported-types.md +++ b/docs/object-serialization/go/supported-types.md @@ -354,7 +354,7 @@ data, _ := f.Serialize(status) | `time.Time` | Instant | datetime | - | - | | `time.Duration` | Duration | timedelta | - | - | -See [Xlang Serialization](xlang.md) for detailed mapping. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for detailed mapping. ## Unsupported Types @@ -370,5 +370,5 @@ Attempting to serialize these types will result in an error. ## Related Topics - [Type Registration](type-registration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [References](references.md) diff --git a/docs/object-serialization/go/thread-safety.md b/docs/object-serialization/go/thread-safety.md index 7d78e3bb9b0..89b53676675 100644 --- a/docs/object-serialization/go/thread-safety.md +++ b/docs/object-serialization/go/thread-safety.md @@ -343,5 +343,5 @@ go func() { ## Related Topics - [Configuration](configuration.md) -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Troubleshooting](troubleshooting.md) diff --git a/docs/object-serialization/go/troubleshooting.md b/docs/object-serialization/go/troubleshooting.md index e9a04edcd42..9c51ba6be01 100644 --- a/docs/object-serialization/go/troubleshooting.md +++ b/docs/object-serialization/go/troubleshooting.md @@ -438,6 +438,6 @@ If you encounter issues not covered here: ## Related Topics - [Configuration](configuration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Schema Evolution](schema-evolution.md) - [Thread Safety](thread-safety.md) diff --git a/docs/object-serialization/go/type-registration.md b/docs/object-serialization/go/type-registration.md index 18abc8ad634..9367affdd32 100644 --- a/docs/object-serialization/go/type-registration.md +++ b/docs/object-serialization/go/type-registration.md @@ -258,7 +258,7 @@ Two types registered with the same ID will conflict. ## Related Topics -- [Basic Serialization](core-api.md) -- [Xlang Serialization](xlang.md) +- [Basic Serialization](basic-serialization.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Supported Types](supported-types.md) - [Troubleshooting](troubleshooting.md) diff --git a/docs/object-serialization/index.md b/docs/object-serialization/index.md index 996a5080dd1..02eeae901eb 100644 --- a/docs/object-serialization/index.md +++ b/docs/object-serialization/index.md @@ -1,5 +1,5 @@ --- -title: Object Serialization +title: Introduction sidebar_position: 0 id: index license: | @@ -24,27 +24,43 @@ collections, polymorphic values, and optional shared references. ## Choose a mode -| Mode | Use it when | Start here | -| ------ | ---------------------------------------------------- | ---------------------------- | -| Xlang | Bytes cross runtime boundaries | [Xlang mode](xlang/index.md) | -| Native | Every writer and reader uses the same runtime family | [Native mode](native.md) | +| Mode | Use it when | Start here | +| --------------- | ---------------------------------------------------- | --------------------------------- | +| Xlang (default) | Bytes cross runtime boundaries | [Xlang Serialization](xlang.md) | +| Native | Every writer and reader uses the same runtime family | [Native serialization](native.md) | Xlang and native are the only object-serialization modes. Row Format is a random-access analytical representation, and Fory JSON is a Java JSON codec; use the [format chooser](../introduction/choose-a-format.md) when object reconstruction is not your goal. +## Read the concepts and modes + +Read these pages before choosing a runtime API: + +1. [Core Concepts](core-concepts.md) explains object graphs, types, schemas, references, + polymorphism, and schema evolution across both modes. +2. [Xlang Serialization](xlang.md) explains the portable format used by default and the rules that + different language runtimes must share. +3. [Native Serialization](native.md) explains when a same-runtime native format is appropriate. + ## Browse by runtime Choose a runtime to find its installation route, lifecycle, exact APIs, configuration, type registration, schema behavior, extensions, platforms, and troubleshooting: -[Browse supported runtimes](runtimes.md) - -## Security - -Before decoding externally supplied bytes, read [Object Serialization Security](security.md). It -covers accepted-type policy, registration, resource limits, transport responsibilities, and -negative verification for both modes. +| Runtime | Modes | Documentation | +| --------------------- | -------------------- | ------------------------------------------------------ | +| Java | xlang and native | [Java runtime](./java/index.md) | +| Python | xlang and native | [Python runtime](./python/index.md) | +| C++ | xlang and native | [C++ runtime](./cpp/index.md) | +| Go | xlang and native | [Go runtime](./go/index.md) | +| Rust | xlang and native | [Rust runtime](./rust/index.md) | +| JavaScript/TypeScript | xlang | [JavaScript/TypeScript runtime](./javascript/index.md) | +| C# | xlang | [C# runtime](./csharp/index.md) | +| Swift | xlang | [Swift runtime](./swift/index.md) | +| Dart | xlang | [Dart runtime](./dart/index.md) | +| Scala | xlang and JVM native | [Scala runtime](./scala/index.md) | +| Kotlin | xlang and JVM native | [Kotlin runtime](./kotlin/index.md) | ## Specifications diff --git a/docs/object-serialization/java/advanced-features.md b/docs/object-serialization/java/advanced-features.md index 1f58d604ff3..ea7cce50bc9 100644 --- a/docs/object-serialization/java/advanced-features.md +++ b/docs/object-serialization/java/advanced-features.md @@ -157,4 +157,4 @@ static { - [Configuration](configuration.md) - All ForyBuilder options - [Native Serialization](native.md) - Java-only serialization, JDK hooks, and zero-copy buffers - [Object Copy](object-copy.md) - Deep copy functionality -- [Xlang Serialization](xlang.md) - Java xlang interoperability +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Java xlang interoperability diff --git a/docs/object-serialization/java/xlang.md b/docs/object-serialization/java/basic-serialization.md similarity index 68% rename from docs/object-serialization/java/xlang.md rename to docs/object-serialization/java/basic-serialization.md index c18b13d5387..70fb3ea7dc9 100644 --- a/docs/object-serialization/java/xlang.md +++ b/docs/object-serialization/java/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,12 +19,120 @@ license: | limitations under the License. --- +This page covers the Java xlang quickstart. Xlang mode is the default Java wire format and is the +right first choice for cross-language payloads. + +## Create a Fory Instance + +For a single-threaded xlang Fory instance, set the mode explicitly: + +```java +import org.apache.fory.Fory; + +Fory fory = Fory.builder() + .withXlang(true) + .requireClassRegistration(true) + .build(); +``` + +For a thread-safe Fory instance, build `ThreadSafeFory` from the same builder: + +```java +import org.apache.fory.ThreadSafeFory; + +ThreadSafeFory fory = Fory.builder() + .withXlang(true) + .requireClassRegistration(true) + .buildThreadSafeFory(); +``` + +Default Java xlang mode also defaults to compatible schema mode, so independently deployed services +can add and remove fields when their schema metadata remains compatible. Use +`withCompatible(false)` only when every reader and writer always uses the same schema and you want +faster serialization and smaller size. Use the `compatible=false` opt-out only after verifying that every language uses the same xlang schema, or when native types are generated from Fory schema IDL. + +## Register Custom Types + +Register application classes with the same type identity on every peer. Numeric IDs are compact and +fast, while name registration is easier to coordinate across independently owned services. + +```java +import org.apache.fory.annotation.ForyField; + +public class User { + @ForyField(id = 0) + public String name; + + @ForyField(id = 1) + public int age; +} + +Fory fory = Fory.builder() + .withXlang(true) + .requireClassRegistration(true) + .build(); + +fory.register(User.class, "example", "User"); +``` + +Use field IDs for long-lived schemas so field identity is stable even if Java field names change. +See [Schema Metadata](schema-metadata.md) for Java annotations, nullability, reference tracking, and +enum metadata. + +## Serialize And Deserialize + +```java +User user = new User(); +user.name = "Alice"; +user.age = 30; + +byte[] bytes = fory.serialize(user); +User decoded = fory.deserialize(bytes, User.class); +``` + +When xlang bytes cross languages, every peer must register the same type identity and compatible +field metadata. The shared rules live in [Cross-Language Interoperability](../xlang.md), and the Java-specific +interoperability requirements are covered below. + +## Use Native Serialization For Java-Only Traffic + +For same-language Java/JVM traffic, native mode is usually the better fit: + +```java +Fory fory = Fory.builder() + .withXlang(false) + .build(); +``` + +Native mode supports the broad Java object serialization surface, including JDK serialization hooks, +object copy, and native-mode zero-copy buffers. See [Native Serialization](native.md). + +## Common Options + +- `withRefTracking(true)` preserves shared references and circular references. +- `requireClassRegistration(true)` keeps the default registered-type policy. +- Compatible mode is enabled by default for native-mode and xlang payloads. Use + `withCompatible(false)` only when every reader and writer uses the same schema and you want faster + serialization and smaller size. For xlang payloads, use the `compatible=false` opt-out only after verifying that every language uses the same schema, or when native types are generated from Fory schema IDL. +- `withAsyncCompilation(true)` enables asynchronous serializer compilation where supported. + +## Best Practices + +1. **Reuse Fory instances**: Creating Fory is expensive, always reuse instances +2. **Use appropriate thread safety**: Choose between single-thread and thread-safe based on your needs +3. **Register classes**: Keep type identity stable across every xlang peer +4. **Configure reference tracking**: Enable it only when the object graph needs identity or cycles + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Apache Fory™ xlang serialization is the Java wire mode for payloads that must be read by Python, Rust, Go, JavaScript/TypeScript, C++, C#, Swift, Dart, Scala, Kotlin, or another non-Java Fory implementation. Java defaults to xlang mode with compatible schema evolution, but examples set the mode explicitly so the payload contract is visible in code. -## Create an Xlang Fory Instance +### Xlang Configuration Use one long-lived `Fory` or `ThreadSafeFory` instance per configuration. Creating a Fory instance is expensive because Fory caches type metadata and generated serializers. @@ -45,12 +153,12 @@ identity or cycles. Disable it for value-shaped schemas. Use [Native Serialization](native.md) instead when every writer and reader is Java and the payload should preserve Java-specific object behavior. -## Register Types +### Register Types Types must be registered with consistent IDs or names across all languages. Fory supports two registration methods. -### Register by ID (Recommended for Performance) +#### Register by ID (Recommended for Performance) ```java public record Person(String name, int age) {} @@ -67,7 +175,7 @@ Benefits: faster serialization and smaller binary size. Trade-off: every service must coordinate IDs so the same logical type uses the same number. -### Register by Name (Recommended for Flexibility) +#### Register by Name (Recommended for Flexibility) ```java public record Person(String name, int age) {} @@ -88,9 +196,9 @@ Trade-off: the payload includes string identity, so it is larger than ID-based r The Java API also supports a single string type name, such as `fory.register(Person.class, "example.Person")`. Use the same logical identity on every peer. -## Java To Python Example +### Java To Python Example -### Java (Serializer) +#### Java (Serializer) ```java import org.apache.fory.Fory; @@ -116,7 +224,7 @@ public class Example { } ``` -### Python (Deserializer) +#### Python (Deserializer) ```python import pyfory @@ -137,7 +245,7 @@ with open("person.bin", "rb") as input_file: print(f"{person.name}, {person.age}") # Output: Bob, 25 ``` -## Handling Circular and Shared References +### Handling Circular and Shared References Xlang mode supports circular and shared references when reference tracking is enabled: @@ -166,7 +274,7 @@ byte[] bytes = fory.serialize(node1); // Python/Rust/Go can correctly deserialize this with circular references preserved ``` -## Type Mapping Considerations +### Type Mapping Considerations Not all Java types have equivalents in other languages. When using xlang mode: @@ -181,7 +289,7 @@ Not all Java types have equivalents in other languages. When using xlang mode: - See [Type Mapping Guide](../../specification/xlang_type_mapping.md) for the complete compatibility matrix. -### Lists and Dense Arrays +#### Lists and Dense Arrays Java primitive arrays are dense `array` carriers, except plain `byte[]`, which defaults to `bytes`. General Java collections and Fory primitive-list @@ -212,7 +320,7 @@ private @UInt32Type int[] ids; private @BFloat16Type short[] values; ``` -### Compatible Types +#### Compatible Types ```java public record UserData( @@ -223,7 +331,7 @@ public record UserData( ) {} ``` -### Problematic Types +#### Problematic Types ```java public record UserData( @@ -233,7 +341,7 @@ public record UserData( ) {} ``` -## Performance Considerations +### Performance Considerations Xlang mode has additional overhead compared to Java native mode: @@ -246,45 +354,45 @@ Xlang mode has additional overhead compared to Java native mode: - **Disable reference tracking** if you don't need circular references (`withRefTracking(false)`) - **Use native mode** (`withXlang(false)`) when only Java serialization is needed -## Best Practices +### Interoperability Best Practices 1. Use explicit type IDs or namespace/type names for every user type. 2. Keep compatible mode for independently deployed services. 3. Test payloads through every peer before relying on a schema in production. 4. Use native serialization for Java-only traffic that needs Java-specific object behavior. -## Troubleshooting +### Interoperability Troubleshooting -### "Type not registered" errors +#### "Type not registered" errors - Verify type is registered with same ID/name on both sides - Check if type name has typos or case differences -### "Type mismatch" errors +#### "Type mismatch" errors - Ensure field types are compatible across languages - Review [Type Mapping Guide](../../specification/xlang_type_mapping.md) -### Data corruption or unexpected values +#### Data corruption or unexpected values - Verify both sides use xlang payloads - Ensure both sides have compatible Fory versions -## See Also +### Specifications and References - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Type Mapping Reference](../../specification/xlang_type_mapping.md) -- [Python Xlang Serialization Guide](../python/xlang.md) -- [Rust Xlang Serialization Guide](../rust/xlang.md) +- [Python Interoperability Guide](../python/basic-serialization.md#cross-language-interoperability) +- [Rust Interoperability Guide](../rust/basic-serialization.md#cross-language-interoperability) -## Related Topics +### Related Guides - [Schema Evolution](schema-evolution.md) - Compatible mode - [Type Registration](type-registration.md) - Registration methods - [Native Serialization](native.md) - Java-only serialization features - [Row Format](../../row-format/java.md) - Cross-language row format -## Built-in values +### Built-in values ```java import org.apache.fory.*; @@ -310,7 +418,7 @@ public class Example1 { } ``` -## Custom values +### Custom values ```java import org.apache.fory.*; @@ -369,3 +477,10 @@ public class Example2 { } } ``` + +## Related Topics + +- [Configuration](configuration.md) - All ForyBuilder options +- [Native Serialization](native.md) - Java-only serialization features +- [Schema Metadata](schema-metadata.md) - Field IDs, nullability, reference tracking, and enum IDs +- [Troubleshooting](troubleshooting.md) - Common API usage issues diff --git a/docs/object-serialization/java/configuration.md b/docs/object-serialization/java/configuration.md index 0f1f44ae810..6463c69b74e 100644 --- a/docs/object-serialization/java/configuration.md +++ b/docs/object-serialization/java/configuration.md @@ -86,43 +86,7 @@ For xlang payloads, call `withCompatible(false)` only after verifying that every ## Security -Keep class registration enabled for production and any untrusted payload source: - -```java -Fory fory = Fory.builder() - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .build(); -``` - -Security-related options: - -- `requireClassRegistration(true)` restricts deserialization to registered classes. -- `withMaxDepth(...)` rejects unexpectedly deep object graphs. -- `withMaxGraphMemoryBytes(...)` sets an approximate gate for materialized graph memory during one - root deserialization. The estimate mainly covers collections, maps, arrays, structs, and objects; - Fory core primitive arrays and primitive lists count their primitive storage from the decoded - length. It skips leaf values such as strings, primitive scalars, and dedicated binary values that - do not use a primitive-array serializer. Actual process memory can be higher than this limit. Leaf - values remain protected by byte-availability checks: if the unread input does not contain enough - bytes, Fory will not read or create that leaf value. The default is a fixed `128 MiB`; set a - positive byte limit when trusted workloads need a larger or smaller gate. -- `withMaxUnbackedContainerItems(...)` limits count-driven collection and map work whose repeated - read bodies do not consume proportional input. The default is `8192`; zero is a strict limit. -- `withMaxTypeFields(...)` and `withMaxTypeMetaBytes(...)` bound the field count - and encoded body size of one received remote metadata body. -- `withMaxSchemaVersionsPerType(...)` and - `withMaxAverageSchemaVersionsPerType(...)` bound accepted remote metadata versions without - changing registration, dynamic loading, or schema-evolution semantics. -- `withDeserializeUnknownClass(false)` avoids materializing unknown classes from metadata. -- `checkJdkClassSerializable(true)` keeps the JDK serializability check for `java.*` classes. -- Class registration warnings can be useful during security audits; use - `suppressClassRegistrationWarnings(false)` when you need to surface unexpected types. - -Use `requireClassRegistration(false)` only for trusted payloads, and pair it with a `TypeChecker` -allow list when dynamic class loading is required. +See [Java Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics diff --git a/docs/object-serialization/java/core-api.md b/docs/object-serialization/java/core-api.md deleted file mode 100644 index a8f10bb452d..00000000000 --- a/docs/object-serialization/java/core-api.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers the Java xlang quickstart. Xlang mode is the default Java wire format and is the -right first choice for cross-language payloads. - -## Create a Fory Instance - -For a single-threaded xlang Fory instance, set the mode explicitly: - -```java -import org.apache.fory.Fory; - -Fory fory = Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .build(); -``` - -For a thread-safe Fory instance, build `ThreadSafeFory` from the same builder: - -```java -import org.apache.fory.ThreadSafeFory; - -ThreadSafeFory fory = Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .buildThreadSafeFory(); -``` - -Default Java xlang mode also defaults to compatible schema mode, so independently deployed services -can add and remove fields when their schema metadata remains compatible. Use -`withCompatible(false)` only when every reader and writer always uses the same schema and you want -faster serialization and smaller size. Use the `compatible=false` opt-out only after verifying that every language uses the same xlang schema, or when native types are generated from Fory schema IDL. - -## Register Custom Types - -Register application classes with the same type identity on every peer. Numeric IDs are compact and -fast, while name registration is easier to coordinate across independently owned services. - -```java -import org.apache.fory.annotation.ForyField; - -public class User { - @ForyField(id = 0) - public String name; - - @ForyField(id = 1) - public int age; -} - -Fory fory = Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .build(); - -fory.register(User.class, "example", "User"); -``` - -Use field IDs for long-lived schemas so field identity is stable even if Java field names change. -See [Schema Metadata](schema-metadata.md) for Java annotations, nullability, reference tracking, and -enum metadata. - -## Serialize And Deserialize - -```java -User user = new User(); -user.name = "Alice"; -user.age = 30; - -byte[] bytes = fory.serialize(user); -User decoded = fory.deserialize(bytes, User.class); -``` - -When xlang bytes cross languages, every peer must register the same type identity and compatible -field metadata. The shared rules live in [Xlang](../xlang/index.md), while Java-specific API calls -are in [Xlang Serialization](xlang.md). - -## Use Native Serialization For Java-Only Traffic - -For same-language Java/JVM traffic, native mode is usually the better fit: - -```java -Fory fory = Fory.builder() - .withXlang(false) - .build(); -``` - -Native mode supports the broad Java object serialization surface, including JDK serialization hooks, -object copy, and native-mode zero-copy buffers. See [Native Serialization](native.md). - -## Common Options - -- `withRefTracking(true)` preserves shared references and circular references. -- `requireClassRegistration(true)` keeps the default registered-type policy. -- Compatible mode is enabled by default for native-mode and xlang payloads. Use - `withCompatible(false)` only when every reader and writer uses the same schema and you want faster - serialization and smaller size. For xlang payloads, use the `compatible=false` opt-out only after verifying that every language uses the same schema, or when native types are generated from Fory schema IDL. -- `withAsyncCompilation(true)` enables asynchronous serializer compilation where supported. - -## Best Practices - -1. **Reuse Fory instances**: Creating Fory is expensive, always reuse instances -2. **Use appropriate thread safety**: Choose between single-thread and thread-safe based on your needs -3. **Register classes**: Keep type identity stable across every xlang peer -4. **Configure reference tracking**: Enable it only when the object graph needs identity or cycles - -## Related Topics - -- [Configuration](configuration.md) - All ForyBuilder options -- [Native Serialization](native.md) - Java-only serialization features -- [Schema Metadata](schema-metadata.md) - Field IDs, nullability, reference tracking, and enum IDs -- [Xlang Serialization](xlang.md) - Java xlang interoperability -- [Troubleshooting](troubleshooting.md) - Common API usage issues diff --git a/docs/object-serialization/java/index.md b/docs/object-serialization/java/index.md index 88a69b87efd..2d5add91f89 100644 --- a/docs/object-serialization/java/index.md +++ b/docs/object-serialization/java/index.md @@ -22,7 +22,7 @@ license: | Apache Fory Java provides high-performance binary object serialization. Use xlang mode for payloads shared with other supported runtimes and native mode for Java/JVM-only object graphs. -This runtime guide is scoped to Binary Object Serialization. For other Java products, use +This runtime guide is scoped to Binary Object Serialization. For other Java capabilities, use [Row Format](../../row-format/java.md), [Fory JSON](../../json/index.md), [Fory IDL and compiler](../../compiler/index.md), or [Fory gRPC](../../grpc/java.md). @@ -169,7 +169,7 @@ Use xlang mode for cross-language payloads and schemas shared with non-Java impl Use native mode for Java-only traffic. Native mode is selected with `.withXlang(false)` and owns Java-specific object behavior such as JDK serialization hooks, `Externalizable`, dynamic object graphs, object copy, and Java native-mode zero-copy buffers. It is optimized for the JVM type system and supports a broader Java object surface than xlang mode. Compatible mode is enabled by default. Set `.withCompatible(false)` only when every reader and writer uses the same class schema and you want faster serialization and smaller size. If you are replacing JDK serialization, Kryo, FST, Hessian, or Java-only Protocol Buffers payloads, start with native mode. -See [Native Serialization](native.md) for Java-only serialization details and [Xlang Serialization](xlang.md) for Java xlang registration and interoperability rules. +See [Native Serialization](native.md) for Java-only serialization details and [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for Java xlang registration and interoperability rules. ### Thread Safety @@ -257,10 +257,10 @@ model-discovery workflows differ from Fory Core. | Group | Pages | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Serialization modes | [Xlang](xlang.md), [Native](native.md) | -| Common | [Core API](core-api.md), [Configuration](configuration.md), [Type Registration](type-registration.md), [Schema Evolution](schema-evolution.md), [Schema Metadata](schema-metadata.md), [Custom Serializers](custom-serializers.md) | +| Serialization modes | [Basic Serialization](basic-serialization.md), [Native Serialization](native.md) | +| Common | [Configuration](configuration.md), [Type Registration](type-registration.md), [Schema Evolution](schema-evolution.md), [Schema Metadata](schema-metadata.md), [Custom Serializers](custom-serializers.md) | | Java-specific features | [Advanced Features](advanced-features.md), [Compression](compression.md), [Object Copy](object-copy.md), [JDK Custom Serialization](jdk-serialization.md), [Static Generated Serializers](static-generated-serializers.md), [Virtual Threads](virtual-threads.md) | | Platform and operate | [Android](android.md), [GraalVM Native Image](graalvm.md), [Troubleshooting](troubleshooting.md) | Before decoding externally supplied binary payloads, read -[Object Serialization Security](../security.md). +[Java Security](security.md). diff --git a/docs/object-serialization/java/native.md b/docs/object-serialization/java/native.md index 6832213e732..6737b2f53db 100644 --- a/docs/object-serialization/java/native.md +++ b/docs/object-serialization/java/native.md @@ -28,7 +28,7 @@ payloads. Native serialization in this page means Fory's `xlang=false` wire mode. It is separate from GraalVM native image support, which is covered in [GraalVM Native Image](graalvm.md). -Use [Xlang Serialization](xlang.md), the default Java mode, when bytes must be read by +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default Java mode, when bytes must be read by non-Java Fory implementations. ## When To Use Native Serialization @@ -326,8 +326,8 @@ Use `JavaSerializer.serializedByJDK(...)` only at the mixed-format boundary, the ## Related Topics -- [Basic Serialization](core-api.md) - Xlang-first Java quickstart -- [Xlang Serialization](xlang.md) - Cross-language Java payloads +- [Basic Serialization](basic-serialization.md) - Xlang-first Java quickstart +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language Java payloads - [Configuration](configuration.md) - Java builder options - [Schema Evolution](schema-evolution.md) - compatible mode and same-schema optimization - [Type Registration](type-registration.md) - Registration and security diff --git a/docs/object-serialization/java/object-copy.md b/docs/object-serialization/java/object-copy.md index 002b79b3349..b0c02b23ecc 100644 --- a/docs/object-serialization/java/object-copy.md +++ b/docs/object-serialization/java/object-copy.md @@ -357,7 +357,7 @@ Fix it by either: ## Related Topics -- [Basic Serialization](core-api.md) - Fory instance creation and core APIs +- [Basic Serialization](basic-serialization.md) - Fory instance creation and core APIs - [Configuration](configuration.md) - Builder options including `withRefCopy` - [Custom Serializers](custom-serializers.md) - Serializer design and registration - [Virtual Threads](virtual-threads.md) - Thread-safe Fory guidance diff --git a/docs/object-serialization/java/schema-evolution.md b/docs/object-serialization/java/schema-evolution.md index 1e71d7e97ed..9a2796306eb 100644 --- a/docs/object-serialization/java/schema-evolution.md +++ b/docs/object-serialization/java/schema-evolution.md @@ -266,5 +266,5 @@ public class SameSchemaMessage { ## Related Topics - [Configuration](configuration.md) - All ForyBuilder options -- [Xlang Serialization](xlang.md) - xlang mode +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - xlang mode - [Troubleshooting](troubleshooting.md) - Common schema issues diff --git a/docs/object-serialization/java/schema-metadata.md b/docs/object-serialization/java/schema-metadata.md index 0e631edf9b9..981e47184a2 100644 --- a/docs/object-serialization/java/schema-metadata.md +++ b/docs/object-serialization/java/schema-metadata.md @@ -710,7 +710,7 @@ public class User { ## Related Topics -- [Basic Serialization](core-api.md) - Getting started with Fory serialization +- [Basic Serialization](basic-serialization.md) - Getting started with Fory serialization - [Configuration](configuration.md) - `ForyBuilder` options - [Schema Evolution](schema-evolution.md) - Compatible mode and schema evolution -- [Xlang Serialization](xlang.md) - Interoperability with Python, Rust, C++, Go +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Interoperability with Python, Rust, C++, Go diff --git a/docs/object-serialization/java/security.md b/docs/object-serialization/java/security.md new file mode 100644 index 00000000000..780d39b2d2b --- /dev/null +++ b/docs/object-serialization/java/security.md @@ -0,0 +1,83 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Java reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Keep class registration enabled for production and any untrusted payload source: + +```java +Fory fory = Fory.builder() + .requireClassRegistration(true) + .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .withMaxUnbackedContainerItems(8192) + .build(); +``` + +Security-related options: + +- `requireClassRegistration(true)` restricts deserialization to registered classes. +- `withMaxDepth(...)` rejects unexpectedly deep object graphs. +- `withMaxGraphMemoryBytes(...)` sets an approximate gate for materialized graph memory during one + root deserialization. The estimate mainly covers collections, maps, arrays, structs, and objects; + Fory core primitive arrays and primitive lists count their primitive storage from the decoded + length. It skips leaf values such as strings, primitive scalars, and dedicated binary values that + do not use a primitive-array serializer. Actual process memory can be higher than this limit. Leaf + values remain protected by byte-availability checks: if the unread input does not contain enough + bytes, Fory will not read or create that leaf value. The default is a fixed `128 MiB`; set a + positive byte limit when trusted workloads need a larger or smaller gate. +- `withMaxUnbackedContainerItems(...)` limits count-driven collection and map work whose repeated + read bodies do not consume proportional input. The default is `8192`; zero is a strict limit. +- `withMaxTypeFields(...)` and `withMaxTypeMetaBytes(...)` bound the field count + and encoded body size of one received remote metadata body. +- `withMaxSchemaVersionsPerType(...)` and + `withMaxAverageSchemaVersionsPerType(...)` bound accepted remote metadata versions without + changing registration, dynamic loading, or schema-evolution semantics. +- `withDeserializeUnknownClass(false)` avoids materializing unknown classes from metadata. +- `checkJdkClassSerializable(true)` keeps the JDK serializability check for `java.*` classes. +- Class registration warnings can be useful during security audits; use + `suppressClassRegistrationWarnings(false)` when you need to surface unexpected types. + +Use `requireClassRegistration(false)` only for trusted payloads, and pair it with a `TypeChecker` +allow list when dynamic class loading is required. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/javascript/xlang.md b/docs/object-serialization/javascript/basic-serialization.md similarity index 54% rename from docs/object-serialization/javascript/xlang.md rename to docs/object-serialization/javascript/basic-serialization.md index 24452ee5594..ce14adb1ce0 100644 --- a/docs/object-serialization/javascript/xlang.md +++ b/docs/object-serialization/javascript/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,6 +19,228 @@ license: | limitations under the License. --- +This guide covers the core serialization APIs in the default xlang mode for Apache Fory JavaScript. + +## Create a `Fory` Instance + +```ts +import Fory from "@apache-fory/core"; + +const fory = new Fory(); +``` + +Create one instance, register your schemas, and reuse it. Fory caches the generated serializers after the first `register` call, so recreating it on every request wastes that work. + +## Define a Schema with `Type.struct` + +The most common path is to define a schema and register it. + +```ts +import Fory, { Type } from "@apache-fory/core"; + +const accountType = Type.struct( + { typeName: "example.account" }, + { + id: Type.int64(), + owner: Type.string(), + active: Type.bool(), + nickname: Type.string().setNullable(true), + }, +); + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(accountType); +``` + +## Serialize and Deserialize + +```ts +const bytes = serialize({ + id: 42n, + owner: "Alice", + active: true, + nickname: null, +}); + +const value = deserialize(bytes); +console.log(value); +// { id: 42n, owner: 'Alice', active: true, nickname: null } +``` + +The returned `bytes` value is a `Uint8Array`/platform buffer and can be sent over the network or written to storage. + +## Root-Level Dynamic Serialization + +`Fory` can also serialize dynamic root values without first binding a schema-specific serializer. + +```ts +const fory = new Fory(); + +const bytes = fory.serialize( + new Map([ + ["name", "Alice"], + ["age", 30], + ]), +); + +const value = fory.deserialize(bytes); +``` + +This is convenient for dynamic payloads, but explicit schemas are usually better for stable interfaces and cross-language contracts. + +## Primitive Values + +```ts +const fory = new Fory(); + +fory.deserialize(fory.serialize(true)); +// true + +fory.deserialize(fory.serialize("hello")); +// 'hello' + +fory.deserialize(fory.serialize(123)); +// 123 + +fory.deserialize(fory.serialize(123n)); +// 123n + +fory.deserialize(fory.serialize(new Date("2021-10-20T09:13:00Z"))); +// Date +``` + +### Number and `bigint` + +JavaScript `number` is a 64-bit float, which cannot exactly represent all 64-bit integers. For cross-language contracts or anywhere exact integer sizes matter, use explicit field types in your schema: + +- `Type.int32()` — 32-bit integer; use JavaScript `number` +- `Type.int64()` — 64-bit integer; use JavaScript `bigint` +- `Type.float32()` / `Type.float64()` — floating-point + +Dynamic root serialization (calling `fory.serialize(someNumber)` without a schema) will infer a type, but the inferred type is not guaranteed by the API. Use a schema for any stable contract. + +## Arrays, Maps, and Sets + +```ts +const inventoryType = Type.struct("example.inventory", { + tags: Type.list(Type.string()), + counts: Type.map(Type.string(), Type.int32()), + labels: Type.set(Type.string()), +}); + +const fory = new Fory({ ref: true }); +const { serialize, deserialize } = fory.register(inventoryType); + +const bytes = serialize({ + tags: ["hot", "new"], + counts: new Map([ + ["apple", 3], + ["pear", 8], + ]), + labels: new Set(["featured", "seasonal"]), +}); + +const value = deserialize(bytes); +``` + +## Nested Structs + +```ts +const addressType = Type.struct("example.address", { + city: Type.string(), + country: Type.string(), +}); + +const userType = Type.struct("example.user", { + name: Type.string(), + address: Type.struct("example.address", { + city: Type.string(), + country: Type.string(), + }), +}); + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(userType); + +const bytes = serialize({ + name: "Alice", + address: { city: "Hangzhou", country: "CN" }, +}); + +const user = deserialize(bytes); +``` + +If a nested value can be missing, mark it nullable: + +```ts +const wrapperType = Type.struct("example.wrapper", { + child: Type.struct("example.child", { + name: Type.string(), + }).setNullable(true), +}); +``` + +## Decorator-Based Registration + +TypeScript decorators are also supported. + +```ts +import Fory, { Type } from "@apache-fory/core"; + +@Type.struct("example.user") +class User { + @Type.int64() + id!: bigint; + + @Type.string() + name!: string; +} + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(User); + +const user = new User(); +user.id = 1n; +user.name = "Alice"; + +const copy = deserialize(serialize(user)); +console.log(copy instanceof User); // true +``` + +## Nullability + +Field nullability is explicit in schema-based structs. + +```ts +const nullableType = Type.struct("example.optional_user", { + name: Type.string(), + email: Type.string().setNullable(true), +}); +``` + +If a field is not marked nullable and you try to write `null`, serialization throws. + +## Debugging Generated Code + +You can inspect generated serializer code with `hooks.afterCodeGenerated`. + +```ts +const fory = new Fory({ + hooks: { + afterCodeGenerated(code) { + console.log(code); + return code; + }, + }, +}); +``` + +This is useful when debugging schema behavior, field ordering, or generated fast paths. + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Fory JavaScript serializes to the same binary format as the Java, Python, C++, Go, Rust, C#, Swift, Dart, Scala, and Kotlin Fory implementations. You can write a message in JavaScript and read it in Java, or any other direction, without a @@ -29,7 +251,7 @@ Things to keep in mind: - Fory JavaScript reads and writes cross-language payloads only; it does not support any native-mode format. - JavaScript does not support out-of-band mode. -## Requirements for a Successful Round Trip +### Requirements for a Successful Round Trip For a message to survive a round trip between JavaScript and another language: @@ -39,7 +261,7 @@ For a message to survive a round trip between JavaScript and another language: 4. Compatible schema evolution on both sides. JavaScript enables it by default. 5. **Same reference tracking config** if your data has shared or circular references. -## Step-by-Step: JavaScript to Another Peer +### Step-by-Step: JavaScript to Another Peer 1. Define the JavaScript schema with the same type name or numeric ID used by the peer. 2. Register the schema in both peers. @@ -75,14 +297,14 @@ On the other side, register the same `example.message` type (same name or same n - [Go guide](../go/index.md) - [Rust guide](../rust/index.md) -## Field Naming +### Field Naming Fory matches fields by name. When models are defined in multiple languages, keep field names consistent — or at minimum use a naming scheme that maps unambiguously across languages (e.g. `snake_case` everywhere). With the default compatible schema evolution, field order differences are tolerated, but the names themselves must still match. -## Numeric Types +### Numeric Types JavaScript `number` is a 64-bit float, which does not map cleanly to every integer type in other languages. Use explicit schema types: @@ -90,7 +312,7 @@ JavaScript `number` is a 64-bit float, which does not map cleanly to every integ - `Type.int64()` with `bigint` values for 64-bit integers (Java `long`, Go `int64`) - `Type.float32()` or `Type.float64()` for floating-point values -## Lists and Dense Arrays +### Lists and Dense Arrays Use `Type.list(T)` for ordinary JavaScript `Array` values and Fory `list` schema. Dense bool/numeric vectors use the explicit array builders @@ -113,13 +335,13 @@ listed below. | `array` | `Type.float32Array()` | | `array` | `Type.float64Array()` | -## Date and Time +### Date and Time - `Type.timestamp()` — a point in time; round-trips as a JavaScript `Date` - `Type.date()` — a date without time; deserializes as `Date` - `Type.duration()` — exposed as a numeric millisecond value in JavaScript -## Polymorphic Fields +### Polymorphic Fields `Type.any()` lets a field hold different concrete types, but it is harder to keep in sync across languages. Prefer explicit field schemas whenever possible. @@ -132,7 +354,7 @@ const wrapperType = Type.struct( ); ``` -## Enums +### Enums Enum member **order** must match across languages. Fory encodes enums by ordinal position, not by value. @@ -144,17 +366,17 @@ fory.register(Type.enum({ typeId: 210 }, Color)); Use the same type ID or type name in every peer. -## Safety Limits +### Safety Limits The `maxDepth` option bounds nested payloads. It does not change the binary format; it only controls what the local `Fory` instance accepts. -## Related Topics +### Related Guides - [Supported Types](supported-types.md) - [Schema Evolution](schema-evolution.md) - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) -## Built-in values +### Built-in values ```javascript import Fory from "@apache-fory/core"; @@ -165,7 +387,7 @@ const result = fory.deserialize(input); console.log(result); ``` -## Custom values +### Custom values ```javascript import Fory, { Type } from "@apache-fory/core"; @@ -184,7 +406,7 @@ const result = deserialize(input); console.log(result); ``` -## Shared and circular references +### Shared and circular references ```javascript import Fory, { Type } from "@apache-fory/core"; @@ -204,3 +426,9 @@ const input = serialize(data); const result = deserialize(input); console.log(result.bar.foo === result.foo); ``` + +## Related Topics + +- [Type Registration](type-registration.md) +- [Supported Types](supported-types.md) +- [References](references.md) diff --git a/docs/object-serialization/javascript/configuration.md b/docs/object-serialization/javascript/configuration.md index f7d76d30e16..c73194882e3 100644 --- a/docs/object-serialization/javascript/configuration.md +++ b/docs/object-serialization/javascript/configuration.md @@ -140,24 +140,11 @@ Leave this unset unless you run on Node.js 20+ and have benchmarked your workloa ## Security -Security-related configuration: - -- Register only the expected schemas before deserializing untrusted payloads. -- Set `maxDepth` for the maximum nesting depth your service accepts. -- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input - bytes. -- Keep `maxTypeFields` and `maxTypeMetaBytes` at their defaults unless the data - is not malicious and a trusted peer sends larger remote metadata. -- Keep `maxSchemaVersionsPerType` and - `maxAverageSchemaVersionsPerType` at their defaults unless the data is not - malicious and a trusted peer sends many remote schema versions. -- Prefer explicit `Type.struct(...)` schemas over `Type.any()` for untrusted input. -- Pass `hps` only from the official package version you deploy with Fory. +See [JavaScript/TypeScript Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Metadata](schema-metadata.md) - [Schema Evolution](schema-evolution.md) - [References](references.md) diff --git a/docs/object-serialization/javascript/core-api.md b/docs/object-serialization/javascript/core-api.md deleted file mode 100644 index 6b4650e8f2e..00000000000 --- a/docs/object-serialization/javascript/core-api.md +++ /dev/null @@ -1,244 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This guide covers the core serialization APIs in Apache Fory JavaScript. - -## Create a `Fory` Instance - -```ts -import Fory from "@apache-fory/core"; - -const fory = new Fory(); -``` - -Create one instance, register your schemas, and reuse it. Fory caches the generated serializers after the first `register` call, so recreating it on every request wastes that work. - -## Define a Schema with `Type.struct` - -The most common path is to define a schema and register it. - -```ts -import Fory, { Type } from "@apache-fory/core"; - -const accountType = Type.struct( - { typeName: "example.account" }, - { - id: Type.int64(), - owner: Type.string(), - active: Type.bool(), - nickname: Type.string().setNullable(true), - }, -); - -const fory = new Fory(); -const { serialize, deserialize } = fory.register(accountType); -``` - -## Serialize and Deserialize - -```ts -const bytes = serialize({ - id: 42n, - owner: "Alice", - active: true, - nickname: null, -}); - -const value = deserialize(bytes); -console.log(value); -// { id: 42n, owner: 'Alice', active: true, nickname: null } -``` - -The returned `bytes` value is a `Uint8Array`/platform buffer and can be sent over the network or written to storage. - -## Root-Level Dynamic Serialization - -`Fory` can also serialize dynamic root values without first binding a schema-specific serializer. - -```ts -const fory = new Fory(); - -const bytes = fory.serialize( - new Map([ - ["name", "Alice"], - ["age", 30], - ]), -); - -const value = fory.deserialize(bytes); -``` - -This is convenient for dynamic payloads, but explicit schemas are usually better for stable interfaces and cross-language contracts. - -## Primitive Values - -```ts -const fory = new Fory(); - -fory.deserialize(fory.serialize(true)); -// true - -fory.deserialize(fory.serialize("hello")); -// 'hello' - -fory.deserialize(fory.serialize(123)); -// 123 - -fory.deserialize(fory.serialize(123n)); -// 123n - -fory.deserialize(fory.serialize(new Date("2021-10-20T09:13:00Z"))); -// Date -``` - -### Number and `bigint` - -JavaScript `number` is a 64-bit float, which cannot exactly represent all 64-bit integers. For cross-language contracts or anywhere exact integer sizes matter, use explicit field types in your schema: - -- `Type.int32()` — 32-bit integer; use JavaScript `number` -- `Type.int64()` — 64-bit integer; use JavaScript `bigint` -- `Type.float32()` / `Type.float64()` — floating-point - -Dynamic root serialization (calling `fory.serialize(someNumber)` without a schema) will infer a type, but the inferred type is not guaranteed by the API. Use a schema for any stable contract. - -## Arrays, Maps, and Sets - -```ts -const inventoryType = Type.struct("example.inventory", { - tags: Type.list(Type.string()), - counts: Type.map(Type.string(), Type.int32()), - labels: Type.set(Type.string()), -}); - -const fory = new Fory({ ref: true }); -const { serialize, deserialize } = fory.register(inventoryType); - -const bytes = serialize({ - tags: ["hot", "new"], - counts: new Map([ - ["apple", 3], - ["pear", 8], - ]), - labels: new Set(["featured", "seasonal"]), -}); - -const value = deserialize(bytes); -``` - -## Nested Structs - -```ts -const addressType = Type.struct("example.address", { - city: Type.string(), - country: Type.string(), -}); - -const userType = Type.struct("example.user", { - name: Type.string(), - address: Type.struct("example.address", { - city: Type.string(), - country: Type.string(), - }), -}); - -const fory = new Fory(); -const { serialize, deserialize } = fory.register(userType); - -const bytes = serialize({ - name: "Alice", - address: { city: "Hangzhou", country: "CN" }, -}); - -const user = deserialize(bytes); -``` - -If a nested value can be missing, mark it nullable: - -```ts -const wrapperType = Type.struct("example.wrapper", { - child: Type.struct("example.child", { - name: Type.string(), - }).setNullable(true), -}); -``` - -## Decorator-Based Registration - -TypeScript decorators are also supported. - -```ts -import Fory, { Type } from "@apache-fory/core"; - -@Type.struct("example.user") -class User { - @Type.int64() - id!: bigint; - - @Type.string() - name!: string; -} - -const fory = new Fory(); -const { serialize, deserialize } = fory.register(User); - -const user = new User(); -user.id = 1n; -user.name = "Alice"; - -const copy = deserialize(serialize(user)); -console.log(copy instanceof User); // true -``` - -## Nullability - -Field nullability is explicit in schema-based structs. - -```ts -const nullableType = Type.struct("example.optional_user", { - name: Type.string(), - email: Type.string().setNullable(true), -}); -``` - -If a field is not marked nullable and you try to write `null`, serialization throws. - -## Debugging Generated Code - -You can inspect generated serializer code with `hooks.afterCodeGenerated`. - -```ts -const fory = new Fory({ - hooks: { - afterCodeGenerated(code) { - console.log(code); - return code; - }, - }, -}); -``` - -This is useful when debugging schema behavior, field ordering, or generated fast paths. - -## Related Topics - -- [Type Registration](type-registration.md) -- [Supported Types](supported-types.md) -- [References](references.md) diff --git a/docs/object-serialization/javascript/index.md b/docs/object-serialization/javascript/index.md index 994a350ccb7..31a158002c4 100644 --- a/docs/object-serialization/javascript/index.md +++ b/docs/object-serialization/javascript/index.md @@ -156,21 +156,23 @@ options; see [Configuration](configuration.md). ## Documentation -| Topic | Description | -| -------------------------------------------- | ------------------------------------------------------- | -| [Basic Serialization](core-api.md) | Core APIs and everyday usage | -| [Configuration](configuration.md) | Fory options, compatible mode, limits, and HPS | -| [Type Registration](type-registration.md) | Numeric IDs, names, decorators, and schema registration | -| [Schema Metadata](schema-metadata.md) | Type builders, field options, and decorators | -| [Supported Types](supported-types.md) | Primitive, collection, time, enum, and struct mappings | -| [References](references.md) | Shared references and circular object graphs | -| [Schema Evolution](schema-evolution.md) | Compatible mode and evolving structs | -| [Xlang Serialization](xlang.md) | Interop guidance and mapping rules | -| [Fory IDL Compiler](../../compiler/index.md) | Generate TypeScript models from `.fdl` schemas | -| [gRPC Support](../../grpc/javascript.md) | Node.js gRPC and browser gRPC-Web generated clients | -| [Troubleshooting](troubleshooting.md) | Common issues, limits, and debugging tips | +| Topic | Description | +| --------------------------------------------- | ------------------------------------------------------- | +| [Basic Serialization](basic-serialization.md) | Default xlang APIs, usage, and interoperability | +| [Configuration](configuration.md) | Fory options, compatible mode, limits, and HPS | +| [Type Registration](type-registration.md) | Numeric IDs, names, decorators, and schema registration | +| [Schema Metadata](schema-metadata.md) | Type builders, field options, and decorators | +| [Supported Types](supported-types.md) | Primitive, collection, time, enum, and struct mappings | +| [References](references.md) | Shared references and circular object graphs | +| [Schema Evolution](schema-evolution.md) | Compatible mode and evolving structs | +| [Fory IDL Compiler](../../compiler/index.md) | Generate TypeScript models from `.fdl` schemas | +| [gRPC Support](../../grpc/javascript.md) | Node.js gRPC and browser gRPC-Web generated clients | +| [Troubleshooting](troubleshooting.md) | Common issues, limits, and debugging tips | ## Related Resources - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Xlang Type Mapping](../../specification/xlang_type_mapping.md) + +Before decoding bytes from outside the application trust boundary, read +[JavaScript/TypeScript Security](security.md). diff --git a/docs/object-serialization/javascript/references.md b/docs/object-serialization/javascript/references.md index 38b496843fb..5f0f118bdbd 100644 --- a/docs/object-serialization/javascript/references.md +++ b/docs/object-serialization/javascript/references.md @@ -106,6 +106,6 @@ Reference tracking is part of the Fory binary protocol and works across language ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Evolution](schema-evolution.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/javascript/schema-evolution.md b/docs/object-serialization/javascript/schema-evolution.md index 22d77da5d4b..f91d96ef5e4 100644 --- a/docs/object-serialization/javascript/schema-evolution.md +++ b/docs/object-serialization/javascript/schema-evolution.md @@ -108,9 +108,9 @@ reads expecting compatible metadata, deserialization will fail. ## Xlang Requirement -Compatible mode only protects you from schema differences in the _fields_ of a type. You still need the same type identity (same numeric ID or same `typeName`) on every side. See [Xlang Serialization](xlang.md). +Compatible mode only protects you from schema differences in the _fields_ of a type. You still need the same type identity (same numeric ID or same `typeName`) on every side. See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability). ## Related Topics - [Type Registration](type-registration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/javascript/security.md b/docs/object-serialization/javascript/security.md new file mode 100644 index 00000000000..6a01ac5b73e --- /dev/null +++ b/docs/object-serialization/javascript/security.md @@ -0,0 +1,60 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a JavaScript/TypeScript reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected schemas before deserializing untrusted payloads. +- Set `maxDepth` for the maximum nesting depth your service accepts. +- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. +- Keep `maxTypeFields` and `maxTypeMetaBytes` at their defaults unless the data + is not malicious and a trusted peer sends larger remote metadata. +- Keep `maxSchemaVersionsPerType` and + `maxAverageSchemaVersionsPerType` at their defaults unless the data is not + malicious and a trusted peer sends many remote schema versions. +- Prefer explicit `Type.struct(...)` schemas over `Type.any()` for untrusted input. +- Pass `hps` only from the official package version you deploy with Fory. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/javascript/supported-types.md b/docs/object-serialization/javascript/supported-types.md index 37cfda8b4e8..57ca8eb011d 100644 --- a/docs/object-serialization/javascript/supported-types.md +++ b/docs/object-serialization/javascript/supported-types.md @@ -172,6 +172,6 @@ For types that need completely custom encoding, use `Type.ext(...)` and pass a c ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [References](references.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/javascript/troubleshooting.md b/docs/object-serialization/javascript/troubleshooting.md index c1aa890a1b4..b215dd37b77 100644 --- a/docs/object-serialization/javascript/troubleshooting.md +++ b/docs/object-serialization/javascript/troubleshooting.md @@ -85,6 +85,6 @@ const fory = new Fory({ ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [References](references.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/javascript/type-registration.md b/docs/object-serialization/javascript/type-registration.md index b109c2f8226..a0c471c0750 100644 --- a/docs/object-serialization/javascript/type-registration.md +++ b/docs/object-serialization/javascript/type-registration.md @@ -154,11 +154,11 @@ Use **names** when: ## Xlang -For a message to round-trip between JavaScript and another language, both sides must use the same identity for a given type: same numeric ID, or same `typeName`. Use `.` inside `typeName` to add a namespace prefix. See [Xlang Serialization](xlang.md). +For a message to round-trip between JavaScript and another language, both sides must use the same identity for a given type: same numeric ID, or same `typeName`. Use `.` inside `typeName` to add a namespace prefix. See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability). ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Metadata](schema-metadata.md) - [Schema Evolution](schema-evolution.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/kotlin/xlang.md b/docs/object-serialization/kotlin/basic-serialization.md similarity index 82% rename from docs/object-serialization/kotlin/xlang.md rename to docs/object-serialization/kotlin/basic-serialization.md index 1f5b86e3bf2..0fdbfb3c04a 100644 --- a/docs/object-serialization/kotlin/xlang.md +++ b/docs/object-serialization/kotlin/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Kotlin Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,6 +19,12 @@ license: | limitations under the License. --- +Xlang is the default serialization mode for Fory Kotlin. This page covers the basic serialization API and interoperability rules for that default mode. + +## Cross-Language Interoperability + +The following sections cover model generation, registration, and cross-language round trips in the default xlang mode. + Kotlin xlang serialization uses the JVM Fory implementation through `ForyKotlin`. Use it when Kotlin payloads must be read by another supported Fory runtime. Register portable model types with the same identity and field schema on every peer. @@ -27,7 +33,7 @@ Kotlin data classes, enums, and sealed-class models use the Kotlin integration a serializers where applicable. Exact portable carrier mappings remain defined by the [xlang type mapping](../../specification/xlang_type_mapping.md). -## Create an xlang instance +### Create a Fory Instance ```kotlin import org.apache.fory.kotlin.ForyKotlin @@ -37,7 +43,7 @@ val fory = ForyKotlin.builder() .build() ``` -## First round trip +### First round trip ```kotlin import org.apache.fory.ThreadSafeFory diff --git a/docs/object-serialization/kotlin/configuration.md b/docs/object-serialization/kotlin/configuration.md index 4d70d499fc6..f58b2d83a94 100644 --- a/docs/object-serialization/kotlin/configuration.md +++ b/docs/object-serialization/kotlin/configuration.md @@ -148,31 +148,4 @@ For xlang payloads, call `withCompatible(false)` only after verifying that every ## Security -Kotlin uses the Java configuration surface. Keep class registration enabled for production -and any untrusted payload source: - -```kotlin -val fory = ForyKotlin.builder() - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .withMaxTypeFields(512) - .withMaxTypeMetaBytes(4096) - .build() -``` - -Security-related configuration: - -- Keep `requireClassRegistration(true)` and register application classes or generated modules. -- Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. -- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input - bytes. -- Keep `withMaxUnbackedContainerItems(...)` at `8192` unless trusted compact codecs require a - larger root allowance. Zero rejects every unbacked item. -- Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits - at their defaults unless the data is not malicious and a trusted peer sends larger metadata or - many schema versions. -- Follow [Object Serialization Security](../security.md) for allow-listing and unknown-class - controls. +See [Kotlin Security](security.md) for trust boundaries, safe reader configuration, and verification. diff --git a/docs/object-serialization/kotlin/index.md b/docs/object-serialization/kotlin/index.md index 76abeddabc6..5b5910fff26 100644 --- a/docs/object-serialization/kotlin/index.md +++ b/docs/object-serialization/kotlin/index.md @@ -117,7 +117,7 @@ See [Configuration](configuration.md) for Kotlin builder setup and [Java Native Fory Kotlin is built on top of Fory Java. Most configuration options, features, and concepts from Fory Java apply directly to Kotlin. Refer to the Java documentation for: - [Configuration](../java/configuration.md) - All ForyBuilder options -- [Basic Serialization](../java/core-api.md) - Serialization patterns and APIs +- [Basic Serialization](../java/basic-serialization.md) - Serialization patterns and APIs - [Type Registration](../java/type-registration.md) - Class registration and security - [Schema Evolution](../java/schema-evolution.md) - Forward/backward compatibility - [Custom Serializers](../java/custom-serializers.md) - Implement custom serializers @@ -126,7 +126,7 @@ Fory Kotlin is built on top of Fory Java. Most configuration options, features, ## Kotlin-Specific Documentation -- [Xlang Serialization](xlang.md) - Kotlin APIs for direct cross-language object serialization +- [Basic Serialization](basic-serialization.md) - Models, APIs, and interoperability in the default xlang mode - [Configuration](configuration.md) - Kotlin-specific Fory setup requirements - [Native Serialization](native.md) - Serializing Kotlin types in JVM-native mode - [Schema Metadata](schema-metadata.md) - Kotlin annotations, nullability, references, and integer metadata @@ -134,3 +134,6 @@ Fory Kotlin is built on top of Fory Java. Most configuration options, features, - [Static Generated Serializers](static-generated-serializers.md) - KSP xlang/schema serializer generation - [Kotlin gRPC Support](../../grpc/kotlin.md) - Coroutine stubs and service bases for Fory IDL services - [Android Support](../java/android.md) - Android setup, R8 behavior, and release-build validation + +Before decoding bytes from outside the application trust boundary, read +[Kotlin Security](security.md). diff --git a/docs/object-serialization/kotlin/native.md b/docs/object-serialization/kotlin/native.md index 5b162785258..99a88b96707 100644 --- a/docs/object-serialization/kotlin/native.md +++ b/docs/object-serialization/kotlin/native.md @@ -20,7 +20,7 @@ license: | --- This page covers serialization of Kotlin-specific JVM types in native mode. For -cross-language Kotlin models, use [Kotlin Xlang Serialization](xlang.md). +cross-language Kotlin models, use [Kotlin Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability). When compatible mode is enabled, Kotlin readers use the JVM compatible-read rules for selected scalar field type changes. A matched field can read between `Boolean`, `String`, numeric scalars, diff --git a/docs/object-serialization/kotlin/security.md b/docs/object-serialization/kotlin/security.md new file mode 100644 index 00000000000..56bbafd5569 --- /dev/null +++ b/docs/object-serialization/kotlin/security.md @@ -0,0 +1,74 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Kotlin reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Kotlin uses the Java configuration surface. Keep class registration enabled for production +and any untrusted payload source: + +```kotlin +val fory = ForyKotlin.builder() + .requireClassRegistration(true) + .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .withMaxUnbackedContainerItems(8192) + .withMaxTypeFields(512) + .withMaxTypeMetaBytes(4096) + .build() +``` + +Security-related configuration: + +- Keep `requireClassRegistration(true)` and register application classes or generated modules. +- Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. +- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. +- Keep `withMaxUnbackedContainerItems(...)` at `8192` unless trusted compact codecs require a + larger root allowance. Zero rejects every unbacked item. +- Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits + at their defaults unless the data is not malicious and a trusted peer sends larger metadata or + many schema versions. +- Follow [Java Security](../java/security.md) for allow-listing and unknown-class + controls. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the Kotlin option reference and +[Java Type Registration](../java/type-registration.md) for the underlying registration API. diff --git a/docs/object-serialization/native.md b/docs/object-serialization/native.md index e85a2a05d45..d5bf9a87dc6 100644 --- a/docs/object-serialization/native.md +++ b/docs/object-serialization/native.md @@ -1,6 +1,6 @@ --- title: Native Serialization -sidebar_position: 2 +sidebar_position: 3 id: native license: | Licensed to the Apache Software Foundation (ASF) under one or more @@ -26,7 +26,7 @@ system. It is not one shared cross-language protocol. Use native mode for same-runtime traffic that needs language-specific object shapes, migration from a host serializer, or a smaller/faster format without xlang type-mapping constraints. Use -[xlang mode](xlang/index.md) whenever a different runtime must read the bytes. +[xlang mode](xlang.md) whenever a different runtime must read the bytes. ## Supported runtime families diff --git a/docs/object-serialization/python/xlang.md b/docs/object-serialization/python/basic-serialization.md similarity index 76% rename from docs/object-serialization/python/xlang.md rename to docs/object-serialization/python/basic-serialization.md index 3f9b53d19c9..489bc77929e 100644 --- a/docs/object-serialization/python/xlang.md +++ b/docs/object-serialization/python/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,12 +19,101 @@ license: | limitations under the License. --- +This page covers the Python xlang quickstart. `pyfory.Fory()` defaults to xlang mode with +compatible schema evolution; examples set `xlang=True` explicitly so the mode choice is visible. + +## Basic Object Serialization + +Serialize and deserialize Python objects with a simple API: + +```python +import pyfory + +fory = pyfory.Fory(xlang=True) + +# Serialize xlang-compatible values +data = fory.dumps({"name": "Alice", "age": 30, "scores": [95, 87, 92]}) + +# Deserialize back to Python object +obj = fory.loads(data) +print(obj) # {'name': 'Alice', 'age': 30, 'scores': [95, 87, 92]} +``` + +**Note**: `dumps()`/`loads()` are aliases for `serialize()`/`deserialize()`. Both APIs are identical, use whichever feels more intuitive. + +## Custom Class Serialization + +Use dataclasses and type annotations for stable xlang payloads: + +```python +import pyfory +from dataclasses import dataclass +from typing import List, Dict + +@dataclass +class Person: + name: str + age: pyfory.Int32 + scores: List[pyfory.Int32] + metadata: Dict[str, str] + +fory = pyfory.Fory(xlang=True, ref=True) +fory.register(Person, name="example.Person") +person = Person("Bob", 25, [88, 92, 85], {"team": "engineering"}) +data = fory.serialize(person) +result = fory.deserialize(data) +print(result) # Person(name='Bob', age=25, ...) +``` + +## Reference Tracking & Circular References + +Handle repeated references safely when the payload uses xlang-compatible types: + +```python +import pyfory + +f = pyfory.Fory(xlang=True, ref=True) + +shared = ["shared"] +value = [shared, shared] + +data = f.serialize(value) +result = f.deserialize(data) +assert result[0] is result[1] +``` + +For arbitrary Python object graphs, local classes, functions, and methods, use +[Native Serialization](native.md). + +## Performance Tips + +1. **Disable `ref=True` if not needed**: Reference tracking has overhead +2. **Use type_id instead of name**: Integer IDs are faster than string names +3. **Reuse Fory instances**: Create once, use many times +4. **Enable Cython**: Make sure `ENABLE_FORY_CYTHON_SERIALIZATION=1` + +```python +# Good: Reuse instance +fory = pyfory.Fory(xlang=True) +for obj in objects: + data = fory.dumps(obj) + +# Bad: Create new instance each time +for obj in objects: + fory = pyfory.Fory(xlang=True) # Wasteful! + data = fory.dumps(obj) +``` + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + `pyfory` supports xlang object graph serialization, allowing you to serialize data in Python and deserialize it in Java, C++, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another supported language. -## Create an Xlang Fory Instance +### Xlang Configuration Python defaults to xlang mode with compatible schema evolution. Set the mode explicitly in xlang examples: @@ -33,9 +122,9 @@ import pyfory fory = pyfory.Fory(xlang=True, ref=False, strict=True) ``` -## Xlang Example +### Xlang Example -### Python (Serializer) +#### Python (Serializer) ```python import pyfory @@ -56,7 +145,7 @@ binary_data = f.serialize(person) # binary_data can now be sent to Java, Go, etc. ``` -### Java (Deserializer) +#### Java (Deserializer) ```java import org.apache.fory.*; @@ -75,7 +164,7 @@ fory.register(Person.class, "example.Person"); Person person = (Person) fory.deserialize(binaryData); ``` -### Rust (Deserializer) +#### Rust (Deserializer) ```rust use fory::Fory; @@ -93,7 +182,7 @@ fory.register_by_name::("example.Person"); let person: Person = fory.deserialize(&binary_data)?; ``` -## Type Annotations for Xlang +### Type Annotations for Xlang Use pyfory type annotations for explicit xlang type mapping: @@ -119,7 +208,7 @@ Nested collection annotations are part of the field schema. Compatible-mode reads consume bytes with the remote schema metadata, then assign only when the decoded value safely satisfies the local schema. -## Reduced-Precision Types +### Reduced-Precision Types `pyfory.Float16` and `pyfory.BFloat16` are reserved annotation markers for xlang reduced-precision fields. They are not value wrapper classes; scalar values deserialize as native @@ -130,7 +219,7 @@ numeric values with `pyfory.Float16Array.from_values([...])` or `pyfory.BFloat16Array.from_values([...])`. Use `from_buffer(...)` and `to_buffer()` only when you already need packed little-endian `uint16` storage and want the raw-buffer fast path. -## Type Mapping +### Type Mapping | Python marker/carrier | Java | Rust | Go | | ---------------------- | -------------- | --------------- | --------------------- | @@ -147,7 +236,7 @@ already need packed little-endian `uint16` storage and want the raw-buffer fast | `list` | `List` | `Vec` | `[]T` | | `dict` | `Map` | `HashMap` | `map[K]V` | -### Lists and Dense Arrays +#### Lists and Dense Arrays Python `List[T]` maps to Fory `list`. Use `pyfory.Array[T]`, `pyfory.NDArray[T]`, or `pyfory.PyArray[T]` only when the schema is the dense @@ -197,24 +286,24 @@ carrier. Use `pyfory.format.from_arrow_schema(...)` and `pyfory.format.to_arrow_schema(...)` to convert between PyArrow schemas and Fory row-format schemas. -## Differences from Python Native Mode +### Differences from Python Native Mode The binary protocol and API are similar to `pyfory`'s Python native mode, but Python native mode can serialize any Python object—including global functions, local functions, lambdas, local classes, and types with custom serialization using `__getstate__/__reduce__/__reduce_ex__`, which are **not allowed** in xlang mode. -## See Also +### Specifications and References - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Type Mapping Reference](../../specification/xlang_type_mapping.md) -- [Java Xlang Serialization Guide](../java/xlang.md) -- [Rust Xlang Serialization Guide](../rust/xlang.md) +- [Java Interoperability Guide](../java/basic-serialization.md#cross-language-interoperability) +- [Rust Interoperability Guide](../rust/basic-serialization.md#cross-language-interoperability) -## Related Topics +### Related Guides - [Configuration](configuration.md) - xlang mode settings - [Schema Evolution](schema-evolution.md) - Compatible mode - [Type Registration](type-registration.md) - Registration patterns -## Read the Java file example +### Read the Java file example ```python import pyfory @@ -236,7 +325,7 @@ print(f"Name: {person.name}, Age: {person.age}") # Output: Name: Alice, Age: 30 ``` -## Built-in values +### Built-in values ```python import pyfory @@ -255,7 +344,7 @@ new_map = fory.deserialize(data) print(new_map) ``` -## Custom values +### Custom values ```python from dataclasses import dataclass @@ -312,7 +401,7 @@ if __name__ == "__main__": print(f.deserialize(data)) ``` -## Shared and circular references +### Shared and circular references ```python from typing import Dict @@ -332,3 +421,10 @@ data = fory.serialize(obj) # bytes can be deserialized by other languages print(fory.deserialize(data)) ``` + +## Related Topics + +- [Configuration](configuration.md) - Fory parameters +- [Type Registration](type-registration.md) - Registration patterns +- [Native Serialization](native.md) - Functions and lambdas +- [Out-of-Band Serialization](out-of-band.md) - Buffer callback APIs diff --git a/docs/object-serialization/python/configuration.md b/docs/object-serialization/python/configuration.md index 57d1cca9452..04652322e70 100644 --- a/docs/object-serialization/python/configuration.md +++ b/docs/object-serialization/python/configuration.md @@ -180,126 +180,10 @@ Use `strict=False` only for trusted data, preferably with a `policy=` deserializ ## Security -Treat native-mode bytes from untrusted sources the same way you would treat untrusted pickle bytes. -Native mode can reconstruct Python objects, import modules, invoke reduction hooks, and rebuild -dynamic classes or functions when `strict=False`. - -### Production Configuration - -Keep `strict=True` for production payloads unless the whole data source is trusted and a -`DeserializationPolicy` owns the remaining trust decisions: - -```python -import pyfory - -fory = pyfory.Fory( - xlang=True, - ref=False, - strict=True, - max_depth=50, - max_type_fields=512, - max_type_meta_bytes=4096, - max_schema_versions_per_type=10, - max_average_schema_versions_per_type=3, - max_graph_memory_bytes=128 * 1024 * 1024, -) - -fory.register(UserModel, name="example.User") -fory.register(OrderModel, name="example.Order") -``` - -Use dynamic native-mode deserialization (`strict=False`) only for trusted Python-only payloads: - -```python -import pyfory - -fory = pyfory.Fory( - xlang=False, - ref=True, - strict=False, - max_depth=100, -) -``` - -Received remote metadata is also limited: - -- `max_type_fields` limits the number of fields accepted in one received struct metadata body. -- `max_type_meta_bytes` limits the encoded body bytes accepted for one received TypeDef body. -- `max_schema_versions_per_type` limits accepted remote metadata versions for one logical type. -- `max_average_schema_versions_per_type` limits the average across accepted remote types. -- `max_graph_memory_bytes` sets an approximate gate for materialized graph memory during one root - deserialization. The estimate mainly covers lists, tuples, sets, dicts, object arrays, structs, - and Python objects. It skips leaf values such as strings, binary data, primitive scalars, and - dense primitive arrays, so actual process memory can be higher than this value. Leaf values remain - protected by byte-availability checks: if the unread input does not contain enough bytes, Fory - will not read or create that leaf value. The default is a fixed `128 MiB` for all root input - forms. Set a positive byte value for trusted payloads that legitimately need a larger or smaller - gate. -- `max_unbacked_container_items` limits collection elements and map entries whose repeated read - bodies do not consume proportional input during one root deserialization. The default is `8192`; - zero is a strict limit. - -These limits do not change `strict`, `policy`, dynamic loading, unknown-class handling, or -schema-evolution semantics. - -### DeserializationPolicy - -When `strict=False` is necessary, use `DeserializationPolicy` to restrict the dynamic types and -hooks accepted during deserialization: - -```python -import pyfory -from pyfory import DeserializationPolicy - -dangerous_modules = {"subprocess", "os", "__builtin__"} - -class SafeDeserializationPolicy(DeserializationPolicy): - def validate_class(self, cls, is_local, **kwargs): - if cls.__module__ in dangerous_modules: - raise ValueError(f"Blocked dangerous class: {cls.__module__}.{cls.__name__}") - - def intercept_reduce_call(self, callable_obj, args, **kwargs): - if getattr(callable_obj, "__name__", "") == "Popen": - raise ValueError("Blocked attempt to invoke subprocess.Popen") - return None - - def intercept_setstate(self, obj, state, **kwargs): - if isinstance(state, dict) and "password" in state: - state["password"] = "***REDACTED***" - return None - -policy = SafeDeserializationPolicy() -fory = pyfory.Fory(xlang=False, ref=True, strict=False, policy=policy) -``` - -Available policy hooks include: - -Reference validation hooks reject by raising exceptions and otherwise leave deserialized references -unchanged. - -| Hook | Description | -| -------------------------------------------- | --------------------------------------------------- | -| `validate_class(cls, is_local)` | Validate or block class types | -| `validate_module(module_name, is_local)` | Validate or block module imports | -| `validate_function(func, is_local)` | Validate or block function references | -| `validate_method(method, is_local)` | Validate or block method references | -| `intercept_reduce_call(callable_obj, args)` | Intercept `__reduce__` invocations | -| `inspect_reduced_object(obj)` | Inspect or replace objects created via `__reduce__` | -| `intercept_setstate(obj, state)` | Sanitize state before `__setstate__` | -| `authorize_instantiation(cls, args, kwargs)` | Control class instantiation | - -### Security Checklist - -- Keep `strict=True` for untrusted data. -- Register all expected application types before deserialization. -- Use `DeserializationPolicy` when `strict=False` is necessary. -- Keep `max_depth` low enough to reject unexpectedly deep payloads. -- Keep `max_graph_memory_bytes` at the fixed `128 MiB` default for most inputs, or set a positive - explicit gate for trusted workloads with different legitimate collection/map/struct sizes. -- Do not treat xlang/native mode choice as a security control. +See [Python Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) - Using configured Fory +- [Basic Serialization](basic-serialization.md) - Using configured Fory - [Type Registration](type-registration.md) - Registration patterns - [Native Serialization](native.md) - Python-only object serialization diff --git a/docs/object-serialization/python/core-api.md b/docs/object-serialization/python/core-api.md deleted file mode 100644 index a14ab689d51..00000000000 --- a/docs/object-serialization/python/core-api.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers the Python xlang quickstart. `pyfory.Fory()` defaults to xlang mode with -compatible schema evolution; examples set `xlang=True` explicitly so the mode choice is visible. - -## Basic Object Serialization - -Serialize and deserialize Python objects with a simple API: - -```python -import pyfory - -fory = pyfory.Fory(xlang=True) - -# Serialize xlang-compatible values -data = fory.dumps({"name": "Alice", "age": 30, "scores": [95, 87, 92]}) - -# Deserialize back to Python object -obj = fory.loads(data) -print(obj) # {'name': 'Alice', 'age': 30, 'scores': [95, 87, 92]} -``` - -**Note**: `dumps()`/`loads()` are aliases for `serialize()`/`deserialize()`. Both APIs are identical, use whichever feels more intuitive. - -## Custom Class Serialization - -Use dataclasses and type annotations for stable xlang payloads: - -```python -import pyfory -from dataclasses import dataclass -from typing import List, Dict - -@dataclass -class Person: - name: str - age: pyfory.Int32 - scores: List[pyfory.Int32] - metadata: Dict[str, str] - -fory = pyfory.Fory(xlang=True, ref=True) -fory.register(Person, name="example.Person") -person = Person("Bob", 25, [88, 92, 85], {"team": "engineering"}) -data = fory.serialize(person) -result = fory.deserialize(data) -print(result) # Person(name='Bob', age=25, ...) -``` - -## Reference Tracking & Circular References - -Handle repeated references safely when the payload uses xlang-compatible types: - -```python -import pyfory - -f = pyfory.Fory(xlang=True, ref=True) - -shared = ["shared"] -value = [shared, shared] - -data = f.serialize(value) -result = f.deserialize(data) -assert result[0] is result[1] -``` - -For arbitrary Python object graphs, local classes, functions, and methods, use -[Native Serialization](native.md). - -## Performance Tips - -1. **Disable `ref=True` if not needed**: Reference tracking has overhead -2. **Use type_id instead of name**: Integer IDs are faster than string names -3. **Reuse Fory instances**: Create once, use many times -4. **Enable Cython**: Make sure `ENABLE_FORY_CYTHON_SERIALIZATION=1` - -```python -# Good: Reuse instance -fory = pyfory.Fory(xlang=True) -for obj in objects: - data = fory.dumps(obj) - -# Bad: Create new instance each time -for obj in objects: - fory = pyfory.Fory(xlang=True) # Wasteful! - data = fory.dumps(obj) -``` - -## Related Topics - -- [Configuration](configuration.md) - Fory parameters -- [Type Registration](type-registration.md) - Registration patterns -- [Native Serialization](native.md) - Functions and lambdas -- [Out-of-Band Serialization](out-of-band.md) - Buffer callback APIs diff --git a/docs/object-serialization/python/custom-serializers.md b/docs/object-serialization/python/custom-serializers.md index e934c3454fc..da907d43028 100644 --- a/docs/object-serialization/python/custom-serializers.md +++ b/docs/object-serialization/python/custom-serializers.md @@ -135,4 +135,4 @@ fory.register(MyClass, name="com.example.MyClass", serializer=MySerializer(fory. - [Type Registration](type-registration.md) - Registration patterns - [Configuration](configuration.md) - Fory parameters -- [Xlang Serialization](xlang.md) - type registration and schema rules for xlang +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - type registration and schema rules for xlang diff --git a/docs/object-serialization/python/index.md b/docs/object-serialization/python/index.md index b356e36b943..cf05c7510a2 100644 --- a/docs/object-serialization/python/index.md +++ b/docs/object-serialization/python/index.md @@ -147,12 +147,11 @@ Use xlang mode for cross-language payloads and dataclass schemas shared with oth Use native mode for Python-only traffic. Native mode is selected with `xlang=False` and owns pickle/cloudpickle-style behavior such as functions, lambdas, classes, methods, `__reduce__`, `__getstate__`, and out-of-band pickle protocol 5 buffers. It is optimized for Python's type system and supports a broader Python object surface than xlang mode, so use it when replacing pickle or cloudpickle. Compatible mode is enabled by default. Set `compatible=False` only when every reader and writer uses the same Python class schema and you want faster serialization and smaller size. -See [Native Serialization](native.md) for Python-only serialization details and [Xlang Serialization](xlang.md) for Python xlang registration and interoperability rules. +See [Native Serialization](native.md) for Python-only serialization details and [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for Python xlang registration and interoperability rules. ## Next Steps -- [Basic Serialization](core-api.md) - Basic usage patterns -- [Xlang Serialization](xlang.md) - xlang mode +- [Basic Serialization](basic-serialization.md) - Default xlang APIs and interoperability - [Native Serialization](native.md) - Python-only serialization - [Configuration](configuration.md) - Fory parameters, modes, and security - [Type Registration](type-registration.md) - User-defined type registration @@ -167,3 +166,6 @@ See [Native Serialization](native.md) for Python-only serialization details and - **PyPI**: https://pypi.org/project/pyfory/ - **Issues**: https://github.com/apache/fory/issues - **Slack**: https://join.slack.com/t/fory-project/shared_invite/zt-36g0qouzm-kcQSvV_dtfbtBKHRwT5gsw + +Before decoding bytes from outside the application trust boundary, read +[Python Security](security.md). diff --git a/docs/object-serialization/python/native.md b/docs/object-serialization/python/native.md index d7d5225516a..e77e24bbeac 100644 --- a/docs/object-serialization/python/native.md +++ b/docs/object-serialization/python/native.md @@ -23,7 +23,7 @@ Python native serialization is the Python-only wire mode selected with `xlang=Fa every writer and reader is Python and the payload should follow Python's object model instead of the portable xlang type system. -Use [Xlang Serialization](xlang.md), the default Python mode, when bytes must be read +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default Python mode, when bytes must be read by Java, C++, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another non-Python Fory implementation. @@ -186,7 +186,7 @@ Keep the payload in native mode. Xlang mode does not execute Python `__reduce__` ## Related Topics -- [Xlang Serialization](xlang.md) - Cross-language Python payloads +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language Python payloads - [Configuration](configuration.md) - Python `Fory` options - [Out-of-Band Serialization](out-of-band.md) - Zero-copy buffer support -- [Object Serialization Security](../security.md) - Deserialization policies +- [Configuration](configuration.md#security) - Deserialization policies diff --git a/docs/object-serialization/python/numpy-integration.md b/docs/object-serialization/python/numpy-integration.md index 4884f6e2d50..8188696ace5 100644 --- a/docs/object-serialization/python/numpy-integration.md +++ b/docs/object-serialization/python/numpy-integration.md @@ -100,4 +100,4 @@ assert np.array_equal(array, result) ## Related Topics - [Out-of-Band Serialization](out-of-band.md) - Zero-copy buffers -- [Basic Serialization](core-api.md) - Standard usage +- [Basic Serialization](basic-serialization.md) - Standard usage diff --git a/docs/object-serialization/python/out-of-band.md b/docs/object-serialization/python/out-of-band.md index 8a3ff0399fa..8db222fecbc 100644 --- a/docs/object-serialization/python/out-of-band.md +++ b/docs/object-serialization/python/out-of-band.md @@ -175,5 +175,5 @@ for buffer_obj in buffer_objects: ## Related Topics - [NumPy Integration](numpy-integration.md) - NumPy array serialization -- [Basic Serialization](core-api.md) - Standard serialization +- [Basic Serialization](basic-serialization.md) - Standard serialization - [Configuration](configuration.md) - Fory parameters diff --git a/docs/object-serialization/python/schema-evolution.md b/docs/object-serialization/python/schema-evolution.md index 52ba957c78d..a8927c92c05 100644 --- a/docs/object-serialization/python/schema-evolution.md +++ b/docs/object-serialization/python/schema-evolution.md @@ -137,5 +137,5 @@ class SameSchemaMessage: ## Related Topics - [Configuration](configuration.md) - Compatible mode settings -- [Xlang Serialization](xlang.md) - Schema evolution across languages +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Schema evolution across languages - [Type Registration](type-registration.md) - Registration patterns diff --git a/docs/object-serialization/python/schema-metadata.md b/docs/object-serialization/python/schema-metadata.md index ad8db45551a..913cafb4f84 100644 --- a/docs/object-serialization/python/schema-metadata.md +++ b/docs/object-serialization/python/schema-metadata.md @@ -524,6 +524,6 @@ class User: ## Related Topics -- [Basic Serialization](core-api.md) - Getting started with Fory serialization +- [Basic Serialization](basic-serialization.md) - Getting started with Fory serialization - [Schema Evolution](schema-evolution.md) - Compatible mode and schema evolution -- [Xlang Serialization](xlang.md) - Interoperability with Java, Rust, C++, Go +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Interoperability with Java, Rust, C++, Go diff --git a/docs/object-serialization/python/security.md b/docs/object-serialization/python/security.md new file mode 100644 index 00000000000..db9e96c980b --- /dev/null +++ b/docs/object-serialization/python/security.md @@ -0,0 +1,163 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Python reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Treat native-mode bytes from untrusted sources the same way you would treat untrusted pickle bytes. +Native mode can reconstruct Python objects, import modules, invoke reduction hooks, and rebuild +dynamic classes or functions when `strict=False`. + +### Production Configuration + +Keep `strict=True` for production payloads unless the whole data source is trusted and a +`DeserializationPolicy` owns the remaining trust decisions: + +```python +import pyfory + +fory = pyfory.Fory( + xlang=True, + ref=False, + strict=True, + max_depth=50, + max_type_fields=512, + max_type_meta_bytes=4096, + max_schema_versions_per_type=10, + max_average_schema_versions_per_type=3, + max_graph_memory_bytes=128 * 1024 * 1024, +) + +fory.register(UserModel, name="example.User") +fory.register(OrderModel, name="example.Order") +``` + +Use dynamic native-mode deserialization (`strict=False`) only for trusted Python-only payloads: + +```python +import pyfory + +fory = pyfory.Fory( + xlang=False, + ref=True, + strict=False, + max_depth=100, +) +``` + +Received remote metadata is also limited: + +- `max_type_fields` limits the number of fields accepted in one received struct metadata body. +- `max_type_meta_bytes` limits the encoded body bytes accepted for one received TypeDef body. +- `max_schema_versions_per_type` limits accepted remote metadata versions for one logical type. +- `max_average_schema_versions_per_type` limits the average across accepted remote types. +- `max_graph_memory_bytes` sets an approximate gate for materialized graph memory during one root + deserialization. The estimate mainly covers lists, tuples, sets, dicts, object arrays, structs, + and Python objects. It skips leaf values such as strings, binary data, primitive scalars, and + dense primitive arrays, so actual process memory can be higher than this value. Leaf values remain + protected by byte-availability checks: if the unread input does not contain enough bytes, Fory + will not read or create that leaf value. The default is a fixed `128 MiB` for all root input + forms. Set a positive byte value for trusted payloads that legitimately need a larger or smaller + gate. +- `max_unbacked_container_items` limits collection elements and map entries whose repeated read + bodies do not consume proportional input during one root deserialization. The default is `8192`; + zero is a strict limit. + +These limits do not change `strict`, `policy`, dynamic loading, unknown-class handling, or +schema-evolution semantics. + +### DeserializationPolicy + +When `strict=False` is necessary, use `DeserializationPolicy` to restrict the dynamic types and +hooks accepted during deserialization: + +```python +import pyfory +from pyfory import DeserializationPolicy + +dangerous_modules = {"subprocess", "os", "__builtin__"} + +class SafeDeserializationPolicy(DeserializationPolicy): + def validate_class(self, cls, is_local, **kwargs): + if cls.__module__ in dangerous_modules: + raise ValueError(f"Blocked dangerous class: {cls.__module__}.{cls.__name__}") + + def intercept_reduce_call(self, callable_obj, args, **kwargs): + if getattr(callable_obj, "__name__", "") == "Popen": + raise ValueError("Blocked attempt to invoke subprocess.Popen") + return None + + def intercept_setstate(self, obj, state, **kwargs): + if isinstance(state, dict) and "password" in state: + state["password"] = "***REDACTED***" + return None + +policy = SafeDeserializationPolicy() +fory = pyfory.Fory(xlang=False, ref=True, strict=False, policy=policy) +``` + +Available policy hooks include: + +Reference validation hooks reject by raising exceptions and otherwise leave deserialized references +unchanged. + +| Hook | Description | +| -------------------------------------------- | --------------------------------------------------- | +| `validate_class(cls, is_local)` | Validate or block class types | +| `validate_module(module_name, is_local)` | Validate or block module imports | +| `validate_function(func, is_local)` | Validate or block function references | +| `validate_method(method, is_local)` | Validate or block method references | +| `intercept_reduce_call(callable_obj, args)` | Intercept `__reduce__` invocations | +| `inspect_reduced_object(obj)` | Inspect or replace objects created via `__reduce__` | +| `intercept_setstate(obj, state)` | Sanitize state before `__setstate__` | +| `authorize_instantiation(cls, args, kwargs)` | Control class instantiation | + +### Security Checklist + +- Keep `strict=True` for untrusted data. +- Register all expected application types before deserialization. +- Use `DeserializationPolicy` when `strict=False` is necessary. +- Keep `max_depth` low enough to reject unexpectedly deep payloads. +- Keep `max_graph_memory_bytes` at the fixed `128 MiB` default for most inputs, or set a positive + explicit gate for trusted workloads with different legitimate collection/map/struct sizes. +- Do not treat xlang/native mode choice as a security control. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/python/troubleshooting.md b/docs/object-serialization/python/troubleshooting.md index bda885242b0..11928d08c5f 100644 --- a/docs/object-serialization/python/troubleshooting.md +++ b/docs/object-serialization/python/troubleshooting.md @@ -192,4 +192,4 @@ ruff check --fix . - [Configuration](configuration.md) - Fory parameters - [Type Registration](type-registration.md) - Registration best practices -- [Object Serialization Security](../security.md) - Security configuration +- [Configuration](configuration.md#security) - Security configuration diff --git a/docs/object-serialization/python/type-registration.md b/docs/object-serialization/python/type-registration.md index 5c6a8f34da8..917537c767d 100644 --- a/docs/object-serialization/python/type-registration.md +++ b/docs/object-serialization/python/type-registration.md @@ -19,7 +19,7 @@ license: | limitations under the License. --- -This page covers Python type registration APIs. Use [Object Serialization Security](../security.md) for +This page covers Python type registration APIs. Use [Configuration](configuration.md#security) for strict-mode policy, max-depth limits, and trusted-data guidance. ## Type Registration @@ -88,5 +88,5 @@ application object. ## Related Topics - [Configuration](configuration.md) - Fory parameters -- [Object Serialization Security](../security.md) - Strict mode, deserialization policies, and maximum read depth +- [Configuration](configuration.md#security) - Strict mode, deserialization policies, and maximum read depth - [Custom Serializers](custom-serializers.md) - Custom serialization diff --git a/docs/object-serialization/runtimes.md b/docs/object-serialization/runtimes.md deleted file mode 100644 index c820c929d97..00000000000 --- a/docs/object-serialization/runtimes.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Object Serialization Runtimes -sidebar_position: 3 -id: runtimes -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Choose the runtime whose API your application uses. Runtime documentation is scoped to -binary object serialization; Row Format, Fory JSON, compiler, and gRPC pages remain with -their product owners. - -| Runtime | Modes | Documentation | -| --------------------- | -------------------- | ------------------------------------------------------ | -| Java | xlang and native | [Java runtime](./java/index.md) | -| Python | xlang and native | [Python runtime](./python/index.md) | -| C++ | xlang and native | [C++ runtime](./cpp/index.md) | -| Go | xlang and native | [Go runtime](./go/index.md) | -| Rust | xlang and native | [Rust runtime](./rust/index.md) | -| JavaScript/TypeScript | xlang | [JavaScript/TypeScript runtime](./javascript/index.md) | -| C# | xlang | [C# runtime](./csharp/index.md) | -| Swift | xlang | [Swift runtime](./swift/index.md) | -| Dart | xlang | [Dart runtime](./dart/index.md) | -| Scala | xlang and JVM native | [Scala runtime](./scala/index.md) | -| Kotlin | xlang and JVM native | [Kotlin runtime](./kotlin/index.md) | diff --git a/docs/object-serialization/rust/xlang.md b/docs/object-serialization/rust/basic-serialization.md similarity index 53% rename from docs/object-serialization/rust/xlang.md rename to docs/object-serialization/rust/basic-serialization.md index c3b573a6399..b8fcf9e873c 100644 --- a/docs/object-serialization/rust/xlang.md +++ b/docs/object-serialization/rust/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,10 +19,204 @@ license: | limitations under the License. --- +This page covers basic object graph serialization and supported types in the default xlang mode for Fory Rust. + +## Object Graph Serialization + +Apache Fory™ provides automatic serialization of complex object graphs, preserving the structure and relationships between objects. The `#[derive(ForyStruct)]` macro generates efficient serialization code at compile time, eliminating reflection overhead. + +**Key capabilities:** + +- Nested struct serialization with arbitrary depth +- Collection types (Vec, HashMap, HashSet, BTreeMap) +- Optional fields with `Option` +- Automatic handling of primitive types and strings +- Efficient binary encoding with variable-length integers + +```rust +use fory::{Fory, Error}; +use fory::ForyStruct; +use std::collections::HashMap; + +#[derive(ForyStruct, Debug, PartialEq)] +struct Person { + name: String, + age: i32, + address: Address, + hobbies: Vec, + metadata: HashMap, +} + +#[derive(ForyStruct, Debug, PartialEq)] +struct Address { + street: String, + city: String, + country: String, +} + +let mut fory = Fory::builder().xlang(true).build(); +fory.register_by_name::
("example.Address").unwrap(); +fory.register_by_name::("example.Person").unwrap(); + +let person = Person { + name: "John Doe".to_string(), + age: 30, + address: Address { + street: "123 Main St".to_string(), + city: "New York".to_string(), + country: "USA".to_string(), + }, + hobbies: vec!["reading".to_string(), "coding".to_string()], + metadata: HashMap::from([ + ("role".to_string(), "developer".to_string()), + ]), +}; + +let bytes = fory.serialize(&person).unwrap(); +let decoded: Person = fory.deserialize(&bytes)?; +assert_eq!(person, decoded); +``` + +## Supported Types + +### Primitive Types + +| Rust Type | Description | +| ------------------------- | --------------------------- | +| `bool` | Boolean | +| `i8`, `i16`, `i32`, `i64` | Signed integers | +| `f32`, `f64` | Floating point | +| `BFloat16` | 16-bit brain floating point | +| `String` | UTF-8 string | + +### Collections + +| Rust Type | Description | +| ---------------- | ------------------ | +| `Vec` | Dynamic array | +| `VecDeque` | Double-ended queue | +| `LinkedList` | Doubly-linked list | +| `HashMap` | Hash map | +| `BTreeMap` | Ordered map | +| `HashSet` | Hash set | +| `BTreeSet` | Ordered set | +| `BinaryHeap` | Binary heap | +| `Option` | Optional value | + +`Vec` is the dense carrier when the schema is `array`. + +### Smart Pointers + +| Rust Type | Description | +| ------------ | ---------------------------------------------------- | +| `Box` | Heap allocation | +| `Rc` | Reference counting (shared refs tracked) | +| `Arc` | Thread-safe reference counting (shared refs tracked) | +| `RcWeak` | Weak reference to `Rc` (breaks circular refs) | +| `ArcWeak` | Weak reference to `Arc` (breaks circular refs) | +| `RefCell` | Interior mutability (runtime borrow checking) | +| `Mutex` | Thread-safe interior mutability | + +### Date and Time + +| Rust Type | Description | +| ----------- | ------------------------------------------------------- | +| `Date` | Date without timezone, stored as epoch days | +| `Timestamp` | Point in time, stored as epoch seconds and nanos | +| `Duration` | Signed duration, stored as seconds and normalized nanos | + +The built-in carriers expose dependency-free constructors, accessors, conversions, and checked +arithmetic: + +```rust +use fory::{Date, Duration, Timestamp}; + +let date = Date::from_epoch_days(19_782); +assert_eq!(date.checked_add_days(1)?.epoch_days(), 19_783); + +let timestamp = Timestamp::from_epoch_millis(-1); +assert_eq!(timestamp.to_epoch_millis()?, -1); + +let duration = Duration::from_parts(1, 1_500_000_000)?; +assert_eq!(duration.to_millis()?, 2_500); +let later = timestamp.checked_add_duration(duration)?; +``` + +`chrono::NaiveDate`, `chrono::NaiveDateTime`, and `chrono::Duration` are supported when the Rust +`chrono` feature is enabled: + +```toml +[dependencies] +fory = { version = "1.5.0", features = ["chrono"] } +``` + +### Custom Types + +Use `#[derive(ForyStruct)]` for object graph serialization. The separate +[Rust Row Format guide](../../row-format/rust.md) documents `#[derive(ForyRow)]` and its supported +type set. + +## Serialization APIs + +```rust +use fory::{Fory, Reader}; + +let mut fory = Fory::builder().xlang(true).build(); +fory.register::(1)?; + +let obj = MyStruct { /* ... */ }; + +// Basic serialize/deserialize +let bytes = fory.serialize(&obj)?; +let decoded: MyStruct = fory.deserialize(&bytes)?; + +// Serialize to existing buffer +let mut buf: Vec = vec![]; +fory.serialize_to(&mut buf, &obj)?; + +// Deserialize from reader +let mut reader = Reader::new(&buf); +let decoded: MyStruct = fory.deserialize_from(&mut reader)?; +``` + +When the Rust value type uses an external structural serializer or custom +serializer, select it explicitly at the root: + +```rust +let bytes = fory.serialize_with::(&user)?; +let decoded: third_party::User = + fory.deserialize_with::(&bytes)?; +``` + +Carrier serializers compose the same selection for a root container: + +```rust +use fory::VecSerializer; + +let bytes = + fory.serialize_with::>(&users)?; +let decoded: Vec = + fory.deserialize_with::>(&bytes)?; +``` + +See [External-Type Serialization](external-types.md) for field annotations, +all supported carriers, and registration. + +## Performance Tips + +- **Buffer Pre-allocation**: Minimizes memory allocations during serialization +- **Compact Encoding**: Variable-length encoding for space efficiency +- **Little-Endian**: Optimized for modern CPU architectures +- **Reference Deduplication**: Shared objects serialized only once + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Apache Fory™ supports seamless data exchange across Java, Python, C++, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. -## Create an Xlang Fory Instance +### Xlang Configuration Rust defaults to xlang mode with compatible schema evolution. Set the mode explicitly in xlang examples: @@ -39,9 +233,9 @@ fory.register::(100)?; // fory.register_by_name::("com.example.MyStruct")?; ``` -## Type Registration for Xlang +### Type Registration for Xlang -### Register by ID +#### Register by ID For fast, compact serialization with consistent IDs across languages: @@ -51,7 +245,7 @@ let mut fory = Fory::builder().xlang(true).build(); fory.register::(100)?; // Same ID in Java, Python, etc. ``` -### Register by Name +#### Register by Name For more flexible type naming: @@ -59,9 +253,9 @@ For more flexible type naming: fory.register_by_name::("com.example.User")?; ``` -## Xlang Example +### Xlang Example -### Rust (Serializer) +#### Rust (Serializer) ```rust use fory::Fory; @@ -86,7 +280,7 @@ let bytes = fory.serialize(&person)?; // bytes can be deserialized by Java, Python, etc. ``` -### Third-Party Rust Types +#### Third-Party Rust Types An external structural serializer gives a third-party Rust type the same xlang schema as an equivalent local derive: @@ -120,7 +314,7 @@ multiple tuple or named fields is supported with `xlang(false)`, but its serializer registration is rejected in xlang mode. See [External-Type Serialization](external-types.md). -### Dynamic Rust Carriers +#### Dynamic Rust Carriers `Box`, `Rc`, `Arc`, and application `dyn Trait` carriers can be used in xlang mode when every selected concrete @@ -128,7 +322,7 @@ target has an xlang-compatible structural or EXT identity. Fory writes the concrete registered target identity; the Rust trait or erased-carrier identity does not appear on the wire. -### Java (Deserializer) +#### Java (Deserializer) ```java import org.apache.fory.*; @@ -149,7 +343,7 @@ fory.register(Person.class, 100); // Same ID as Rust Person person = (Person) fory.deserialize(bytesFromRust); ``` -### Python (Deserializer) +#### Python (Deserializer) ```python import pyfory @@ -166,11 +360,11 @@ fory.register_type(Person, type_id=100) # Same ID as Rust person = fory.deserialize(bytes_from_rust) ``` -## Type Mapping +### Type Mapping See [xlang_type_mapping.md](../../specification/xlang_type_mapping.md) for complete type mapping across languages. -### Common Type Mappings +#### Common Type Mappings | Rust | Java | Python | | --------------- | -------------- | --------------- | @@ -189,7 +383,7 @@ See [xlang_type_mapping.md](../../specification/xlang_type_mapping.md) for compl | `HashMap` | `Map` | `Dict[K,V]` | | `Option` | nullable `T` | `Optional[T]` | -### Lists and Dense Arrays +#### Lists and Dense Arrays Rust `Vec` maps to Fory `list` by default for manual structs. Use an explicit array field attribute when the schema is dense `array`. @@ -211,28 +405,28 @@ explicit array field attribute when the schema is dense `array`. | `array` | `#[fory(array)] Vec` | | `array` | `#[fory(array)] Vec` | -## Best Practices +### Interoperability Best Practices 1. **Use consistent type IDs** across all languages 2. **Keep compatible mode** for schema evolution 3. **Register all types** before serialization 4. **Test cross-language** compatibility during development -## See Also +### Specifications and References - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Type Mapping Reference](../../specification/xlang_type_mapping.md) -- [Java Xlang Serialization Guide](../java/xlang.md) -- [Python Xlang Serialization Guide](../python/xlang.md) +- [Java Interoperability Guide](../java/basic-serialization.md#cross-language-interoperability) +- [Python Interoperability Guide](../python/basic-serialization.md#cross-language-interoperability) -## Related Topics +### Related Guides - [Configuration](configuration.md) - xlang mode configuration - [Schema Evolution](schema-evolution.md) - Compatible mode - [Type Registration](type-registration.md) - Registration methods - [External-Type Serialization](external-types.md) - Third-party values in xlang mode -## Built-in values +### Built-in values ```rust use fory::Fory; @@ -245,7 +439,7 @@ fn run() { } ``` -## Custom values +### Custom values ```rust use chrono::{NaiveDate, NaiveDateTime}; @@ -309,6 +503,14 @@ fn complex_struct() { } ``` -## Shared and circular references +### Shared and circular references Circular references cannot be implemented in Rust due to ownership restrictions. + +## Related Topics + +- [Type Registration](type-registration.md) - Registering types +- [References](references.md) - Shared and circular references +- [Custom Serializers](custom-serializers.md) - Custom serialization +- [External-Type Serialization](external-types.md) - Third-party values and carrier roots +- [Row Format](../../row-format/rust.md) - Standard Row Format and zero-copy borrowed views diff --git a/docs/object-serialization/rust/configuration.md b/docs/object-serialization/rust/configuration.md index 8edeb023d0a..f5fb04eefc5 100644 --- a/docs/object-serialization/rust/configuration.md +++ b/docs/object-serialization/rust/configuration.md @@ -201,21 +201,10 @@ Use `.compatible(false)` only when the schema used to deserialize every payload ## Security -Security-related configuration: - -- Register application structs and trait-object implementations before deserializing untrusted - payloads. -- Use `max_dyn_depth(...)` to reject unexpectedly deep dynamic object graphs. -- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a - positive byte gate for trusted workloads with different legitimate collection/map/struct sizes. -- Keep `max_unbacked_container_items(...)` at `8192` unless trusted compact codecs require a - larger root allowance. Zero rejects every unbacked item. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer concrete typed fields over `dyn Any` or broad trait-object fields for untrusted input. +See [Rust Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) - Using configured Fory +- [Basic Serialization](basic-serialization.md) - Using configured Fory - [Schema Evolution](schema-evolution.md) - Compatible mode details -- [Xlang Serialization](xlang.md) - xlang mode +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - xlang mode diff --git a/docs/object-serialization/rust/core-api.md b/docs/object-serialization/rust/core-api.md deleted file mode 100644 index 5bce4c68e4d..00000000000 --- a/docs/object-serialization/rust/core-api.md +++ /dev/null @@ -1,218 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers basic object graph serialization and supported types. - -## Object Graph Serialization - -Apache Fory™ provides automatic serialization of complex object graphs, preserving the structure and relationships between objects. The `#[derive(ForyStruct)]` macro generates efficient serialization code at compile time, eliminating reflection overhead. - -**Key capabilities:** - -- Nested struct serialization with arbitrary depth -- Collection types (Vec, HashMap, HashSet, BTreeMap) -- Optional fields with `Option` -- Automatic handling of primitive types and strings -- Efficient binary encoding with variable-length integers - -```rust -use fory::{Fory, Error}; -use fory::ForyStruct; -use std::collections::HashMap; - -#[derive(ForyStruct, Debug, PartialEq)] -struct Person { - name: String, - age: i32, - address: Address, - hobbies: Vec, - metadata: HashMap, -} - -#[derive(ForyStruct, Debug, PartialEq)] -struct Address { - street: String, - city: String, - country: String, -} - -let mut fory = Fory::builder().xlang(true).build(); -fory.register_by_name::
("example.Address").unwrap(); -fory.register_by_name::("example.Person").unwrap(); - -let person = Person { - name: "John Doe".to_string(), - age: 30, - address: Address { - street: "123 Main St".to_string(), - city: "New York".to_string(), - country: "USA".to_string(), - }, - hobbies: vec!["reading".to_string(), "coding".to_string()], - metadata: HashMap::from([ - ("role".to_string(), "developer".to_string()), - ]), -}; - -let bytes = fory.serialize(&person).unwrap(); -let decoded: Person = fory.deserialize(&bytes)?; -assert_eq!(person, decoded); -``` - -## Supported Types - -### Primitive Types - -| Rust Type | Description | -| ------------------------- | --------------------------- | -| `bool` | Boolean | -| `i8`, `i16`, `i32`, `i64` | Signed integers | -| `f32`, `f64` | Floating point | -| `BFloat16` | 16-bit brain floating point | -| `String` | UTF-8 string | - -### Collections - -| Rust Type | Description | -| ---------------- | ------------------ | -| `Vec` | Dynamic array | -| `VecDeque` | Double-ended queue | -| `LinkedList` | Doubly-linked list | -| `HashMap` | Hash map | -| `BTreeMap` | Ordered map | -| `HashSet` | Hash set | -| `BTreeSet` | Ordered set | -| `BinaryHeap` | Binary heap | -| `Option` | Optional value | - -`Vec` is the dense carrier when the schema is `array`. - -### Smart Pointers - -| Rust Type | Description | -| ------------ | ---------------------------------------------------- | -| `Box` | Heap allocation | -| `Rc` | Reference counting (shared refs tracked) | -| `Arc` | Thread-safe reference counting (shared refs tracked) | -| `RcWeak` | Weak reference to `Rc` (breaks circular refs) | -| `ArcWeak` | Weak reference to `Arc` (breaks circular refs) | -| `RefCell` | Interior mutability (runtime borrow checking) | -| `Mutex` | Thread-safe interior mutability | - -### Date and Time - -| Rust Type | Description | -| ----------- | ------------------------------------------------------- | -| `Date` | Date without timezone, stored as epoch days | -| `Timestamp` | Point in time, stored as epoch seconds and nanos | -| `Duration` | Signed duration, stored as seconds and normalized nanos | - -The built-in carriers expose dependency-free constructors, accessors, conversions, and checked -arithmetic: - -```rust -use fory::{Date, Duration, Timestamp}; - -let date = Date::from_epoch_days(19_782); -assert_eq!(date.checked_add_days(1)?.epoch_days(), 19_783); - -let timestamp = Timestamp::from_epoch_millis(-1); -assert_eq!(timestamp.to_epoch_millis()?, -1); - -let duration = Duration::from_parts(1, 1_500_000_000)?; -assert_eq!(duration.to_millis()?, 2_500); -let later = timestamp.checked_add_duration(duration)?; -``` - -`chrono::NaiveDate`, `chrono::NaiveDateTime`, and `chrono::Duration` are supported when the Rust -`chrono` feature is enabled: - -```toml -[dependencies] -fory = { version = "1.5.0", features = ["chrono"] } -``` - -### Custom Types - -Use `#[derive(ForyStruct)]` for object graph serialization. The separate -[Rust Row Format guide](../../row-format/rust.md) documents `#[derive(ForyRow)]` and its supported -type set. - -## Serialization APIs - -```rust -use fory::{Fory, Reader}; - -let mut fory = Fory::builder().xlang(true).build(); -fory.register::(1)?; - -let obj = MyStruct { /* ... */ }; - -// Basic serialize/deserialize -let bytes = fory.serialize(&obj)?; -let decoded: MyStruct = fory.deserialize(&bytes)?; - -// Serialize to existing buffer -let mut buf: Vec = vec![]; -fory.serialize_to(&mut buf, &obj)?; - -// Deserialize from reader -let mut reader = Reader::new(&buf); -let decoded: MyStruct = fory.deserialize_from(&mut reader)?; -``` - -When the Rust value type uses an external structural serializer or custom -serializer, select it explicitly at the root: - -```rust -let bytes = fory.serialize_with::(&user)?; -let decoded: third_party::User = - fory.deserialize_with::(&bytes)?; -``` - -Carrier serializers compose the same selection for a root container: - -```rust -use fory::VecSerializer; - -let bytes = - fory.serialize_with::>(&users)?; -let decoded: Vec = - fory.deserialize_with::>(&bytes)?; -``` - -See [External-Type Serialization](external-types.md) for field annotations, -all supported carriers, and registration. - -## Performance Tips - -- **Buffer Pre-allocation**: Minimizes memory allocations during serialization -- **Compact Encoding**: Variable-length encoding for space efficiency -- **Little-Endian**: Optimized for modern CPU architectures -- **Reference Deduplication**: Shared objects serialized only once - -## Related Topics - -- [Type Registration](type-registration.md) - Registering types -- [References](references.md) - Shared and circular references -- [Custom Serializers](custom-serializers.md) - Custom serialization -- [External-Type Serialization](external-types.md) - Third-party values and carrier roots -- [Row Format](../../row-format/rust.md) - Standard Row Format and zero-copy borrowed views diff --git a/docs/object-serialization/rust/index.md b/docs/object-serialization/rust/index.md index 00a29191c86..6f1156060e1 100644 --- a/docs/object-serialization/rust/index.md +++ b/docs/object-serialization/rust/index.md @@ -101,7 +101,7 @@ Use xlang mode for cross-language payloads and schemas shared with other Fory im Use native mode for Rust-only traffic. Native mode is selected with `.xlang(false)` and keeps Rust object serialization in Rust-native form. It supports native-only concrete targets and data-enum shapes that have no xlang representation. Dynamic `Any`, application trait, and shared-reference carriers can also be used in xlang mode when every selected concrete target is xlang-compatible. Compatible mode is enabled by default. Set `.compatible(false)` only when every reader and writer uses the same Rust schema and you want faster serialization and smaller size. -See [Xlang Serialization](xlang.md) for Rust xlang registration and interoperability rules, and [Native Serialization](native.md) for Rust-only payloads. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for Rust xlang registration and interoperability rules, and [Native Serialization](native.md) for Rust-only payloads. ## Thread Safety @@ -158,8 +158,7 @@ fn main() -> Result<(), Error> { ## Next Steps - [Configuration](configuration.md) - Fory builder options and modes -- [Basic Serialization](core-api.md) - Object graph serialization -- [Xlang Serialization](xlang.md) - xlang mode +- [Basic Serialization](basic-serialization.md) - Default xlang object graphs and interoperability - [Native Serialization](native.md) - Rust-only serialization - [References](references.md) - Shared and circular references - [Polymorphism](polymorphism.md) - Trait object serialization @@ -168,3 +167,6 @@ fn main() -> Result<(), Error> { plus carrier composition - [Row Format](../../row-format/rust.md) - Standard Row Format with borrowed views - [gRPC Support](../../grpc/rust.md) - Fory payloads over tonic + +Before decoding bytes from outside the application trust boundary, read +[Rust Security](security.md). diff --git a/docs/object-serialization/rust/native.md b/docs/object-serialization/rust/native.md index feaaf0cf33a..084d6ab118d 100644 --- a/docs/object-serialization/rust/native.md +++ b/docs/object-serialization/rust/native.md @@ -23,7 +23,7 @@ Rust native serialization is the Rust-only wire mode selected with `.xlang(false every writer and reader is Rust and the payload should preserve Rust object-graph behavior instead of the portable xlang type system. -Use [Xlang Serialization](xlang.md), the default Rust mode, when bytes must be read +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default Rust mode, when bytes must be read by Java, Python, C++, Go, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another non-Rust Fory implementation. @@ -115,7 +115,7 @@ Native serialization owns the Rust-specific object surface: is appropriate. - Date and time carriers, including optional `chrono` support. -Use [Basic Serialization](core-api.md), [References](references.md), and +Use [Basic Serialization](basic-serialization.md), [References](references.md), and [Trait Object Serialization](polymorphism.md) for focused examples. ## Shared And Circular References @@ -264,9 +264,9 @@ Native serialization defaults to compatible mode. Keep that default when schemas ## Related Topics -- [Xlang Serialization](xlang.md) - Cross-language Rust payloads +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language Rust payloads - [Configuration](configuration.md) - Builder options -- [Basic Serialization](core-api.md) - Object graph serialization +- [Basic Serialization](basic-serialization.md) - Object graph serialization - [Shared & Circular References](references.md) - `Rc`, `Arc`, and weak pointers - [Trait Object Serialization](polymorphism.md) - Trait objects and dynamic dispatch - [Schema Evolution](schema-evolution.md) - Compatible mode diff --git a/docs/object-serialization/rust/references.md b/docs/object-serialization/rust/references.md index 7bc0e584c30..4cbbc01f794 100644 --- a/docs/object-serialization/rust/references.md +++ b/docs/object-serialization/rust/references.md @@ -208,6 +208,6 @@ for child in &decoded.lock().unwrap().children { ## Related Topics -- [Basic Serialization](core-api.md) - Supported types +- [Basic Serialization](basic-serialization.md) - Supported types - [Polymorphism](polymorphism.md) - Trait objects with Rc/Arc - [Configuration](configuration.md) - Reference tracking options diff --git a/docs/object-serialization/rust/schema-evolution.md b/docs/object-serialization/rust/schema-evolution.md index 29e3624be67..81505ac2bf9 100644 --- a/docs/object-serialization/rust/schema-evolution.md +++ b/docs/object-serialization/rust/schema-evolution.md @@ -278,5 +278,5 @@ assert_eq!(data, decoded); - [Configuration](configuration.md) - Compatible mode settings - [Polymorphism](polymorphism.md) - Trait objects with schema evolution -- [Xlang Serialization](xlang.md) - Schema evolution across languages +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Schema evolution across languages - [External-Type Serialization](external-types.md) - Compatible schemas for third-party values diff --git a/docs/object-serialization/rust/schema-metadata.md b/docs/object-serialization/rust/schema-metadata.md index ebf226e1954..249f9a906a9 100644 --- a/docs/object-serialization/rust/schema-metadata.md +++ b/docs/object-serialization/rust/schema-metadata.md @@ -497,7 +497,7 @@ struct User { ## Related Topics -- [Basic Serialization](core-api.md) - Getting started with Fory serialization +- [Basic Serialization](basic-serialization.md) - Getting started with Fory serialization - [Schema Evolution](schema-evolution.md) - Compatible mode and schema evolution -- [Xlang Serialization](xlang.md) - Interoperability with Java, C++, Go, Python +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Interoperability with Java, C++, Go, Python - [External-Type Serialization](external-types.md) - Selecting serializers for third-party fields diff --git a/docs/object-serialization/rust/security.md b/docs/object-serialization/rust/security.md new file mode 100644 index 00000000000..7a17d95fecf --- /dev/null +++ b/docs/object-serialization/rust/security.md @@ -0,0 +1,58 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Rust reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register application structs and trait-object implementations before deserializing untrusted + payloads. +- Use `max_dyn_depth(...)` to reject unexpectedly deep dynamic object graphs. +- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a + positive byte gate for trusted workloads with different legitimate collection/map/struct sizes. +- Keep `max_unbacked_container_items(...)` at `8192` unless trusted compact codecs require a + larger root allowance. Zero rejects every unbacked item. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer concrete typed fields over `dyn Any` or broad trait-object fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/rust/type-registration.md b/docs/object-serialization/rust/type-registration.md index 3efe057ae30..162eb4a9541 100644 --- a/docs/object-serialization/rust/type-registration.md +++ b/docs/object-serialization/rust/type-registration.md @@ -136,6 +136,6 @@ let handles: Vec<_> = (0..4) ## Related Topics - [Configuration](configuration.md) - Fory builder options -- [Xlang Serialization](xlang.md) - xlang mode registration +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - xlang mode registration - [Custom Serializers](custom-serializers.md) - Custom serialization - [External-Type Serialization](external-types.md) - Third-party targets and carrier roots diff --git a/docs/object-serialization/scala/xlang.md b/docs/object-serialization/scala/basic-serialization.md similarity index 93% rename from docs/object-serialization/scala/xlang.md rename to docs/object-serialization/scala/basic-serialization.md index e49a86a4b93..c1768d45e56 100644 --- a/docs/object-serialization/scala/xlang.md +++ b/docs/object-serialization/scala/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Scala Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,11 +19,17 @@ license: | limitations under the License. --- +Xlang is the default serialization mode for Fory Scala. This page covers the basic serialization API and interoperability rules for that default mode. + +## Cross-Language Interoperability + +The following sections cover model generation, registration, and cross-language round trips in the default xlang mode. + The Fory schema IDL Scala target generates Scala 3 source for xlang payloads. The Fory Scala artifact remains cross-built for Scala 2.13 and Scala 3; only the schema IDL output and quoted macro derivation require Scala 3. -## Setup +### Setup Generated Scala code uses the public macro API in `org.apache.fory.scala` and the shared JVM annotations in `org.apache.fory.annotation`. Macro internals live @@ -55,7 +61,7 @@ serializers or Scala-specific registration state in Java core. Enums and unions are registered with their serializers directly because their derived serializers own case dispatch. -## Generated Messages +### Generated Messages Acyclic messages generate case classes: @@ -107,7 +113,7 @@ supported mutable collection interfaces such as `scala.collection.Seq` and `scala.collection.Map`, but concrete mutable collection classes are outside the schema IDL surface unless explicitly generated. -## Generated Enums +### Generated Enums IDL enums generate Scala 3 enums only. The compiler does not emit Java enum files. @@ -127,7 +133,7 @@ enum Status { Generated registration uses `ScalaSerializers.registerEnum(...)` so the stable Fory enum IDs from case-level `@ForyEnumId` metadata are used in xlang mode. -## Generated Unions +### Generated Unions IDL unions generate Scala 3 ADT enums with macro-derived serializers: @@ -167,7 +173,7 @@ locally. The macro writes the existing xlang union envelope directly. It does not allocate temporary Java `Union` carriers. -## Manual Scala 3 Derivation +### Manual Scala 3 Derivation Manual Scala 3 models can derive the same serializer typeclass: @@ -192,7 +198,7 @@ constructor-owned value that participates in the cycle, such as a Scala enum case or case class, the serializer fails with a clear error because no copied identity can be published until construction has completed. -## First round trip +### First round trip ```scala import org.apache.fory.Fory diff --git a/docs/object-serialization/scala/configuration.md b/docs/object-serialization/scala/configuration.md index ae81c3cf661..2524eaeb0e1 100644 --- a/docs/object-serialization/scala/configuration.md +++ b/docs/object-serialization/scala/configuration.md @@ -192,31 +192,4 @@ For xlang payloads, call `withCompatible(false)` only after verifying that every ## Security -Scala uses the Java configuration surface. Keep class registration enabled for production -and any untrusted payload source: - -```scala -val fory = ForyScala.builder() - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .withMaxTypeFields(512) - .withMaxTypeMetaBytes(4096) - .build() -``` - -Security-related configuration: - -- Keep `requireClassRegistration(true)` and register application classes or generated modules. -- Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. -- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input - bytes. -- Keep `withMaxUnbackedContainerItems(...)` at `8192` unless trusted compact codecs require a - larger root allowance. Zero rejects every unbacked item. -- Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits - at their defaults unless the data is not malicious and a trusted peer sends larger metadata or - many schema versions. -- Follow [Object Serialization Security](../security.md) for allow-listing and unknown-class - controls. +See [Scala Security](security.md) for trust boundaries, safe reader configuration, and verification. diff --git a/docs/object-serialization/scala/index.md b/docs/object-serialization/scala/index.md index 0d60f59af67..de1d3efec94 100644 --- a/docs/object-serialization/scala/index.md +++ b/docs/object-serialization/scala/index.md @@ -105,7 +105,7 @@ See [Configuration](configuration.md) for Scala builder setup and [Java Native S Fory Scala is built on top of Fory Java. Most configuration options, features, and concepts from Fory Java apply directly to Scala. Refer to the Java documentation for: - [Configuration](../java/configuration.md) - All ForyBuilder options -- [Basic Serialization](../java/core-api.md) - Serialization patterns and APIs +- [Basic Serialization](../java/basic-serialization.md) - Serialization patterns and APIs - [Type Registration](../java/type-registration.md) - Class registration and security - [Schema Evolution](../java/schema-evolution.md) - Forward/backward compatibility - [Custom Serializers](../java/custom-serializers.md) - Implement custom serializers @@ -118,5 +118,8 @@ Fory Scala is built on top of Fory Java. Most configuration options, features, a - [Native Serialization](native.md) - Serializing Scala types in JVM-native mode - [Schema Metadata](schema-metadata.md) - Scala annotations, references, enum IDs, and union metadata - [Default Values](default-values.md) - Scala class default values support -- [Xlang Serialization](xlang.md) - Scala 3 generated models and macro-derived xlang serializers +- [Basic Serialization](basic-serialization.md) - Models, APIs, and interoperability in the default xlang mode - [gRPC Support](../../grpc/scala.md) - Scala 3 generated gRPC service companions + +Before decoding bytes from outside the application trust boundary, read +[Scala Security](security.md). diff --git a/docs/object-serialization/scala/native.md b/docs/object-serialization/scala/native.md index e3992c87730..20fd963b978 100644 --- a/docs/object-serialization/scala/native.md +++ b/docs/object-serialization/scala/native.md @@ -21,7 +21,7 @@ license: | This page covers serialization of Scala-specific JVM types in native mode. For cross-language Scala models, use the xlang path described in -[Xlang Serialization](xlang.md). +[Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability). When compatible mode is enabled, Scala readers use the JVM compatible-read rules for selected scalar field type changes. A matched field can read between `Boolean`, `String`, numeric scalars, diff --git a/docs/object-serialization/scala/schema-metadata.md b/docs/object-serialization/scala/schema-metadata.md index 1d04dce8ab9..d630690824c 100644 --- a/docs/object-serialization/scala/schema-metadata.md +++ b/docs/object-serialization/scala/schema-metadata.md @@ -119,6 +119,6 @@ reflection is not the source of truth for generated Scala metadata. ## Related Topics -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Configuration](configuration.md) - [Default Values](default-values.md) diff --git a/docs/object-serialization/scala/security.md b/docs/object-serialization/scala/security.md new file mode 100644 index 00000000000..f5f40c61c78 --- /dev/null +++ b/docs/object-serialization/scala/security.md @@ -0,0 +1,74 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Scala reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Scala uses the Java configuration surface. Keep class registration enabled for production +and any untrusted payload source: + +```scala +val fory = ForyScala.builder() + .requireClassRegistration(true) + .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .withMaxUnbackedContainerItems(8192) + .withMaxTypeFields(512) + .withMaxTypeMetaBytes(4096) + .build() +``` + +Security-related configuration: + +- Keep `requireClassRegistration(true)` and register application classes or generated modules. +- Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. +- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. +- Keep `withMaxUnbackedContainerItems(...)` at `8192` unless trusted compact codecs require a + larger root allowance. Zero rejects every unbacked item. +- Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits + at their defaults unless the data is not malicious and a trusted peer sends larger metadata or + many schema versions. +- Follow [Java Security](../java/security.md) for allow-listing and unknown-class + controls. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the Scala option reference and +[Java Type Registration](../java/type-registration.md) for the underlying registration API. diff --git a/docs/object-serialization/security.md b/docs/object-serialization/security.md deleted file mode 100644 index 28789dc4661..00000000000 --- a/docs/object-serialization/security.md +++ /dev/null @@ -1,210 +0,0 @@ ---- -title: Object Serialization Security -sidebar_position: 4 -id: security -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This guide defines the trust boundary and safe operating model for Fory binary object serialization in xlang and native mode. Contributor-facing classification rules live in the [deserialization security model](deserialization-security-model.md). - -Fory is an in-process serialization library. Applications link Fory into their -own process, configure serializers and type policies, and call Fory APIs to -serialize application-owned objects or deserialize encoded Fory data. Fory does -not provide a standalone network service, daemon, authentication system, or -transport protocol. - -## Trust Boundaries - -Fory's primary security boundary is encoded bytes or streams passed to -deserialization APIs from untrusted or partially trusted sources. The embedding -application owns where those bytes come from and which Fory configuration, -registered types, schemas, and policies are used to read them. - -The adversary model for untrusted deserialization is a sender that can craft -encoded bytes or stream behavior presented to a Fory read API. It does not assume -the sender can change the embedding application's Fory configuration, registered -type set, `TypeChecker` or equivalent allow-list policy, schema definitions, -classloader, or other active policy objects unless the application itself exposes -those controls. - -Fory security boundaries include: - -- Runtime safety, including avoiding crashes, panics, undefined behavior, and - out-of-bounds memory access. -- Resource ownership, including memory, CPU progress, stream buffers, native - allocations, callbacks, and retained read-side state. -- Explicit Fory policy checks, such as class, type, function, method, - registration, or deserialization policies that restrict what may be - materialized. -- Cleanup boundaries, where state created during a failed root operation must - not leak into later operations. - -Runtime serializer code generation and JIT compilation are not paths for -executing encoded input. They operate on types and schemas after the active -registration check, `TypeChecker`, schema check, or policy check has accepted the -type surface. When class registration is disabled, `TypeChecker` or an -equivalent allow-list policy is the relevant gate. Generated serializer code is -derived from checked type descriptors rather than from attacker-controlled byte -contents. - -The [deserialization security model](deserialization-security-model.md) defines how to -classify these boundaries for untrusted deserialization paths. - -## Non-Goals - -Fory does not provide: - -- Encoded-data authenticity, integrity, confidentiality, signing, MACs, or - encryption. -- Transport security or protection for bytes while they are stored or moved - outside Fory. -- Application-level authorization or validation for the business meaning of a - successfully deserialized value. -- A sandbox for user-registered classes, functions, constructors, setters, - finalizers, or other application-owned logic. - -Applications that receive Fory data from untrusted sources should authenticate -or integrity-check those bytes before passing them to Fory when authenticity or -tamper resistance matters. - -## Downstream Responsibilities - -Applications are responsible for: - -- Choosing whether a byte source is trusted enough for the configured - deserialization mode. -- Keeping class or type registration enabled for untrusted data unless another - explicit Fory policy owns the accepted type surface. -- Registering only types and serializers that are safe for the application's - trust boundary. -- Configuring depth and resource limits for the largest data shape the - application intends to accept. -- Treating cross-language peers and schemas as part of the application's trust - relationship. - -Disabling registration or using dynamic deserialization on trusted data is a -configuration choice. For untrusted data, bypassing an explicit Fory policy, -crashing, leaking resources, retaining attacker-controlled state, or allocating -disproportionately remains security-relevant as described in the -[deserialization security model](deserialization-security-model.md). - -## Resource Limits - -### Depth Limits - -Set the runtime's depth limit to the deepest graph the application deliberately -accepts. Some runtimes apply this limit to every nested value; others apply a -separate dynamic-object depth limit. Use the selected runtime's configuration -page for its exact scope and default. A depth limit prevents excessively nested -input from turning into unbounded recursion, but it is not a byte or memory -quota. - -### Graph Memory Limit - -`maxGraphMemoryBytes`, or the runtime-equivalent option, is an approximate gate -for graph owners materialized by one root deserialization operation. The fixed -default is 128 MiB, and explicit values must be positive. Each root operation -starts with the full configured budget, including after a failed read. - -The budget covers runtime-owned collections, maps, arrays, structs, and objects -according to each implementation's storage model. It is not exact heap -accounting, an input-size limit, or a replacement for readable-byte checks. -Actual process memory may be higher. Keep external body or file-size limits at -the boundary that receives the bytes. - -### Remote Schema Metadata Limits - -Compatible mode may receive remote metadata (`TypeDef` or `TypeMeta`) for types that are not already -known by the reader. Fory limits how many distinct remote metadata versions can be accepted, and -also limits the size of each received metadata body: - -- `maxSchemaVersionsPerType`: maximum accepted remote metadata versions for one logical type. The - default is `10`. -- `maxAverageSchemaVersionsPerType`: average accepted remote metadata versions across all accepted - remote types. The default is `3`; the effective global floor is `8192` metadata entries. -- `maxTypeFields`: maximum fields declared by one received struct metadata body. The default is - `512`. -- `maxTypeMetaBytes`: maximum encoded metadata body bytes for one received TypeDef or TypeMeta body, - excluding the 8-byte header and any extended-size varint. The default is `4096`. - -These limits are resource protections. They do not change wire format, registration requirements, -dynamic type loading, unknown-type handling, or schema-evolution compatibility. - -Raise these values only when a known peer deliberately sends larger metadata or -many schema versions. - -### Count-Driven Container Work Limit - -Every runtime limits collection elements and map entries whose repeated read -bodies do not consume proportional input. The default root allowance is `8192`. -Zero is a strict limit, and negative values are rejected. Raise the limit only -for trusted payloads that intentionally use compact zero-byte element codecs or -empty Struct bodies. This is a reader resource limit and does not change the -wire format or writer behavior. - -## Configure a Runtime - -Keep registration enabled for untrusted input, choose the wire mode explicitly, -and set limits to values derived from the endpoint's accepted models. A minimal -Java boundary looks like this: - -```java -Fory fory = - Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .build(); -``` - -Register only the application types that the endpoint accepts. If registration -is disabled, configure the runtime's explicit type checker or allow-list before -reading external data. - -Exact option names, defaults, and mode-specific behavior belong to the runtime -configuration guides: - -| Runtime | Configuration | -| --------------------- | ------------------------------------------------------- | -| Java | [Java configuration](java/configuration.md) | -| Python | [Python configuration](python/configuration.md) | -| C++ | [C++ configuration](cpp/configuration.md) | -| Go | [Go configuration](go/configuration.md) | -| Rust | [Rust configuration](rust/configuration.md) | -| JavaScript/TypeScript | [JavaScript configuration](javascript/configuration.md) | -| C# | [C# configuration](csharp/configuration.md) | -| Swift | [Swift configuration](swift/configuration.md) | -| Dart | [Dart configuration](dart/configuration.md) | -| Scala | [Scala configuration](scala/configuration.md) | -| Kotlin | [Kotlin configuration](kotlin/configuration.md) | - -## Verify the Boundary - -Add negative tests alongside the normal round trip. Verify that the configured -reader rejects: - -- an unregistered or disallowed application type; -- a graph deeper than the accepted model; -- a graph that exceeds the configured memory budget; -- excessive remote schema versions or metadata size in compatible xlang mode; -- excessive count-driven container work; and -- a malformed root followed by a valid root on the same reusable runtime. - -Also verify the application's external authentication, integrity, request-size, -timeout, and domain-validation controls independently of Fory. diff --git a/docs/object-serialization/swift/xlang.md b/docs/object-serialization/swift/basic-serialization.md similarity index 55% rename from docs/object-serialization/swift/xlang.md rename to docs/object-serialization/swift/basic-serialization.md index b49b2e798ed..1f1eaadb3f7 100644 --- a/docs/object-serialization/swift/xlang.md +++ b/docs/object-serialization/swift/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,17 +19,177 @@ license: | limitations under the License. --- +This page covers object graph serialization and core API usage in the default xlang mode for Fory Swift. + +## Object Graph Serialization + +Use `@ForyStruct`, `@ForyEnum`, or `@ForyUnion`, register types, then serialize and deserialize. + +```swift +import Foundation +import Fory + +@ForyStruct +struct Address: Equatable { + var street: String = "" + var zip: Int32 = 0 +} + +@ForyStruct +struct Person: Equatable { + var id: Int64 = 0 + var name: String = "" + var nickname: String? = nil + var tags: Set = [] + var scores: [Int32] = [] + var addresses: [Address] = [] + var metadata: [Int8: Int32?] = [:] +} + +let fory = Fory() +try fory.register(Address.self, id: 100) +try fory.register(Person.self, id: 101) + +let person = Person( + id: 42, + name: "Alice", + nickname: nil, + tags: ["swift", "xlang"], + scores: [10, 20, 30], + addresses: [Address(street: "Main", zip: 94107)], + metadata: [1: 100, 2: nil] +) + +let data = try fory.serialize(person) +let decoded: Person = try fory.deserialize(data) +assert(decoded == person) +``` + +## Working with Existing Buffers + +Append serialized bytes to an existing `Data` and deserialize from `ByteBuffer`. + +```swift +var output = Data() +try fory.serialize(person, to: &output) + +let inputBuffer = ByteBuffer(data: output) +let fromBuffer: Person = try fory.deserialize(from: inputBuffer) +assert(fromBuffer == person) +``` + +## Selecting a Serializer + +A type that implements `Serializer` with `Target == Self` selects itself: + +```swift +let data = try fory.serialize(person) +let decoded: Person = try fory.deserialize(data) +``` + +This implicit selection composes through generated fields and ordinary +optionals, arrays, sets, and dictionaries. It also applies when an application +intentionally gives an external type one retroactive self-target conformance. + +When a separate serializer targets the value, select it with `with`: + +```swift +try fory.register(UserSerializer.self, id: 200) + +let data = try fory.serialize( + externalUser, + with: UserSerializer.self +) +let decoded = try fory.deserialize( + data, + with: UserSerializer.self +) +``` + +The same selection works with existing buffers: + +```swift +var output = Data() +try fory.serialize( + externalUser, + with: UserSerializer.self, + to: &output +) + +let input = ByteBuffer(data: output) +let decoded = try fory.deserialize( + from: input, + with: UserSerializer.self +) +``` + +See [External-Type Serialization](external-types.md) for structural +serializers and recursive carrier roots. See +[Custom Serializers](custom-serializers.md) for serializers implemented +directly by a type, retroactive conformances, and separate custom serializers. + +## Built-in Supported Types + +### Primitive and scalar + +- `Bool` +- `Int8`, `Int16`, `Int32`, `Int64`, `Int` +- `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt` +- `Float`, `Double` +- `String` +- `Data` + +### Date and time + +- `Date` +- `LocalDate` +- `Duration` + +Use `Date` for timestamp values and `LocalDate` for day-only dates. `LocalDate` +supports epoch-day and `Date` conversions through `fromEpochDay(_:)`, +`toEpochDay()`, `init(utcDate:)`, and `toUTCDate()`. + +### Collections + +- Optionals and arrays whose values directly implement `Serializer` +- Sets whose elements directly implement `Serializer` and are `Hashable` +- Dictionaries whose keys and values directly implement `Serializer`, with + `Hashable` keys + +Children that use a separate serializer compose with: + +- `OptionalSerializer` +- `ArraySerializer` +- `SetSerializer` +- `DictionarySerializer` + +### Dynamic + +- `Any` and `AnyObject` +- `AnyHashable` +- Arbitrary application protocol values +- Supported heterogeneous arrays and dictionaries + +`Any` and `AnyObject` roots use direct root APIs. Arbitrary application +protocol roots and dynamic values nested in carriers use explicit `with:` +selection. +See [Polymorphism and Dynamic Types](polymorphism.md). + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Fory Swift can exchange payloads with other Fory implementations using the xlang protocol. -## Recommended Xlang Configuration +### Recommended Xlang Configuration ```swift let fory = Fory() ``` -## Register Types with Shared Identity +### Register Types with Shared Identity -### ID-based registration +#### ID-based registration ```swift @ForyStruct @@ -42,13 +202,13 @@ let fory = Fory() try fory.register(Order.self, id: 100) ``` -### Name-based registration +#### Name-based registration ```swift try fory.register(Order.self, name: "com.example.Order") ``` -## Xlang Rules +### Xlang Rules - Keep type registration mapping consistent across languages - Keep compatible mode enabled when independently evolving schemas. Swift enables it by default. @@ -58,7 +218,7 @@ try fory.register(Order.self, name: "com.example.Order") intentional retroactive self-target conformance for a type owned by another module -## Lists and Dense Arrays +### Lists and Dense Arrays Swift `Array` fields map to Fory `list` unless field metadata explicitly requests dense `array`. Use `array` only for one-dimensional bool or @@ -84,7 +244,7 @@ numeric data. An array that uses a separate element serializer still uses normal list encoding. Use `@ArrayField` only for supported dense bool or numeric arrays. -## External Targets +### External Targets External structural serializers produce the same xlang STRUCT, ENUM, or UNION schema and value bytes as an equivalent ordinary Swift model: @@ -110,7 +270,7 @@ Swift has no native serialization mode. A known `@ForyUnion` case has zero or one associated value. Use a struct payload for a union alternative with multiple logical fields. -## Swift IDL Workflow +### Swift IDL Workflow Generate Swift models directly from Fory IDL/Proto/FBS inputs: @@ -135,7 +295,7 @@ let payload = try fory.serialize(book) let decoded: Addressbook.AddressBook = try fory.deserialize(payload) ``` -### Run Swift IDL Integration Tests +#### Run Swift IDL Integration Tests ```bash cd integration_tests/idl_tests @@ -144,7 +304,7 @@ cd integration_tests/idl_tests This runs Swift roundtrip matrix tests and Java peer roundtrip checks (`IDL_PEER_LANG=swift`). -## Debugging Xlang Tests +### Debugging Xlang Tests Enable debug output when running xlang tests: @@ -152,7 +312,7 @@ Enable debug output when running xlang tests: ENABLE_FORY_DEBUG_OUTPUT=1 FORY_SWIFT_JAVA_CI=1 mvn -T16 test -Dtest=org.apache.fory.xlang.SwiftXlangTest ``` -## First round trip +### First round trip ```swift import Fory @@ -175,7 +335,7 @@ print("\(result.name) \(result.age)") For more cross-language rules and examples, see: -- [Cross-Language Serialization Guide](../xlang/index.md) +- [Cross-Language Interoperability](../xlang.md) - [Java Guide](../java/index.md) - [Python Guide](../python/index.md) - [Dart Guide](../dart/index.md) diff --git a/docs/object-serialization/swift/configuration.md b/docs/object-serialization/swift/configuration.md index cffcedab157..1d3bae58ad0 100644 --- a/docs/object-serialization/swift/configuration.md +++ b/docs/object-serialization/swift/configuration.md @@ -157,13 +157,4 @@ let fory = Fory(compatible: false) ## Security -Security-related configuration: - -- Register only the expected generated models before deserializing untrusted payloads. -- Use `checkClassVersion` with `compatible: false` for intentional same-schema payloads. -- Set `maxDepth` for the largest dynamic `Any` nesting depth your service accepts. -- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, class, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input - bytes. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. +See [Swift Security](security.md) for trust boundaries, safe reader configuration, and verification. diff --git a/docs/object-serialization/swift/core-api.md b/docs/object-serialization/swift/core-api.md deleted file mode 100644 index bdff028ddeb..00000000000 --- a/docs/object-serialization/swift/core-api.md +++ /dev/null @@ -1,176 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers object graph serialization and core API usage in Swift. - -## Object Graph Serialization - -Use `@ForyStruct`, `@ForyEnum`, or `@ForyUnion`, register types, then serialize and deserialize. - -```swift -import Foundation -import Fory - -@ForyStruct -struct Address: Equatable { - var street: String = "" - var zip: Int32 = 0 -} - -@ForyStruct -struct Person: Equatable { - var id: Int64 = 0 - var name: String = "" - var nickname: String? = nil - var tags: Set = [] - var scores: [Int32] = [] - var addresses: [Address] = [] - var metadata: [Int8: Int32?] = [:] -} - -let fory = Fory() -try fory.register(Address.self, id: 100) -try fory.register(Person.self, id: 101) - -let person = Person( - id: 42, - name: "Alice", - nickname: nil, - tags: ["swift", "xlang"], - scores: [10, 20, 30], - addresses: [Address(street: "Main", zip: 94107)], - metadata: [1: 100, 2: nil] -) - -let data = try fory.serialize(person) -let decoded: Person = try fory.deserialize(data) -assert(decoded == person) -``` - -## Working with Existing Buffers - -Append serialized bytes to an existing `Data` and deserialize from `ByteBuffer`. - -```swift -var output = Data() -try fory.serialize(person, to: &output) - -let inputBuffer = ByteBuffer(data: output) -let fromBuffer: Person = try fory.deserialize(from: inputBuffer) -assert(fromBuffer == person) -``` - -## Selecting a Serializer - -A type that implements `Serializer` with `Target == Self` selects itself: - -```swift -let data = try fory.serialize(person) -let decoded: Person = try fory.deserialize(data) -``` - -This implicit selection composes through generated fields and ordinary -optionals, arrays, sets, and dictionaries. It also applies when an application -intentionally gives an external type one retroactive self-target conformance. - -When a separate serializer targets the value, select it with `with`: - -```swift -try fory.register(UserSerializer.self, id: 200) - -let data = try fory.serialize( - externalUser, - with: UserSerializer.self -) -let decoded = try fory.deserialize( - data, - with: UserSerializer.self -) -``` - -The same selection works with existing buffers: - -```swift -var output = Data() -try fory.serialize( - externalUser, - with: UserSerializer.self, - to: &output -) - -let input = ByteBuffer(data: output) -let decoded = try fory.deserialize( - from: input, - with: UserSerializer.self -) -``` - -See [External-Type Serialization](external-types.md) for structural -serializers and recursive carrier roots. See -[Custom Serializers](custom-serializers.md) for serializers implemented -directly by a type, retroactive conformances, and separate custom serializers. - -## Built-in Supported Types - -### Primitive and scalar - -- `Bool` -- `Int8`, `Int16`, `Int32`, `Int64`, `Int` -- `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt` -- `Float`, `Double` -- `String` -- `Data` - -### Date and time - -- `Date` -- `LocalDate` -- `Duration` - -Use `Date` for timestamp values and `LocalDate` for day-only dates. `LocalDate` -supports epoch-day and `Date` conversions through `fromEpochDay(_:)`, -`toEpochDay()`, `init(utcDate:)`, and `toUTCDate()`. - -### Collections - -- Optionals and arrays whose values directly implement `Serializer` -- Sets whose elements directly implement `Serializer` and are `Hashable` -- Dictionaries whose keys and values directly implement `Serializer`, with - `Hashable` keys - -Children that use a separate serializer compose with: - -- `OptionalSerializer` -- `ArraySerializer` -- `SetSerializer` -- `DictionarySerializer` - -### Dynamic - -- `Any` and `AnyObject` -- `AnyHashable` -- Arbitrary application protocol values -- Supported heterogeneous arrays and dictionaries - -`Any` and `AnyObject` roots use direct root APIs. Arbitrary application -protocol roots and dynamic values nested in carriers use explicit `with:` -selection. -See [Polymorphism and Dynamic Types](polymorphism.md). diff --git a/docs/object-serialization/swift/index.md b/docs/object-serialization/swift/index.md index 718bea78605..135c02a5e6e 100644 --- a/docs/object-serialization/swift/index.md +++ b/docs/object-serialization/swift/index.md @@ -52,8 +52,7 @@ targets: [ ## Guide Contents - [Configuration](configuration.md) -- [Basic Serialization](core-api.md) -- [Xlang Serialization](xlang.md) +- [Basic Serialization](basic-serialization.md) - [Schema Metadata](schema-metadata.md) - [Type Registration](type-registration.md) - [External-Type Serialization](external-types.md) @@ -83,3 +82,6 @@ let output: User = try fory.deserialize(data) assert(input == output) ``` + +Before decoding bytes from outside the application trust boundary, read +[Swift Security](security.md). diff --git a/docs/object-serialization/swift/security.md b/docs/object-serialization/swift/security.md new file mode 100644 index 00000000000..a03e6c58b56 --- /dev/null +++ b/docs/object-serialization/swift/security.md @@ -0,0 +1,56 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Swift reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected generated models before deserializing untrusted payloads. +- Use `checkClassVersion` with `compatible: false` for intentional same-schema payloads. +- Set `maxDepth` for the largest dynamic `Any` nesting depth your service accepts. +- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, class, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/xlang.md b/docs/object-serialization/xlang.md new file mode 100644 index 00000000000..bfd8ea7b4f3 --- /dev/null +++ b/docs/object-serialization/xlang.md @@ -0,0 +1,367 @@ +--- +title: Xlang Serialization +sidebar_position: 2 +id: xlang +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Xlang is Fory's default object serialization mode. It uses one portable binary format across Java, +Python, C++, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. Each runtime's +[Basic Serialization](#runtime-guides) page owns its API and model examples; this page explains the +rules that peers must share. + +Read [Core Concepts](core-concepts.md) first for the object graph, schema, reference, and +polymorphism concepts shared by xlang and native modes. + +## Overview + +Use xlang serialization when bytes cross runtime boundaries, including polyglot services, data +pipelines, and frontend/backend communication. It provides: + +- Direct serialization of native language models without requiring an IDL. +- Coordinated numeric or named identities for application types. +- Compatible schema evolution for independently deployed peers. +- Optional shared-reference and circular-reference preservation. +- Polymorphic values when every concrete type has a portable mapping. +- Out-of-band buffers for large binary and numeric data where the runtime supports them. + +Use [Native Serialization](native.md) instead when every writer and reader uses the same supported +runtime and the object graph needs language-specific behavior such as Java serialization hooks or +Python pickle-compatible objects. + +### Supported Runtimes + +| Runtime | Package or target | Modes | +| --------------------- | ------------------------------------------ | ------------ | +| Java | `org.apache.fory:fory-core` | xlang/native | +| Python | `pyfory` | xlang/native | +| C++ | Fory C++ CMake or Bazel target | xlang/native | +| Go | `github.com/apache/fory/go/fory` | xlang/native | +| Rust | `fory` crate | xlang/native | +| JavaScript/TypeScript | `@apache-fory/core` | xlang | +| C# | `Apache.Fory` | xlang | +| Swift | `Fory` Swift Package Manager target | xlang | +| Dart | `fory` package | xlang | +| Scala | `org.apache.fory:fory-scala` | xlang/native | +| Kotlin | `org.apache.fory:fory-kotlin` and Java API | xlang/native | + +### First Cross-Language Round Trip + +Register the same logical type identity and compatible fields on every peer. The following example +uses a shared type name. + +Rust producer: + +```rust +use fory::{Fory, ForyStruct}; + +#[derive(ForyStruct)] +struct Person { + name: String, + age: i32, +} + +let mut fory = Fory::builder().xlang(true).build(); +fory.register_by_name::("example.Person").unwrap(); + +let person = Person { + name: "Alice".to_string(), + age: 30, +}; +let bytes = fory.serialize(&person).unwrap(); +``` + +Python consumer: + +```python +from dataclasses import dataclass +import pyfory + +@dataclass +class Person: + name: str + age: pyfory.Int32 + +fory = pyfory.Fory(xlang=True) +fory.register_type(Person, name="example.Person") +person = fory.deserialize(bytes_from_rust) +``` + +Although xlang is the default, examples often select it explicitly so the transport contract is +visible in application code. + +### Native Models or Fory IDL + +| Approach | Use when | +| --------------------------------- | ---------------------------------------------------------------------- | +| Native language models | The contract is small and teams want to start without a compiler step | +| [Fory IDL](../compiler/index.md) | Many messages or teams need one schema and generated models | +| Native first, then migrate to IDL | A small contract is becoming a long-lived, multi-team service boundary | + +A minimal Fory IDL message looks like this: + +```protobuf +package example; + +message Person { + string name = 1; + int32 age = 2; + optional string email = 3; +} +``` + +Generate the required runtime targets with `foryc`; generated models use consistent field and type +metadata across those targets. + +## Type System and Type Identity + +### Built-in and Custom Types + +Primitive numeric values, strings, binary values, temporal values, lists, sets, maps, dense numeric +arrays, enums, structs, and unions have shared xlang schemas. Built-in values do not require user +registration. Application structs, enums, unions, and extension types require a coordinated +identity. + +The normative [type mapping](../specification/xlang_type_mapping.md) defines the exact host-language +carrier for every xlang type. Important cases include: + +- Python uses markers such as `pyfory.Int32`, `pyfory.Float16`, and `pyfory.BFloat16` when the native + Python type does not express the required width. +- Java, Dart, and other runtimes use annotations or schema metadata where one host type can represent + multiple xlang types. +- Reduced-precision `float16` and `bfloat16` values and dense arrays use runtime-specific carriers. +- `list` and dense `array` are distinct schemas. In compatible mode, a direct struct field may + adapt between a list and dense bool/numeric array when the element domain is compatible and the + actual list contains no unrepresentable null or reference-tracked element. + +Use the mapping specification instead of inferring compatibility from similar host-language names. + +### Coordinate Type Identity + +Every peer must register a custom type with either the same numeric ID or the same namespace and type +name. Numeric IDs produce smaller metadata; names are easier to coordinate across independently +owned services. Do not register one peer by ID and another by name for the same contract. + +Registration must happen before the first root serialization or deserialization operation. Keep a +small contract registry or use generated Fory IDL modules when multiple teams own the peers. + +### Static and Dynamic Fields + +A statically known field uses its declared serializer without writing a concrete runtime type. A +dynamic field carries enough type information to select the concrete registered type. Dynamic +metadata is needed for interfaces, abstract types, trait objects, and other polymorphic positions; +it is unnecessary for primitives and exact final types. + +| Runtime | Dynamic field model | +| ------- | ----------------------------------------------------------------- | +| Java | `@ForyField(dynamic = ...)` controls automatic or forced metadata | +| Python | `pyfory.field(dynamic=...)` controls object-field metadata | +| C++ | `fory::F(...).dynamic(...)` overrides automatic detection | +| Go | Interface fields express dynamic values | +| Rust | Trait-object carriers express dynamic values | + +Writing dynamic metadata costs space and type-resolution work. Disable it only when the field can +never contain another concrete type. Exact annotation and registration examples belong to each +runtime's schema metadata, type registration, and polymorphism pages. + +## Nullability and Reference Tracking + +Nullability and reference tracking solve different problems: + +| Concern | Purpose | +| ------------------ | ------------------------------------------------------------- | +| Nullability | Allows a field or value position to contain no value | +| Reference tracking | Preserves repeated object identity and supports object cycles | + +The wire framing is defined by the +[xlang serialization specification](../specification/xlang_serialization_spec.md). Applications +should configure the semantic behavior through runtime APIs rather than depend on flag values. + +### Nullability + +Xlang struct fields are non-nullable by default. Non-nullable fields are smaller and make required +data explicit. Nullable or optional carriers include Java boxed/annotated values, Python +`Optional[T]`, C++ `std::optional`, Go pointers, Rust `Option`, and Scala `Option[T]`. + +Keep corresponding fields consistently nullable across peers. In same-schema mode, changing +nullability changes the schema and is incompatible. Compatible mode supports the documented +nullable and scalar adaptations, but a remote null still cannot be materialized into a local type +that has no valid null or missing-value behavior. + +### Shared and Circular References + +Enable reference tracking when the graph contains the same object more than once or contains a +cycle. Leave it disabled for value-shaped data to avoid identity-table overhead. + +```java +import org.apache.fory.Fory; +import org.apache.fory.annotation.Ref; + +public class Node { + public String value; + @Ref public Node next; +} + +Node first = new Node(); +Node second = new Node(); +first.next = second; +second.next = first; + +Fory fory = Fory.builder() + .withXlang(true) + .withRefTracking(true) + .build(); +``` + +Global reference tracking enables the runtime mechanism; field metadata selects which positions +participate. Common field-level controls are Java and Scala `@Ref`, Go `fory:"ref"` tags, Rust +`#[fory(ref = true)]`, and C++ smart-pointer or `fory::F().ref()` metadata. Consult the runtime guide +because default tracking differs by carrier and language. + +Reference support also follows the host ownership model. For example, Rust can preserve supported +shared-reference carriers, while cycles require representable ownership and weak-reference shapes. + +## Polymorphism + +Xlang polymorphism preserves a value's concrete registered type when its declared field, collection +element, or root type is broader. Every receiving peer must: + +1. Register the same concrete type identity. +2. Provide a compatible field schema for that concrete type. +3. Mark or model the position as dynamic when the runtime cannot infer it. +4. Use a concrete type that has a portable xlang mapping. + +Host-language inheritance alone does not make a type portable. If a shape has no xlang mapping, use +the runtime's native mode for same-language traffic or define a portable model. See the runtime +polymorphism pages for interfaces, trait objects, unions, and generated-code syntax. + +## Schema Evolution + +Compatible mode is the xlang default. It carries schema metadata so independently deployed readers +can tolerate supported field additions, removals, reordering, and documented compatible type +adaptations. + +Keep compatible mode when peers may deploy separately. Select same-schema mode only when every +reader and writer uses the same: + +- Type identity and field IDs or names. +- Field types and nested generic shapes. +- Nullability and reference metadata. +- Polymorphic alternatives. + +Same-schema mode reduces metadata and payload size, but any mismatch can produce a schema hash or +type error. Generated Fory IDL models make exact coordination easier when all peers are released +together. Normative compatibility behavior lives in the +[xlang serialization specification](../specification/xlang_serialization_spec.md). + +## Zero-Copy Serialization + +Some runtimes can move large binary or numeric buffers out of the main serialized byte stream. This +avoids copying those buffers into one contiguous payload. + +The transport flow is: + +1. Serialize the object graph and collect selected buffer objects through a callback. +2. Send the main metadata bytes and the collected buffers separately. +3. Provide the buffers in the same order during deserialization. + +Java: + +```java +Collection objects = new ArrayList<>(); +byte[] metadata = fory.serialize(value, object -> !objects.add(object)); +List buffers = objects.stream() + .map(BufferObject::toBuffer) + .toList(); +Object decoded = fory.deserialize(metadata, buffers); +``` + +Python: + +```python +objects = [] +metadata = fory.serialize(value, buffer_callback=objects.append) +buffers = [obj.to_buffer() for obj in objects] +decoded = fory.deserialize(metadata, buffers=buffers) +``` + +Go exposes the equivalent callback-buffer flow through its serialization and buffer APIs. Use the +runtime documentation for the current method names and supported buffer carriers. + +Out-of-band serialization helps when buffers are large and the transport can send them without an +additional copy. For small arrays, callback and multi-buffer transport overhead may cost more than +copying. The application owns buffer ordering, lifetime, and transport framing. See +[Python Out-of-Band Serialization](python/out-of-band.md) for Python and NumPy details. + +## Troubleshooting + +| Symptom | Likely cause | Resolution | +| ---------------------------------------- | --------------------------------------------------- | -------------------------------------------------------------- | +| Type is not registered | Registration missing or performed too late | Register every custom type before the first root operation | +| Type ID or name mismatch | Peers use different identities | Use the same numeric ID or the same namespace and type name | +| Integer overflow or float precision loss | Host carriers use different numeric widths | Follow the type mapping and use explicit width metadata | +| Fields decode incorrectly | Field IDs, names, or types differ | Align field metadata or regenerate all peers from the same IDL | +| Stack overflow on a cyclic graph | Reference tracking is disabled | Enable global and field-level reference tracking | +| Shared objects become duplicates | The value position does not track references | Enable reference tracking for that carrier or field | +| Unsupported host type | The type has no portable xlang representation | Use a portable model or native mode for same-language traffic | +| Schema/hash mismatch | Same-schema peers have different schemas | Align every peer or restore compatible mode | +| Failure after an upgrade | Peers run incompatible protocol versions | Align supported Fory versions and review release notes | +| Payload rejected immediately | One peer wrote native bytes and another reads xlang | Keep all peers on xlang for a cross-language contract | + +### Diagnostic Checklist + +1. Confirm that every peer uses xlang mode and a mutually supported Fory version. +2. Compare the registered type identity, field IDs or names, numeric widths, nullability, and + reference metadata. +3. Reproduce a same-runtime round trip before testing the cross-runtime direction. +4. Test both directions for every language pair used in production. +5. Reduce the value to one type and field, then add fields back until the mismatch appears. +6. Inspect the runtime-specific troubleshooting page for generated-code, platform, or API errors. + +When diagnosing binary layout, use the specifications and runtime debug facilities. Do not treat a +hex dump or internal flag value as a stable application API. + +## Runtime Guides + +- [Java](java/basic-serialization.md#cross-language-interoperability) +- [Python](python/basic-serialization.md#cross-language-interoperability) +- [C++](cpp/basic-serialization.md#cross-language-interoperability) +- [Go](go/basic-serialization.md#cross-language-interoperability) +- [Rust](rust/basic-serialization.md#cross-language-interoperability) +- [JavaScript/TypeScript](javascript/basic-serialization.md#cross-language-interoperability) +- [C#](csharp/basic-serialization.md#cross-language-interoperability) +- [Swift](swift/basic-serialization.md#cross-language-interoperability) +- [Dart](dart/basic-serialization.md#cross-language-interoperability) +- [Scala](scala/basic-serialization.md#cross-language-interoperability) +- [Kotlin](kotlin/basic-serialization.md#cross-language-interoperability) + +## Related Documentation + +- [Xlang Serialization Format](../specification/xlang_serialization_spec.md) — normative wire format +- [Xlang Type Mapping](../specification/xlang_type_mapping.md) — exact runtime carrier mappings +- [Fory IDL and Compiler](../compiler/index.md) — schema-first models and code generation +- [Getting Started](../start/index.md) — installation and first serialization for each runtime +- [Row Format](../row-format/index.md) — random-access analytical rows for trusted data + +## Operational Best Practices + +1. Coordinate one type identity and field contract across every peer. +2. Keep compatible mode unless all readers and writers deploy the same schema together. +3. Enable reference tracking only for identity-bearing or cyclic graphs. +4. Reuse configured Fory instances instead of rebuilding them for each operation. +5. Validate every production language pair in both directions before deployment. +6. Prefer Fory IDL once a contract spans many messages, services, or independently owned teams. diff --git a/docs/object-serialization/xlang/_category_.json b/docs/object-serialization/xlang/_category_.json deleted file mode 100644 index d2418b5e5e0..00000000000 --- a/docs/object-serialization/xlang/_category_.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "position": 1, - "label": "Xlang", - "collapsible": true, - "collapsed": true -} diff --git a/docs/object-serialization/xlang/index.md b/docs/object-serialization/xlang/index.md deleted file mode 100644 index 2777ce9d598..00000000000 --- a/docs/object-serialization/xlang/index.md +++ /dev/null @@ -1,192 +0,0 @@ ---- -title: Xlang Serialization Guide -sidebar_position: 0 -id: index -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Apache Fory™ xlang serialization is the default wire format for cross-language payloads. Serialize -data in one language and deserialize it in another without manual conversion. You can use direct -language model types for small contracts, or use Fory IDL and code generation when a schema-first -workflow is a better fit. - -## Features - -- **No IDL required**: Serialize objects directly with language model types. -- **Multi-language support**: Java, Python, C++, Go, Rust, - JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin interoperate through - the same xlang format. -- **Reference support**: Shared and circular references work across language boundaries when reference tracking is enabled in each peer. -- **Schema evolution**: Compatible mode is the xlang default so readers can tolerate added, removed, or reordered fields. -- **Out-of-band buffers**: Language implementations can expose zero-copy buffer paths for large binary data. -- **High performance**: Fory implementations use generated serializers, JIT serializers, or optimized code paths where available. - -## Supported Languages - -| Language | Status | Package or target | -| --------------------- | --------- | -------------------------------- | -| Java | Supported | `org.apache.fory:fory-core` | -| Python | Supported | `pyfory` | -| C++ | Supported | Bazel/CMake build | -| Go | Supported | `github.com/apache/fory/go/fory` | -| Rust | Supported | `fory` crate | -| JavaScript/TypeScript | Supported | `@apache-fory/core` | -| C# | Supported | `Apache.Fory` | -| Swift | Supported | Swift Package Manager target | -| Dart | Supported | `fory` package | -| Scala | Supported | `org.apache.fory:fory-scala` | -| Kotlin | Supported | `org.apache.fory:fory-kotlin` | - -## When to Use Xlang Mode - -Use xlang mode when: - -- Building multi-language microservices -- Creating polyglot data pipelines -- Sharing data between frontend JavaScript/TypeScript and backend services such - as Java, Python, Go, C#, Scala, or Kotlin - -Use native mode for same-language traffic in Java, Scala, Kotlin, Python, C++, -Go, or Rust: - -- All serialization/deserialization happens in the same language -- You need language-specific features such as Python pickle-style objects or Java serialization hooks -- You want native-mode payloads for same-language services - -## Quick Example - -### Java (Producer) - -```java -import org.apache.fory.*; -import org.apache.fory.config.*; - -public class Person { - public String name; - public int age; -} - -Fory fory = Fory.builder().withXlang(true).build(); -fory.register(Person.class, "example.Person"); - -Person person = new Person(); -person.name = "Alice"; -person.age = 30; -byte[] bytes = fory.serialize(person); -// Send bytes to Python, Go, Rust, etc. -``` - -### Python (Consumer) - -```python -import pyfory -from dataclasses import dataclass - -@dataclass -class Person: - name: str - age: pyfory.Int32 - -fory = pyfory.Fory(xlang=True) -fory.register_type(Person, name="example.Person") - -# Receive bytes from Java -person = fory.deserialize(bytes_from_java) -print(f"{person.name}, {person.age}") # Alice, 30 -``` - -## Fory IDL - -For schema-first projects, Fory also provides **Fory IDL** and code generation. - -- Compiler docs: [Fory IDL Overview](../../compiler/index.md) -- Best for large multi-language message contracts and long-lived schemas - -### Minimal IDL Example - -Create `person.fdl`: - -```protobuf -package example; - -message Person { - string name = 1; - int32 age = 2; - optional string email = 3; -} -``` - -Generate code: - -```bash -foryc person.fdl --lang java,python,cpp,go,rust,javascript,csharp,swift,dart,scala,kotlin --output ./generated -``` - -This generates native language types with consistent field/type mappings across all targets. - -## When to Fory IDL - -| Option | Use When | Why | -| ---------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -| Native xlang types (no IDL) | You only have a few message types and want to move quickly | Avoids the integration/setup cost of introducing and operating the compiler | -| Fory IDL (schema-first + codegen) | You have many messages across multiple languages/teams/services | Provides a single contract, stronger consistency, and easier long-term evolution | -| Hybrid (start native, move to IDL) | Project starts small but message count and cross-team dependency grows | Lets you keep early velocity, then standardize once schema complexity increases | - -## Documentation - -| Topic | Description | -| --------------------------------------------------------- | ----------------------------------------------- | -| [Getting Started](../../start/index.md) | Installation and basic setup for all languages | -| [Type Mapping](../../specification/xlang_type_mapping.md) | Xlang type mapping reference | -| [Type System](type-system.md) | Built-in types and cross-runtime type behavior | -| [Type Identity](type-identity.md) | Coordinate names and numeric IDs across peers | -| [Nullability](nullability.md) | Nullable field behavior and configuration | -| [References](references.md) | Shared and circular object references | -| [Polymorphism](polymorphism.md) | Runtime type selection and registration | -| [Schema Evolution](schema-evolution.md) | Compatible and same-schema mode choices | -| [Zero-Copy](zero-copy.md) | Out-of-band serialization for large data | -| [Row Format](../../row-format/index.md) | Cache-friendly binary format with random access | -| [Troubleshooting](troubleshooting.md) | Common issues and solutions | - -## Language-Specific Guides - -For language-specific details and API reference: - -- [Java Xlang Serialization Guide](../java/xlang.md) -- [Python Xlang Serialization Guide](../python/xlang.md) -- [C++ Xlang Serialization Guide](../cpp/xlang.md) -- [Go Xlang Serialization Guide](../go/xlang.md) -- [Rust Xlang Serialization Guide](../rust/xlang.md) -- [JavaScript/TypeScript Xlang Serialization Guide](../javascript/xlang.md) -- [C# Xlang Serialization Guide](../csharp/xlang.md) -- [Swift Xlang Serialization Guide](../swift/xlang.md) -- [Dart Xlang Serialization Guide](../dart/xlang.md) -- [Scala Xlang Serialization Guide](../scala/xlang.md) -- [Kotlin Xlang Serialization Guide](../kotlin/xlang.md) - -## Specifications - -- [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - Binary protocol details -- [Type Mapping Specification](../../specification/xlang_type_mapping.md) - Complete type mapping reference - -## Operational best practices - -1. **Use consistent type names**: Ensure all languages use the same type name or ID -2. **Enable reference tracking**: If your data has circular or shared references -3. **Reuse Fory instances**: Creating Fory is expensive; reuse instances -4. **Use type annotations**: In Python, use markers such as `pyfory.Int32` for precise type mapping -5. **Test cross-language**: Verify serialization works across all target languages diff --git a/docs/object-serialization/xlang/nullability.md b/docs/object-serialization/xlang/nullability.md deleted file mode 100644 index e7b7fe2d72f..00000000000 --- a/docs/object-serialization/xlang/nullability.md +++ /dev/null @@ -1,267 +0,0 @@ ---- -title: Field Nullability -sidebar_position: 4 -id: nullability -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page explains how Fory handles field nullability in cross-language (xlang) serialization mode. - -## Default Behavior - -In xlang mode, **fields are non-nullable by default**. This means: - -- Values must always be present (non-null) -- No null flag byte is written for the field -- Serialization is more compact - -The following types are nullable by default: - -- `Optional` (Java, C++) -- Java boxed types (`Integer`, `Long`, `Double`, etc.) -- Go pointer types (`*int32`, `*string`, etc.) -- Rust `Option` -- Python `Optional[T]` -- Scala `Option[T]` - -| Field Type | Default Nullable | Null Flag Written | -| ------------------------------------------ | ---------------- | ----------------- | -| Primitives (`int`, `bool`, `float`, etc.) | No | No | -| `String` | No | No | -| `List`, `Map`, `Set` | No | No | -| Custom structs | No | No | -| Enums | No | No | -| Java boxed types (`Integer`, `Long`, etc.) | Yes | Yes | -| Go pointer types (`*int32`, `*string`) | Yes | Yes | -| `Optional` / `Option` | Yes | Yes | - -## Wire Format - -The nullable flag controls whether a **null flag byte** is written before the field value: - -``` -Non-nullable field: [value data] -Nullable field: [null_flag] [value data if not null] -``` - -Where `null_flag` is: - -- `-1` (NULL_FLAG): Value is null -- `-2` (NOT_NULL_VALUE_FLAG): Value is present - -## Nullable vs Reference Tracking - -These are related but distinct concepts: - -| Concept | Purpose | Flag Values | -| ---------------------- | ------------------------------------ | ------------------------------------------- | -| **Nullable** | Allow null values for a field | `-1` (null), `-2` (not null) | -| **Reference Tracking** | Deduplicate shared object references | `-1` (null), `-2` (not null), `≥0` (ref ID) | - -Key differences: - -- **Nullable only**: Writes `-1` or `-2` flag, no reference deduplication -- **Reference tracking**: Extends nullable semantics with reference IDs (`≥0`) for previously seen objects -- Both use the same flag byte position—ref tracking is a superset of nullable - -When `refTracking=true`, the null flag byte doubles as a ref flag: - -``` -ref_flag = -1 → null value -ref_flag = -2 → new object (first occurrence) -ref_flag >= 0 → reference to object at index ref_flag -``` - -For detailed reference tracking behavior, see [Reference Tracking](references.md). - -## Language-Specific Examples - -### Java - -```java -public class Person { - // Non-nullable by default in xlang mode - String name; // Must not be null - int age; // Primitive, always non-nullable - List tags; // Must not be null - - // Explicitly nullable - @Nullable - String nickname; // Can be null - - // Optional wrapper - nullable by default - Optional bio; // Can be empty/null -} - -Fory fory = Fory.builder() - .withXlang(true) - .build(); -fory.register(Person.class, "example.Person"); -``` - -### Python - -```python -from dataclasses import dataclass -from typing import Optional, List -import pyfory - -@dataclass -class Person: - # Non-nullable by default - name: str # Must have a value - age: pyfory.Int32 # Primitive - tags: List[str] # Must not be None - - # Optional makes it nullable - nickname: Optional[str] = None # Can be None - bio: Optional[str] = None # Can be None - -fory = pyfory.Fory(xlang=True) -fory.register_type(Person, name="example.Person") -``` - -### Rust - -```rust -use fory::{Fory, ForyStruct}; - -#[derive(ForyStruct)] -struct Person { - // Non-nullable by default - name: String, - age: i32, - tags: Vec, - - // Option is nullable - nickname: Option, // Can be None - bio: Option, // Can be None -} -``` - -### Go - -```go -type Person struct { - // Non-nullable by default - Name string - Age int32 - Tags []string - - // Pointer types for nullable fields - Nickname *string // Can be nil - Bio *string // Can be nil -} - -fory := forygo.NewFory(forygo.WithXlang(true)) -fory.RegisterStructByName(Person{}, "example.Person") -``` - -### C++ - -```cpp -struct Person { - // Non-nullable by default - std::string name; - int32_t age; - std::vector tags; - - // std::optional for nullable - std::optional nickname; - std::optional bio; -}; -FORY_STRUCT(Person, name, age, tags, nickname, bio); -``` - -## Customizing Nullability - -### Java: @Nullable Annotation - -```java -public class Config { - @Nullable - String optionalSetting; // Explicitly nullable - - String requiredSetting; // Explicitly non-nullable (default) -} -``` - -### C++: FORY_STRUCT Field Config - -```cpp -struct Config { - std::optional optional_setting; - std::string required_setting; -}; - -FORY_STRUCT(Config, - (optional_setting, fory::F(1)), - (required_setting, fory::F(2)) -); -``` - -For nullable pointer carriers, opt in with `.nullable()`: - -```cpp -struct ConfigRef { - std::shared_ptr optional_setting; - std::shared_ptr required_setting; -}; - -FORY_STRUCT(ConfigRef, - (optional_setting, fory::F(1).nullable()), - (required_setting, fory::F(2)) -); -``` - -## Null Value Handling - -When a non-nullable field receives a null value: - -| Language | Behavior | -| -------- | ---------------------------------------------------- | -| Java | Throws `NullPointerException` or serialization error | -| Python | Raises `TypeError` or serialization error | -| Rust | Compile-time error (non-Option types can't be None) | -| Go | Zero value is used (empty string, 0, etc.) | -| C++ | Default-constructed value or undefined behavior | - -## Schema Compatibility - -The nullable flag is part of the struct schema fingerprint. When compatible mode is disabled, changing a field's nullability is a **breaking change** that will cause schema version mismatch errors. - -``` -Schema A: { name: String (non-nullable) } -Schema B: { name: String (nullable) } -// These have different fingerprints when compatible mode is disabled -``` - -In compatible mode, top-level scalar fields can still be matched when their scalar type is otherwise compatible and the nullability or optional wrapper differs. Present values are read through compatible scalar conversion and must satisfy the normal lossless conversion checks. Remote null values follow the compatible-read null/default behavior for the local field. - -## Best Practices - -1. **Use non-nullable by default**: Only make fields nullable when null is a valid semantic value -2. **Use Optional/Option wrappers**: Instead of raw types with nullable annotation -3. **Be consistent across languages**: Use the same nullability for corresponding fields -4. **Document nullable fields**: Make it clear which fields can be null in your API - -## See Also - -- [Reference Tracking](references.md) - Shared and circular reference handling -- [Xlang Overview](index.md) - Cross-language serialization workflow and runtime guides -- [Type Mapping](../../specification/xlang_type_mapping.md) - Cross-language type mapping reference -- [Xlang Specification](../../specification/xlang_serialization_spec.md) - Binary protocol details diff --git a/docs/object-serialization/xlang/polymorphism.md b/docs/object-serialization/xlang/polymorphism.md deleted file mode 100644 index 7851e178295..00000000000 --- a/docs/object-serialization/xlang/polymorphism.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Polymorphism -sidebar_position: 6 -id: polymorphism -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Xlang polymorphism preserves a value's concrete registered type when a field, element, or root is -declared through a broader type. Every peer must agree on the concrete type identity and compatible -field schema. - -## Runtime rules - -Use the selected runtime's schema metadata, registration, and polymorphism pages for exact syntax. -The [xlang serialization specification](../../specification/xlang_serialization_spec.md) defines -the shared behavior and limitations. - -Do not infer xlang support from host-language inheritance alone. A concrete subtype must have a -portable xlang mapping and coordinated registration on every peer that can receive it. diff --git a/docs/object-serialization/xlang/references.md b/docs/object-serialization/xlang/references.md deleted file mode 100644 index 6e12328c4e6..00000000000 --- a/docs/object-serialization/xlang/references.md +++ /dev/null @@ -1,303 +0,0 @@ ---- -title: Reference Tracking -sidebar_position: 5 -id: references -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page explains how Fory handles reference tracking for shared and circular references in cross-language serialization. - -## Overview - -Reference tracking enables: - -- **Shared references**: Same object referenced multiple times is serialized once -- **Circular references**: Objects that reference themselves or form cycles -- **Memory efficiency**: No duplicate data for repeated objects - -## Enabling Reference Tracking - -### Java - -```java -Fory fory = Fory.builder() - .withXlang(true) - .withRefTracking(true) - .build(); -``` - -### Python - -```python -fory = pyfory.Fory(xlang=True, ref=True) -``` - -### Go - -```go -fory := forygo.NewFory( - forygo.WithXlang(true), - forygo.WithTrackRef(true), -) -``` - -### C++ - -```cpp -auto fory = fory::serialization::Fory::builder().xlang(true).track_ref(true).build(); -``` - -### Rust - -```rust -let fory = Fory::builder() - .xlang(true) - .track_ref(true).build(); -``` - -### Scala - -```scala -import org.apache.fory.scala.ForyScala - -val fory = ForyScala.builder() - .withXlang(true) - .withRefTracking(true) - .build() -``` - -## Wire Format - -When reference tracking is enabled, nullable fields write a **ref flag byte** before the value: - -``` -[ref_flag] [value data if not null/ref] -``` - -Where `ref_flag` is: - -| Value | Meaning | -| -------------------------- | ----------------------------------------------------- | -| `-1` (NULL_FLAG) | Value is null | -| `-2` (NOT_NULL_VALUE_FLAG) | Value is present, first occurrence | -| `≥0` | Reference ID pointing to previously serialized object | - -## Reference Tracking vs Nullability - -These are **independent** concepts: - -| Concept | Purpose | Controlled By | -| ---------------------- | ------------------------------------------ | ---------------------------------------- | -| **Nullability** | Whether a field can hold null values | Field type (`Optional`) or annotation | -| **Reference Tracking** | Whether duplicate objects are deduplicated | Global `refTracking` option | - -Key behavior: - -- Ref flag bytes are **only written for nullable fields** -- Non-nullable fields skip ref flags entirely, even with `refTracking=true` -- Reference deduplication only applies to objects that appear multiple times - -```java -// Reference tracking enabled, but non-nullable fields still skip ref flags -Fory fory = Fory.builder() - .withXlang(true) - .withRefTracking(true) - .build(); -``` - -## Per-Field Reference Tracking - -By default, **most fields do not track references** even when global `refTracking=true`. Only specific pointer/smart pointer types track references by default. - -### Default Behavior by Language - -| Language | Default Ref Tracking | Types That Track Refs by Default | -| -------- | -------------------- | ---------------------------------------------------------- | -| Java | No | None (use annotation to enable) | -| Python | No | None (use annotation to enable) | -| Go | No | None (use `fory:"ref"` to enable) | -| C++ | Yes | `std::shared_ptr`, `fory::serialization::SharedWeak` | -| Rust | No | `Rc`, `Arc`, `Weak` | -| Scala | No | None (use `@Ref` to enable) | - -### Customizing Per-Field Ref Tracking - -#### Java: @Ref Annotation - -```java -public class Document { - // Default: no ref tracking - String title; - - // Enable ref tracking for this field - @Ref - Author author; - - // Shared across documents, track refs to avoid duplicates - List<@Ref Tag> tags; -} -``` - -#### C++: FORY_STRUCT Field Config - -```cpp -struct Document { - std::string title; - - // shared_ptr/SharedWeak track refs by default - std::shared_ptr author; - fory::serialization::SharedWeak data; - - std::shared_ptr tag_owner; -}; -FORY_STRUCT(Document, - title, - author, - data, - (tag_owner, fory::F().ref()) -); -``` - -To disable reference tracking for C++ entirely, set -`Fory::builder().xlang(true).track_ref(false).build()` on the serializer. - -#### Rust: Field Attributes - -```rust -use fory::ForyStruct; -use std::rc::Rc; - -#[derive(ForyStruct)] -struct Document { - title: String, - - // Rc/Arc track refs by default - author: Rc, - - // Explicitly enable ref tracking - #[fory(ref = true)] - tags: Vec, -} -``` - -#### Scala: @Ref Annotation - -Scala schema IDL and Scala 3 macro derivation use the same shared JVM `@Ref` -annotation: - -```scala -import org.apache.fory.annotation.{ForyField, ForyStruct, Ref} -import org.apache.fory.scala.ForySerializer - -@ForyStruct -final class Node() derives ForySerializer { - @ForyField(id = 1) - var children: List[Node @Ref] = List.empty - - @Ref - @ForyField(id = 2) - var parent: Option[Node] = None -} -``` - -For Scala, top-level field reference tracking is owned by `@Ref` on the field or -constructor parameter. Type-use `T @Ref` is for nested element/value/payload -references, such as `List[Node @Ref]`. - -#### Go: Struct Tags - -```go -type Document struct { - Title string - - // Enable ref tracking for pointer to struct - Author *Author `fory:"ref"` - - // Enable ref tracking for slice - Tags []Tag `fory:"ref"` -} -``` - -### When to Enable Per-Field Ref Tracking - -Enable ref tracking for fields that: - -- May contain the same object instance multiple times -- Are part of circular reference chains -- Hold large objects that might be shared - -Disable (or leave default) for fields that: - -- Always contain unique values -- Are primitives or simple value types -- Don't participate in object sharing - -## Example: Shared References - -```java -public class Container { - List data; - List sameData; // Points to same list -} - -Container obj = new Container(); -obj.data = Arrays.asList("a", "b", "c"); -obj.sameData = obj.data; // Shared reference - -// With refTracking=true: data serialized once, sameData stores reference ID -// With refTracking=false: data serialized twice (duplicate) -``` - -## Example: Circular References - -```java -public class Node { - String value; - Node next; -} - -Node a = new Node("A"); -Node b = new Node("B"); -a.next = b; -b.next = a; // Circular reference - -// With refTracking=true: works correctly -// With refTracking=false: infinite recursion error -``` - -## Language Support - -| Language | Shared Refs | Circular Refs | -| ---------- | ----------- | -------------------- | -| Java | Yes | Yes | -| Python | Yes | Yes | -| Go | Yes | Yes | -| C++ | Yes | Yes | -| JavaScript | Yes | Yes | -| Rust | Yes | No (ownership rules) | - -## Performance Considerations - -- **Overhead**: Reference tracking adds a hash map lookup per object -- **When to enable**: Use when data has shared/circular references -- **When to disable**: Use for simple data structures without sharing - -## See Also - -- [Field Nullability](nullability.md) - How nullability affects serialization -- [Xlang Overview](index.md) - Cross-language serialization workflow and runtime guides -- [Xlang Specification](../../specification/xlang_serialization_spec.md) - Binary protocol details diff --git a/docs/object-serialization/xlang/schema-evolution.md b/docs/object-serialization/xlang/schema-evolution.md deleted file mode 100644 index 6e519da3c0f..00000000000 --- a/docs/object-serialization/xlang/schema-evolution.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Schema Evolution -sidebar_position: 3 -id: schema-evolution -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Xlang compatible mode carries schema metadata so readers can tolerate supported field additions, -removals, and reordering. Same-schema mode reduces metadata only when every reader and writer uses -the same schema. - -## Choose compatible or same-schema mode - -Keep compatible mode for independently deployed peers. Use same-schema mode only after verifying -that every runtime uses the same field identities, nullability, reference metadata, and types. - -Runtime APIs and examples live in each runtime's `schema-evolution.md` page. Normative schema -metadata and compatibility behavior are defined by the -[xlang serialization specification](../../specification/xlang_serialization_spec.md). diff --git a/docs/object-serialization/xlang/troubleshooting.md b/docs/object-serialization/xlang/troubleshooting.md deleted file mode 100644 index e6514c15899..00000000000 --- a/docs/object-serialization/xlang/troubleshooting.md +++ /dev/null @@ -1,321 +0,0 @@ ---- -title: Troubleshooting -sidebar_position: 90 -id: troubleshooting -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers common issues and solutions when using cross-language serialization. - -## Type Registration Errors - -### "Type not registered" Error - -**Symptom:** - -``` -Error: Type 'example.Person' is not registered -``` - -**Cause:** The type was not registered before deserialization, or the type name doesn't match. - -**Solution:** - -1. Ensure the type is registered with the same name on both sides: - - ```java - // Java - fory.register(Person.class, "example.Person"); - ``` - - ```python - # Python - fory.register_type(Person, name="example.Person") - ``` - -2. Check for typos or case differences in type names - -3. Register types before any serialization/deserialization calls - -### "Type ID mismatch" Error - -**Symptom:** - -``` -Error: Expected type ID 100, got 101 -``` - -**Cause:** Different type IDs used across languages. - -**Solution:** Use consistent type IDs: - -```java -// Java -fory.register(Person.class, 100); -fory.register(Address.class, 101); -``` - -```python -# Python -fory.register_type(Person, type_id=100) -fory.register_type(Address, type_id=101) -``` - -## Type Mapping Issues - -### Integer Overflow - -**Symptom:** Values are truncated or wrapped unexpectedly. - -**Cause:** Using different integer sizes across languages. - -**Solution:** - -1. In Python, use explicit type annotations: - - ```python - @dataclass - class Data: - value: pyfory.Int32 # Not just 'int' - ``` - -2. Ensure integer ranges are compatible: - - `int8`: -128 to 127 - - `int16`: -32,768 to 32,767 - - `int32`: -2,147,483,648 to 2,147,483,647 - -### Float Precision Loss - -**Symptom:** Float values have unexpected precision. - -**Cause:** Mixing `float32` and `float64` types. - -**Solution:** - -1. Use consistent float types: - - ```python - @dataclass - class Data: - value: pyfory.Float32 # Explicit 32-bit float - ``` - -2. Be aware that Python's `float` maps to `float64` by default - -### String Encoding Errors - -**Symptom:** - -``` -Error: Invalid UTF-8 sequence -``` - -**Cause:** Non-UTF-8 encoded strings. - -**Solution:** - -1. Ensure all strings are valid UTF-8 -2. In Python, decode bytes before serialization: - - ```python - text = raw_bytes.decode('utf-8') - ``` - -## Field Order Issues - -### "Field mismatch" Error - -**Symptom:** Deserialized objects have wrong field values. - -**Cause:** Field order differs between languages. - -**Solution:** Fory sorts fields by their snake_cased names. Ensure field names are consistent: - -```java -// Java - fields will be sorted: age, email, name -public class Person { - public String name; - public int age; - public String email; -} -``` - -```python -# Python - same field order -@dataclass -class Person: - name: str - age: pyfory.Int32 - email: str -``` - -## Reference Tracking Issues - -### Stack Overflow with Circular References - -**Symptom:** - -``` -StackOverflowError or RecursionError -``` - -**Cause:** Reference tracking is disabled but data has circular references. - -**Solution:** Enable reference tracking: - -```java -// Java -Fory fory = Fory.builder() - .withRefTracking(true) - .build(); -``` - -```python -# Python -fory = pyfory.Fory(ref=True) -``` - -### Duplicate Objects - -**Symptom:** Shared objects are duplicated after deserialization. - -**Cause:** Reference tracking is disabled. - -**Solution:** Enable reference tracking if objects are shared within the graph. - -## Xlang Type Issues - -### Incompatible Types in Xlang Mode - -**Symptom:** - -``` -Error: Type 'Optional' is not supported in xlang mode -``` - -**Cause:** Using Java-specific types that don't have cross-language equivalents. - -**Solution:** Use compatible types: - -```java -// Instead of Optional -public String email; // nullable - -// Instead of BigDecimal -public double amount; - -// Instead of EnumSet -public Set statuses; -``` - -## Version Compatibility - -### Schema Hash Mismatch - -**Symptom:** Deserialization fails with an error such as `class version hash mismatch`, -`schema version mismatch`, `struct version mismatch`, or `hash mismatch`. - -**Cause:** The writer and reader have disabled compatible mode while their struct/class schemas -differ. In xlang mode this can happen even when each language made a reasonable local change, -because field names, type annotations, field IDs, nullability, and generated schema metadata must -still align exactly. - -**Solution:** - -1. Align the schemas carefully on every service and language: field names or field IDs, field order, - type annotations, nullability, and type registration IDs/names. -2. Xlang mode defaults to compatible mode in current implementations. If a peer has explicitly selected - `compatible=false`, remove that override or enable compatible mode on every peer. - Compatible mode writes extra schema metadata, so payloads are larger, but it is recommended - for xlang services that may evolve independently. -3. Set `compatible=false` only when every reader and writer always uses the same schema. For xlang payloads, do this only after verifying that every language uses that schema, or when native types are generated from Fory schema IDL. - -### Serialization Format Changed - -**Symptom:** Deserialization fails after upgrading Fory. - -**Cause:** Breaking changes in serialization format. - -**Solution:** - -1. Ensure all services use compatible Fory versions -2. Check release notes for breaking changes -3. Consider using schema evolution (compatible mode) for gradual upgrades - -## Debugging Tips - -### Enable Debug Logging - -**Java:** - -```java -// Add to JVM options --Dfory.debug=true -``` - -**Python:** - -```python -import logging -logging.getLogger('pyfory').setLevel(logging.DEBUG) -``` - -### Inspect Serialized Data - -Use hex dump to inspect the binary format: - -```python -data = fory.serialize(obj) -print(data.hex()) -``` - -### Test Round-Trip - -Always test round-trip serialization in each language: - -```java -byte[] bytes = fory.serialize(obj); -Object result = fory.deserialize(bytes); -assert obj.equals(result); -``` - -### Cross-Language Testing - -Test serialization across all target languages before deployment: - -```bash -# Serialize in Java -java -jar serializer.jar > data.bin - -# Deserialize in Python -python deserializer.py data.bin -``` - -## Common Mistakes - -1. **Not registering types**: Always register custom types before use -2. **Inconsistent type names/IDs**: Use the same names/IDs across all languages -3. **Mixing xlang and native payloads**: Keep every peer on the xlang wire format -4. **Wrong type annotations**: Use markers such as `pyfory.Int32` in Python -5. **Ignoring reference tracking**: Enable for circular/shared references - -## See Also - -- [Type Mapping](../../specification/xlang_type_mapping.md) - Cross-language type mapping reference -- [Getting Started](../../start/index.md) - Runtime setup guides -- [Java Troubleshooting](../java/troubleshooting.md) - Java-specific issues -- [Python Troubleshooting](../python/troubleshooting.md) - Python-specific issues diff --git a/docs/object-serialization/xlang/type-identity.md b/docs/object-serialization/xlang/type-identity.md deleted file mode 100644 index 844b7e4b525..00000000000 --- a/docs/object-serialization/xlang/type-identity.md +++ /dev/null @@ -1,298 +0,0 @@ ---- -title: Type Identity and Dynamic Fields -sidebar_position: 2 -id: type-identity -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Field type meta configuration controls whether type information is written during serialization for struct fields. This is essential for supporting polymorphism where the actual concrete type may differ from the declared field type. - -## Overview - -When serializing a struct field, Fory needs to determine whether to write type metadata: - -- **Static typing**: Use the declared field type's serializer directly (no type info written) -- **Dynamic typing**: Write type information to support subtypes - -## When Type Meta Is Needed - -Type metadata is required when: - -1. **Interface/abstract fields**: The declared type is abstract, so concrete type must be recorded -2. **Polymorphic fields**: The concrete type may be a subclass of the declared type -3. **Cross-language compatibility**: When the receiver needs type information to deserialize correctly - -Type metadata is NOT needed when: - -1. **Final/concrete types**: The declared type is final/sealed and cannot be subclassed -2. **Primitive types**: Type is known at compile time -3. **Performance optimization**: When you know the concrete type always matches the declared type - -## Language-Specific Configuration - -### Java - -Java requires explicit configuration because concrete classes can be subclassed unless marked `final`. - -Use the `@ForyField` annotation with the `dynamic` parameter: - -```java -import org.apache.fory.annotation.ForyField; -import org.apache.fory.annotation.ForyField.Dynamic; - -public class Container { - // AUTO (default): Interface types write type info, concrete types don't - @ForyField(id = 0) - private Shape shape; // Interface - type info written - - // FALSE: Never write type info (use declared type's serializer) - @ForyField(id = 1, dynamic = Dynamic.FALSE) - private Circle circle; // Always treated as Circle - - // TRUE: Always write type info (support subtypes) - @ForyField(id = 2, dynamic = Dynamic.TRUE) - private Shape concreteShape; // Type info written even if concrete -} -``` - -**Dynamic Options**: - -| Value | Behavior | -| ------- | ------------------------------------------------------ | -| `AUTO` | Interface/abstract types are dynamic, concrete are not | -| `FALSE` | Never write type info, use declared type's serializer | -| `TRUE` | Always write type info to support subtypes | - -**Use Cases**: - -- `AUTO`: Default behavior, suitable for most cases -- `FALSE`: Performance optimization when you know the exact type -- `TRUE`: When a concrete field may hold subclass instances - -### C++ - -C++ uses the `.dynamic(bool)` builder method inside `FORY_STRUCT`: - -```cpp -#include "fory/serialization/fory.h" - -// Abstract base class with pure virtual methods -struct Animal { - virtual ~Animal() = default; - virtual std::string speak() const = 0; -}; - -struct Zoo { - // Auto: type info written because Animal is polymorphic (std::is_polymorphic) - std::shared_ptr animal; - - // Force non-dynamic: skip type info even though Animal is polymorphic - std::shared_ptr fixed_animal; - - // Force dynamic: write type info even for non-polymorphic types - std::shared_ptr polymorphic_data; -}; -FORY_STRUCT(Zoo, - (animal, fory::F(0).nullable()), // Auto-detect polymorphism - (fixed_animal, fory::F(1).nullable().dynamic(false)), // Skip type info - (polymorphic_data, fory::F(2).dynamic(true)) // Force type info -); -``` - -**Default Behavior**: Fory auto-detects polymorphism via `std::is_polymorphic`. Types with pure virtual methods are treated as dynamic by default. - -### Go and Rust - -Go and Rust do **not** require explicit dynamic configuration because: - -- **Go**: Interface types are inherently dynamic - Fory can determine from the type whether it's an interface -- **Rust**: Trait objects (`dyn Trait`) are explicitly marked in the type system - -The type system in these languages already indicates whether a field is polymorphic: - -```go -// Go: interface types are automatically dynamic -type Container struct { - Shape Shape // Interface - type info written automatically - Circle Circle // Concrete struct - no type info needed -} -``` - -```rust -// Rust: trait objects are explicitly marked -struct Container { - shape: Box, // Trait object - type info written automatically - circle: Circle, // Concrete type - no type info needed -} -``` - -### Python - -Use `pyfory.field()` with the `dynamic` parameter: - -```python -from dataclasses import dataclass -from abc import ABC, abstractmethod -import pyfory - -class Shape(ABC): - @abstractmethod - def area(self) -> float: - pass - -@dataclass -class Circle(Shape): - radius: float = 0.0 - - def area(self) -> float: - return 3.14159 * self.radius * self.radius - -@dataclass -class Container: - # Abstract class: dynamic is always True (type info written) - shape: Shape = pyfory.field(id=0) - - # Concrete type with explicit dynamic=True (force type info) - circle: Circle = pyfory.field(id=1, dynamic=True) - - # Concrete type with explicit dynamic=False (skip type info) - fixed_circle: Circle = pyfory.field(id=2, dynamic=False) -``` - -**Default Behavior**: - -| Mode | Abstract Class | Concrete Object Types | Numeric/str/time Types | -| ----------- | -------------- | --------------------- | ---------------------- | -| Native mode | `True` | `True` | `False` | -| Xlang mode | `True` | `False` | `False` | - -- **Abstract classes**: `dynamic` is always `True` (type info must be written) -- **Native mode**: `dynamic` defaults to `True` for object types, `False` for numeric/str/time types -- **Xlang mode**: `dynamic` defaults to `False` for concrete types - -## Default Behavior - -| Language | Interface/Abstract Types | Concrete Types | -| -------- | ------------------------ | ---------------- | -| Java | Dynamic (write type) | Static (no type) | -| C++ | Dynamic (virtual) | Static | -| Go | Dynamic (interface) | Static (struct) | -| Rust | Dynamic (dyn Trait) | Static | -| Python | Dynamic (all objects) | Dynamic | - -## Performance Considerations - -Writing type metadata has overhead: - -- **Space**: Type information adds bytes to serialized output -- **Time**: Type resolution during serialization/deserialization - -Use `dynamic = FALSE` (Java) or `dynamic(false)` (C++) when: - -- You're certain the concrete type matches the declared type -- Performance is critical and polymorphism is not needed -- The field type is effectively final - -## Cross-Language Compatibility - -When serializing data for cross-language consumption: - -1. **Use consistent type registration**: Register types with the same ID across languages -2. **Prefer explicit configuration**: Use `dynamic = TRUE` when unsure about receiver's expectations -3. **Document polymorphic fields**: Make it clear which fields may contain subtypes - -## Example: Polymorphic Container - -### Java - -```java -public interface Animal { - String speak(); -} - -public class Dog implements Animal { - private String name; - - @Override - public String speak() { return "Woof!"; } -} - -public class Cat implements Animal { - private String name; - - @Override - public String speak() { return "Meow!"; } -} - -public class Zoo { - // Type info written because Animal is an interface - @ForyField(id = 0) - private Animal animal; - - // Force type info for concrete type that may hold subtypes - @ForyField(id = 1, dynamic = Dynamic.TRUE) - private Dog maybeMixedBreed; -} -``` - -### C++ - -```cpp -// Abstract base class with pure virtual methods -class Animal { -public: - virtual std::string speak() const = 0; - virtual ~Animal() = default; -}; - -class Dog : public Animal { -public: - std::string name; - std::string speak() const override { return "Woof!"; } -}; - -struct Zoo { - std::shared_ptr animal; - std::shared_ptr maybe_mixed_breed; -}; - -FORY_STRUCT(Zoo, - (animal, fory::F(0).nullable()), // Auto-detect (Animal is polymorphic) - (maybe_mixed_breed, fory::F(1).dynamic(true)) // Force dynamic for concrete type -); -``` - -## Related Topics - -- [Field Nullability](nullability.md) - Controlling null handling for fields -- [Reference Tracking](references.md) - Managing shared/circular references -- [Type Mapping](../../specification/xlang_type_mapping.md) - Cross-language type compatibility - -## Coordinate Type Identity Across Peers - -Every peer must register a custom type with the same numeric ID or the same -namespace and type name. Numeric IDs produce smaller metadata; names are -easier to coordinate across independently deployed services. Do not mix an -ID registration on one peer with a name registration on another. - -Use the selected runtime guide for the exact registration API: -[Java](../java/xlang.md), [Python](../python/xlang.md), -[C++](../cpp/xlang.md), [Go](../go/xlang.md), -[Rust](../rust/xlang.md), [JavaScript](../javascript/xlang.md), -[C#](../csharp/xlang.md), [Swift](../swift/xlang.md), -[Dart](../dart/xlang.md), [Scala](../scala/xlang.md), and -[Kotlin](../kotlin/xlang.md). diff --git a/docs/object-serialization/xlang/type-system.md b/docs/object-serialization/xlang/type-system.md deleted file mode 100644 index ac711295e8f..00000000000 --- a/docs/object-serialization/xlang/type-system.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: Type System -sidebar_position: 1 -id: type-system -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -## Serialize Built-in Types - -Common types can be serialized automatically without registration: primitive numeric types, string, binary, array, list, map, and more. - -Reduced-precision floating-point values are also part of the built-in xlang type system: - -- `float16` and `array` -- `bfloat16` and `array` - -Use the language-specific carrier types documented in the type mapping reference. Python uses `pyfory.Float16` and `pyfory.BFloat16` as annotation markers only; scalar values are native Python `float`, and dense reduced-precision arrays use `pyfory.Float16Array` and `pyfory.BFloat16Array`. Go uses the `float16` and `bfloat16` packages for scalar, slice, and array carriers; JavaScript uses `number` for scalar `float16` and `bfloat16`, and dense array carriers `BoolArray`, `Float16Array`, and `BFloat16Array` for the corresponding `array` schemas. Dart uses `double` plus `Float16Type` or `Bfloat16Type` metadata for scalar fields, and `Float16List` / `Bfloat16List` for dense arrays. Java uses `@ArrayType` on supported reduced-precision carriers for `array` / `array` schema, while general object arrays stay on the `list` path; C++, Rust, and C# provide their own dedicated scalar and array carriers. - -When `compatible=true`, a direct struct/class field can evolve between `list` and `array` for dense bool/numeric `T`. Integer list element encodings in the same signedness and width domain match the corresponding dense array element domain. This applies only to the immediate matched field schema. It does not apply to nested collection, map, array, union, or generic positions. A peer `list` schema can be read into a local `array` field when the actual payload has no null elements. If the payload carries a null element or ref-tracked element encoding, reading it into a local `array` field raises a compatible-read error. - -## Serialize Custom Types - -User-defined types must be registered using the register API to establish the mapping relationship between types in different languages. Use consistent type names across all languages. - -## Exact mappings - -The normative [xlang type mapping](../../specification/xlang_type_mapping.md) defines the exact -carrier mapping for every runtime. Runtime pages show the API syntax and examples for that mapping. diff --git a/docs/object-serialization/xlang/zero-copy.md b/docs/object-serialization/xlang/zero-copy.md deleted file mode 100644 index 49c6f07a644..00000000000 --- a/docs/object-serialization/xlang/zero-copy.md +++ /dev/null @@ -1,203 +0,0 @@ ---- -title: Zero-Copy Serialization -sidebar_position: 7 -id: zero-copy -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Zero-copy serialization allows large binary data (byte arrays, numeric arrays) to be serialized out-of-band, avoiding memory copies and reducing serialization overhead. - -## When to Use Zero-Copy - -Use zero-copy serialization when: - -- Serializing large byte arrays or binary blobs -- Working with numeric arrays (int[], double[], etc.) -- Transferring data over high-performance networks -- Memory efficiency is critical - -## How It Works - -1. **Serialization**: Large buffers are extracted and returned separately via a callback -2. **Transport**: The main serialized data and buffer objects are transmitted separately -3. **Deserialization**: Buffers are provided back to reconstruct the original object - -This avoids copying large data into the main serialization buffer. - -## Java - -```java -import org.apache.fory.*; -import org.apache.fory.config.*; -import org.apache.fory.serializer.BufferObject; -import org.apache.fory.memory.MemoryBuffer; - -import java.util.*; -import java.util.stream.Collectors; - -public class ZeroCopyExample { - public static void main(String[] args) { - Fory fory = Fory.builder().withXlang(true).build(); - - // Data with large arrays - List list = List.of( - "str", - new byte[1000], // Large byte array - new int[100], // Large int array - new double[100] // Large double array - ); - - // Collect buffer objects during serialization - Collection bufferObjects = new ArrayList<>(); - byte[] bytes = fory.serialize(list, e -> !bufferObjects.add(e)); - - // Convert to buffers for transport - List buffers = bufferObjects.stream() - .map(BufferObject::toBuffer) - .collect(Collectors.toList()); - - // Deserialize with buffers - Object result = fory.deserialize(bytes, buffers); - System.out.println(result); - } -} -``` - -## Python - -```python -import array -import pyfory -import numpy as np - -fory = pyfory.Fory(xlang=True) - -# Data with large arrays -data = [ - "str", - bytes(bytearray(1000)), # Large byte array - array.array("i", range(100)), # Large int array - np.full(100, 0.0, dtype=np.double) # Large numpy array -] - -# Collect buffer objects during serialization -serialized_objects = [] -serialized_data = fory.serialize(data, buffer_callback=serialized_objects.append) - -# Convert to buffers for transport -buffers = [obj.to_buffer() for obj in serialized_objects] - -# Deserialize with buffers -result = fory.deserialize(serialized_data, buffers=buffers) -print(result) -``` - -## Go - -```go -package main - -import forygo "github.com/apache/fory/go/fory" -import "fmt" - -func main() { - serializer := forygo.NewFory(forygo.WithXlang(true)) - - // Data with large arrays - list := []any{ - "str", - make([]byte, 1000), // Large byte array - } - - buf := forygo.NewByteBuffer(nil) - var bufferObjects []forygo.BufferObject - - // Collect buffer objects during serialization - if err := serializer.SerializeWithCallback(buf, list, func(o forygo.BufferObject) bool { - bufferObjects = append(bufferObjects, o) - return false - }); err != nil { - panic(err) - } - - // Convert to buffers for transport - var buffers []*forygo.ByteBuffer - for _, o := range bufferObjects { - buffers = append(buffers, o.ToBuffer()) - } - - // Deserialize with buffers - var newList []any - if err := serializer.DeserializeWithCallbackBuffers(buf, &newList, buffers); err != nil { - panic(err) - } - fmt.Println(newList) -} -``` - -## Use Cases - -### High-Performance Data Transfer - -When sending large datasets over the network: - -```java -// Sender -Collection buffers = new ArrayList<>(); -byte[] metadata = fory.serialize(dataObject, e -> !buffers.add(e)); - -// Send metadata and buffers separately -network.sendMetadata(metadata); -for (BufferObject buf : buffers) { - network.sendBuffer(buf.toBuffer()); -} - -// Receiver -byte[] metadata = network.receiveMetadata(); -List buffers = network.receiveBuffers(); -Object data = fory.deserialize(metadata, buffers); -``` - -### Memory-Mapped Files - -Zero-copy works well with memory-mapped files: - -```java -// Write -Collection buffers = new ArrayList<>(); -byte[] data = fory.serialize(largeObject, e -> !buffers.add(e)); -writeToFile("data.bin", data); -for (int i = 0; i < buffers.size(); i++) { - writeToFile("buffer" + i + ".bin", buffers.get(i).toBuffer()); -} - -// Read -byte[] data = readFromFile("data.bin"); -List buffers = readBufferFiles(); -Object result = fory.deserialize(data, buffers); -``` - -## Performance Considerations - -1. **Threshold**: Small arrays may not benefit from zero-copy due to callback overhead -2. **Network**: Zero-copy is most beneficial when buffers can be sent without copying -3. **Memory**: Reduces peak memory usage by avoiding buffer copies - -## See Also - -- [Xlang Overview](index.md) - Standard serialization workflow and runtime guides -- [Python Out-of-Band Guide](../python/out-of-band.md) - Python-specific zero-copy details diff --git a/docs/row-format/compact.md b/docs/row-format/compact.md deleted file mode 100644 index dbe2104691a..00000000000 --- a/docs/row-format/compact.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -title: Compact Row -sidebar_position: 2 -id: compact -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Compact Row is a Java-only row encoding that reduces fixed-slot and null-bitmap overhead. It is not -wire-compatible with Standard Row. - -## Create a compact encoder - -```java -RowEncoder encoder = - Encoders.buildBeanCodec(MyBean.class) - .compactEncoding() - .build() - .get(); - -BinaryRow row = encoder.toRow(value); -MyBean decoded = encoder.fromRow(row); -``` - -Reuse the encoder within one thread. Create separate encoders for concurrent threads. - -## Layout tradeoffs - -- Fixed-size fields use their natural widths instead of eight-byte Standard Row slots. -- Fields are sorted by alignment to reduce padding. -- The null bitmap is omitted when no field is nullable. -- Fixed-size nested structs can be stored inline. - -Choose Compact Row only when every reader is Java and the space reduction justifies the -Java-specific layout. Use [Standard Row](standard.md) for Java/Python/C++/Rust interchange. - -See the [Row Format specification](../specification/row_format_spec.md) for the exact binary layout. diff --git a/docs/row-format/cpp.md b/docs/row-format/cpp.md index 9132a330c54..f34b25c6ea2 100644 --- a/docs/row-format/cpp.md +++ b/docs/row-format/cpp.md @@ -504,6 +504,6 @@ int32_t id = row.get_int32(0); ## Related Topics - [C++ Row example](https://github.com/apache/fory/tree/main/examples/cpp/hello_row) - Complete runnable example -- [Basic Serialization](../object-serialization/cpp/core-api.md) - Object graph serialization +- [Basic Serialization](../object-serialization/cpp/basic-serialization.md) - Object graph serialization - [C++ Object Serialization Configuration](../object-serialization/cpp/configuration.md) - Builder options - [C++ Object Serialization Supported Types](../object-serialization/cpp/supported-types.md) - Object serialization types diff --git a/docs/row-format/index.md b/docs/row-format/index.md index 1ebcf6d913b..8c901cd0744 100644 --- a/docs/row-format/index.md +++ b/docs/row-format/index.md @@ -1,5 +1,5 @@ --- -title: Row Format +title: Introduction sidebar_position: 0 id: index license: | @@ -19,27 +19,81 @@ license: | limitations under the License. --- -Row Format stores typed values in a cache-friendly binary layout for random and partial access -without reconstructing a complete object graph. Use it for analytical and -in-memory data processing. +Row Format is a cache-friendly binary format for efficient random access and partial +deserialization. Unlike object graph serialization, it lets readers access individual fields +without reconstructing the complete object. -## Choose a row family +Row Format is intended only for trusted analytical data, including memory-mapped data, selective +field access, and data pipelines. Use [Object Serialization](../object-serialization/index.md) when +the application needs general object graphs, shared or circular references, or complete object +reconstruction as its primary access pattern. -| Family | Runtime support | Compatibility | -| --------------------------- | ----------------------- | -------------------------------- | -| [Standard Row](standard.md) | Java, Python, C++, Rust | Shared Standard Row layout | -| [Compact Row](compact.md) | Java | Java-only, space-oriented layout | +## Choose a Layout -Use Binary Object Serialization when the goal is complete object reconstruction, references, or -general application messaging. Use Row Format when a workload reads selected fields, nested arrays, -or maps directly from encoded data. +| Layout | Runtime support | Compatibility | +| ------------ | ----------------------- | -------------------------------- | +| Standard Row | Java, Python, C++, Rust | Shared Standard Row layout | +| Compact Row | Java | Java-only, space-oriented layout | -## Runtime guides +## Standard Row -- [Java](java.md) -- [Python](python.md) -- [C++](cpp.md) -- [Rust](rust.md) +Standard Row is the interoperable layout for Java, Python, C++, and Rust. -The normative [Row Format specification](../specification/row_format_spec.md) defines Standard and -Compact layouts. +### Features + +- **Zero-copy random access**: Read selected fields directly from encoded data. +- **Partial deserialization**: Reconstruct only the values an application needs. +- **Cross-language compatibility**: Share Standard Row bytes between Java, Python, C++, and Rust. +- **Apache Arrow integration**: Convert rows to Arrow data in Java and Python. + +### Layout + +Standard Row stores fixed-width values inline and variable-width values by offset and size. Rows, +arrays, and maps use a schema to resolve field positions and element types. The normative byte +layout, alignment rules, type table, and endianness are defined by the +[Row Format specification](../specification/row_format_spec.md). + +### Runtime Support + +| Runtime | Standard Row compatibility | Runtime guide | Additional integration | +| ------- | -------------------------- | ------------------- | ------------------------------------------------------ | +| Java | Compatible | [Java](java.md) | Arrow conversion; interface and extension-type mapping | +| Python | Compatible | [Python](python.md) | PyArrow schema and table conversion | +| C++ | Compatible | [C++](cpp.md) | Native row readers and writers | +| Rust | Compatible | [Rust](rust.md) | Borrowed struct, array, and map views | + +Use the runtime guides for installation, schema construction, encoding, random access, partial +reads, and language-specific integrations. + +## Compact Row + +Compact Row is a Java-only row encoding that reduces fixed-slot and null-bitmap overhead. It is not +wire-compatible with Standard Row. + +### Create a Compact Encoder + +```java +RowEncoder encoder = + Encoders.buildBeanCodec(MyBean.class) + .compactEncoding() + .build() + .get(); + +BinaryRow row = encoder.toRow(value); +MyBean decoded = encoder.fromRow(row); +``` + +Reuse the encoder within one thread. Create separate encoders for concurrent threads. + +### Layout Tradeoffs + +- Fixed-size fields use their natural widths instead of eight-byte Standard Row slots. +- Fields are sorted by alignment to reduce padding. +- The null bitmap is omitted when no field is nullable. +- Fixed-size nested structs can be stored inline. + +Choose Compact Row only when every reader is Java and the space reduction justifies the +Java-specific layout. Use Standard Row for Java, Python, C++, and Rust interchange. + +See the [Row Format specification](../specification/row_format_spec.md) for the exact Standard and +Compact binary layouts. diff --git a/docs/row-format/java.md b/docs/row-format/java.md index 8679cadd8cd..04a87955143 100644 --- a/docs/row-format/java.md +++ b/docs/row-format/java.md @@ -279,6 +279,6 @@ Parent decoded = encoder.fromRow(row); ## Related Topics -- [Xlang Serialization](../object-serialization/java/xlang.md) - xlang mode +- [Cross-Language Interoperability](../object-serialization/java/basic-serialization.md#cross-language-interoperability) - xlang mode - [Java Advanced Features](../object-serialization/java/advanced-features.md) - Zero-copy object serialization - [Row Format Specification](https://fory.apache.org/docs/specification/row_format_spec) - Protocol details diff --git a/docs/row-format/python.md b/docs/row-format/python.md index 0963df2fb16..4852a16c562 100644 --- a/docs/row-format/python.md +++ b/docs/row-format/python.md @@ -195,6 +195,6 @@ pip install pyfory[format] ## Related Topics -- [Xlang Serialization](../object-serialization/python/xlang.md) - xlang mode -- [Basic Serialization](../object-serialization/python/core-api.md) - Object serialization +- [Cross-Language Interoperability](../object-serialization/python/basic-serialization.md#cross-language-interoperability) - xlang mode +- [Basic Serialization](../object-serialization/python/basic-serialization.md) - Object serialization - [Row Format Specification](https://fory.apache.org/docs/specification/row_format_spec) - Protocol details diff --git a/docs/row-format/rust.md b/docs/row-format/rust.md index 33efc184ac8..debd04d7d10 100644 --- a/docs/row-format/rust.md +++ b/docs/row-format/rust.md @@ -185,6 +185,6 @@ assert!(scores.get(scores.len()).is_err()); ## Related Topics -- [Basic Serialization](../object-serialization/rust/core-api.md) - Object graph serialization -- [Standard Row Format](standard.md) - Shared layout for Java, Python, C++, and Rust +- [Basic Serialization](../object-serialization/rust/basic-serialization.md) - Object graph serialization +- [Standard Row Format](index.md#standard-row) - Shared layout for Java, Python, C++, and Rust - [Row Format Specification](../specification/row_format_spec.md) - Protocol details diff --git a/docs/row-format/standard.md b/docs/row-format/standard.md deleted file mode 100644 index b420f1e9f55..00000000000 --- a/docs/row-format/standard.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: Standard Row Format -sidebar_position: 1 -id: standard -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Fory Row Format is a cache-friendly binary format for efficient random access -and partial deserialization. Unlike object graph serialization, it lets readers -access individual fields without reconstructing the complete object. - -## Features - -- **Zero-copy random access**: Read selected fields directly from encoded data. -- **Partial deserialization**: Reconstruct only the values an application needs. -- **Cross-language compatibility**: Share Standard Row bytes between Java, - Python, C++, and Rust. -- **Apache Arrow integration**: Convert rows to Arrow data in Java and Python. - -## Format Boundary - -Standard Row stores fixed-width values inline and variable-width values by -offset and size. Rows, arrays, and maps use a schema to resolve field positions -and element types. The normative byte layout, alignment rules, type table, and -endianness are defined by the -[Row Format Specification](../specification/row_format_spec.md). - -Row Format is intended only for trusted analytics data, including memory-mapped data, selective field -access, and data pipelines. Use [Object Serialization](../object-serialization/index.md) -when the application needs general object graphs, shared or circular references, -or complete object reconstruction as its primary access pattern. - -## Implementations - -| Runtime | Standard Row compatibility | Runtime guide | Additional integration | -| ------- | -------------------------- | ------------------- | ------------------------------------------------------ | -| Java | Compatible | [Java](java.md) | Arrow conversion; interface and extension-type mapping | -| Python | Compatible | [Python](python.md) | PyArrow schema and table conversion | -| C++ | Compatible | [C++](cpp.md) | Native row readers and writers | -| Rust | Compatible | [Rust](rust.md) | Borrowed struct, array, and map views | - -Use the runtime guides for installation, schema construction, encoding, random -access, partial reads, and language-specific integrations. diff --git a/docs/start/cpp.md b/docs/start/cpp.md index 7e48ce01ab0..bc8384134b8 100644 --- a/docs/start/cpp.md +++ b/docs/start/cpp.md @@ -95,7 +95,7 @@ cmake --build build --parallel See [C++ Object Serialization](../object-serialization/cpp/index.md) for Bazel, Windows, error handling, and thread-safe instances; then continue to -[xlang](../object-serialization/cpp/xlang.md) or +[xlang](../object-serialization/cpp/basic-serialization.md#cross-language-interoperability) or [native mode](../object-serialization/cpp/native.md). ## Other Capabilities diff --git a/docs/start/csharp.md b/docs/start/csharp.md index 91ebefcb2f7..0167f6b0adf 100644 --- a/docs/start/csharp.md +++ b/docs/start/csharp.md @@ -71,7 +71,7 @@ dotnet run C# uses xlang mode. Continue with [C# Object Serialization](../object-serialization/csharp/index.md), -[xlang types](../object-serialization/csharp/xlang.md), +[xlang types](../object-serialization/csharp/basic-serialization.md#cross-language-interoperability), [configuration](../object-serialization/csharp/configuration.md), and [schema evolution](../object-serialization/csharp/schema-evolution.md). diff --git a/docs/start/go.md b/docs/start/go.md index dc545e8f868..969fb106ee7 100644 --- a/docs/start/go.md +++ b/docs/start/go.md @@ -81,7 +81,7 @@ func main() { go run . ``` -Use [xlang mode](../object-serialization/go/xlang.md) for cross-language data +Use [xlang mode](../object-serialization/go/basic-serialization.md#cross-language-interoperability) for cross-language data and [native mode](../object-serialization/go/native.md) for Go-only data. Continue with [Go Object Serialization](../object-serialization/go/index.md), [configuration](../object-serialization/go/configuration.md), and diff --git a/docs/start/index.md b/docs/start/index.md index 3575ff456a9..ea48248b39a 100644 --- a/docs/start/index.md +++ b/docs/start/index.md @@ -1,5 +1,5 @@ --- -title: Getting Started +title: Overview sidebar_position: 0 id: index license: | @@ -68,4 +68,4 @@ round trip for an application project, and the next capability-specific steps: Object Serialization uses xlang mode for portable cross-language data. Java, Python, C++, Go, Rust, Scala, and Kotlin also offer native mode for same-runtime data. Use [Choose a Format](../introduction/choose-a-format.md) -when the product choice is not yet clear. +when you have not yet chosen a format. diff --git a/docs/start/java.md b/docs/start/java.md index 1bf4ecd4c46..bc38eedafde 100644 --- a/docs/start/java.md +++ b/docs/start/java.md @@ -87,7 +87,7 @@ Reuse a `Fory` instance within one thread instead of rebuilding it for every value. `Fory` is not thread-safe; use `ThreadSafeFory` for shared concurrent access. Continue with [Java Object Serialization](../object-serialization/java/index.md), -[xlang mode](../object-serialization/java/xlang.md), +[xlang mode](../object-serialization/java/basic-serialization.md#cross-language-interoperability), [native mode](../object-serialization/java/native.md), or [configuration](../object-serialization/java/configuration.md). diff --git a/docs/start/javascript.md b/docs/start/javascript.md index 693b78632c4..c2577997d56 100644 --- a/docs/start/javascript.md +++ b/docs/start/javascript.md @@ -65,7 +65,7 @@ node example.cjs JavaScript uses xlang mode. Continue with [JavaScript/TypeScript Object Serialization](../object-serialization/javascript/index.md), -[xlang types](../object-serialization/javascript/xlang.md), +[xlang types](../object-serialization/javascript/basic-serialization.md#cross-language-interoperability), [configuration](../object-serialization/javascript/configuration.md), and [schema evolution](../object-serialization/javascript/schema-evolution.md). diff --git a/docs/start/kotlin.md b/docs/start/kotlin.md index ad9768d65d3..e2fe8f74ab0 100644 --- a/docs/start/kotlin.md +++ b/docs/start/kotlin.md @@ -70,7 +70,7 @@ If the project applies Gradle's `application` plugin, run its application task: Use xlang mode for data shared with other Fory runtimes or native mode for Kotlin/JVM-only data. Continue with [Kotlin Object Serialization](../object-serialization/kotlin/index.md), -[xlang](../object-serialization/kotlin/xlang.md), or +[xlang](../object-serialization/kotlin/basic-serialization.md#cross-language-interoperability), or [native mode](../object-serialization/kotlin/native.md). ## Other Capabilities diff --git a/docs/start/python.md b/docs/start/python.md index 4f3d03a9d29..d7c7387bf9b 100644 --- a/docs/start/python.md +++ b/docs/start/python.md @@ -60,7 +60,7 @@ decoded = fory.deserialize(data) print(decoded) ``` -Use [xlang mode](../object-serialization/python/xlang.md) for cross-language +Use [xlang mode](../object-serialization/python/basic-serialization.md#cross-language-interoperability) for cross-language data. Use [native mode](../object-serialization/python/native.md) for Python-only objects, including Python callables and serialization hooks. Continue with the [Python guide](../object-serialization/python/index.md), diff --git a/docs/start/rust.md b/docs/start/rust.md index 30f4303c749..431c9441594 100644 --- a/docs/start/rust.md +++ b/docs/start/rust.md @@ -63,7 +63,7 @@ fn main() -> Result<(), Error> { } ``` -Use [xlang mode](../object-serialization/rust/xlang.md) for cross-language data +Use [xlang mode](../object-serialization/rust/basic-serialization.md#cross-language-interoperability) for cross-language data and [native mode](../object-serialization/rust/native.md) for Rust-only data. Continue with [Rust Object Serialization](../object-serialization/rust/index.md), [configuration](../object-serialization/rust/configuration.md), and diff --git a/docs/start/scala.md b/docs/start/scala.md index 974038f1a59..5e8a83c51bd 100644 --- a/docs/start/scala.md +++ b/docs/start/scala.md @@ -69,7 +69,7 @@ sbt run Use xlang mode for data shared with other Fory runtimes or native mode for Scala/JVM-only data. Continue with [Scala Object Serialization](../object-serialization/scala/index.md), -[xlang](../object-serialization/scala/xlang.md), or +[xlang](../object-serialization/scala/basic-serialization.md#cross-language-interoperability), or [native mode](../object-serialization/scala/native.md). ## Other Capabilities diff --git a/docs/start/swift.md b/docs/start/swift.md index 7118e7d9c72..56bb072fd80 100644 --- a/docs/start/swift.md +++ b/docs/start/swift.md @@ -37,7 +37,7 @@ Create an executable package: swift package init --type executable --name ForyExample ``` -Add the released package and `Fory` product to the generated `Package.swift`: +Add the released package and depend on its `Fory` library in the generated `Package.swift`: ```swift title="Package.swift" dependencies: [ @@ -77,7 +77,7 @@ swift run Swift uses xlang mode. Continue with [Swift Object Serialization](../object-serialization/swift/index.md), -[xlang types](../object-serialization/swift/xlang.md), +[xlang types](../object-serialization/swift/basic-serialization.md#cross-language-interoperability), [configuration](../object-serialization/swift/configuration.md), and [schema evolution](../object-serialization/swift/schema-evolution.md). diff --git a/docusaurus.config.ts b/docusaurus.config.ts index cda3314067e..79263e79e74 100644 --- a/docusaurus.config.ts +++ b/docusaurus.config.ts @@ -47,6 +47,8 @@ const config: Config = { '@docusaurus/preset-classic', { docs: { + // Internal security models and the retired shared image tree must not become website pages. + exclude: ['security/**', 'images/**'], sidebarCollapsible: true, lastVersion: '1.5.0', versions: { diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmarks/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmarks/index.md index d63eefe2401..c196520e505 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmarks/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/benchmarks/index.md @@ -1,5 +1,5 @@ --- -title: 基准测试 +title: 基准结果 sidebar_position: 1 id: index license: | diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/community/how_to_release.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/community/how_to_release.md index 4c4acc258cf..227d59f5f53 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/community/how_to_release.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/community/how_to_release.md @@ -428,7 +428,7 @@ To learn more about Fory, please see https://fory.apache.org/ [ ] No compiled archives bundled in source archive. [ ] Can compile from source. -How to Build and Test, please refer to: https://github.com/apache/fory/blob/main/docs/development/building.md +How to Build and Test, please refer to: https://github.com/apache/fory/blob/main/docs/development/index.md Thanks, diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/compiler/generated-code/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/compiler/generated-code/index.md index 5dd28a10f34..e730d92252c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/compiler/generated-code/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/compiler/generated-code/index.md @@ -1,5 +1,5 @@ --- -title: 生成的代码 +title: 概述 sidebar_position: 1 id: index license: | diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/building.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/building.md deleted file mode 100644 index b7e336c2882..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/building.md +++ /dev/null @@ -1,134 +0,0 @@ ---- -title: 开发 -sidebar_position: 20 -id: building -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -## 如何构建 Apache Fory™ - -从 https://github.com/apache/fory 克隆源代码树。 - -### 构建 Apache Fory™ Java - -```bash -cd java -mvn -T16 package -``` - -#### 环境要求 - -- JDK 17+ -- Maven 3.6.3+ - -### 构建 Apache Fory™ Python - -```bash -cd python -pip install -v -e . - -# Optional: build Cython extension (replace X.Y with your Python version) -bazel build //:cp_fory_so --@rules_python//python/config_settings:python_version=X.Y -``` - -#### 环境要求 - -- CPython 3.8+ -- Bazel 8+（构建 Cython 扩展时必需） - -### 构建 Apache Fory™ C++ - -```bash -cd cpp -bazel build //cpp/... -``` - -#### 环境要求 - -- C++17 编译器 -- Bazel 8+ - -### 构建 Apache Fory™ Go - -```bash -cd go/fory -go test -v ./... -``` - -从 Java 测试模块运行 Go xlang 测试： - -```bash -cd java -mvn -T16 install -DskipTests -cd fory-core -FORY_GO_JAVA_CI=1 ENABLE_FORY_DEBUG_OUTPUT=1 mvn test -Dtest=org.apache.fory.xlang.GoXlangTest -``` - -#### 环境要求 - -- Go 1.24+ - -### 构建 Apache Fory™ Rust - -```bash -cd rust -cargo build -cargo test --features tests - -# Debug a specific test -RUST_BACKTRACE=1 FORY_PANIC_ON_ERROR=1 ENABLE_FORY_DEBUG_OUTPUT=1 \ - cargo test --test mod $dir$::$test_file::$test_method -- --nocapture -``` - -#### 环境要求 - -- 通过 rustup 安装的 Rust 工具链 -- `cargo-expand`（可选，用于调试宏展开） - -### 构建 Apache Fory™ JavaScript - -```bash -cd javascript -npm install - -npm run build -node ./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit -``` - -#### 环境要求 - -- Node.js (LTS) -- npm - -### 检查 Markdown 文档 - -```bash -cd docs -npx prettier --write "**/*.md" -``` - -#### 环境要求 - -- Node.js (LTS) -- npm - -## 参与贡献 - -有关贡献的详细信息，请参阅[如何为 Apache Fory™ 做贡献](https://github.com/apache/fory/blob/main/CONTRIBUTING.md)。 -对于 AI 辅助的贡献，请遵循 -[AI 贡献政策](https://github.com/apache/fory/blob/main/AI_POLICY.md)，包括大量使用 AI 辅助时 -所需的自我审查、双审查者 AI 审查流程、披露和验证证据。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/cpp-debugging.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/cpp-debugging.md index 9610822aa14..15e2f31a7e9 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/cpp-debugging.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/cpp-debugging.md @@ -18,9 +18,9 @@ license: | limitations under the License. --- -import JumpGeneratedFile from "@site/docs/images/jmp_generate_file.png"; -import VscodeDebugFory from "@site/docs/images/vscode_debug_fory.jpg"; -import VscodeSelectDebugRun from "@site/docs/images/vscode_select_debug_run.png"; +import JumpGeneratedFile from "@site/docs/development/jmp_generate_file.png"; +import VscodeDebugFory from "@site/docs/development/vscode_debug_fory.jpg"; +import VscodeSelectDebugRun from "@site/docs/development/vscode_select_debug_run.png"; ## 调试 C++ diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/index.md index 9e547b6e14b..b70e93b825a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/development/index.md @@ -1,5 +1,5 @@ --- -title: 开发 +title: 构建与测试 sidebar_position: 1 id: index license: | @@ -22,11 +22,120 @@ license: | 贡献者文档涵盖仓库设置、构建、测试、调试和面向发布的工作流程。 这部分文档与已发布构件的用户指南相互独立。 -## 从这里开始 +从 https://github.com/apache/fory 克隆源代码树。 -- [构建并测试仓库](building.md) -- [调试 C++ 实现](cpp-debugging.md) -- [贡献指南](https://github.com/apache/fory/blob/main/CONTRIBUTING.md) +## Java + +```bash +cd java +mvn -T16 package +``` + +环境要求： + +- JDK 17+ +- Maven 3.6.3+ + +## Python + +```bash +cd python +pip install -v -e . + +# Optional: build Cython extension (replace X.Y with your Python version) +bazel build //:cp_fory_so --@rules_python//python/config_settings:python_version=X.Y +``` + +环境要求： + +- CPython 3.8+ +- Bazel 8+（构建 Cython 扩展时必需） + +## C++ + +```bash +cd cpp +bazel build //cpp/... +``` + +环境要求： + +- C++17 编译器 +- Bazel 8+ + +## Go + +```bash +cd go/fory +go test -v ./... +``` + +从 Java 测试模块运行 Go xlang 测试： + +```bash +cd java +mvn -T16 install -DskipTests +cd fory-core +FORY_GO_JAVA_CI=1 ENABLE_FORY_DEBUG_OUTPUT=1 mvn test -Dtest=org.apache.fory.xlang.GoXlangTest +``` + +环境要求：Go 1.24+。 + +## Rust + +```bash +cd rust +cargo build +cargo test --features tests + +# Debug a specific test +RUST_BACKTRACE=1 FORY_PANIC_ON_ERROR=1 ENABLE_FORY_DEBUG_OUTPUT=1 \ + cargo test --test mod $dir$::$test_file::$test_method -- --nocapture +``` + +环境要求： + +- 通过 rustup 安装的 Rust 工具链 +- `cargo-expand`（可选，用于调试宏展开） + +## JavaScript + +```bash +cd javascript +npm install + +npm run build +node ./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit +``` + +环境要求： + +- Node.js (LTS) +- npm + +## Markdown + +```bash +cd docs +npx prettier --write "**/*.md" +``` + +环境要求： + +- Node.js (LTS) +- npm + +## 调试 + +仓库中的 VS Code、Bazel、LLDB 和 GDB 配置请参阅[调试 C++](cpp-debugging.md)。 不属于共享仓库构建的运行时特定贡献者说明，仍保留在各运行时的源代码树和贡献者 README 中。 + +## 参与贡献 + +有关贡献的详细信息，请参阅 +[如何为 Apache Fory™ 做贡献](https://github.com/apache/fory/blob/main/CONTRIBUTING.md)。 +对于 AI 辅助的贡献，请遵循 +[AI 贡献政策](https://github.com/apache/fory/blob/main/AI_POLICY.md)，包括大量使用 AI 辅助时 +所需的自我审查、双审查者 AI 审查流程、披露和验证证据。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/architecture.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/architecture.md deleted file mode 100644 index 9e2b817e1ed..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/architecture.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -title: 架构 -sidebar_position: 2 -id: architecture -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -生成的服务配套代码使用标准 gRPC server、channel、方法描述符、deadline、状态码、interceptor -和流式 API。Fory 生成的 marshaller 负责对生成的请求和响应模型进行编码和解码。 - -## 职责边界 - -Fory 可以为应用提供的 gRPC 运行时生成服务配套代码。这些代码为请求和响应对象提供 Fory 序列化； -listener、channel、credential、身份认证、授权、deadline、重试和传输生命周期仍由应用和 gRPC 技术栈负责。 - -Fory 软件包不会将某个 gRPC 实现作为强制依赖。应用负责选择和配置运行时的 gRPC 库。 - -## 生成的服务接口 - -编译器会生成符合运行时习惯的 service base、client 或 stub、方法元数据和 Fory marshaller。 -模型生成详见[生成代码](../compiler/generated-code/index.md)；各运行时页面介绍 server 与 client 集成。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/index.md index 0d7538b872d..eb86718878c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/index.md @@ -1,5 +1,5 @@ --- -title: Fory gRPC +title: 概述 sidebar_position: 1 id: index license: | @@ -51,8 +51,44 @@ Python 默认生成异步 `grpc.aio` 配套代码，并支持通过 `--grpc-pyth JavaScript 在 Node.js 中使用 `@grpc/grpc-js`；浏览器客户端通过 `--grpc-web` 单独生成， 并使用 `grpc-web`。 +## 架构 + +生成的服务配套代码使用标准 gRPC server、channel、方法描述符、deadline、状态码、interceptor +和流式 API。Fory 生成的 marshaller 负责对生成的请求和响应模型进行编码和解码。 + +### 职责边界 + +Fory 可以为应用提供的 gRPC 运行时生成服务配套代码。这些代码为请求和响应对象提供 Fory 序列化； +listener、channel、credential、身份认证、授权、deadline、重试和传输生命周期仍由应用和 gRPC 技术栈负责。 + +Fory 软件包不会将某个 gRPC 实现作为强制依赖。应用负责选择和配置运行时的 gRPC 库。 + +### 生成的服务接口 + +编译器会生成符合运行时习惯的 service base、client 或 stub、方法元数据和 Fory marshaller。 +模型生成详见[生成代码](../compiler/generated-code/index.md)；各运行时页面介绍 server 与 client 集成。 + +## 互操作性 + +只有使用同一份生成服务契约、匹配的 Fory 类型标识以及兼容的生成模型 Schema，Fory gRPC 对端 +才能互操作。 + +### 协议边界 + +传输协议是 gRPC，但消息字节是 Fory 载荷。通用 protobuf 客户端和 server reflection 工具无法将 +这些载荷解码为 protobuf 消息。每个对端都必须通过受支持的 Fory 编译器前端生成。 + +### 验证 + +至少测试一个一元调用，以及服务使用的每种流式调用形式。protobuf `UNIMPLEMENTED` 或解码失败 +通常表示对端使用了普通 protobuf stub，或使用了不同的生成服务契约。 + ## 运行时指南 +Java、Python、C++、Go、Rust、JavaScript/TypeScript、C#、Dart、Scala 和 Kotlin 均有相应的 +gRPC 配套代码文档。当前依赖和流式调用支持请参阅[支持矩阵](../introduction/support-matrix.md) +以及所选运行时页面。 + | 运行时 | 指南 | | --------------------- | ------------------------------------ | | Java | [Java](java.md) | @@ -65,5 +101,3 @@ JavaScript 在 Node.js 中使用 `@grpc/grpc-js`；浏览器客户端通过 `--g | Dart | [Dart](dart.md) | | Scala | [Scala](scala.md) | | Kotlin | [Kotlin](kotlin.md) | - -载荷和传输的职责边界请参阅[架构](architecture.md)，对端兼容性请参阅[互操作性](interoperability.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/interoperability.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/interoperability.md deleted file mode 100644 index 5d1f4d93007..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/grpc/interoperability.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -title: 互操作性 -sidebar_position: 3 -id: interoperability -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -只有使用同一份生成服务契约、匹配的 Fory 类型标识以及兼容的生成模型 Schema，Fory gRPC 对端 -才能互操作。 - -## 协议边界 - -传输协议是 gRPC，但消息字节是 Fory 载荷。通用 protobuf 客户端和 server reflection 工具无法将 -这些载荷解码为 protobuf 消息。每个对端都必须通过受支持的 Fory 编译器前端生成。 - -## 支持的生成配套代码 - -Java、Python、C++、Go、Rust、JavaScript/TypeScript、C#、Dart、Scala 和 Kotlin 均有相应的 -gRPC 配套代码文档。当前依赖和流式调用支持请参阅[支持矩阵](../introduction/support-matrix.md) -以及所选运行时页面。 - -## 验证 - -至少测试一个一元调用，以及服务使用的每种流式调用形式。protobuf `UNIMPLEMENTED` 或解码失败 -通常表示对端使用了普通 protobuf stub，或使用了不同的生成服务契约。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/choose-a-format.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/choose-a-format.md index 30f64384f5a..a4e762440e5 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/choose-a-format.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/choose-a-format.md @@ -19,12 +19,15 @@ license: | limitations under the License. --- -| 格式 | 适用场景 | 入门文档 | -| ------------- | ------------------------------------ | ---------------------------------------------------- | -| Xlang 二进制 | 数据需要跨语言传输 | [跨语言指南](../object-serialization/xlang/index.md) | -| Native 二进制 | 生产端和消费端使用同一种语言 | 相应语言指南 | -| Row Format | 需要随机字段访问或分析场景的部分读取 | [Row Format 规范](../row-format/index.md) | -| Fory JSON | Java 应用需要高性能标准 JSON | [Fory JSON 指南](../json/index.md) | +| 格式或模式 | 数据模型 | 适用场景 | 互操作范围 | 入门文档 | +| ------------- | --------------------------- | ------------------------------------ | ------------------------------------------------------------------- | ---------------------------------------------- | +| Xlang 二进制 | 可移植对象图 | 数据需要跨语言传输 | 支持的运行时共享同一种线格式 | [跨语言指南](../object-serialization/xlang.md) | +| Native 二进制 | 运行时原生对象图 | 生产端和消费端使用同一种语言 | 仅限同一运行时家族 | [对象序列化](../object-serialization/index.md) | +| Row Format | 可随机访问的二进制行 | 需要随机字段访问或分析场景的部分读取 | Standard Row 由 Java、Python、C++ 和 Rust 共享；Compact 仅支持 Java | [Row Format 指南](../row-format/index.md) | +| Fory JSON | 映射到 Java 对象的标准 JSON | Java 应用需要高性能标准 JSON | 标准 JSON 文本 | [Fory JSON 指南](../json/index.md) | + +Xlang 和 native 是对象序列化的两种并列模式，适用于接收端需要重建对象图的场景。Row Format +和 Fory JSON 是独立格式，不是额外的对象序列化模式。 对于 Java、Scala、Kotlin、Python、C++、Go 和 Rust，同语言通信应使用 native 模式。它避开 xlang 的跨语言类型映射和元数据限制，更贴近各语言的原生类型系统，并支持更丰富的语言特定 @@ -51,3 +54,10 @@ pickle 和 cloudpickle。 需要重建对象图时选择 xlang 或 native 模式。对于可信的分析数据，如果随机字段访问能带来 收益，请选择 Row Format。Java 应用需要标准 JSON 时选择 Fory JSON。多个团队需要统一的 Schema 优先契约时，请使用 Fory IDL 和编译器；它会生成使用相应 Fory 能力的模型。 + +## 相关能力 + +[Fory IDL 和编译器](../compiler/index.md)可以为支持的运行时生成原生模型。服务定义还可以生成 +[Fory gRPC](../grpc/index.md)代码。这两项能力都不会定义新的序列化格式。 + +协议实现者应遵循 [Specifications](../specification/xlang_serialization_spec.md) 中的规范性格式文档。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/index.md index f854e086ee4..07770b0cc71 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/index.md @@ -1,5 +1,5 @@ --- -title: 简介 +title: 概述 sidebar_position: 1 id: index license: | diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/support-matrix.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/support-matrix.md index d2245f59cdf..91c44820b72 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/support-matrix.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/introduction/support-matrix.md @@ -20,7 +20,7 @@ license: | --- 选择能力前，请使用此矩阵确认文档所覆盖的 API。存在某个运行时页面，并不表示该运行时 -支持所有 Fory 产品。 +支持所有 Fory 能力。 | 能力 | 已有文档的运行时 | 互操作能力 | | ------------------- | ---------------------------------------------------------------------------------- | -------------------------------------- | @@ -34,4 +34,4 @@ license: | [Android](../object-serialization/java/android.md) 和 [GraalVM Native Image](../object-serialization/java/graalvm.md) 等平台限制，由相应的 Java -对象序列化和 Fory JSON 产品文档说明。 +对象序列化和 Fory JSON 指南说明。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/json/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/json/index.md index 8e86d5b9bc0..71bc8b99d4e 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/json/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/json/index.md @@ -1,5 +1,5 @@ --- -title: Fory JSON +title: 概述 sidebar_position: 1 id: index license: | @@ -58,5 +58,5 @@ Fory core 的 native 或 xlang 二进制协议。 ## 相关 Java 指南 二进制序列化请从 [Java 对象序列化](../object-serialization/java/index.md)开始，并选择 -[xlang](../object-serialization/java/xlang.md) 或 [native](../object-serialization/java/native.md)。 +[xlang](../object-serialization/java/basic-serialization.md#cross-language-interoperability) 或 [native](../object-serialization/java/native.md)。 二进制 builder 选项另见 [Java 配置](../object-serialization/java/configuration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/core-concepts.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/core-concepts.md new file mode 100644 index 00000000000..c3654cf1d0a --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/core-concepts.md @@ -0,0 +1,105 @@ +--- +title: 核心概念 +sidebar_position: 1 +id: core-concepts +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Fory 对象序列化把对象图转换为字节，并在读取时重建对象图。默认的 +[xlang 模式](xlang.md)和受支持的[原生模式](native.md)都使用这些核心概念；所选模式决定 +可使用的类型和线格式规则。 + +## 对象图 + +一个根值可以包含标量字段、集合、Map、嵌套对象、重复引用和循环引用。序列化从根值开始遍历 +对象图，反序列化则根据编码的类型和字段数据创建新的对象图。 + +这与序列化一行数据或 JSON 文档不同。对象序列化可以保留运行时类型和对象身份，使读取端能够 +重建应用对象，而不只是读取值。可信的分析数据应使用[行格式](../row-format/index.md)，JSON +交换应使用 [Fory JSON](../json/index.md)。 + +## 运行时实例和注册 + +Fory 实例持有序列化模式、Schema 行为、引用设置、已注册类型、自定义序列化器和读取限制。 +应在第一次根序列化或反序列化之前完成配置和注册，然后复用该实例。第一次根操作开始后，注册 +会被冻结，从而保证同一个实例始终以相同方式解析类型。 + +各运行时的线程安全方式不同。有些运行时提供线程安全包装器或对象池，另一些要求每个线程或任务 +使用独立实例。共享普通实例之前，请先查看对应语言文档中的并发约定。 + +## 类型和类型标识 + +内置类型的标识由 Fory 定义。应用的 Struct、Class、Enum、Union 和扩展类型使用注册的数字 ID +或名称。类型标识决定“由哪个序列化器和模型读取该值”，字段 Schema 则描述“模型包含哪些数据”。 + +静态已知字段可以直接使用其声明类型。动态字段还会携带具体运行时类型，用于接口、抽象类、 +Trait Object、宽泛对象类型或异构值。动态类型更灵活，但所有可能的具体类型都必须完成注册， +并且受到所选模式支持。 + +xlang 模式要求所有通信方协调一致的可移植类型标识和类型映射；原生模式可以使用运行时特有的 +类型与标识。可移植规则请参阅 [Xlang 序列化](xlang.md)，具体注册 API 请参阅各语言的类型注册页面。 + +## Schema 和演进 + +Schema 描述结构化值的字段及其嵌套类型。兼容模式携带 Schema 元数据，使读取端能够处理受支持的 +字段增加、删除、重排和类型适配。读写端可能独立部署时，应使用兼容模式。 + +同 Schema 模式假设两端拥有完全一致的类型标识、字段、嵌套类型、可空性和引用元数据。它可以减少 +元数据和字节大小，但 Schema 不一致会导致错误。只有同一个发布流程能保证所有读写端始终同步时 +才应使用该模式。 + +契约发布后，应保持字段 ID 或名称稳定。重命名或复用标识可能使一次预期的演进被解释为另一个字段 +或类型。 + +## 可空性 + +可空性决定一个值位置是否允许没有值。不同语言通过可空引用、Option 类型、指针、注解或 Schema +元数据表达可空性。可空字段与“字段值恰好等于默认值”不是同一概念。 + +读写端应保持可空性一致。兼容模式可以处理文档明确支持的可空和缺失字段情况，但不能把远端的 +null 放入本地没有合法 null 或缺失值表示的载体。 + +## 引用跟踪 + +引用跟踪用于保留对象身份。当对象图多次包含同一个对象，或包含循环引用时，应启用引用跟踪。 +如果未启用，重复值可能被重建为多个不同对象，循环引用也可能递归直到操作失败。 + +对于不关心对象身份的无环值型数据，应保持引用跟踪关闭，以避免额外的对象元数据和查找开销。 +有些运行时同时使用全局设置和字段级元数据，准确行为请参阅对应语言的“引用”或“基础序列化”页面。 + +## 多态 + +当字段声明类型比实际值更宽泛时，多态序列化会保存该值的具体类型。读取端必须知道并接受该具体 +类型，而且该类型必须能由所选模式表示。 + +仅有宿主语言继承关系并不会自动形成可移植契约。跨语言数据只能使用所有通信方都有 xlang 映射的 +候选类型。同运行时数据可以通过原生模式使用更多语言特有的 Class、Trait 或钩子行为。 + +## 自定义序列化器 + +当内置 Schema 推断无法表达某个类型时，可以使用自定义序列化器。注册操作把自定义序列化器与 +应用类型关联起来。自定义序列化器必须遵守所选模式的规则：xlang 序列化器需要可移植表示，原生 +序列化器可以使用运行时特有的数据和钩子。 + +如果内置序列化器或生成模型已经能表达该类型，应优先使用它们，以便更容易理解 Schema 演进、 +引用处理和跨语言行为。 + +## 继续选择模式 + +- [Xlang 序列化](xlang.md)是默认模式；不同语言运行时交换字节时必须使用它。 +- [原生序列化](native.md)用于受支持的同运行时场景，适合需要原生类型或原生行为的数据。 +- 选择模式后，再进入具体语言目录查看安装、API、配置、注册、平台、安全和故障排除文档。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/core-api.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/basic-serialization.md similarity index 50% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/core-api.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/basic-serialization.md index 7ab2c46965d..1edad9254a7 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/core-api.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: 基本序列化 -sidebar_position: 3 -id: core-api +title: 基础序列化 +sidebar_position: 1 +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,7 +19,7 @@ license: | limitations under the License. --- -本文介绍基本对象图序列化和核心序列化 API。 +本文介绍 Fory C++ 默认 xlang 模式下的基本对象图序列化和核心序列化 API。 ## 对象图序列化 @@ -343,6 +343,281 @@ fory.register_struct(2); - **禁用跟踪**：不需要引用时使用 `track_ref(false)` - **紧凑编码**：使用变长编码提高空间效率 +## 跨语言互操作 {#cross-language-interoperability} + +以下内容说明默认 xlang 格式的跨语言类型映射、类型标识和互操作要求。 + +本文说明如何在 C++ 与其他语言之间使用 Fory 跨语言序列化。 + +### 概述 + +Apache Fory™ 支持 C++、Java、Python、Go、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala 和 Kotlin 之间的无缝数据交换。跨语言模式确保所有受支持语言之间的二进制兼容性。 + +### Xlang 配置 + +C++ 默认使用跨语言模式，兼容 Schema 演进也是该模式的默认设置。跨语言示例中应显式设置模式： + +```cpp +#include "fory/serialization/fory.h" + +using namespace fory::serialization; + +auto fory = Fory::builder().xlang(true).build(); +``` + +### 跨语言示例 + +#### C++ 生产端 + +```cpp +#include "fory/serialization/fory.h" +#include + +using namespace fory::serialization; + +struct Message { + std::string topic; + int64_t timestamp; + std::map headers; + std::vector payload; + + bool operator==(const Message &other) const { + return topic == other.topic && timestamp == other.timestamp && + headers == other.headers && payload == other.payload; + } +}; +FORY_STRUCT(Message, topic, timestamp, headers, payload); + +int main() { + auto fory = Fory::builder().xlang(true).build(); + fory.register_struct(100); + + Message msg{ + "events.user", + 1699999999000, + {{"content-type", "application/json"}}, + {'h', 'e', 'l', 'l', 'o'} + }; + + auto result = fory.serialize(msg); + if (result.ok()) { + auto bytes = std::move(result).value(); + // write to file, send over network, etc. + std::ofstream file("message.bin", std::ios::binary); + file.write(reinterpret_cast(bytes.data()), bytes.size()); + } + return 0; +} +``` + +#### Java 消费端 + +```java +import org.apache.fory.Fory; + +public class Message { + public String topic; + public long timestamp; + public Map headers; + public byte[] payload; +} + +public class Consumer { + public static void main(String[] args) throws Exception { + Fory fory = Fory.builder() + .withXlang(true) + .build(); + fory.register(Message.class, 100); // Same ID as C++ + + byte[] bytes = Files.readAllBytes(Path.of("message.bin")); + Message msg = (Message) fory.deserialize(bytes); + + System.out.println("Topic: " + msg.topic); + System.out.println("Timestamp: " + msg.timestamp); + } +} +``` + +#### Python 消费端 + +```python +import pyfory + +class Message: + topic: str + timestamp: int + headers: dict[str, str] + payload: bytes + +fory = pyfory.Fory(xlang=True) +fory.register(Message, type_id=100) # Same ID as C++ + +with open("message.bin", "rb") as f: + data = f.read() + +msg = fory.deserialize(data) +print(f"Topic: {msg.topic}") +print(f"Timestamp: {msg.timestamp}") +``` + +### 类型映射 + +#### 原始类型 + +| C++ Type | Java Type | Python Type | Go Type | Rust Type | +| ------------------ | ---------- | ----------------- | ------------------- | ---------- | +| `bool` | `boolean` | `bool` | `bool` | `bool` | +| `int8_t` | `byte` | `int` | `int8` | `i8` | +| `int16_t` | `short` | `int` | `int16` | `i16` | +| `int32_t` | `int` | `int` | `int32` | `i32` | +| `int64_t` | `long` | `int` | `int64` | `i64` | +| `float` | `float` | `float` | `float32` | `f32` | +| `double` | `double` | `float` | `float64` | `f64` | +| `fory::float16_t` | `Float16` | `pyfory.Float16` | `float16.Float16` | `Float16` | +| `fory::bfloat16_t` | `BFloat16` | `pyfory.BFloat16` | `bfloat16.BFloat16` | `BFloat16` | + +#### 字符串类型 + +| C++ Type | Java Type | Python Type | Go Type | Rust Type | +| ------------- | --------- | ----------- | -------- | --------- | +| `std::string` | `String` | `str` | `string` | `String` | + +#### 集合类型 + +| C++ Type | Java Type | Python Type | Go Type | Rust Type | +| ------------------------------------------- | -------------- | --------------- | --------------------- | --------------- | +| `std::vector` | `List` | `list` | `[]T` | `Vec` | +| `std::vector` | `Float16List` | `Float16Array` | `[]float16.Float16` | `Vec` | +| `std::vector` | `BFloat16List` | `BFloat16Array` | `[]bfloat16.BFloat16` | `Vec` | +| `std::set` | `Set` | `set` | `map[T]struct{}` | `HashSet` | +| `std::map` / `std::unordered_map` | `Map` | `dict` | `map[K]V` | `HashMap` | + +#### 列表与稠密数组 + +在手写 C++ 结构体中，`std::vector` 默认映射到 Fory `list`。当 Schema 是稠密 `array` 时，请使用字段元数据 DSL 的数组节点。 + +| Fory Schema | C++ 元数据示例 | +| ----------------- | ---------------------------------------- | +| `list` | `fory::F(id).list(fory::T::int32())` | +| `array` | `fory::F(id).array(fory::T::bool_())` | +| `array` | `fory::F(id).array(fory::T::int8())` | +| `array` | `fory::F(id).array(fory::T::int16())` | +| `array` | `fory::F(id).array(fory::T::int32())` | +| `array` | `fory::F(id).array(fory::T::int64())` | +| `array` | `fory::F(id).array(fory::T::uint8())` | +| `array` | `fory::F(id).array(fory::T::uint16())` | +| `array` | `fory::F(id).array(fory::T::uint32())` | +| `array` | `fory::F(id).array(fory::T::uint64())` | +| `array` | `fory::F(id).array(fory::T::float16())` | +| `array` | `fory::F(id).array(fory::T::bfloat16())` | +| `array` | `fory::F(id).array(fory::T::float32())` | +| `array` | `fory::F(id).array(fory::T::float64())` | + +#### 时间类型 + +| C++ Type | Java Type | Python Type | Go Type | +| ----------------- | ----------- | --------------- | --------------- | +| `fory::Timestamp` | `Instant` | `datetime` | `time.Time` | +| `fory::Duration` | `Duration` | `timedelta` | `time.Duration` | +| `fory::Date` | `LocalDate` | `datetime.date` | `time.Time` | + +### 字段顺序要求 + +**重要：**字段按 snake_case 字段名称排序，转换后的名称必须在各语言之间一致。 + +#### C++ + +```cpp +struct Person { + std::string name; // Field 0 + int32_t age; // Field 1 + std::string email; // Field 2 +}; +FORY_STRUCT(Person, name, age, email); // Order matters! +``` + +#### Java + +```java +public class Person { + public String name; // Field 0 + public int age; // Field 1 + public String email; // Field 2 +} +``` + +#### Python + +```python +class Person: + name: str # Field 0 + age: int # Field 1 + email: str # Field 2 +``` + +### 类型 ID 一致性 + +所有语言都必须使用相同的类型 ID： + +```cpp +// C++ +fory.register_struct(100); +fory.register_struct
(101); +fory.register_struct(102); +``` + +```java +// Java +fory.register(Person.class, 100); +fory.register(Address.class, 101); +fory.register(Order.class, 102); +``` + +```python +# Python +fory.register(Person, type_id=100) +fory.register(Address, type_id=101) +fory.register(Order, type_id=102) +``` + +### 兼容模式 + +跨语言模式默认已使用兼容 Schema 演进。对于可能独立演进的 Schema，请保留该默认设置： + +```cpp +auto fory = Fory::builder().xlang(true).build(); +``` + +兼容模式允许： + +- 添加新字段（带默认值） +- 删除未使用的字段 +- 重排字段 + +### 互操作故障排查 + +#### 类型不匹配错误 + +``` +Error: Type mismatch: expected 100, got 101 +``` + +**解决方案：**确保所有语言的类型 ID 一致。 + +#### 编码错误 + +``` +Error: Invalid UTF-8 sequence +``` + +**解决方案：**确保所有语言中的字符串都是有效的 UTF-8。 + +### 相关指南 + +- [配置](configuration.md) - 构建器选项 +- [类型注册](type-registration.md) - 注册类型 +- [支持的类型](supported-types.md) - 类型兼容性 + ## 相关主题 - [配置](configuration.md) - 构建器选项 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/configuration.md index 41de723e1ba..8fbd8edfea3 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/configuration.md @@ -233,17 +233,10 @@ auto fory = Fory::builder().build_thread_safe(); // Returns ThreadSafeFory ## 安全 -安全相关配置： - -- 反序列化不可信载荷之前，注册所有结构体和多态实现。 -- 对有意使用相同 Schema 的载荷，将 `check_struct_version(true)` 与 `compatible(false)` 配合使用。 -- 对大多数输入，将 `max_graph_memory_bytes(...)` 保持为固定默认值 `128 MiB`；只有可信工作负载需要不同的集合/映射/结构体限制时才设置正值。 -- 在模型允许范围内尽量降低 `max_dyn_depth(...)`，以拒绝意外过深的多态对象图。 -- 除非数据确定无恶意且可信对等端会发送更大的元数据或大量 Schema 版本，否则请保留远端 Schema 元数据限制的默认值。 -- 对不可信输入，优先使用具体字段而非宽泛的多态字段。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [C++ 安全](security.md)。 ## 相关主题 -- [基本序列化](core-api.md) - 使用已配置的 Fory -- [跨语言序列化](xlang.md) - 跨语言模式详情 +- [基本序列化](basic-serialization.md) - 使用已配置的 Fory +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言模式详情 - [类型注册](type-registration.md) - 注册类型 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/custom-serializers.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/custom-serializers.md index 2894517bf72..6a312b080e3 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/custom-serializers.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/custom-serializers.md @@ -366,6 +366,6 @@ static MyType read_data(ReadContext &ctx) { ## 相关主题 - [类型注册](type-registration.md) - 注册序列化器 -- [基本序列化](core-api.md) - 使用 FORY_STRUCT 宏 +- [基本序列化](basic-serialization.md) - 使用 FORY_STRUCT 宏 - [Schema 演进](schema-evolution.md) - 兼容模式 -- [跨语言序列化](xlang.md) - 跨语言序列化 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言序列化 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/index.md index e2bb83c3b10..24fdcfd9d8a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/index.md @@ -217,7 +217,7 @@ struct Derived : Base { 仅限 C++ 的通信应使用原生模式。通过 `.xlang(false)` 选择原生模式，它会让 C++ 对象序列化保持 C++ 原生形式。载荷不会离开 C++ 时，该模式针对 C++ 类型进行了优化，并避免可移植跨语言类型映射的约束。兼容模式默认启用。只有每个读取端和写入端都使用相同的 C++ Schema，并且希望获得更快的序列化和更小的体积时，才设置 `.compatible(false)`。 -C++ 跨语言注册和互操作规则参见[跨语言序列化](xlang.md)，仅限 C++ 的载荷参见[原生序列化](native.md)。 +C++ 跨语言注册和互操作规则参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)，仅限 C++ 的载荷参见[原生序列化](native.md)。 ## 线程安全 @@ -259,8 +259,7 @@ std::thread t2([&]() { ## 后续步骤 - [配置](configuration.md) - 构建器选项和模式 -- [基本序列化](core-api.md) - 对象图序列化 -- [跨语言序列化](xlang.md) - 跨语言模式和互操作性 +- [基本序列化](basic-serialization.md) - 对象图序列化 - [原生序列化](native.md) - 仅限 C++ 的序列化 - [Schema 元数据](schema-metadata.md) - 字段级元数据（可空、引用跟踪） - [Schema 演进](schema-evolution.md) - 兼容模式和 Schema 变更 @@ -269,3 +268,5 @@ std::thread t2([&]() { - [自定义序列化器](custom-serializers.md) - 扩展序列化行为 - [行格式](../../row-format/cpp.md) - 零拷贝行格式 - [gRPC 支持](../../grpc/cpp.md) - 通过 gRPC C++ 传输 Fory 载荷 + +解码来自应用信任边界之外的字节之前，请阅读 [C++ 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/native.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/native.md index ca5b91fc80b..088d5b9f7b7 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/native.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/native.md @@ -21,7 +21,7 @@ license: | C++ 原生序列化是通过 `.xlang(false)` 选择、仅限 C++ 的编码模式。当所有写入端和读取端都是 C++，并且载荷应遵循 C++ 类型行为而非可移植的跨语言类型系统时，请使用该模式。 -如果字节需要由 Java、Python、Go、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala、Kotlin 或其他非 C++ Fory 实现读取，请使用 C++ 默认模式[跨语言序列化](xlang.md)。 +如果字节需要由 Java、Python、Go、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala、Kotlin 或其他非 C++ Fory 实现读取，请使用 C++ 默认模式[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ## 何时使用原生序列化 @@ -186,9 +186,9 @@ auto unsigned_value = fory.deserialize(unsigned_bytes).value(); ## 相关主题 -- [跨语言序列化](xlang.md) - 跨语言 C++ 载荷 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言 C++ 载荷 - [配置](configuration.md) - 构建器选项 -- [基本序列化](core-api.md) - 对象图序列化 +- [基本序列化](basic-serialization.md) - 对象图序列化 - [支持的类型](supported-types.md) - C++ 类型支持 - [多态序列化](polymorphism.md) - 多态对象模型 - [Schema 演进](schema-evolution.md) - 兼容模式 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/polymorphism.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/polymorphism.md index 85fd4b9fa35..fe2e5276695 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/polymorphism.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/polymorphism.md @@ -479,4 +479,4 @@ if (!decoded_result.ok()) { - [Schema 元数据](schema-metadata.md) - 字段级元数据和选项 - [支持的类型](supported-types.md) - 智能指针和集合 - [配置](configuration.md) - `max_dyn_depth` 和其他设置 -- [基本序列化](core-api.md) - 核心序列化概念 +- [基本序列化](basic-serialization.md) - 核心序列化概念 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/schema-evolution.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/schema-evolution.md index afeb3cba216..2d2d5f4e351 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/schema-evolution.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/schema-evolution.md @@ -411,4 +411,4 @@ Fory fory = Fory.builder() - [配置](configuration.md) - 启用兼容模式 - [类型注册](type-registration.md) - 类型 ID 管理 -- [跨语言序列化](xlang.md) - 跨语言注意事项 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言注意事项 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/security.md new file mode 100644 index 00000000000..6c36bb5f2fa --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/security.md @@ -0,0 +1,51 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 C++ 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +安全相关配置： + +- 反序列化不可信载荷之前，注册所有结构体和多态实现。 +- 对有意使用相同 Schema 的载荷，将 `check_struct_version(true)` 与 `compatible(false)` 配合使用。 +- 对大多数输入，将 `max_graph_memory_bytes(...)` 保持为固定默认值 `128 MiB`；只有可信工作负载需要不同的集合/映射/结构体限制时才设置正值。 +- 在模型允许范围内尽量降低 `max_dyn_depth(...)`，以拒绝意外过深的多态对象图。 +- 除非数据确定无恶意且可信对等端会发送更大的元数据或大量 Schema 版本，否则请保留远端 Schema 元数据限制的默认值。 +- 对不可信输入，优先使用具体字段而非宽泛的多态字段。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[类型注册](type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/supported-types.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/supported-types.md index f855746c3c1..e8d543c65a3 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/supported-types.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/supported-types.md @@ -287,6 +287,6 @@ auto decoded = fory.deserialize(bytes).value(); ## 相关主题 -- [基本序列化](core-api.md) - 使用这些类型 +- [基本序列化](basic-serialization.md) - 使用这些类型 - [类型注册](type-registration.md) - 注册类型 -- [跨语言序列化](xlang.md) - 跨语言兼容性 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言兼容性 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/type-registration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/type-registration.md index c0c5f16ab69..793cb01d0be 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/type-registration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/type-registration.md @@ -249,6 +249,6 @@ if (!result.ok()) { ## 相关主题 -- [基本序列化](core-api.md) - 使用已注册类型 -- [跨语言序列化](xlang.md) - 跨语言注意事项 +- [基本序列化](basic-serialization.md) - 使用已注册类型 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言注意事项 - [支持的类型](supported-types.md) - 所有支持的类型 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/xlang.md deleted file mode 100644 index 11d1a2d1b87..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/cpp/xlang.md +++ /dev/null @@ -1,291 +0,0 @@ ---- -title: 跨语言序列化 -sidebar_position: 1 -id: xlang -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本文说明如何在 C++ 与其他语言之间使用 Fory 跨语言序列化。 - -## 概述 - -Apache Fory™ 支持 C++、Java、Python、Go、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala 和 Kotlin 之间的无缝数据交换。跨语言模式确保所有受支持语言之间的二进制兼容性。 - -## 创建跨语言 Fory 实例 - -C++ 默认使用跨语言模式，兼容 Schema 演进也是该模式的默认设置。跨语言示例中应显式设置模式： - -```cpp -#include "fory/serialization/fory.h" - -using namespace fory::serialization; - -auto fory = Fory::builder().xlang(true).build(); -``` - -## 跨语言示例 - -### C++ 生产端 - -```cpp -#include "fory/serialization/fory.h" -#include - -using namespace fory::serialization; - -struct Message { - std::string topic; - int64_t timestamp; - std::map headers; - std::vector payload; - - bool operator==(const Message &other) const { - return topic == other.topic && timestamp == other.timestamp && - headers == other.headers && payload == other.payload; - } -}; -FORY_STRUCT(Message, topic, timestamp, headers, payload); - -int main() { - auto fory = Fory::builder().xlang(true).build(); - fory.register_struct(100); - - Message msg{ - "events.user", - 1699999999000, - {{"content-type", "application/json"}}, - {'h', 'e', 'l', 'l', 'o'} - }; - - auto result = fory.serialize(msg); - if (result.ok()) { - auto bytes = std::move(result).value(); - // write to file, send over network, etc. - std::ofstream file("message.bin", std::ios::binary); - file.write(reinterpret_cast(bytes.data()), bytes.size()); - } - return 0; -} -``` - -### Java 消费端 - -```java -import org.apache.fory.Fory; - -public class Message { - public String topic; - public long timestamp; - public Map headers; - public byte[] payload; -} - -public class Consumer { - public static void main(String[] args) throws Exception { - Fory fory = Fory.builder() - .withXlang(true) - .build(); - fory.register(Message.class, 100); // Same ID as C++ - - byte[] bytes = Files.readAllBytes(Path.of("message.bin")); - Message msg = (Message) fory.deserialize(bytes); - - System.out.println("Topic: " + msg.topic); - System.out.println("Timestamp: " + msg.timestamp); - } -} -``` - -### Python 消费端 - -```python -import pyfory - -class Message: - topic: str - timestamp: int - headers: dict[str, str] - payload: bytes - -fory = pyfory.Fory(xlang=True) -fory.register(Message, type_id=100) # Same ID as C++ - -with open("message.bin", "rb") as f: - data = f.read() - -msg = fory.deserialize(data) -print(f"Topic: {msg.topic}") -print(f"Timestamp: {msg.timestamp}") -``` - -## 类型映射 - -### 原始类型 - -| C++ Type | Java Type | Python Type | Go Type | Rust Type | -| ------------------ | ---------- | ----------------- | ------------------- | ---------- | -| `bool` | `boolean` | `bool` | `bool` | `bool` | -| `int8_t` | `byte` | `int` | `int8` | `i8` | -| `int16_t` | `short` | `int` | `int16` | `i16` | -| `int32_t` | `int` | `int` | `int32` | `i32` | -| `int64_t` | `long` | `int` | `int64` | `i64` | -| `float` | `float` | `float` | `float32` | `f32` | -| `double` | `double` | `float` | `float64` | `f64` | -| `fory::float16_t` | `Float16` | `pyfory.Float16` | `float16.Float16` | `Float16` | -| `fory::bfloat16_t` | `BFloat16` | `pyfory.BFloat16` | `bfloat16.BFloat16` | `BFloat16` | - -### 字符串类型 - -| C++ Type | Java Type | Python Type | Go Type | Rust Type | -| ------------- | --------- | ----------- | -------- | --------- | -| `std::string` | `String` | `str` | `string` | `String` | - -### 集合类型 - -| C++ Type | Java Type | Python Type | Go Type | Rust Type | -| ------------------------------------------- | -------------- | --------------- | --------------------- | --------------- | -| `std::vector` | `List` | `list` | `[]T` | `Vec` | -| `std::vector` | `Float16List` | `Float16Array` | `[]float16.Float16` | `Vec` | -| `std::vector` | `BFloat16List` | `BFloat16Array` | `[]bfloat16.BFloat16` | `Vec` | -| `std::set` | `Set` | `set` | `map[T]struct{}` | `HashSet` | -| `std::map` / `std::unordered_map` | `Map` | `dict` | `map[K]V` | `HashMap` | - -### 列表与稠密数组 - -在手写 C++ 结构体中，`std::vector` 默认映射到 Fory `list`。当 Schema 是稠密 `array` 时，请使用字段元数据 DSL 的数组节点。 - -| Fory Schema | C++ 元数据示例 | -| ----------------- | ---------------------------------------- | -| `list` | `fory::F(id).list(fory::T::int32())` | -| `array` | `fory::F(id).array(fory::T::bool_())` | -| `array` | `fory::F(id).array(fory::T::int8())` | -| `array` | `fory::F(id).array(fory::T::int16())` | -| `array` | `fory::F(id).array(fory::T::int32())` | -| `array` | `fory::F(id).array(fory::T::int64())` | -| `array` | `fory::F(id).array(fory::T::uint8())` | -| `array` | `fory::F(id).array(fory::T::uint16())` | -| `array` | `fory::F(id).array(fory::T::uint32())` | -| `array` | `fory::F(id).array(fory::T::uint64())` | -| `array` | `fory::F(id).array(fory::T::float16())` | -| `array` | `fory::F(id).array(fory::T::bfloat16())` | -| `array` | `fory::F(id).array(fory::T::float32())` | -| `array` | `fory::F(id).array(fory::T::float64())` | - -### 时间类型 - -| C++ Type | Java Type | Python Type | Go Type | -| ----------------- | ----------- | --------------- | --------------- | -| `fory::Timestamp` | `Instant` | `datetime` | `time.Time` | -| `fory::Duration` | `Duration` | `timedelta` | `time.Duration` | -| `fory::Date` | `LocalDate` | `datetime.date` | `time.Time` | - -## 字段顺序要求 - -**重要：**字段按 snake_case 字段名称排序，转换后的名称必须在各语言之间一致。 - -### C++ - -```cpp -struct Person { - std::string name; // Field 0 - int32_t age; // Field 1 - std::string email; // Field 2 -}; -FORY_STRUCT(Person, name, age, email); // Order matters! -``` - -### Java - -```java -public class Person { - public String name; // Field 0 - public int age; // Field 1 - public String email; // Field 2 -} -``` - -### Python - -```python -class Person: - name: str # Field 0 - age: int # Field 1 - email: str # Field 2 -``` - -## 类型 ID 一致性 - -所有语言都必须使用相同的类型 ID： - -```cpp -// C++ -fory.register_struct(100); -fory.register_struct
(101); -fory.register_struct(102); -``` - -```java -// Java -fory.register(Person.class, 100); -fory.register(Address.class, 101); -fory.register(Order.class, 102); -``` - -```python -# Python -fory.register(Person, type_id=100) -fory.register(Address, type_id=101) -fory.register(Order, type_id=102) -``` - -## 兼容模式 - -跨语言模式默认已使用兼容 Schema 演进。对于可能独立演进的 Schema，请保留该默认设置： - -```cpp -auto fory = Fory::builder().xlang(true).build(); -``` - -兼容模式允许： - -- 添加新字段（带默认值） -- 删除未使用的字段 -- 重排字段 - -## 故障排查 - -### 类型不匹配错误 - -``` -Error: Type mismatch: expected 100, got 101 -``` - -**解决方案：**确保所有语言的类型 ID 一致。 - -### 编码错误 - -``` -Error: Invalid UTF-8 sequence -``` - -**解决方案：**确保所有语言中的字符串都是有效的 UTF-8。 - -## 相关主题 - -- [配置](configuration.md) - 构建器选项 -- [类型注册](type-registration.md) - 注册类型 -- [支持的类型](supported-types.md) - 类型兼容性 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/core-api.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/basic-serialization.md similarity index 56% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/core-api.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/basic-serialization.md index 04d27b400fe..2c16aae5f4e 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/core-api.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: 基本序列化 -sidebar_position: 3 -id: core-api +title: 基础序列化 +sidebar_position: 1 +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,7 +19,7 @@ license: | limitations under the License. --- -本页介绍 Apache Fory™ C# 的类型化序列化 API。 +本页介绍 Apache Fory™ C# 默认 xlang 模式下的类型化序列化 API。 ## 对象图序列化 @@ -148,6 +148,119 @@ fory.Serialize(dynamicWriter, value); - 原始类型和集合不需要用户注册。 - 显式注册由 `[ForyStruct]`、`[ForyEnum]`、`[ForyUnion]`、外部结构化序列化器或自定义序列化器处理的用户类型。 +## 跨语言互操作 {#cross-language-interoperability} + +以下内容说明默认 xlang 格式的跨语言类型映射、类型标识和互操作要求。 + +Apache Fory™ C# 支持与其他 Fory 实现进行跨语言序列化。 + +### Xlang 配置 + +C# 始终写入和读取 xlang 帧头。它没有模式开关，因此互操作代码只需配置兼容模式和引用跟踪等其余设置。 + +```csharp +Fory fory = Fory.Builder() + .Build(); +``` + +### 使用稳定 ID 注册 + +```csharp +[ForyStruct] +public sealed class Person +{ + public string Name { get; set; } = string.Empty; + public int Age { get; set; } +} + +Fory fory = Fory.Builder() + .Build(); + +fory.Register(100); +``` + +所有语言使用相同的 ID 映射。 + +第三方 class、struct 和 enum 可以使用[外部类型序列化](external-types.md)。请使用其他语言通信方所用的相同 ID 或名称注册目标类型，而不是本地序列化器声明。 + +### 按名称注册 + +```csharp +fory.Register("com.example.Person"); +``` + +### 跨语言示例 + +#### C#（序列化端） + +```csharp +Person person = new() { Name = "Alice", Age = 30 }; +byte[] payload = fory.Serialize(person); +``` + +#### Java（反序列化端） + +```java +Fory fory = Fory.builder() + .withXlang(true) + .withRefTracking(true) + .build(); + +fory.register(Person.class, 100); +Person value = (Person) fory.deserialize(payloadFromCSharp); +``` + +#### Python（反序列化端） + +```python +import pyfory + +fory = pyfory.Fory(xlang=True, ref=True) +fory.register_type(Person, type_id=100) +value = fory.deserialize(payload_from_csharp) +``` + +### 类型映射参考 + +完整映射参见[跨语言指南](../xlang.md)。 + +对于低精度数值载荷，使用 `Half` / `Half[]` 或 `List` 表示 xlang `float16`，使用 `BFloat16` / `BFloat16[]` 或 `List` 表示 xlang `bfloat16`。 + +### 列表和密集数组 + +C# `List` 映射到 Fory `list`。请使用 Schema 标记 `Apache.Fory.Schema.Types.Array` 表示密集 `array` 字段。 + +| Fory Schema | C# Schema 标记示意 | +| ----------------- | --------------------- | +| `list` | `S.List` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | + +### 互操作最佳实践 + +1. 保持类型 ID 稳定并记录在文档中。 +2. 滚动升级期间保持启用兼容模式。 +3. 在读写双方注册所有用户类型。 +4. 使用真实载荷的往返测试验证集成。 + +### 相关指南 + +- [类型注册](type-registration.md) +- [外部类型](external-types.md) +- [Schema 演进](schema-evolution.md) +- [支持的类型](supported-types.md) + ## 相关主题 - [类型注册](type-registration.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/configuration.md index 6d2f89febb4..6e24e0cc193 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/configuration.md @@ -186,17 +186,10 @@ ThreadSafeFory fory = Fory.Builder() ## 安全 -安全相关配置： - -- 反序列化不可信载荷前，只注册预期类型。 -- 对有意使用的相同 Schema 载荷，使用 `CheckStructVersion(true)` 并配合 `Compatible(false)`。 -- 设置 `MaxDepth(...)`，拒绝深度超出预期的动态对象图。 -- 将 `MaxGraphMemoryBytes(...)` 设为包含大量 collection、map、array、struct 和 object 的载荷的近似限制。它并非精确的堆上限；叶子值受剩余输入字节限制。 -- 保持远程 Schema 元数据限制的默认值，除非数据可信且可信通信方会发送更大的元数据或大量 Schema 版本。 -- 对不可信输入，优先使用生成或注册的具体模型，而不是宽泛的动态字段。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [C# 安全](security.md)。 ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [Schema 演进](schema-evolution.md) - [线程安全](thread-safety.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/external-types.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/external-types.md index bbd9efb498f..dcb42216aa6 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/external-types.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/external-types.md @@ -186,7 +186,7 @@ public sealed class LocalRecord : ThirdParty.VendorRecord 只要每个带注解的具体子类都具有合法的无参构造路径，`BaseOnly` 目标就可以是抽象类或缺少无参构造函数。 -第一方基类应改为直接使用 `[ForyStruct]` 注解。参见[类继承](core-api.md#class-inheritance)。 +第一方基类应改为直接使用 `[ForyStruct]` 注解。参见[类继承](basic-serialization.md#class-inheritance)。 ## 声明和目标要求 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/index.md index aba61391e13..608ba1122fd 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/index.md @@ -87,8 +87,7 @@ User decoded = fory.Deserialize(payload); | 主题 | 说明 | | --------------------------------------- | ------------------------------------- | | [配置](configuration.md) | 构建器选项和模式设置 | -| [基本序列化](core-api.md) | 类型化和动态序列化 API | -| [跨语言序列化](xlang.md) | 互操作指南 | +| [基本序列化](basic-serialization.md) | 类型化和动态序列化 API | | [Schema 元数据](schema-metadata.md) | `[ForyField]` ID 和 Schema 类型描述符 | | [类型注册](type-registration.md) | 注册用户类型和自定义序列化器 | | [外部类型](external-types.md) | 第三方类型的序列化器 | @@ -103,5 +102,7 @@ User decoded = fory.Deserialize(payload); ## 相关资源 - [跨语言序列化规范](../../specification/xlang_serialization_spec.md) -- [跨语言指南](../xlang/index.md) +- [跨语言指南](../xlang.md) - [C# 源码目录](https://github.com/apache/fory/tree/main/csharp) + +解码来自应用信任边界之外的字节之前，请阅读 [C# 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/references.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/references.md index 34cfffa6df7..04035c2ffea 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/references.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/references.md @@ -72,6 +72,6 @@ C# union 包装器不可变，并在读取其 case 载荷后创建。不支持 ## 相关主题 - [配置](configuration.md) -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [外部类型](external-types.md) - [线程安全](thread-safety.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/security.md new file mode 100644 index 00000000000..e29b9ce33f6 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/security.md @@ -0,0 +1,51 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 C# 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +安全相关配置： + +- 反序列化不可信载荷前，只注册预期类型。 +- 对有意使用的相同 Schema 载荷，使用 `CheckStructVersion(true)` 并配合 `Compatible(false)`。 +- 设置 `MaxDepth(...)`，拒绝深度超出预期的动态对象图。 +- 将 `MaxGraphMemoryBytes(...)` 设为包含大量 collection、map、array、struct 和 object 的载荷的近似限制。它并非精确的堆上限；叶子值受剩余输入字节限制。 +- 保持远程 Schema 元数据限制的默认值，除非数据可信且可信通信方会发送更大的元数据或大量 Schema 版本。 +- 对不可信输入，优先使用生成或注册的具体模型，而不是宽泛的动态字段。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[类型注册](type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/supported-types.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/supported-types.md index 271780824e0..d8d50a49971 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/supported-types.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/supported-types.md @@ -100,11 +100,11 @@ license: | ## 注意事项 - 应显式注册用户定义类型。 -- 跨语言用法请遵循[跨语言指南](../xlang/index.md)。 +- 跨语言用法请遵循[跨语言指南](../xlang.md)。 ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [外部类型](external-types.md) - [类型注册](type-registration.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/type-registration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/type-registration.md index 97c793ee828..a71a519e0f0 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/type-registration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/type-registration.md @@ -87,7 +87,7 @@ fory.Register(101); ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [外部类型](external-types.md) - [自定义序列化器](custom-serializers.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/xlang.md deleted file mode 100644 index 00a874ad386..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/csharp/xlang.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -title: 跨语言序列化 -sidebar_position: 1 -id: xlang -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Apache Fory™ C# 支持与其他 Fory 实现进行跨语言序列化。 - -## 跨语言 Fory 实例 - -C# 始终写入和读取 xlang 帧头。它没有模式开关，因此互操作代码只需配置兼容模式和引用跟踪等其余设置。 - -```csharp -Fory fory = Fory.Builder() - .Build(); -``` - -## 使用稳定 ID 注册 - -```csharp -[ForyStruct] -public sealed class Person -{ - public string Name { get; set; } = string.Empty; - public int Age { get; set; } -} - -Fory fory = Fory.Builder() - .Build(); - -fory.Register(100); -``` - -所有语言使用相同的 ID 映射。 - -第三方 class、struct 和 enum 可以使用[外部类型序列化](external-types.md)。请使用其他语言通信方所用的相同 ID 或名称注册目标类型，而不是本地序列化器声明。 - -## 按名称注册 - -```csharp -fory.Register("com.example.Person"); -``` - -## 跨语言示例 - -### C#（序列化端） - -```csharp -Person person = new() { Name = "Alice", Age = 30 }; -byte[] payload = fory.Serialize(person); -``` - -### Java（反序列化端） - -```java -Fory fory = Fory.builder() - .withXlang(true) - .withRefTracking(true) - .build(); - -fory.register(Person.class, 100); -Person value = (Person) fory.deserialize(payloadFromCSharp); -``` - -### Python（反序列化端） - -```python -import pyfory - -fory = pyfory.Fory(xlang=True, ref=True) -fory.register_type(Person, type_id=100) -value = fory.deserialize(payload_from_csharp) -``` - -## 类型映射参考 - -完整映射参见[跨语言指南](../xlang/index.md)。 - -对于低精度数值载荷，使用 `Half` / `Half[]` 或 `List` 表示 xlang `float16`，使用 `BFloat16` / `BFloat16[]` 或 `List` 表示 xlang `bfloat16`。 - -## 列表和密集数组 - -C# `List` 映射到 Fory `list`。请使用 Schema 标记 `Apache.Fory.Schema.Types.Array` 表示密集 `array` 字段。 - -| Fory Schema | C# Schema 标记示意 | -| ----------------- | --------------------- | -| `list` | `S.List` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | - -## 最佳实践 - -1. 保持类型 ID 稳定并记录在文档中。 -2. 滚动升级期间保持启用兼容模式。 -3. 在读写双方注册所有用户类型。 -4. 使用真实载荷的往返测试验证集成。 - -## 相关主题 - -- [类型注册](type-registration.md) -- [外部类型](external-types.md) -- [Schema 演进](schema-evolution.md) -- [支持的类型](supported-types.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/basic-serialization.md similarity index 69% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/xlang.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/basic-serialization.md index c5c838b18a6..b00c64dec5f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/xlang.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang 序列化 +title: 基础序列化 sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,9 +19,132 @@ license: | limitations under the License. --- +本页介绍如何使用 Apache Fory™ Dart 的默认 xlang 模式序列化和反序列化值。 + +## 创建 `Fory` 实例 + +创建并复用一个实例；每次调用都创建新的 `Fory` 会浪费资源。 + +```dart +import 'package:fory/fory.dart'; + +final fory = Fory(); +``` + +## 序列化和反序列化带注解类型 + +```dart +import 'package:fory/fory.dart'; + +part 'person.fory.dart'; + +@ForyStruct() +class Person { + Person(); + + String name = ''; + + @ForyField(type: Int32Type()) + int age = 0; +} + +void main() { + final fory = Fory(); + PersonForyModule.register( + fory, + Person, + name: 'example.Person', + ); + + final person = Person() + ..name = 'Ada' + ..age = 36; + + final bytes = fory.serialize(person); + final roundTrip = fory.deserialize(bytes); + print(roundTrip.name); +} +``` + +`deserialize` 返回转换为 `T` 的解码值。如果载荷描述的类型与 `T` 不同，它会抛出异常。 + +## Null 值 + +支持直接序列化 `null`： + +```dart +final fory = Fory(); +final bytes = fory.serialize(null); +final value = fory.deserialize(bytes); +``` + +## 序列化集合和动态载荷 + +可以直接序列化集合值： + +```dart +final fory = Fory(); +final bytes = fory.serialize([ + 'hello', + 42, + true, +]); +final value = fory.deserialize>(bytes); +``` + +对于异构集合，请反序列化为 `Object?`、`List` 或 `Map`。 + +## 引用跟踪 + +默认情况下，Fory 不跟踪对象标识；如果同一个对象在列表中出现两次，它会被序列化两次。当数据包含共享引用或循环结构时，请启用引用跟踪。 + +对于顶层集合： + +```dart +final fory = Fory(); +final shared = String.fromCharCodes('shared'.codeUnits); +final bytes = fory.serialize([shared, shared], trackRef: true); +final roundTrip = fory.deserialize>(bytes); +print(identical(roundTrip[0], roundTrip[1])); // true +``` + +对于生成 struct 内的字段，请改为在该字段上使用 `@ForyField(ref: true)`。 + +## 复用缓冲区 + +如果希望避免每次调用都分配新的 `Uint8List`，请将 `serializeTo` 和 `deserializeFrom` 与显式 `Buffer` 配合使用： + +```dart +final fory = Fory(); +final buffer = Buffer(); + +fory.serializeTo('Ada', buffer); +final value = fory.deserializeFrom(buffer); +``` + +这是一项优化。对于大多数应用，默认的 `serialize`/`deserialize` 组合已经足够。 + +## 序列化前注册类型 + +序列化自定义 class 或 enum 前，需要向 `Fory` 注册。生成代码可以简化此操作： + +```dart +PersonForyModule.register( + fory, + Person, + id: 100, +); +``` + +如果跳过注册，反序列化会以 `Type ... is not registered` 失败。参见[类型注册](type-registration.md)和[代码生成](code-generation.md)。 + +## 跨语言互操作 {#cross-language-interoperability} + +以下内容说明默认 xlang 格式的跨语言类型映射、类型标识和互操作要求。 + Apache Fory™ Dart 与 Java、Go、C#、Python、Rust 和 Swift 的 Fory 实现使用相同的二进制格式进行序列化。你可以在 Dart 中写入消息并在 Java 中读取，反之亦然，无需任何转换层。 -## 设置 +### Xlang 配置 按常规方式创建 `Fory` 实例。Dart 中无需启用单独的 xlang 选项： @@ -31,11 +154,11 @@ final fory = Fory(); // xlang payloads with compatible schema evolution 关键要求是两端使用相同身份注册同一类型。 -## 注册身份 +### 注册身份 最重要的规则是：**所有端都使用相同的类型身份**。你有两种选择： -### 数字 ID +#### 数字 ID 更适合规模较小、协作紧密的团队： @@ -44,7 +167,7 @@ final fory = Fory(); // xlang payloads with compatible schema evolution ModelsForyModule.register(fory, Person, id: 100); ``` -### 命名空间 + 类型名称 +#### 命名空间 + 类型名称 更适合多个团队独立定义类型的场景： @@ -59,7 +182,7 @@ ModelsForyModule.register( 不要在不同实现中为同一类型混用这两种策略。 -## 外部类型 +### 外部类型 对于由另一个 Dart 软件包拥有的结构体类，请定义 [外部结构化序列化器](external-types.md)，并使用所有对端都采用的相同 ID 或名称注册目标： @@ -84,9 +207,9 @@ ExternalSerializersForyModule.register( 声明中的字段 ID、名称、可空性和编码宽度注解定义了 Dart 端的 xlang Schema。外部声明 可以显式列出可访问的继承目标属性，但 Fory 不会自动扫描外部目标的层次结构。 -## Dart 到 Java 示例 +### Dart 到 Java 示例 -### Dart +#### Dart ```dart import 'package:fory/fory.dart'; @@ -110,7 +233,7 @@ final bytes = fory.serialize(Person() ..age = 30); ``` -### Java +#### Java ```java Fory fory = Fory.builder() @@ -121,9 +244,9 @@ fory.register(Person.class, 100); Person value = (Person) fory.deserialize(bytesFromDart); ``` -## Dart 到 C# 示例 +### Dart 到 C# 示例 -### Dart +#### Dart ```dart final fory = Fory(); @@ -133,7 +256,7 @@ final bytes = fory.serialize(Person() ..age = 30); ``` -### CSharp +#### CSharp ```csharp [ForyStruct] @@ -150,9 +273,9 @@ fory.Register(100); Person person = fory.Deserialize(payloadFromDart); ``` -## Dart 到 Go 示例 +### Dart 到 Go 示例 -### Dart +#### Dart ```dart final fory = Fory(); @@ -162,7 +285,7 @@ final bytes = fory.serialize(Person() ..age = 30); ``` -### Go +#### Go ```go type Person struct { @@ -177,7 +300,7 @@ var person Person _ = f.Deserialize(bytesFromDart, &person) ``` -## 字段匹配规则 +### 字段匹配规则 Fory 按名称或稳定的字段 ID 匹配字段。为了实现稳健的跨语言互操作： @@ -197,7 +320,7 @@ Schema 中。父类和子类字段共享同一个字段 ID 命名空间和同一 已包含的继承 `@ForyField(ref: true)` 和嵌套容器引用元数据，与直接声明在子类上的字段 使用相同的引用行为。继承不会改变 xlang 引用成帧方式，也不会添加父类级别的引用状态。 -## Dart 类型映射说明 +### Dart 类型映射说明 由于 Dart `int` 本身并不保证确切的 xlang 编码宽度，因此在需要精确跨语言解释时，应优先使用显式字段元数据： @@ -209,7 +332,7 @@ Schema 中。父类和子类字段共享同一个字段 ID 命名空间和同一 - 16 位浮点数组载荷使用 `Float16List` 和 `Bfloat16List` - 明确的时间语义使用 `Timestamp`、`LocalDate` 和 `Duration` -### 列表与稠密数组 +#### 列表与稠密数组 除非字段具有显式数组元数据，否则 `List` 始终表示 Fory `list`。仅对一维稠密 bool 或数值数据使用 `array`。 @@ -233,7 +356,7 @@ bool 或数值数据使用 `array`。 请参阅[支持的类型](supported-types.md)和 [xlang 类型映射](../../specification/xlang_type_mapping.md)。 -## 验证 +### 验证 在生产环境中依赖跨语言契约之前，请使用你支持的每种实现对载荷执行端到端测试。 @@ -245,10 +368,16 @@ dart analyze dart test ``` -## 相关主题 +### 相关指南 - [结构体继承](inheritance.md) - [类型注册](type-registration.md) - [外部类型序列化](external-types.md) - [Schema 演进](schema-evolution.md) -- [Xlang 指南](../xlang/index.md) +- [跨语言互操作](../xlang.md) + +## 相关主题 + +- [配置](configuration.md) +- [类型注册](type-registration.md) +- [Schema 元数据](schema-metadata.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/configuration.md index b4c9db16dc6..7a40964aee5 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/configuration.md @@ -149,17 +149,10 @@ final fory = Fory(maxUnbackedContainerItems: 8192); ## 安全 -安全相关配置： - -- 反序列化不可信载荷前，只注册预期的生成模型。 -- 对有意使用的相同 Schema 载荷，将 `checkStructVersion: true` 与 `compatible: false` 结合使用。 -- 设置 `maxDepth`，拒绝深度超出预期的载荷结构。 -- 对大多数输入保持 `maxGraphMemoryBytes` 默认值；对于已知可信且包含大量 collection/map/struct 的载荷，也可以设置显式正字节数限制。 -- 保持远程 Schema 元数据限制的默认值，除非数据可信且可信通信方会发送更大的元数据或大量 Schema 版本。 -- 对不可信输入，优先使用生成的 Schema 和显式字段元数据，而不是宽泛的动态字段。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [Dart 安全](security.md)。 ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [Schema 演进](schema-evolution.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/core-api.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/core-api.md deleted file mode 100644 index 34e82230452..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/core-api.md +++ /dev/null @@ -1,145 +0,0 @@ ---- -title: 基本序列化 -sidebar_position: 2 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本页介绍如何使用 Apache Fory™ Dart 序列化和反序列化值。 - -## 创建 `Fory` 实例 - -创建并复用一个实例；每次调用都创建新的 `Fory` 会浪费资源。 - -```dart -import 'package:fory/fory.dart'; - -final fory = Fory(); -``` - -## 序列化和反序列化带注解类型 - -```dart -import 'package:fory/fory.dart'; - -part 'person.fory.dart'; - -@ForyStruct() -class Person { - Person(); - - String name = ''; - - @ForyField(type: Int32Type()) - int age = 0; -} - -void main() { - final fory = Fory(); - PersonForyModule.register( - fory, - Person, - name: 'example.Person', - ); - - final person = Person() - ..name = 'Ada' - ..age = 36; - - final bytes = fory.serialize(person); - final roundTrip = fory.deserialize(bytes); - print(roundTrip.name); -} -``` - -`deserialize` 返回转换为 `T` 的解码值。如果载荷描述的类型与 `T` 不同，它会抛出异常。 - -## Null 值 - -支持直接序列化 `null`： - -```dart -final fory = Fory(); -final bytes = fory.serialize(null); -final value = fory.deserialize(bytes); -``` - -## 序列化集合和动态载荷 - -可以直接序列化集合值： - -```dart -final fory = Fory(); -final bytes = fory.serialize([ - 'hello', - 42, - true, -]); -final value = fory.deserialize>(bytes); -``` - -对于异构集合，请反序列化为 `Object?`、`List` 或 `Map`。 - -## 引用跟踪 - -默认情况下，Fory 不跟踪对象标识；如果同一个对象在列表中出现两次，它会被序列化两次。当数据包含共享引用或循环结构时，请启用引用跟踪。 - -对于顶层集合： - -```dart -final fory = Fory(); -final shared = String.fromCharCodes('shared'.codeUnits); -final bytes = fory.serialize([shared, shared], trackRef: true); -final roundTrip = fory.deserialize>(bytes); -print(identical(roundTrip[0], roundTrip[1])); // true -``` - -对于生成 struct 内的字段，请改为在该字段上使用 `@ForyField(ref: true)`。 - -## 复用缓冲区 - -如果希望避免每次调用都分配新的 `Uint8List`，请将 `serializeTo` 和 `deserializeFrom` 与显式 `Buffer` 配合使用： - -```dart -final fory = Fory(); -final buffer = Buffer(); - -fory.serializeTo('Ada', buffer); -final value = fory.deserializeFrom(buffer); -``` - -这是一项优化。对于大多数应用，默认的 `serialize`/`deserialize` 组合已经足够。 - -## 序列化前注册类型 - -序列化自定义 class 或 enum 前，需要向 `Fory` 注册。生成代码可以简化此操作： - -```dart -PersonForyModule.register( - fory, - Person, - id: 100, -); -``` - -如果跳过注册，反序列化会以 `Type ... is not registered` 失败。参见[类型注册](type-registration.md)和[代码生成](code-generation.md)。 - -## 相关主题 - -- [配置](configuration.md) -- [类型注册](type-registration.md) -- [Schema 元数据](schema-metadata.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/custom-serializers.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/custom-serializers.md index 7a4f87752fe..fe226081ad4 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/custom-serializers.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/custom-serializers.md @@ -137,5 +137,5 @@ return value; - [类型注册](type-registration.md) - [外部类型序列化](external-types.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - [故障排查](troubleshooting.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/index.md index 165d9e00c42..b9cb3aa8ac1 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/index.md @@ -132,11 +132,10 @@ dart run build_runner build | 主题 | 说明 | | --------------------------------------- | ---------------------------------------------- | | [配置](configuration.md) | Fory 选项、兼容模式和安全限制 | -| [基本序列化](core-api.md) | `serialize`、`deserialize`、生成注册和根对象图 | +| [基本序列化](basic-serialization.md) | `serialize`、`deserialize`、生成注册和根对象图 | | [代码生成](code-generation.md) | `@ForyStruct`、build runner 和生成模块 | | [Struct 继承](inheritance.md) | 父类、mixin、私有字段和构造函数 | | [外部类型序列化](external-types.md) | 为其他包拥有的类生成序列化器 | -| [跨语言序列化](xlang.md) | 互操作规则和字段对齐 | | [Schema 元数据](schema-metadata.md) | `@ForyField`、字段 ID、可空性、引用和多态 | | [类型注册](type-registration.md) | 基于 ID 与基于名称的注册及注册规则 | | [自定义序列化器](custom-serializers.md) | 自定义 `Serializer` 实现和 union | @@ -150,5 +149,7 @@ dart run build_runner build - [跨语言序列化规范](../../specification/xlang_serialization_spec.md) - [跨语言实现指南](../../specification/xlang_implementation_guide.md) -- [跨语言指南](../xlang/index.md) +- [跨语言指南](../xlang.md) - [Dart 实现源码目录](https://github.com/apache/fory/tree/main/dart) + +解码来自应用信任边界之外的字节之前，请阅读 [Dart 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/schema-evolution.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/schema-evolution.md index 2d2ac380fc4..5ee1fb35026 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/schema-evolution.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/schema-evolution.md @@ -105,4 +105,4 @@ final fory = Fory(compatible: false); - [配置](configuration.md) - [外部类型序列化](external-types.md) - [Schema 元数据](schema-metadata.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/schema-metadata.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/schema-metadata.md index 53f1295a037..020b6509baa 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/schema-metadata.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/schema-metadata.md @@ -147,4 +147,4 @@ Map> metrics = >{}; - [代码生成](code-generation.md) - [外部类型序列化](external-types.md) - [Schema 演进](schema-evolution.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/security.md new file mode 100644 index 00000000000..a6f4946595c --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/security.md @@ -0,0 +1,51 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 Dart 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +安全相关配置： + +- 反序列化不可信载荷前，只注册预期的生成模型。 +- 对有意使用的相同 Schema 载荷，将 `checkStructVersion: true` 与 `compatible: false` 结合使用。 +- 设置 `maxDepth`，拒绝深度超出预期的载荷结构。 +- 对大多数输入保持 `maxGraphMemoryBytes` 默认值；对于已知可信且包含大量 collection/map/struct 的载荷，也可以设置显式正字节数限制。 +- 保持远程 Schema 元数据限制的默认值，除非数据可信且可信通信方会发送更大的元数据或大量 Schema 版本。 +- 对不可信输入，优先使用生成的 Schema 和显式字段元数据，而不是宽泛的动态字段。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[类型注册](type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/supported-types.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/supported-types.md index 95c055bbf57..9ab030fd720 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/supported-types.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/supported-types.md @@ -139,5 +139,5 @@ Fory 支持 `List`、`Set` 和 `Map`。元素和 key 类型也必须 - [Struct 继承](inheritance.md) - [Schema 元数据](schema-metadata.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - [Schema 演进](schema-evolution.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/troubleshooting.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/troubleshooting.md index 07a0c9c3506..e9f6af47ff3 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/troubleshooting.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/troubleshooting.md @@ -201,7 +201,7 @@ dart test ## 相关主题 - [结构体继承](inheritance.md) -- [Xlang 序列化](xlang.md) +- [跨语言互操作](basic-serialization.md#cross-language-interoperability) - [代码生成](code-generation.md) - [自定义序列化器](custom-serializers.md) - [Web 平台支持](web-platform-support.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/type-registration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/type-registration.md index f5800d50ee5..a2281ade99b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/type-registration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/dart/type-registration.md @@ -104,12 +104,12 @@ fory.registerSerializer( ## 跨语言要求 -读写该类型的每个通信方必须使用相同数字 ID 或名称。示例参见[跨语言序列化](xlang.md)。 +读写该类型的每个通信方必须使用相同数字 ID 或名称。示例参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ## 相关主题 - [Struct 继承](inheritance.md) - [代码生成](code-generation.md) - [外部类型序列化](external-types.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - [自定义序列化器](custom-serializers.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/deserialization-security-model.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/deserialization-security-model.md deleted file mode 100644 index 70b9a4e8919..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/deserialization-security-model.md +++ /dev/null @@ -1,532 +0,0 @@ ---- -title: 反序列化安全模型 -sidebar_position: 99 -id: deserialization-security-model -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本文档定义 Apache Fory 反序列化的安全模型。它是用于分类反序列化行为并确定何处需要 -验证的公开安全参考。它不是漏洞披露，不描述利用技术，也不记录实现历史。 - -该模型有意保持较窄的范围。Fory 应防止不可信输入导致的资源和策略失效，但当验证无法 -保护 Fory 安全边界时，不应在热路径上添加只用于强制字节形式严格性的验证。 - -## 范围 - -该模型仅适用于反序列化来自不可信或部分可信来源的 Fory 二进制对象序列化数据。Fory -JSON 使用不同的读取器、策略、codec 和资源核算规则，因此有单独的 -[安全指南](../json/security.md)。 - -该模型不将成功反序列化值的语义内容视为 Fory 安全边界。发送方始终可以构造协议有效、 -但值由其自行选择的数据。应用授权、对象级业务规则和领域特有验证仍由应用负责。所选择 -的业务不变量仍属于应用策略，而不是 Fory 协议安全边界。 - -除非某个运行时通过不可信反序列化 API 显式公开内存格式路径，否则该模型不管理此类路径。 - -## 信任边界 - -在接受外部字节或流的 API 边界上，Fory 反序列化应将编码输入视为不可信。 - -Fory 安全边界包括： - -- 资源所有权，例如内存、CPU 进度、流缓冲、文件句柄、原生内存分配、回调和保留的 - 读取端表。 -- 运行时安全，例如避免崩溃、panic、未定义行为和越界读写。 -- 显式 Fory 策略检查，例如旨在限制可实体化内容的类型、函数、方法、类或注册策略。 -- 清理边界，即读取失败期间创建的状态必须在下一次根操作前释放或重置。 - -Fory 安全边界不包括： - -- 协议有效值的业务含义。 -- 一个值使用了哪一种协议允许的字节形式。 -- map、set、对象或元数据值是否使用某一种特定编码形状，除非拒绝其他形状属于显式 - 所有者策略，或能够保护上述边界之一。 - -## 类型与类策略 - -当类型、类、函数、方法、注册和反序列化策略旨在限制不可信字节可以实体化的内容时， -这些策略就是安全边界。 - -对于不可信数据，如果编码字节能够实体化当前 Fory 策略本应拒绝的类型、函数、方法、类 -或动态对象，则绕过行为与安全相关。这包括绕过类或类型注册、白名单检查器、严格模式 -检查或语言特有的反序列化策略。 - -应用注册一个类或为该类注册序列化器时，即显式信任该类。在类注册策略下，这两项操作 -都是配置时信任决策。在反序列化调用处显式选择静态根序列化器或静态根目标，也是应用 -对该根路径的授权决策。静态选择根的授权不依赖单独的注册查找；访问已注册身份或注册 -支持的元数据所需的任何注册，仍由实际访问驱动。 - -显式声明或选择静态字段 codec 本身就是应用对该字段的授权决策；该 codec 无需为了授权 -而单独注册。注册外围类或 Schema 也会授权属于该已注册所有者的静态声明字段 codec 和 -序列化器。这同样适用于声明的 Array、Set、Map、Struct 及其他静态组合字段路径。这些 -已声明字段路径不会仅仅因为其主体无需再次查找类型即可解码，就需要独立注册。同样， -如果编码的声明类型标记只能调用已由授权根或外围 Schema 选择的 codec，它也不会造成 -注册绕过。 - -这些静态授权路径不会授权由编码类型元数据选择的任意替代项。输入选择的动态或多态类型 -仍必须通过该类型当前有效的注册和反序列化策略检查。Fory 仅仅发现或生成的序列化器， -如果不是通过显式选择的静态根或已注册外围所有者到达，就只是序列化机制，本身不会授权 -动态选择的类。 - -对可信数据禁用注册或动态类型检查属于调用方配置选择。该选择只会移除相应策略对任意 -类型实体化提供的保证；它不会移除 Fory 对不可信反序列化路径的运行时安全、资源、清理、 -保留状态或无进度循环要求。 - -Fory 不是应用拥有类型的沙箱。如果当前策略允许某个已注册类型或序列化器，则该类型的 -构造、hook、setter、finalizer 或其他逻辑对于应用信任边界是否安全，由应用负责。 - -允许经过策略批准的构造或可调用对象执行时，资源核算不应声称能够约束 Fory 所有权之外 -的任意代码。Fory 拥有的核算只能覆盖由 Fory 自身明确创建或复制，并且仍可从实体化对象图 -访问的对象和存储。临时辅助分配和用户代码内部资源仍在该核算边界之外。 - -## 深度与进度 - -通过对象、元数据、容器或引用进行递归的反序列化路径，应在恶意构造的嵌套耗尽调用栈或 -绕过清理之前，强制执行运行时配置的深度限制。超过配置深度的格式错误输入应使根操作 -失败，而不是继续无界递归。 - -消耗编码数据的循环应保证字节进度、逻辑进度或终止错误。即使不分配内存，能让读取器 -陷入无进度循环的输入也与安全相关。 - -## 安全不变量 - -反序列化代码必须防止不可信输入导致以下结果： - -- 崩溃、panic、未定义行为或越界内存访问。 -- 与已经提供或确认可读的字节相比发生 OOM 或不成比例的分配。 -- 无进度循环，包括格式错误输入之后既不能保证逻辑进度也不能保证字节进度的循环。 -- 在准确读取或跳过相应字节之前，将流缓冲区增长到攻击者声明的大小。 -- 资源泄漏，包括无法执行清理的原生内存分配、句柄、回调或已注册清理工作。 -- 失败后保留攻击者控制的状态，且该状态会影响后续根操作或跨操作增长。 -- 成功绕过显式 Fory 策略边界。 - -如果某条路径无法产生上述任何结果，更早拒绝格式错误的字节通常属于正确性或互操作性 -选择，而不是安全要求。 - -## 健壮性范围关卡 - -报告或修复反序列化健壮性问题之前，请先确认当前实现中存在具体后果： - -- 崩溃、panic、未定义行为或越界访问。 -- 不成比例的分配、CPU 工作量或流增长。 -- 无进度循环。 -- 持久状态、引用表或缓存污染。 -- 后续根操作损坏或失败根操作的清理泄漏。 -- 具体的类型、注册、可调用对象或反序列化策略违规。 - -单纯的协议严格性不属于此关卡。不要仅仅因为格式错误或非规范的标志、枚举值、标记、 -长度形式或保留值被接受、较晚拒绝、以不同方式解码或产生不够精确的错误就修改代码。 -只有当此类验证能防止上述具体后果之一，或实现显式公共契约时，才值得采取行动。 - -## 受控反序列化错误 - -当解码器确定输入对当前所有者路径无效时，根操作必须返回错误并执行正常的失败清理。 -这是结果要求，而不是错误分类体系要求。 - -除非公共 API 或规范另有明确承诺，否则 Fory 不要求特定异常类型、错误码、消息、检测层、 -输入偏移量或尽可能早的检测点。现有的有界下游缓冲区下溢、类型、引用、深度或序列化器 -错误都是有效拒绝。解码器无需仅仅为了将这种受控失败替换为更具体或更统一的错误而添加 -新的局部检查。 - -格式错误输入的测试应证明根操作失败、清理仍然正确，并且相关安全不变量得到保留。如果 -固定确切错误类型或消息需要增加无法保护任何安全边界的成功路径验证，则不应这样做。 - -## 非安全语义 - -以下模式默认不属于漏洞： - -- 协议允许的集合分块、map 分块和字段顺序。 -- 根据目标数据结构或所属序列化器语义折叠的重复键、set 元素或兼容字段。 -- 最终会产生读取错误的格式错误 ref、null 或类型标志。 -- 线性消耗并最终产生读取错误的格式错误标量字节。 -- 在之后的形状验证前读取编码主体，只要操作最终返回错误且不会造成安全不变量失效。 -- 实体化组件类型是已被允许作为类 token 的接口的数组。分配引用数组不会实例化或执行 - 该接口，每个非空元素仍必须针对其具体类型通过当前策略。只有当数组路径绕过该具体 - 元素检查、调用策略禁止的回调，或违反 Fory 所有的运行时安全或资源不变量时，才将其 - 视为与安全相关。 - -Fory 仍可出于规范严格性或互操作性拒绝格式错误的形式。只有当协议所有者要求、在相关 -路径上几乎没有成本，或能保护上述安全不变量时，才应添加此类验证。不要仅仅为了拒绝 -只会增加解码成本的标量字节形式而添加协议层验证。 - -### 携带值的 ref 标志 - -某些读取路径有意为多个携带值的标志共享处理。例如，当 `NotNullValue` 和 `RefValue` 都 -表示后续是编码值时，读取器可以合并其热路径处理。这本身不是格式错误标志 bug。只有 -当合并处理丢失必需的引用语义、跨越显式所有者策略返回成功，或造成资源或运行时安全 -失效时，才将其视为 bug。 - -## 分配与字节可用性 - -在所需字节可用或已被准确读取之前，Fory 不应根据攻击者声明的长度进行大额分配。 - -对于缓冲区支持的输入： - -- 固定大小的二进制值和基本类型稠密数组应在分配目标之前，针对所需编码字节大小调用 - 字节所有者的可读性检查。对于缓冲区支持的输入，这通常是剩余字节比较。 -- 多字节元素数组应在分配前通过溢出检查计算所需字节大小。 -- 根据声明的逻辑元素计数分配后备存储或提供大小提示的容器读取器，应在分配后备存储 - 或预留容量前，针对该计数调用字节所有者的可读性检查。这不是完整的容器主体验证； - 它用于证明在读取器根据计数预分配之前，发送方至少提供了成比例的输入字节。估算的 - 内存预算核算可以在此字节检查前预留预算，因为它不分配后备存储。 -- 当目标逐步增长且每个条目的读取仍使用常规字节所有者检查时，读取器不应仅仅因为循环 - 将读取相应数量的值，就添加基于计数的可读性检查。安全边界是根据不可信计数直接 - 预分配，而不是存在计数循环本身。 - -对于流支持的输入： - -- 读取或跳过较大的字节区域，就是这些字节存在的证明。 -- 按字节计数的可变长度结果分配应在分配前使用字节所有者的可读性检查。跳过路径可以 - 使用有界跳过，而无需实体化被跳过的值。 -- 从流中读取完整请求编码主体后，流支持的缓冲区可以保存该主体。但在输入字节证明 - 攻击者声明的长度确实存在之前，不得预留该长度。 -- 流支持的填充缓冲区应从当前已证明的缓冲区大小按几何级数增长，例如将当前容量翻倍。 - 增长不得限制为当前填充目标：对于较小填充，目标仅略高于当前容量，因此限制到目标 - 会退化为固定大小的增长步骤，在每次小读取时复制整个缓冲区，使流反序列化总体成为 - O(n^2)。当流实现本身是调用方拥有的可信代码时，字节所有者可以使用所有者局部可用性 - 信号作为一次性增长提示，随后可以一次预留完整的当前目标，同时至少保留几何增长步长； - 如果该提示不存在或不足，读取器必须回退到基于已缓冲字节的有界几何增长。序列化器 - 不应添加自己的可用性分支。 -- 截断的流应在分配最终反序列化值前失败，并且只为实际读取的字节加上有界备用容量 - 进行分配。 - -字节所有者应保持面向字节。缓冲区、读取器或读取上下文 API 可以公开字节读取和字节跳过 -操作，但字符串解码、十进制解析、基本类型数组编码、压缩模式和集合容量策略属于各自的 -所属序列化器。 - -## 集合与 Map 容量 - -允许较大的有效集合输入。如果输入包含许多编码元素，应进行成比例的反序列化工作。 - -安全要求是在足够输入字节证明该容量合理之前，避免根据声明的逻辑计数进行不成比例的 -预分配。当重复的元素或条目主体经证明至少消耗一个字节时，根据声明计数进行分配或预留 -的读取器应在分配前调用 `checkReadableBytes(logicalCount)` 或运行时等价方法。当主体可能 -不消耗字节时，可读字节要求可以排除根操作剩余的无输入支撑容器余量。读取器在读取容器 -时仍必须核算实际输入进度。字节检查不会解码整个容器、验证元素语义，也不能替代分块 -验证。不根据逻辑计数预分配的读取器，仍可随实际读取元素而成比例增长。 - -只有当缺少验证会导致无进度循环、无界资源增长、保留状态或跨越 Fory 策略边界返回成功 -时，map 或集合分块验证才与安全相关。协议允许的分块划分属于正常输入，本身不是安全问题。 - -## 无输入支撑容器工作量预算 - -运行时会对由计数驱动、且重复读取主体没有输入进度支撑的集合元素和 map 条目执行根操作 -范围限制。公共选项名为 `maxUnbackedContainerItems`，或使用相应语言中的等价拼写。 -默认值为 `8192`；值必须非负，零是严格限制，而不是表示无限制的哨兵值。 - -一次根读取中的所有嵌套集合、map 和兼容字段跳过操作共享此余量。集合读取器每完成 1024 -个元素以及在最终不完整窗口处核算已完成条目。Map 读取器在现有协议分块边界处核算。 -重复条目主体实际消耗的字节会抵消同一窗口内的已完成条目数。该预算不会添加成帧、在 -写入时拒绝值、改变引用发布，也不会替代对象图内存核算。 - -如果读取器确切的重复操作已知至少消耗一个字节，则保留直接循环和成比例的可读字节检查。 -生成和编译的序列化器应从这些已证明为正的路径中移除预算访问和周期性分支。 - -## 对象图内存预算 - -运行时应对一个实体化对象图创建的估算内存执行每操作近似限制。这是对一次顶层反序列化 -操作创建的对象图所有者进行累计核算；它不是精确的堆测量，也不是原始元素槽位限制。 -实际进程内存可能高于配置的限制。 - -公共配置为 `maxGraphMemoryBytes`。所有输入形式的默认值固定为 `128 MiB`；用户配置的正值 -会覆盖默认值。显式的非正配置无效，应在创建运行时时拒绝。预算不从输入大小派生，流预算 -也不应依赖动态已读字节核算。 - -对象图预算核算应： - -- 在顶层读取状态中初始化，并由顶层反序列化 `finally` 负责清理； -- 只核算 Fory 创建并由返回值对象图保留的对象或存储；仅在构造期间使用的临时辅助对象 - 不属于对象图预算； -- 不声称能够预算显式策略允许相应代码后执行的任意构造函数、可调用对象、描述符、 - finalizer 或状态恢复内部资源； -- 将读取上下文/读取状态限制为原始字节预留；计数运算以及集合、map、数组、结构体和对象 - 存储公式属于具体序列化器或生成序列化器所有者； -- 在比较预算或分配前拒绝算术溢出； -- 估算浅层所有者存储的下限：引用支持或堆实体化的集合、map、set 和引用数组预留非零 - 浅层自身成本加上后备/引用/内联存储；引用支持或堆实体化的 struct、record、POJO、 - tuple/product、兼容、生成和动态对象所有者预留非零浅层自身成本加上浅层字段存储； -- 当实际引用槽位大小无法低成本或可靠查询时，使用 4 字节引用槽位；内联存储使用基本 - 类型/值字段宽度； -- 在分配后备存储或预留容量前保留现有字节可用性检查； -- 跳过作为独立所有者的 enum/union，并跳过专用的 string、binary、基本类型标量、基本 - 类型数组和基本类型稠密数组叶子所有者，除非运行时特有所有者章节明确将其纳入。 - -被跳过的叶子所有者仍必须受剩余输入字节限制。如果未读输入没有足够字节表示 string、 -binary 值、基本类型标量、基本类型数组或基本类型稠密数组，运行时不得读取或创建该叶子值。 - -每个运行时都必须在选择公式前检查具体所有者路径。在存储、装箱或分配值的所有者处恰好 -预留一次自身存储。反序列化 facade 可以为每次操作重置预算，但不得预留顶层结果类型、 -自身字节或值存储。引用支持的路径预留父所有者自身成本加上引用存储，而每个被引用的堆 -所有者在实体化时预留自己的浅层自身成本。内联/值路径在持有者/分配所有者中预留内联 -元素、字段或装箱存储；顶层值序列化器和生成的 struct/product 读取路径不得计算自身存储。 -对于内联/值集合或 map 运行时，反序列化 facade 或容器序列化器不得仅仅因为顶层值容器 -是返回值就对其计费。嵌套值容器作为父持有者的内联槽位，或作为实际拥有这些槽位的外层 -集合后备存储元素计费。指针、box、智能指针或类型擦除实体化路径为其分配的堆值预留浅层 -存储。父所有者不得递归包含子对象、集合、map、string、binary 或基本类型稠密数组的内容； -子所有者在实体化时预留自己的浅层内存。 - -### Java Fory 核心 - -Java Fory 核心基本类型数组序列化器预留可移植数组头部，加上逻辑长度乘以基本类型存储 -宽度。基本类型 list 序列化器预留返回 list 的浅层所有者、后备数组头部以及相同的基本 -类型存储。这些已知长度路径在现有成比例可读字节检查后、分配前预留一次；它们不使用 -增量批次。压缩输入使用解压后的逻辑长度，而临时压缩数组仍是保留对象图预算之外的构造 -暂存。Float16 和 BFloat16 稠密数组载体还包括其包装器的浅层所有者。当装箱 list 转换 -首先解码基本类型数组时，数组预留仍作为最终 list 估算的抵扣，转换只预留正的剩余差值。 - -### 生成的结构化目标 - -编码成员和物理存储是不同的输入。属性、访问器、接口和逻辑 Schema 别名不是物理字段， -不得作为存储计费。既序列化又存储的字段只计数一次。仅存储声明贡献其字段宽度，但不得 -进入编码元数据或生成的读写操作。 - -对于 C# 普通类，每个直接带注解的类拥有该类声明的物理实例字段。可继承类提供方发布 -累计父提供方值加上这些直接字段。密封具体序列化器在内部使用相同累计表达式。具体后代 -使用可立即访问的提供方值及其自身直接字段；不得枚举被引用的私有元数据或重建父存储。 -具体对象序列化器预留一个浅层对象所有者加上此累计字段存储。 - -C# 外部类声明拥有其列出的确切第三方物理字段。确切字段映射贡献存储。可见属性映射不 -贡献存储，因此其后备字段必须单独列出。忽略的映射必须标识一个确切类字段，且仅用于 -存储。外部 struct 声明仅支持可见成员映射。可发现但未映射的公共实例字段可以添加一次。 -`BaseOnly` 声明可以拥有普通子类使用的完整目标和目标祖先前缀。它必须列出该前缀中的 -每个非公共物理字段；生成器不会扫描被引用 assembly 的私有布局。 - -确切外部私有身份是固定到版本的软件包 ABI 断言。运行时编码访问使用确切访问器，不得 -回退到反射、布局探测或其他成员。仅存储的私有声明没有运行时访问器，因此应用必须根据 -固定的软件包版本验证它们。 - -Dart 生成器还可以包含生成时在目标上可见的公共实例字段。Swift 宏无法检查其他类型的 -存储布局，因此只使用外部声明。在每个运行时中，这些公式都在生成期间解析，不得向 -反序列化热路径添加反射、布局探测、分配或字段枚举。常规所有者规则仍然适用：引用目标 -预留其浅层所有者和字段存储，而内联值目标由拥有其存储的持有者计费。 - -### 运行时特有所有者说明 - -#### C++ - -C++ 普通 struct、product 和标准库容器属于值存储，除非指针、智能指针或类型擦除所有者 -在堆上分配它们。顶层反序列化会初始化剩余对象图预算，但不会为返回值预留 `sizeof(T)`。 -普通值序列化器不得仅仅因为正在读取值，就预留自身的 `sizeof(T)`。 - -泛型集合和 map 序列化器预留容器路径拥有的元素、键和值存储下限。嵌套值容器头部作为 -父对象的内联槽位或外层容器后备存储中的元素时计费。智能指针和类型擦除实体化路径在 -发布或返回其分配的堆值前，为该值预留浅层存储。泛型 C++ 路径不得臆造标准库头部、 -节点、桶、分配器或调试布局开销。 - -#### Rust - -Rust struct、tuple、enum 和集合值属于内联值存储，除非由 `Box`、`Rc`、`Arc` 或类型擦除 -所有者分配。顶层和派生值读取路径初始化或消耗预算，但不为正在读取的值预留 -`size_of::()`。`Vec`、`HashMap`、`BTreeMap` 及类似序列化器预留其根据计数分配的 -后备或条目值存储；嵌套值容器头部作为父内联字段或外层后备元素计费。 - -装箱、引用计数和类型擦除实体化路径为其创建的堆载荷预留 `size_of::()`。这些分配 -所有者可以使用编译期 `size_of::()` 公式，但值序列化器不应为同一个 `T` 添加并行 -自身预留。 - -在根据计数分配前，如果 Rust 所有者的确切重复操作已证明至少消耗一个字节，则保留完整 -可读字节关卡。不确定的所有者只需对计数中未被剩余无输入支撑条目余量覆盖的部分要求 -可读字节。在分配所有者处恰好应用一次所选关卡；读取共享元数据后不要重复。写入端继续 -编码合法的紧凑或空主体，不执行此读取端余量。 - -固定数组不根据经过验证的编码计数进行分配，因此省略分配关卡。对于零大小元素，`Vec`、 -`VecDeque` 和 `BinaryHeap` 也省略该关卡，因为在这种情况下它们不会创建根据计数派生的 -后备分配。在声明计数驱动分配时，节点、桶和条目所有者保留该关卡。实现不得以猜测的 -分配成本、填充字节、全局紧凑主体绕过或第二个集合/map codec 取而代之。 - -#### Swift - -Swift struct、enum、tuple 和集合值属于值存储。顶层值读取和嵌套值序列化器不应预留 -自身存储。拥有该值的持有者（例如 struct 字段、数组后备存储、dictionary 条目存储或 -装箱/动态实体化路径）拥有相应的对象图预算预留。 - -数组、dictionary 和 set 序列化器分配或预留后备存储时，可以使用稳定的 Swift 类型大小 -信息（例如 `MemoryLayout.stride`）预留后备存储下限。当 Fory 创建保留的对象或 box -时，类、existential 或装箱实体化路径预留所有者存储。不应向读取热路径添加运行时对象 -布局探测。 - -#### Go - -Go struct 以及 slice 或 map 头部属于值存储，除非由指针、接口实体化或其他堆所有者分配。 -顶层反序列化和 struct 值序列化器不应自行预留返回的 struct 或嵌套内联 struct。指针 -序列化器在分配保留的 `*T` 时预留具体 struct 存储。 - -Slice、array、map 和 set 序列化器预留其根据声明计数分配的后备或条目存储。元素和条目 -宽度应尽可能来自序列化器或解析器捕获的稳定类型信息；如果所有者已经知道具体类型，读取 -循环不应重新计算反射大小信息。接口或动态路径只预留 -Fory 明确实体化并保留的存储。 - -#### C\# - -C# 同时包含引用所有者和内联值类型。类、数组、list、dictionary、hash set 和其他堆容器 -预留非零浅层所有者成本，加上直接后备、引用槽位或内联元素存储。即使 dictionary 的键 -或值类型是 struct，dictionary 也是引用类型容器，因此 dictionary 所有者仍与其条目存储 -分开计费。 - -值 struct 内联读取时不预留自身存储；存储该 struct 的持有者（例如对象字段、数组元素、 -list 后备存储、dictionary 条目、box 或动态实体化路径）拥有该预留。当 Fory 创建保留的 -box 时，装箱、`object` 和动态实体化路径预留一个装箱所有者。所有者常量应是相关 C# -对象或容器形状真正可移植的下限，而不是占位标记。 - -运行时不应猜测对象头部、数组头部、分配器头部、调试模式字段、hash 桶、树链接、hash 链 -链接、节点头部、map 条目对象、备用块或运行时表布局，除非所有者路径具有低成本、稳定、 -显式的存储下限信号并记录了公式。所有者常量应是所有者形状的真实下限，而不是占位标记。 - -## 跳过语义 - -跳过未知或不兼容数据时，应根据具体影响进行分类，而不是根据运行时是否实体化临时值。 - -当直接消耗编码内容很简单且由当前运行时路径负责时，这样做很有用。对于 list、set 和 map -等复杂字段，这不是安全要求。如果能够保留现有序列化器所有权模型，运行时可以实体化值 -然后将其丢弃。 - -对于扩展、动态或用户拥有的类型，所属运行时可能不总是有足够信息在不调用已注册序列化器 -的情况下跳过。此时应根据具体影响分类行为： - -- 资源泄漏、保留状态、无进度循环或策略绕过与安全相关。 -- 允许有界实体化后返回错误或丢弃，除非它造成显著内存或 CPU 压力。 -- 仅要求被跳过值使用某一种特定编码形状的严格性，不是安全问题。 - -## 元数据与类型解析 - -当元数据解析影响保留的读取端状态、类型分派或策略决策时，它对安全敏感。 - -元数据读取器应： - -- 避免嵌套元数据结构中的无界递归。 -- 避免攻击者控制的元数据流导致表无界增长。 -- 在使用元数据主体绕过或替换现有策略决策前验证它们。 -- 对于 Java 元数据路径，将 `TypeChecker` 等名称级检查和禁止类列表保持在 - `Class.forName` 之前；为此，通过现有 `TypeResolver.loadClass` 所有者路由远端类名加载。不要通过直接从 - TypeDef 或 TypeMeta 名称加载类来绕过该所有者。被拒绝的输入名称不得导致类加载。 - 将此决策移动到加载前时，应保留注册、动态加载和未知类型语义。需要实体化 - `Class` 的检查仍在加载后执行；不要用仅基于字符串的近似检查替代它们。 -- 将完整的输入数组描述符传给 `TypeChecker`。输入最多可从已接受组件类派生六维数组。 - 更高维数组要求精确可信的完整数组注册或经过检查的名称缓存条目，使输入无法让 JVM - 派生无界的数组类系列。 -- 在正确的根操作边界重置或释放元数据状态。 - -不可信反序列化可以访问的类解析缓存，只能从显式可信配置发布条目，或在当前类策略接受 -解析出的类后发布条目。因此，缓存命中表示已经可信且验证过的 `Class`，应直接使用 -缓存类，而不重复类加载或名称级 `TypeChecker` 工作。只有缓存未命中才执行这些名称级 -检查并发布接受的结果。需要实体化 `Class` 的检查仍由其现有调用方负责。存储纯数据 -未知类占位符的缓存条目可以在精确命中时返回同一个占位符，但不得授权加载最初未命中的 -编码名称。对于 ID 注册和名称注册，精确的已注册名称表命中都是可信的；经过检查的名称 -缓存精确命中也是可信的。两个精确查找都未命中后，读取器不得从反向注册、以类为键的 -状态或 `Class.getName()` 推断另一个已接受名称。自定义名称注册本身不会将 Java 类名发布 -为额外别名；ID 注册会发布 Java 类名。 - -能够创建持久读取状态的远端元数据，必须在保留该状态前受到限制。该检查仅用于资源控制： -不得改变编码兼容性、类型注册、动态类加载、未知类型处理、反序列化策略或 Schema 演进 -语义。失败或不兼容的元数据不得消耗 Schema 版本限制；元数据缓存命中或生成的字段读取器 -不得为这些限制添加验证、hash、分配或策略工作。元数据解析、缓存发布、精确本地匹配和 -计数的具体顺序属于 [xlang 实现指南](../specification/xlang_implementation_guide.md)。 - -经过检查的元数据缓存，是判断接收的 TypeDef 或 TypeMeta 头部是否已经验证的唯一所有者。 -元数据缓存命中意味着该头部先前已被解析、通过主体/hash 验证和策略检查，并由所属缓存 -发布，因此读取器必须跳过剩余元数据主体并使用缓存元数据，不重复主体验证、hash 验证、 -限制检查、精确本地检查或策略工作。元数据缓存未命中是唯一会解析元数据主体、验证其 -hash 和形状、执行元数据限制、进行精确本地字节比较并发布到缓存的路径。不要添加单独的 -可空标志、哨兵头部、每 TypeInfo 接受标记或并行状态来表示此决策。 - -只有实际作为 TypeDef 或 TypeMeta 主体携带的元数据才受元数据主体和 Schema 版本限制。 -兼容的命名 enum、ext 和 union 元数据通常只有一个版本，但当它作为共享元数据发送时， -仍计入远端元数据总量限制。纯基于 ID 的 enum、ext 和 typed-union 值使用 type id 加 user -type id，不得移到此元数据主体路径。 - -远端元数据主体和 struct 字段列表也必须在冷元数据解析路径上受到限制。 -`maxTypeMetaBytes` 限制一个接收的 TypeDef 或 TypeMeta 主体中的编码元数据主体字节数， -不包括 8 字节头部和任何扩展大小 varint。`maxTypeFields` 限制一个接收的 struct 元数据 -主体声明的字段数量。对于 Java 原生 TypeDef 类层级，字段限制适用于该 TypeDef 所有类 -层级的字段总数。这些限制在根据攻击者声明的元数据大小或字段计数进行复制、解压、预留 -或分配之前检查。 - -默认限制为 `maxTypeFields = 512` 和 `maxTypeMetaBytes = 4096`。运行时应将限制失败报告为 -可能的恶意数据，并告知用户只有在数据没有恶意时才提高确切选项。这些限制不得在元数据 -缓存命中、生成的序列化器或已解析 type-id 热路径上引入验证。 - -单纯的元数据字节形式严格性不是安全要求。只有当所有者需要这种严格性，或该形状会改变 -类型身份、保留状态、资源使用或策略行为时，拒绝元数据形状才有意义。 - -## 引用跟踪 - -引用跟踪是编码协议的一部分，并且对性能敏感。读取器可以使用哨兵值和共享的携带值分支， -以保持热路径紧凑。 - -当格式错误输入能够导致以下结果时，引用跟踪验证与安全相关： - -- 访问越界引用而不报告错误。 -- 根操作失败后留下保留的引用状态。 -- 在引用值可用之前注册无界回调或解析器状态。 -- 导致无进度循环或崩溃。 - -不能仅仅因为格式错误的标志没有在最早可能的字节处被拒绝，就要求进行引用跟踪验证。 -只要根操作仍返回错误且没有违反安全不变量，延迟拒绝就是可接受的。下游错误无需是专用 -引用协议错误。 - -## 错误传播与清理 - -Fory 运行时可以有意使用延迟错误传播。读取记录错误后，后续读取步骤可以继续，直到外层 -操作观察到并返回该错误。 - -如果继续执行的工作不会导致以下结果，则可以接受： - -- 崩溃或 panic。 -- 分配或保留攻击者控制的状态。 -- 泄漏资源。 -- 绕过必需的清理。 -- 跨越显式验证或策略边界返回成功。 - -只有当外层根操作清理无法覆盖嵌套路径拥有的状态或资源时，才应添加嵌套 -`try`/`finally` 或等价清理。 - -## 性能要求 - -安全验证必须保持 Fory 热路径性能。如果验证仅用于严格性且会引入以下内容，请勿添加： - -- 每元素对象分配。 -- 热循环中的动态分派或回调。 -- 成功路径上的包装对象或结果载体。 -- 缓冲区支持的 string、binary 或基本类型数组读取所需的额外复制。 -- 无法保护安全不变量的分支。 -- 唯一目的只是统一最终错误类型、消息、位置或时机的辅助调用或生成代码膨胀。 - -优先使用可以内联、且已经利用当前序列化器可用信息的所有者局部检查。不要将序列化器 -拥有的语义移入通用读取上下文辅助方法。 - -## 分类指南 - -审查反序列化行为时，请使用以下问题： - -1. 此输入能否导致崩溃、panic 或越界内存访问？ -2. 较小或未经证明的输入长度能否导致不成比例的分配？ -3. 流支持的读取器能否在精确读取或跳过证明字节存在之前增长缓冲区？ -4. 循环能否在没有字节进度或逻辑进度的情况下继续？ -5. 根操作失败后，该路径能否保留攻击者控制的状态？ -6. 该路径能否泄漏资源或跳过必需的清理？ -7. 该路径能否跨越显式 Fory 策略边界返回成功？ -8. 所提议的验证在相关热路径上是否几乎没有成本？ - -如果前七个问题的答案都是否，该问题通常不属于安全问题。如果验证并非几乎没有成本， -除非协议所有者明确要求，否则请避免添加。 - -## 文档边界 - -安全模型文档不得包含利用样例、CVE 叙述、行级潜在漏洞、分支历史、迁移时间线或清理计划。 -请根据需要将这些细节保留在私有报告、issue 或 pull request 中。 - -公开安全文档应描述持久边界和不变量，而不是实现如何达到这些状态的历史。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/basic-serialization.md similarity index 55% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/xlang.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/basic-serialization.md index 591eff98eb0..1ff24ed131a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/xlang.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: 跨语言序列化 +title: 基础序列化 sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,9 +19,392 @@ license: | limitations under the License. --- +本指南介绍 Fory Go 默认 xlang 模式下的核心序列化 API。 + +## 创建 Fory 实例 + +在序列化前创建 Fory 实例并注册类型： + +```go +import "github.com/apache/fory/go/fory" + +f := fory.New(fory.WithXlang(true)) + +// Register struct with a type ID +f.RegisterStruct(User{}, 1) +f.RegisterStruct(Order{}, 2) + +// Or register with a name (more flexible, less prone to ID conflicts, but higher serialization cost) +f.RegisterStructByName(User{}, "example.User") + +// Register enum types +f.RegisterEnum(Color(0), 3) +``` + +`fory.New()` 使用带兼容 Schema 演进的跨语言模式。示例显式设置 `fory.WithXlang(true)`，以清楚展示模式选择。对于需要原生模式的 Go 专属载荷，请在原生模式示例中显式配置 `fory.WithXlang(false)`。 + +**重要**：应在多次序列化调用间复用 Fory 实例。创建新实例需要分配内部缓冲区、类型缓存和解析器，成本较高。默认 Fory 实例不是线程安全的；并发使用时请使用线程安全包装器（参见[线程安全](thread-safety.md)）。 + +更多详情参见[类型注册](type-registration.md)。 + +## 核心 API + +### 序列化与反序列化 + +主要序列化 API： + +```go +// Serialize any value +data, err := f.Serialize(value) +if err != nil { + // Handle error +} + +// Deserialize into target +var result MyType +err = f.Deserialize(data, &result) +if err != nil { + // Handle error +} +``` + +### Marshal 与 Unmarshal + +`Serialize` 和 `Deserialize` 的别名（Go 开发者更熟悉）： + +```go +data, err := f.Marshal(value) +err = f.Unmarshal(data, &result) +``` + +## 序列化原始类型 + +```go +// Integers +data, _ := f.Serialize(int64(42)) +var i int64 +f.Deserialize(data, &i) // i = 42 + +// Floats +data, _ = f.Serialize(float64(3.14)) +var fl float64 +f.Deserialize(data, &fl) // fl = 3.14 + +// Strings +data, _ = f.Serialize("hello") +var s string +f.Deserialize(data, &s) // s = "hello" + +// Booleans +data, _ = f.Serialize(true) +var b bool +f.Deserialize(data, &b) // b = true +``` + +## 序列化集合 + +### 切片 + +```go +// String slice +strs := []string{"a", "b", "c"} +data, _ := f.Serialize(strs) + +var result []string +f.Deserialize(data, &result) +// result = ["a", "b", "c"] + +// Integer slice +nums := []int64{1, 2, 3} +data, _ = f.Serialize(nums) + +var intResult []int64 +f.Deserialize(data, &intResult) +// intResult = [1, 2, 3] +``` + +### 映射 + +```go +// String to string map +m := map[string]string{"key": "value"} +data, _ := f.Serialize(m) + +var result map[string]string +f.Deserialize(data, &result) +// result = {"key": "value"} + +// String to int map +m2 := map[string]int64{"count": 42} +data, _ = f.Serialize(m2) + +var result2 map[string]int64 +f.Deserialize(data, &result2) +// result2 = {"count": 42} +``` + +## 序列化结构体 + +### 基本结构体序列化 + +只序列化**导出字段**（以大写字母开头）： + +```go +type User struct { + ID int64 // Serialized + Name string // Serialized + password string // NOT serialized (unexported) +} + +f.RegisterStruct(User{}, 1) + +user := &User{ID: 1, Name: "Alice", password: "secret"} +data, _ := f.Serialize(user) + +var result User +f.Deserialize(data, &result) +// result.ID = 1, result.Name = "Alice", result.password = "" +``` + +### 嵌套结构体 + +```go +type Address struct { + City string + Country string +} + +type Person struct { + Name string + Address Address +} + +f.RegisterStruct(Address{}, 1) +f.RegisterStruct(Person{}, 2) + +person := &Person{ + Name: "Alice", + Address: Address{City: "NYC", Country: "USA"}, +} + +data, _ := f.Serialize(person) + +var result Person +f.Deserialize(data, &result) +// result.Address.City = "NYC" +``` + +### 指针字段 + +```go +type Node struct { + Value int32 + Child *Node +} + +// Use WithTrackRef for pointer fields +f := fory.New(fory.WithXlang(true), fory.WithTrackRef(true)) +f.RegisterStruct(Node{}, 1) + +root := &Node{ + Value: 1, + Child: &Node{Value: 2, Child: nil}, +} + +data, _ := f.Serialize(root) + +var result Node +f.Deserialize(data, &result) +// result.Child.Value = 2 +``` + +## 流式 API + +适用于需要控制缓冲区的场景： + +### SerializeTo + +序列化到现有缓冲区： + +```go +buf := fory.NewByteBuffer(nil) + +// Serialize multiple values to same buffer +f.SerializeTo(buf, value1) +f.SerializeTo(buf, value2) + +// Get all serialized data +data := buf.GetByteSlice(0, buf.WriterIndex()) +``` + +### DeserializeFrom + +从现有缓冲区反序列化： + +```go +buf := fory.NewByteBuffer(data) + +var result1, result2 MyType +f.DeserializeFrom(buf, &result1) +f.DeserializeFrom(buf, &result2) +``` + +## 泛型 API（类型安全） + +Fory Go 提供用于类型安全序列化的泛型函数： + +```go +import "github.com/apache/fory/go/fory" + +type User struct { + ID int64 + Name string +} + +// Type-safe serialization +user := &User{ID: 1, Name: "Alice"} +data, err := fory.Serialize(f, user) + +// Type-safe deserialization +var result User +err = fory.Deserialize(f, data, &result) +``` + +泛型 API： + +- 在编译期推断类型 +- 提供更好的类型安全性 +- 可能带来性能收益 + +## 错误处理 + +始终检查序列化操作返回的错误： + +```go +data, err := f.Serialize(value) +if err != nil { + switch e := err.(type) { + case fory.Error: + fmt.Printf("Fory error: %s (kind: %d)\n", e.Error(), e.Kind()) + default: + fmt.Printf("Unknown error: %v\n", err) + } + return +} + +err = f.Deserialize(data, &result) +if err != nil { + // Handle deserialization error +} +``` + +常见错误类型： + +- `ErrKindBufferOutOfBound`：读写超出缓冲区边界 +- `ErrKindTypeMismatch`：反序列化期间类型 ID 不匹配 +- `ErrKindUnknownType`：遇到未知类型 +- `ErrKindMaxDepthExceeded`：超出递归深度限制 +- `ErrKindHashMismatch`：结构体哈希不匹配（Schema 已更改） + +错误解决方法参见[故障排查](troubleshooting.md)。 + +## Nil 处理 + +### Nil 指针 + +```go +var ptr *User = nil +data, _ := f.Serialize(ptr) + +var result *User +f.Deserialize(data, &result) +// result = nil +``` + +### 空集合 + +```go +// Nil slice +var slice []string = nil +data, _ := f.Serialize(slice) + +var result []string +f.Deserialize(data, &result) +// result = nil + +// Empty slice (different from nil) +empty := []string{} +data, _ = f.Serialize(empty) + +f.Deserialize(data, &result) +// result = [] (empty, not nil) +``` + +## 完整示例 + +```go +package main + +import ( + "fmt" + "github.com/apache/fory/go/fory" +) + +type Order struct { + ID int64 + Customer string + Items []Item + Total float64 +} + +type Item struct { + Name string + Quantity int32 + Price float64 +} + +func main() { + f := fory.New(fory.WithXlang(true)) + f.RegisterStruct(Order{}, 1) + f.RegisterStruct(Item{}, 2) + + order := &Order{ + ID: 12345, + Customer: "Alice", + Items: []Item{ + {Name: "Widget", Quantity: 2, Price: 9.99}, + {Name: "Gadget", Quantity: 1, Price: 24.99}, + }, + Total: 44.97, + } + + // Serialize + data, err := f.Serialize(order) + if err != nil { + panic(err) + } + fmt.Printf("Serialized %d bytes\n", len(data)) + + // Deserialize + var result Order + if err := f.Deserialize(data, &result); err != nil { + panic(err) + } + + fmt.Printf("Order ID: %d\n", result.ID) + fmt.Printf("Customer: %s\n", result.Customer) + fmt.Printf("Items: %d\n", len(result.Items)) + fmt.Printf("Total: %.2f\n", result.Total) +} +``` + +## 跨语言互操作 {#cross-language-interoperability} + +以下内容说明默认 xlang 格式的跨语言类型映射、类型标识和互操作要求。 + Fory Go 支持与 Java、Python、C++、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala 和 Kotlin 无缝交换数据。本指南介绍跨语言兼容性和类型映射。 -## 创建跨语言 Fory 实例 +### Xlang 配置 Go 默认使用带兼容 Schema 演进的跨语言模式。跨语言示例中应显式设置模式： @@ -29,11 +412,11 @@ Go 默认使用带兼容 Schema 演进的跨语言模式。跨语言示例中应 f := fory.New(fory.WithXlang(true)) ``` -## 跨语言类型注册 +### 跨语言类型注册 所有语言使用一致的类型 ID： -### Go +#### Go ```go type User struct { @@ -46,7 +429,7 @@ f.RegisterStruct(User{}, 1) data, _ := f.Serialize(&User{ID: 1, Name: "Alice"}) ``` -### Java +#### Java ```java public class User { @@ -58,7 +441,7 @@ fory.register(User.class, 1); User user = fory.deserialize(data, User.class); ``` -### Python +#### Python ```python from dataclasses import dataclass @@ -74,11 +457,11 @@ fory.register(User, type_id=1) user = fory.deserialize(data) ``` -## 类型映射 +### 类型映射 各语言的详细类型映射参见[类型映射规范](../../specification/xlang_type_mapping.md)。 -## 字段顺序 +### 字段顺序 跨语言序列化要求字段顺序一致。Fory 按字段的 snake_case 名称以字母顺序排序。 @@ -104,9 +487,9 @@ type Example struct { } ``` -## 示例 +### 示例 -### Go 到 Java +#### Go 到 Java **Go（序列化器）**： @@ -147,7 +530,7 @@ fory.register(Order.class, 1); Order order = fory.deserialize(data, Order.class); ``` -### Python 到 Go +#### Python 到 Go **Python（序列化器）**： @@ -185,11 +568,11 @@ f.Deserialize(data, &msg) fmt.Println(msg.Content) // "Hello from Python" ``` -### 嵌套结构 +#### 嵌套结构 跨语言嵌套结构要求注册所有类型： -## 列表与稠密数组 +### 列表与稠密数组 Go 切片通常是 `list` 载体，除非字段标签显式请求稠密 `array` Schema。`array` 仅用于一维布尔或数值数据。 @@ -247,9 +630,9 @@ fory.register(Address.class, 1); fory.register(Company.class, 2); ``` -## 常见问题 +### 常见问题 -### 字段名称不匹配 +#### 字段名称不匹配 Go 使用 PascalCase，其他语言可能使用 camelCase 或 snake_case。字段按转换后的 snake_case 名称匹配： @@ -265,7 +648,7 @@ public class User { } ``` -### 类型解释 +#### 类型解释 Go 无符号类型映射到位模式相同的 Java 有符号类型： @@ -275,7 +658,7 @@ var value uint64 = 18446744073709551615 // Max uint64 Java 的 `long` 保存相同位，但解释为 -1。如果需要无符号解释，请在 Java 中使用 `Long.toUnsignedString()`。 -### Nil 与 Null +#### Nil 与 Null Go nil 切片或映射会根据配置采用不同方式序列化： @@ -286,7 +669,7 @@ var slice []string = nil 确保其他语言正确处理 null。 -## 最佳实践 +### 互操作最佳实践 1. **使用一致的类型 ID**：所有语言中的同一类型使用相同数字 ID 2. **注册所有类型**：包括嵌套结构体类型 @@ -294,7 +677,7 @@ var slice []string = nil 4. **测试跨语言互操作**：尽早并经常运行集成测试 5. **处理类型差异**：注意有符号和无符号解释差异 -## 相关主题 +### 相关指南 - [类型注册](type-registration.md) - [支持的类型](supported-types.md) @@ -302,7 +685,7 @@ var slice []string = nil - [跨语言序列化规范](../../specification/xlang_serialization_spec.md) - [类型映射规范](../../specification/xlang_type_mapping.md) -## 内置值 +### 内置值 ```go package main @@ -340,7 +723,7 @@ func main() { } ``` -## 自定义值 +### 自定义值 ```go package main @@ -403,7 +786,7 @@ func main() { } ``` -## 共享引用与循环引用 +### 共享引用与循环引用 ```go package main @@ -436,3 +819,10 @@ func main() { fmt.Println(newValue) } ``` + +## 相关主题 + +- [配置](configuration.md) +- [类型注册](type-registration.md) +- [支持的类型](supported-types.md) +- [引用](references.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/configuration.md index b9c0cc4a85b..1b123656f84 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/configuration.md @@ -405,16 +405,11 @@ for req := range requests { ## 安全 -安全相关配置： - -- 反序列化不可信数据前，只注册预期的结构体。 -- 使用 `WithMaxDepth(...)` 拒绝意外过深的载荷。 -- 除非数据确定无恶意且可信对等端会发送更大的元数据或大量 Schema 版本，否则请保留远端 Schema 元数据限制的默认值。 -- 对不可信输入，优先使用具体结构体字段，而非宽泛的 `any` 或接口类型字段。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [Go 安全](security.md)。 ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [引用](references.md) - [Schema 演进](schema-evolution.md) - [线程安全](thread-safety.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/core-api.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/core-api.md deleted file mode 100644 index 3375168b826..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/core-api.md +++ /dev/null @@ -1,406 +0,0 @@ ---- -title: 基本序列化 -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本指南介绍 Fory Go 的核心序列化 API。 - -## 创建 Fory 实例 - -在序列化前创建 Fory 实例并注册类型： - -```go -import "github.com/apache/fory/go/fory" - -f := fory.New(fory.WithXlang(true)) - -// Register struct with a type ID -f.RegisterStruct(User{}, 1) -f.RegisterStruct(Order{}, 2) - -// Or register with a name (more flexible, less prone to ID conflicts, but higher serialization cost) -f.RegisterStructByName(User{}, "example.User") - -// Register enum types -f.RegisterEnum(Color(0), 3) -``` - -`fory.New()` 使用带兼容 Schema 演进的跨语言模式。示例显式设置 `fory.WithXlang(true)`，以清楚展示模式选择。对于需要原生模式的 Go 专属载荷，请在原生模式示例中显式配置 `fory.WithXlang(false)`。 - -**重要**：应在多次序列化调用间复用 Fory 实例。创建新实例需要分配内部缓冲区、类型缓存和解析器，成本较高。默认 Fory 实例不是线程安全的；并发使用时请使用线程安全包装器（参见[线程安全](thread-safety.md)）。 - -更多详情参见[类型注册](type-registration.md)。 - -## 核心 API - -### 序列化与反序列化 - -主要序列化 API： - -```go -// Serialize any value -data, err := f.Serialize(value) -if err != nil { - // Handle error -} - -// Deserialize into target -var result MyType -err = f.Deserialize(data, &result) -if err != nil { - // Handle error -} -``` - -### Marshal 与 Unmarshal - -`Serialize` 和 `Deserialize` 的别名（Go 开发者更熟悉）： - -```go -data, err := f.Marshal(value) -err = f.Unmarshal(data, &result) -``` - -## 序列化原始类型 - -```go -// Integers -data, _ := f.Serialize(int64(42)) -var i int64 -f.Deserialize(data, &i) // i = 42 - -// Floats -data, _ = f.Serialize(float64(3.14)) -var fl float64 -f.Deserialize(data, &fl) // fl = 3.14 - -// Strings -data, _ = f.Serialize("hello") -var s string -f.Deserialize(data, &s) // s = "hello" - -// Booleans -data, _ = f.Serialize(true) -var b bool -f.Deserialize(data, &b) // b = true -``` - -## 序列化集合 - -### 切片 - -```go -// String slice -strs := []string{"a", "b", "c"} -data, _ := f.Serialize(strs) - -var result []string -f.Deserialize(data, &result) -// result = ["a", "b", "c"] - -// Integer slice -nums := []int64{1, 2, 3} -data, _ = f.Serialize(nums) - -var intResult []int64 -f.Deserialize(data, &intResult) -// intResult = [1, 2, 3] -``` - -### 映射 - -```go -// String to string map -m := map[string]string{"key": "value"} -data, _ := f.Serialize(m) - -var result map[string]string -f.Deserialize(data, &result) -// result = {"key": "value"} - -// String to int map -m2 := map[string]int64{"count": 42} -data, _ = f.Serialize(m2) - -var result2 map[string]int64 -f.Deserialize(data, &result2) -// result2 = {"count": 42} -``` - -## 序列化结构体 - -### 基本结构体序列化 - -只序列化**导出字段**（以大写字母开头）： - -```go -type User struct { - ID int64 // Serialized - Name string // Serialized - password string // NOT serialized (unexported) -} - -f.RegisterStruct(User{}, 1) - -user := &User{ID: 1, Name: "Alice", password: "secret"} -data, _ := f.Serialize(user) - -var result User -f.Deserialize(data, &result) -// result.ID = 1, result.Name = "Alice", result.password = "" -``` - -### 嵌套结构体 - -```go -type Address struct { - City string - Country string -} - -type Person struct { - Name string - Address Address -} - -f.RegisterStruct(Address{}, 1) -f.RegisterStruct(Person{}, 2) - -person := &Person{ - Name: "Alice", - Address: Address{City: "NYC", Country: "USA"}, -} - -data, _ := f.Serialize(person) - -var result Person -f.Deserialize(data, &result) -// result.Address.City = "NYC" -``` - -### 指针字段 - -```go -type Node struct { - Value int32 - Child *Node -} - -// Use WithTrackRef for pointer fields -f := fory.New(fory.WithXlang(true), fory.WithTrackRef(true)) -f.RegisterStruct(Node{}, 1) - -root := &Node{ - Value: 1, - Child: &Node{Value: 2, Child: nil}, -} - -data, _ := f.Serialize(root) - -var result Node -f.Deserialize(data, &result) -// result.Child.Value = 2 -``` - -## 流式 API - -适用于需要控制缓冲区的场景： - -### SerializeTo - -序列化到现有缓冲区： - -```go -buf := fory.NewByteBuffer(nil) - -// Serialize multiple values to same buffer -f.SerializeTo(buf, value1) -f.SerializeTo(buf, value2) - -// Get all serialized data -data := buf.GetByteSlice(0, buf.WriterIndex()) -``` - -### DeserializeFrom - -从现有缓冲区反序列化： - -```go -buf := fory.NewByteBuffer(data) - -var result1, result2 MyType -f.DeserializeFrom(buf, &result1) -f.DeserializeFrom(buf, &result2) -``` - -## 泛型 API（类型安全） - -Fory Go 提供用于类型安全序列化的泛型函数： - -```go -import "github.com/apache/fory/go/fory" - -type User struct { - ID int64 - Name string -} - -// Type-safe serialization -user := &User{ID: 1, Name: "Alice"} -data, err := fory.Serialize(f, user) - -// Type-safe deserialization -var result User -err = fory.Deserialize(f, data, &result) -``` - -泛型 API： - -- 在编译期推断类型 -- 提供更好的类型安全性 -- 可能带来性能收益 - -## 错误处理 - -始终检查序列化操作返回的错误： - -```go -data, err := f.Serialize(value) -if err != nil { - switch e := err.(type) { - case fory.Error: - fmt.Printf("Fory error: %s (kind: %d)\n", e.Error(), e.Kind()) - default: - fmt.Printf("Unknown error: %v\n", err) - } - return -} - -err = f.Deserialize(data, &result) -if err != nil { - // Handle deserialization error -} -``` - -常见错误类型： - -- `ErrKindBufferOutOfBound`：读写超出缓冲区边界 -- `ErrKindTypeMismatch`：反序列化期间类型 ID 不匹配 -- `ErrKindUnknownType`：遇到未知类型 -- `ErrKindMaxDepthExceeded`：超出递归深度限制 -- `ErrKindHashMismatch`：结构体哈希不匹配（Schema 已更改） - -错误解决方法参见[故障排查](troubleshooting.md)。 - -## Nil 处理 - -### Nil 指针 - -```go -var ptr *User = nil -data, _ := f.Serialize(ptr) - -var result *User -f.Deserialize(data, &result) -// result = nil -``` - -### 空集合 - -```go -// Nil slice -var slice []string = nil -data, _ := f.Serialize(slice) - -var result []string -f.Deserialize(data, &result) -// result = nil - -// Empty slice (different from nil) -empty := []string{} -data, _ = f.Serialize(empty) - -f.Deserialize(data, &result) -// result = [] (empty, not nil) -``` - -## 完整示例 - -```go -package main - -import ( - "fmt" - "github.com/apache/fory/go/fory" -) - -type Order struct { - ID int64 - Customer string - Items []Item - Total float64 -} - -type Item struct { - Name string - Quantity int32 - Price float64 -} - -func main() { - f := fory.New(fory.WithXlang(true)) - f.RegisterStruct(Order{}, 1) - f.RegisterStruct(Item{}, 2) - - order := &Order{ - ID: 12345, - Customer: "Alice", - Items: []Item{ - {Name: "Widget", Quantity: 2, Price: 9.99}, - {Name: "Gadget", Quantity: 1, Price: 24.99}, - }, - Total: 44.97, - } - - // Serialize - data, err := f.Serialize(order) - if err != nil { - panic(err) - } - fmt.Printf("Serialized %d bytes\n", len(data)) - - // Deserialize - var result Order - if err := f.Deserialize(data, &result); err != nil { - panic(err) - } - - fmt.Printf("Order ID: %d\n", result.ID) - fmt.Printf("Customer: %s\n", result.Customer) - fmt.Printf("Items: %d\n", len(result.Items)) - fmt.Printf("Total: %.2f\n", result.Total) -} -``` - -## 相关主题 - -- [配置](configuration.md) -- [类型注册](type-registration.md) -- [支持的类型](supported-types.md) -- [引用](references.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/custom-serializers.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/custom-serializers.md index e64e6ee1d65..577490be139 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/custom-serializers.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/custom-serializers.md @@ -282,4 +282,4 @@ func TestMySerializer(t *testing.T) { - [类型注册](type-registration.md) - [支持的类型](supported-types.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/index.md index 10a23059aec..a70f95d3c6a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/index.md @@ -90,7 +90,7 @@ func main() { 仅限 Go 的通信应使用原生模式。通过 `fory.WithXlang(false)` 选择原生模式，它会让 Go 对象序列化保持 Go 原生形式。该模式针对不需要可移植跨语言映射的 Go 结构体、指针、接口和 Go 专属类型行为进行了优化。兼容模式默认启用。只有每个读取端和写入端都使用相同的 Go 结构体 Schema，并且希望获得更快序列化和更小体积时，才设置 `fory.WithCompatible(false)`。 -Go 跨语言注册和互操作规则参见[跨语言序列化](xlang.md)，仅限 Go 的载荷参见[原生序列化](native.md)。 +Go 跨语言注册和互操作规则参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)，仅限 Go 的载荷参见[原生序列化](native.md)。 ## 配置 @@ -130,14 +130,13 @@ data, _ := f.Serialize(&User{ID: 1, Name: "Alice"}) // 'data' can be deserialized by Java, Python, etc. ``` -类型映射和兼容性详情参见[跨语言序列化](xlang.md)。 +类型映射和兼容性详情参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ## 文档 | 主题 | 说明 | | --------------------------------------- | --------------------------- | -| [基本序列化](core-api.md) | 核心 API 和使用模式 | -| [跨语言序列化](xlang.md) | 多语言序列化 | +| [基本序列化](basic-serialization.md) | 核心 API 和使用模式 | | [原生序列化](native.md) | 仅限 Go 的序列化 | | [配置](configuration.md) | 选项和设置 | | [Schema 元数据](schema-metadata.md) | 字段级配置 | @@ -155,3 +154,5 @@ data, _ := f.Serialize(&User{ID: 1, Name: "Alice"}) - [跨语言序列化规范](../../specification/xlang_serialization_spec.md) - [跨语言类型映射](../../specification/xlang_type_mapping.md) - [GitHub 仓库](https://github.com/apache/fory) + +解码来自应用信任边界之外的字节之前，请阅读 [Go 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/native.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/native.md index 655d30cffd0..d7b4de4a111 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/native.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/native.md @@ -21,7 +21,7 @@ license: | Go 原生序列化是通过 `fory.WithXlang(false)` 选择、仅限 Go 的编码模式。当每个写入端和读取端都是 Go 服务，并且载荷应遵循 Go 类型系统而非可移植的跨语言类型系统时，请使用该模式。 -如果字节需要由 Java、Python、C++、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala、Kotlin 或其他非 Go Fory 实现读取，请使用 Go 默认模式[跨语言序列化](xlang.md)。 +如果字节需要由 Java、Python、C++、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala、Kotlin 或其他非 Go Fory 实现读取，请使用 Go 默认模式[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ## 何时使用原生序列化 @@ -192,7 +192,7 @@ _ = data ## 相关主题 -- [跨语言序列化](xlang.md) - 跨语言 Go 载荷 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言 Go 载荷 - [配置](configuration.md) - Go 选项 - [类型注册](type-registration.md) - 结构体和枚举注册 - [引用](references.md) - 共享引用与循环引用 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/references.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/references.md index 5f7fc73f5fe..27c9b2c62e6 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/references.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/references.md @@ -353,4 +353,4 @@ func main() { - [配置](configuration.md) - [结构体标签](schema-metadata.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/schema-evolution.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/schema-evolution.md index 0dc3db1e4ff..9d6f3df4952 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/schema-evolution.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/schema-evolution.md @@ -370,5 +370,5 @@ func main() { ## 相关主题 - [配置](configuration.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - [故障排查](troubleshooting.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/schema-metadata.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/schema-metadata.md index c367ed239a6..72e1d7f2dfe 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/schema-metadata.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/schema-metadata.md @@ -387,5 +387,5 @@ type Session struct { ## 相关主题 - [引用](references.md) -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [Schema 演进](schema-evolution.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/security.md new file mode 100644 index 00000000000..7da27f2bda0 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/security.md @@ -0,0 +1,49 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 Go 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +安全相关配置： + +- 反序列化不可信数据前，只注册预期的结构体。 +- 使用 `WithMaxDepth(...)` 拒绝意外过深的载荷。 +- 除非数据确定无恶意且可信对等端会发送更大的元数据或大量 Schema 版本，否则请保留远端 Schema 元数据限制的默认值。 +- 对不可信输入，优先使用具体结构体字段，而非宽泛的 `any` 或接口类型字段。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[类型注册](type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/supported-types.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/supported-types.md index 2788bd5cbe0..6c15ee49b37 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/supported-types.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/supported-types.md @@ -351,7 +351,7 @@ data, _ := f.Serialize(status) | `time.Time` | Instant | datetime | - | - | | `time.Duration` | Duration | timedelta | - | - | -详细映射参见[跨语言序列化](xlang.md)。 +详细映射参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ## 不支持的类型 @@ -367,5 +367,5 @@ data, _ := f.Serialize(status) ## 相关主题 - [类型注册](type-registration.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - [引用](references.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/thread-safety.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/thread-safety.md index 568009b7b87..3a80134bb93 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/thread-safety.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/thread-safety.md @@ -343,5 +343,5 @@ go func() { ## 相关主题 - [配置](configuration.md) -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [故障排查](troubleshooting.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/troubleshooting.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/troubleshooting.md index d6551b3221f..16c0dfe6d05 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/troubleshooting.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/troubleshooting.md @@ -438,6 +438,6 @@ func TestSchemaEvolution(t *testing.T) { ## 相关主题 - [配置](configuration.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - [Schema 演进](schema-evolution.md) - [线程安全](thread-safety.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/type-registration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/type-registration.md index 961e64f8cbe..4f15830aa85 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/type-registration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/go/type-registration.md @@ -258,7 +258,7 @@ error: unknown type encountered ## 相关主题 -- [基本序列化](core-api.md) -- [跨语言序列化](xlang.md) +- [基本序列化](basic-serialization.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - [支持的类型](supported-types.md) - [故障排查](troubleshooting.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/index.md index 658e097eaca..8e988066296 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/index.md @@ -1,5 +1,5 @@ --- -title: 对象序列化 +title: 介绍 sidebar_position: 0 id: index license: | @@ -23,26 +23,42 @@ license: | ## 选择模式 -| 模式 | 适用场景 | 从这里开始 | -| ------ | -------------------------------------- | ---------------------------- | -| Xlang | 字节需要跨越运行时边界 | [Xlang 模式](xlang/index.md) | -| Native | 所有写入端和读取端都使用同一运行时系列 | [原生模式](native.md) | +| 模式 | 适用场景 | 从这里开始 | +| ------------- | -------------------------------------- | ------------------------ | +| Xlang（默认） | 字节需要跨越运行时边界 | [Xlang 序列化](xlang.md) | +| Native | 所有写入端和读取端都使用同一运行时系列 | [原生序列化](native.md) | Xlang 和原生模式是仅有的两种对象序列化模式。Row Format 是可随机访问的分析表示形式， Fory JSON 是 Java JSON codec；如果目标不是重建对象，请使用 [格式选择指南](../introduction/choose-a-format.md)。 +## 阅读核心概念和序列化模式 + +选择具体运行时 API 之前，请按以下顺序阅读： + +1. [核心概念](core-concepts.md)介绍两种模式共用的对象图、类型、Schema、引用、多态和 + Schema 演进。 +2. [Xlang 序列化](xlang.md)介绍默认使用的可移植格式，以及不同语言运行时必须共同遵守的规则。 +3. [原生序列化](native.md)介绍适合使用同运行时原生格式的场景。 + ## 按运行时浏览 选择运行时以查看其安装方式、生命周期、具体 API、配置、类型注册、Schema 行为、扩展、 平台支持和故障排除信息： -[浏览支持的运行时](runtimes.md) - -## 安全性 - -解码外部提供的字节之前，请阅读[对象序列化安全性](security.md)。其中介绍了两种模式下的 -可接受类型策略、注册、资源限制、传输责任和负向验证。 +| 运行时 | 模式 | 文档 | +| --------------------- | ----------------- | ----------------------------------------------------- | +| Java | xlang 和原生 | [Java 运行时](./java/index.md) | +| Python | xlang 和原生 | [Python 运行时](./python/index.md) | +| C++ | xlang 和原生 | [C++ 运行时](./cpp/index.md) | +| Go | xlang 和原生 | [Go 运行时](./go/index.md) | +| Rust | xlang 和原生 | [Rust 运行时](./rust/index.md) | +| JavaScript/TypeScript | xlang | [JavaScript/TypeScript 运行时](./javascript/index.md) | +| C# | xlang | [C# 运行时](./csharp/index.md) | +| Swift | xlang | [Swift 运行时](./swift/index.md) | +| Dart | xlang | [Dart 运行时](./dart/index.md) | +| Scala | xlang 和 JVM 原生 | [Scala 运行时](./scala/index.md) | +| Kotlin | xlang 和 JVM 原生 | [Kotlin 运行时](./kotlin/index.md) | ## 规范 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/advanced-features.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/advanced-features.md index abd0b25b16c..2e47a9c393a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/advanced-features.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/advanced-features.md @@ -156,4 +156,4 @@ static { - [配置](configuration.md) - 所有 ForyBuilder 选项 - [原生序列化](native.md) - 仅限 Java 的序列化、JDK 钩子和零拷贝缓冲区 - [对象复制](object-copy.md) - 深拷贝功能 -- [跨语言序列化](xlang.md) - Java 跨语言互操作 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - Java 跨语言互操作 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/basic-serialization.md similarity index 69% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/xlang.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/basic-serialization.md index aa995628890..a042f0adfa4 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/xlang.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: 跨语言序列化 +title: 基础序列化 sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,9 +19,106 @@ license: | limitations under the License. --- +本页介绍 Java 跨语言模式快速入门。跨语言模式是 Java 的默认编码格式，也是处理跨语言载荷时的首选。 + +## 创建 Fory 实例 + +创建单线程跨语言 Fory 实例时，请显式设置模式： + +```java +import org.apache.fory.Fory; + +Fory fory = Fory.builder() + .withXlang(true) + .requireClassRegistration(true) + .build(); +``` + +创建线程安全的 Fory 实例时，使用同一个构建器构建 `ThreadSafeFory`： + +```java +import org.apache.fory.ThreadSafeFory; + +ThreadSafeFory fory = Fory.builder() + .withXlang(true) + .requireClassRegistration(true) + .buildThreadSafeFory(); +``` + +Java 默认的跨语言模式也默认启用兼容 Schema 模式，因此独立部署的服务可以在 Schema 元数据保持兼容时增删字段。仅当所有读取端和写入端始终使用相同 Schema，并且希望获得更快的序列化速度和更小的体积时，才使用 `withCompatible(false)`。只有在确认所有语言使用相同的跨语言 Schema，或原生类型由 Fory Schema IDL 生成时，才应选择 `compatible=false`。 + +## 注册自定义类型 + +在每个对等端以相同的类型标识注册应用类。数字 ID 紧凑且快速，而名称注册更便于在独立维护的服务之间协调。 + +```java +import org.apache.fory.annotation.ForyField; + +public class User { + @ForyField(id = 0) + public String name; + + @ForyField(id = 1) + public int age; +} + +Fory fory = Fory.builder() + .withXlang(true) + .requireClassRegistration(true) + .build(); + +fory.register(User.class, "example", "User"); +``` + +长期使用的 Schema 应采用字段 ID，这样即使 Java 字段名发生变化，字段标识仍保持稳定。Java 注解、可空性、引用跟踪和枚举元数据参见 [Schema 元数据](schema-metadata.md)。 + +## 序列化与反序列化 + +```java +User user = new User(); +user.name = "Alice"; +user.age = 30; + +byte[] bytes = fory.serialize(user); +User decoded = fory.deserialize(bytes, User.class); +``` + +跨语言字节在不同语言之间传递时，每个对等端都必须注册相同的类型标识和兼容的字段元数据。 +共享规则参见[跨语言互操作](../xlang.md)，Java 特定的互操作要求见下文。 + +## 对仅限 Java 的流量使用原生序列化 + +对于同语言的 Java/JVM 流量，原生模式通常更合适： + +```java +Fory fory = Fory.builder() + .withXlang(false) + .build(); +``` + +原生模式支持广泛的 Java 对象序列化能力，包括 JDK 序列化钩子、对象复制和原生模式零拷贝缓冲区。参见[原生序列化](native.md)。 + +## 常用选项 + +- `withRefTracking(true)` 保留共享引用和循环引用。 +- `requireClassRegistration(true)` 保持默认的已注册类型策略。 +- 原生模式和跨语言载荷默认启用兼容模式。仅当所有读取端和写入端使用相同 Schema，且希望获得更快速度和更小体积时，才使用 `withCompatible(false)`。对于跨语言载荷，只有在确认所有语言使用相同 Schema，或原生类型由 Fory Schema IDL 生成时，才应选择 `compatible=false`。 +- `withAsyncCompilation(true)` 在受支持的平台上启用异步序列化器编译。 + +## 最佳实践 + +1. **复用 Fory 实例**：创建 Fory 的开销较大，应始终复用实例 +2. **选择合适的线程安全方式**：根据需求选择单线程或线程安全实现 +3. **注册类**：在每个跨语言对等端保持稳定的类型标识 +4. **配置引用跟踪**：仅当对象图需要保持对象标识或循环引用时启用 + +## 跨语言互操作 {#cross-language-interoperability} + +以下内容说明默认 xlang 格式的跨语言类型映射、类型标识和互操作要求。 + Apache Fory™ 跨语言序列化是 Java 为必须由 Python、Rust、Go、JavaScript/TypeScript、C++、C#、Swift、Dart、Scala、Kotlin 或其他非 Java Fory 实现读取的载荷提供的编码模式。Java 默认使用支持兼容 Schema 演进的跨语言模式，但示例会显式设置模式，以便在代码中清楚表达载荷契约。 -## 创建跨语言 Fory 实例 +### Xlang 配置 每种配置使用一个长期存活的 `Fory` 或 `ThreadSafeFory` 实例。Fory 会缓存类型元数据和生成的序列化器，因此创建实例开销较高。 @@ -39,11 +136,11 @@ Fory fory = Fory.builder() 如果所有写入端和读取端都是 Java，并且载荷需要保留 Java 专用对象行为，请改用[原生序列化](native.md)。 -## 注册类型 +### 注册类型 所有语言都必须使用一致的 ID 或名称注册类型。Fory 支持两种注册方法。 -### 按 ID 注册（追求性能时推荐） +#### 按 ID 注册（追求性能时推荐） ```java public record Person(String name, int age) {} @@ -60,7 +157,7 @@ byte[] bytes = fory.serialize(person); 权衡：每个服务都必须协调 ID，确保同一逻辑类型使用相同数字。 -### 按名称注册（追求灵活性时推荐） +#### 按名称注册（追求灵活性时推荐） ```java public record Person(String name, int age) {} @@ -79,9 +176,9 @@ byte[] bytes = fory.serialize(person); Java API 也支持单个字符串类型名，例如 `fory.register(Person.class, "example.Person")`。每个对等端都应使用相同的逻辑标识。 -## Java 到 Python 示例 +### Java 到 Python 示例 -### Java（序列化端） +#### Java（序列化端） ```java import org.apache.fory.Fory; @@ -107,7 +204,7 @@ public class Example { } ``` -### Python（反序列化端） +#### Python（反序列化端） ```python import pyfory @@ -128,7 +225,7 @@ with open("person.bin", "rb") as input_file: print(f"{person.name}, {person.age}") # Output: Bob, 25 ``` -## 处理循环引用和共享引用 +### 处理循环引用和共享引用 启用引用跟踪时，跨语言模式支持循环引用和共享引用： @@ -157,7 +254,7 @@ byte[] bytes = fory.serialize(node1); // Python/Rust/Go can correctly deserialize this with circular references preserved ``` -## 类型映射注意事项 +### 类型映射注意事项 并非所有 Java 类型在其他语言中都有对应类型。使用跨语言模式时： @@ -168,7 +265,7 @@ byte[] bytes = fory.serialize(node1); - 除非每种目标语言都有约定的映射，否则避免使用 `Optional`、`BigDecimal` 和 `EnumSet` 等 Java 专用类型。 - 完整兼容性矩阵参见[类型映射指南](../../specification/xlang_type_mapping.md)。 -### 列表与稠密数组 +#### 列表与稠密数组 Java 原始类型数组是稠密 `array` 载体，但普通 `byte[]` 默认映射为 `bytes`。常规 Java 集合以及 `Int32List`、`Float16List`、`BFloat16List` 等 Fory 原始类型列表载体使用 `list`，除非字段具有显式 `@ArrayType` 元数据。 @@ -196,7 +293,7 @@ private @UInt32Type int[] ids; private @BFloat16Type short[] values; ``` -### 兼容类型 +#### 兼容类型 ```java public record UserData( @@ -207,7 +304,7 @@ public record UserData( ) {} ``` -### 可能有问题的类型 +#### 可能有问题的类型 ```java public record UserData( @@ -217,7 +314,7 @@ public record UserData( ) {} ``` -## 性能注意事项 +### 性能注意事项 与 Java 原生模式相比，跨语言模式存在额外开销： @@ -230,45 +327,45 @@ public record UserData( - 如果不需要循环引用，**禁用引用跟踪**（`withRefTracking(false)`） - 仅需要 Java 序列化时，**使用原生模式**（`withXlang(false)`） -## 最佳实践 +### 互操作最佳实践 1. 为每个用户类型使用显式类型 ID 或命名空间/类型名称。 2. 独立部署的服务保持兼容模式启用。 3. 在生产环境依赖某个 Schema 之前，通过每个对等端测试载荷。 4. 需要 Java 专用对象行为的仅限 Java 流量使用原生序列化。 -## 故障排除 +### 互操作故障排除 -### “类型未注册”错误 +#### “类型未注册”错误 - 验证两端是否使用相同 ID/名称注册类型 - 检查类型名是否有拼写错误或大小写差异 -### “类型不匹配”错误 +#### “类型不匹配”错误 - 确保不同语言中的字段类型兼容 - 查看[类型映射指南](../../specification/xlang_type_mapping.md) -### 数据损坏或值不符合预期 +#### 数据损坏或值不符合预期 - 验证两端都使用跨语言载荷 - 确保两端使用兼容的 Fory 版本 -## 另请参阅 +### 规范与参考 - [跨语言序列化规范](../../specification/xlang_serialization_spec.md) - [类型映射参考](../../specification/xlang_type_mapping.md) -- [Python 跨语言序列化指南](../python/xlang.md) -- [Rust 跨语言序列化指南](../rust/xlang.md) +- [Python 跨语言序列化指南](../python/basic-serialization.md#cross-language-interoperability) +- [Rust 跨语言序列化指南](../rust/basic-serialization.md#cross-language-interoperability) -## 相关主题 +### 相关指南 - [Schema 演进](schema-evolution.md) - 兼容模式 - [类型注册](type-registration.md) - 注册方法 - [原生序列化](native.md) - 仅限 Java 的序列化功能 - [Row Format](../../row-format/java.md) - 跨语言行格式 -## 内置值 +### 内置值 ```java import org.apache.fory.*; @@ -294,7 +391,7 @@ public class Example1 { } ``` -## 自定义值 +### 自定义值 ```java import org.apache.fory.*; @@ -353,3 +450,10 @@ public class Example2 { } } ``` + +## 相关主题 + +- [配置](configuration.md) - 所有 ForyBuilder 选项 +- [原生序列化](native.md) - 仅限 Java 的序列化功能 +- [Schema 元数据](schema-metadata.md) - 字段 ID、可空性、引用跟踪和枚举 ID +- [故障排除](troubleshooting.md) - 常见 API 使用问题 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/configuration.md index 4237f83cc9d..782df3cecaa 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/configuration.md @@ -82,31 +82,7 @@ Fory fory = Fory.builder() ## 安全 -在生产环境以及处理任何不可信载荷来源时，请保持类注册启用： - -```java -Fory fory = Fory.builder() - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .build(); -``` - -安全相关选项： - -- `requireClassRegistration(true)` 将反序列化限制为已注册类。 -- `withMaxDepth(...)` 拒绝深度异常的对象图。 -- `withMaxGraphMemoryBytes(...)` 为单次根反序列化期间实例化的对象图内存设置近似门限。估算主要覆盖集合、映射、数组、结构体和对象；Fory core 原始类型数组和原始类型列表根据解码长度计入其原始存储。它会跳过字符串、原始标量和不使用原始类型数组序列化器的专用二进制值等叶子值。实际进程内存可能高于该限制。叶子值仍受字节可用性检查保护：如果未读取的输入没有足够字节，Fory 不会读取或创建该叶子值。默认值固定为 `128 MiB`；可信工作负载需要更大或更小门限时，请设置正数字节限制。 -- `withMaxUnbackedContainerItems(...)` 限制由数量驱动、但重复读取正文没有消耗相应输入的集合和映射工作。默认值为 `8192`；零表示严格限制。 -- `withMaxTypeFields(...)` 和 `withMaxTypeMetaBytes(...)` 限制单个已接收远程元数据正文的字段数与编码正文大小。 -- `withMaxSchemaVersionsPerType(...)` 和 - `withMaxAverageSchemaVersionsPerType(...)` 限制可接受的远程元数据版本，而不改变注册、动态加载或 Schema 演进语义。 -- `withDeserializeUnknownClass(false)` 避免根据元数据实例化未知类。 -- `checkJdkClassSerializable(true)` 保留对 `java.*` 类的 JDK 可序列化性检查。 -- 类注册警告可用于安全审计；需要暴露意外类型时，请使用 `suppressClassRegistrationWarnings(false)`。 - -仅对可信载荷使用 `requireClassRegistration(false)`；需要动态类加载时，应同时配置 `TypeChecker` 允许列表。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [Java 安全](security.md)。 ## 相关主题 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/core-api.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/core-api.md deleted file mode 100644 index 60c1e3049c5..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/core-api.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -title: 基础序列化 -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本页介绍 Java 跨语言模式快速入门。跨语言模式是 Java 的默认编码格式，也是处理跨语言载荷时的首选。 - -## 创建 Fory 实例 - -创建单线程跨语言 Fory 实例时，请显式设置模式： - -```java -import org.apache.fory.Fory; - -Fory fory = Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .build(); -``` - -创建线程安全的 Fory 实例时，使用同一个构建器构建 `ThreadSafeFory`： - -```java -import org.apache.fory.ThreadSafeFory; - -ThreadSafeFory fory = Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .buildThreadSafeFory(); -``` - -Java 默认的跨语言模式也默认启用兼容 Schema 模式，因此独立部署的服务可以在 Schema 元数据保持兼容时增删字段。仅当所有读取端和写入端始终使用相同 Schema，并且希望获得更快的序列化速度和更小的体积时，才使用 `withCompatible(false)`。只有在确认所有语言使用相同的跨语言 Schema，或原生类型由 Fory Schema IDL 生成时，才应选择 `compatible=false`。 - -## 注册自定义类型 - -在每个对等端以相同的类型标识注册应用类。数字 ID 紧凑且快速，而名称注册更便于在独立维护的服务之间协调。 - -```java -import org.apache.fory.annotation.ForyField; - -public class User { - @ForyField(id = 0) - public String name; - - @ForyField(id = 1) - public int age; -} - -Fory fory = Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .build(); - -fory.register(User.class, "example", "User"); -``` - -长期使用的 Schema 应采用字段 ID，这样即使 Java 字段名发生变化，字段标识仍保持稳定。Java 注解、可空性、引用跟踪和枚举元数据参见 [Schema 元数据](schema-metadata.md)。 - -## 序列化与反序列化 - -```java -User user = new User(); -user.name = "Alice"; -user.age = 30; - -byte[] bytes = fory.serialize(user); -User decoded = fory.deserialize(bytes, User.class); -``` - -跨语言字节在不同语言之间传递时，每个对等端都必须注册相同的类型标识和兼容的字段元数据。共享规则参见[跨语言序列化](../xlang/index.md)，Java 专用 API 调用参见 [Java 跨语言序列化](xlang.md)。 - -## 对仅限 Java 的流量使用原生序列化 - -对于同语言的 Java/JVM 流量，原生模式通常更合适： - -```java -Fory fory = Fory.builder() - .withXlang(false) - .build(); -``` - -原生模式支持广泛的 Java 对象序列化能力，包括 JDK 序列化钩子、对象复制和原生模式零拷贝缓冲区。参见[原生序列化](native.md)。 - -## 常用选项 - -- `withRefTracking(true)` 保留共享引用和循环引用。 -- `requireClassRegistration(true)` 保持默认的已注册类型策略。 -- 原生模式和跨语言载荷默认启用兼容模式。仅当所有读取端和写入端使用相同 Schema，且希望获得更快速度和更小体积时，才使用 `withCompatible(false)`。对于跨语言载荷，只有在确认所有语言使用相同 Schema，或原生类型由 Fory Schema IDL 生成时，才应选择 `compatible=false`。 -- `withAsyncCompilation(true)` 在受支持的平台上启用异步序列化器编译。 - -## 最佳实践 - -1. **复用 Fory 实例**：创建 Fory 的开销较大，应始终复用实例 -2. **选择合适的线程安全方式**：根据需求选择单线程或线程安全实现 -3. **注册类**：在每个跨语言对等端保持稳定的类型标识 -4. **配置引用跟踪**：仅当对象图需要保持对象标识或循环引用时启用 - -## 相关主题 - -- [配置](configuration.md) - 所有 ForyBuilder 选项 -- [原生序列化](native.md) - 仅限 Java 的序列化功能 -- [Schema 元数据](schema-metadata.md) - 字段 ID、可空性、引用跟踪和枚举 ID -- [跨语言序列化](xlang.md) - Java 跨语言互操作 -- [故障排除](troubleshooting.md) - 常见 API 使用问题 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/index.md index b474077a515..708370bbe17 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/index.md @@ -21,7 +21,7 @@ license: | Apache Fory Java 提供高性能二进制对象序列化。与其他受支持运行时共享的载荷使用跨语言模式，仅供 Java/JVM 使用的对象图则使用原生模式。 -本运行时指南仅介绍二进制对象序列化。其他 Java 产品参见 [Row Format](../../row-format/java.md)、[Fory JSON](../../json/index.md)、[Fory IDL 与编译器](../../compiler/index.md)或 [Fory gRPC](../../grpc/java.md)。 +本运行时指南仅介绍二进制对象序列化。其他 Java 能力参见 [Row Format](../../row-format/java.md)、[Fory JSON](../../json/index.md)、[Fory IDL 与编译器](../../compiler/index.md)或 [Fory gRPC](../../grpc/java.md)。 ## 二进制对象序列化 @@ -156,7 +156,7 @@ public class Example { 仅限 Java 的流量应使用原生模式。通过 `.withXlang(false)` 选择原生模式；该模式负责 JDK 序列化钩子、`Externalizable`、动态对象图、对象复制和 Java 原生模式零拷贝缓冲区等 Java 专用对象行为。它针对 JVM 类型系统优化，支持比跨语言模式更广泛的 Java 对象。兼容模式默认启用。仅当每个读取端和写入端都使用相同的类 Schema，且希望获得更快速度和更小体积时，才设置 `.withCompatible(false)`。如果要替换 JDK 序列化、Kryo、FST、Hessian 或仅限 Java 的 Protocol Buffers 载荷，请从原生模式开始。 -仅限 Java 的序列化详情参见[原生序列化](native.md)，Java 跨语言注册与互操作规则参见[跨语言序列化](xlang.md)。 +仅限 Java 的序列化详情参见[原生序列化](native.md)，Java 跨语言注册与互操作规则参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ### 线程安全 @@ -235,9 +235,9 @@ Fory JSON 具有独立的 [Android](../../json/android.md) 和 [GraalVM 原生 | 分组 | 页面 | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 序列化模式 | [跨语言](xlang.md)、[原生](native.md) | -| 通用内容 | [核心 API](core-api.md)、[配置](configuration.md)、[类型注册](type-registration.md)、[Schema 演进](schema-evolution.md)、[Schema 元数据](schema-metadata.md)、[自定义序列化器](custom-serializers.md) | +| 序列化模式 | [基础序列化](basic-serialization.md)、[原生序列化](native.md) | +| 通用 | [配置](configuration.md)、[类型注册](type-registration.md)、[Schema 演进](schema-evolution.md)、[Schema 元数据](schema-metadata.md)、[自定义序列化器](custom-serializers.md) | | Java 专用功能 | [高级功能](advanced-features.md)、[压缩](compression.md)、[对象复制](object-copy.md)、[JDK 自定义序列化](jdk-serialization.md)、[静态生成的序列化器](static-generated-serializers.md)、[虚拟线程](virtual-threads.md) | | 平台与运维 | [Android](android.md)、[GraalVM 原生镜像](graalvm.md)、[故障排除](troubleshooting.md) | -解码外部提供的二进制载荷前，请阅读[对象序列化安全](../security.md)。 +解码外部提供的二进制载荷前，请阅读 [Java 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/native.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/native.md index ec91d531081..55710819dac 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/native.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/native.md @@ -23,7 +23,7 @@ Java 原生序列化是通过 `withXlang(false)` 选择的仅限 Java 编码格 本页所称原生序列化是指 Fory 的 `xlang=false` 编码模式，与 GraalVM 原生镜像支持不同；后者参见 [GraalVM 原生镜像](graalvm.md)。 -如果字节需要由非 Java Fory 实现读取，请使用 Java 默认的[跨语言序列化](xlang.md)模式。 +如果字节需要由非 Java Fory 实现读取，请使用 Java 默认的[跨语言序列化](basic-serialization.md#cross-language-interoperability)模式。 ## 何时使用原生序列化 @@ -280,8 +280,8 @@ Fory fory = Fory.builder() ## 相关主题 -- [基础序列化](core-api.md) - 跨语言优先的 Java 快速入门 -- [跨语言序列化](xlang.md) - 跨语言 Java 载荷 +- [基础序列化](basic-serialization.md) - 跨语言优先的 Java 快速入门 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言 Java 载荷 - [配置](configuration.md) - Java 构建器选项 - [Schema 演进](schema-evolution.md) - 兼容模式与相同 Schema 优化 - [类型注册](type-registration.md) - 注册与安全 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/object-copy.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/object-copy.md index 7da25faad98..466eb67d57c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/object-copy.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/object-copy.md @@ -344,7 +344,7 @@ Fory fory = Fory.builder().withXlang(false) ## 相关主题 -- [基础序列化](core-api.md) - Fory 实例创建和核心 API +- [基础序列化](basic-serialization.md) - Fory 实例创建和核心 API - [配置](configuration.md) - 包括 `withRefCopy` 在内的构建器选项 - [自定义序列化器](custom-serializers.md) - 序列化器设计与注册 - [虚拟线程](virtual-threads.md) - 线程安全的 Fory 使用指南 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/schema-evolution.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/schema-evolution.md index 4e60937eab0..7f20c7d2a96 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/schema-evolution.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/schema-evolution.md @@ -246,5 +246,5 @@ public class SameSchemaMessage { ## 相关主题 - [配置](configuration.md) - 所有 ForyBuilder 选项 -- [跨语言序列化](xlang.md) - 跨语言模式 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言模式 - [故障排除](troubleshooting.md) - 常见 Schema 问题 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/schema-metadata.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/schema-metadata.md index 574a00fadac..837461a6e35 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/schema-metadata.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/schema-metadata.md @@ -694,7 +694,7 @@ public class User { ## 相关主题 -- [基础序列化](core-api.md) - Fory 序列化入门 +- [基础序列化](basic-serialization.md) - Fory 序列化入门 - [配置](configuration.md) - `ForyBuilder` 选项 - [Schema 演进](schema-evolution.md) - 兼容模式与 Schema 演进 -- [跨语言序列化](xlang.md) - 与 Python、Rust、C++、Go 互操作 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 与 Python、Rust、C++、Go 互操作 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/security.md new file mode 100644 index 00000000000..f4c2f408fdd --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/java/security.md @@ -0,0 +1,68 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 Java 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +在生产环境以及处理任何不可信载荷来源时，请保持类注册启用： + +```java +Fory fory = Fory.builder() + .requireClassRegistration(true) + .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .withMaxUnbackedContainerItems(8192) + .build(); +``` + +安全相关选项： + +- `requireClassRegistration(true)` 将反序列化限制为已注册类。 +- `withMaxDepth(...)` 拒绝深度异常的对象图。 +- `withMaxGraphMemoryBytes(...)` 为单次根反序列化期间实例化的对象图内存设置近似门限。估算主要覆盖集合、映射、数组、结构体和对象；Fory core 原始类型数组和原始类型列表根据解码长度计入其原始存储。它会跳过字符串、原始标量和不使用原始类型数组序列化器的专用二进制值等叶子值。实际进程内存可能高于该限制。叶子值仍受字节可用性检查保护：如果未读取的输入没有足够字节，Fory 不会读取或创建该叶子值。默认值固定为 `128 MiB`；可信工作负载需要更大或更小门限时，请设置正数字节限制。 +- `withMaxUnbackedContainerItems(...)` 限制由数量驱动、但重复读取正文没有消耗相应输入的集合和映射工作。默认值为 `8192`；零表示严格限制。 +- `withMaxTypeFields(...)` 和 `withMaxTypeMetaBytes(...)` 限制单个已接收远程元数据正文的字段数与编码正文大小。 +- `withMaxSchemaVersionsPerType(...)` 和 + `withMaxAverageSchemaVersionsPerType(...)` 限制可接受的远程元数据版本，而不改变注册、动态加载或 Schema 演进语义。 +- `withDeserializeUnknownClass(false)` 避免根据元数据实例化未知类。 +- `checkJdkClassSerializable(true)` 保留对 `java.*` 类的 JDK 可序列化性检查。 +- 类注册警告可用于安全审计；需要暴露意外类型时，请使用 `suppressClassRegistrationWarnings(false)`。 + +仅对可信载荷使用 `requireClassRegistration(false)`；需要动态类加载时，应同时配置 `TypeChecker` 允许列表。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[类型注册](type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/basic-serialization.md similarity index 54% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/xlang.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/basic-serialization.md index 75a66d92730..6d27626843a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/xlang.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: 跨语言序列化 +title: 基础序列化 sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,6 +19,228 @@ license: | limitations under the License. --- +本指南介绍 Apache Fory JavaScript 默认 xlang 模式下的核心序列化 API。 + +## 创建 `Fory` 实例 + +```ts +import Fory from "@apache-fory/core"; + +const fory = new Fory(); +``` + +创建一个实例，注册 Schema 后复用它。Fory 会在首次调用 `register` 后缓存生成的序列化器，因此每次请求都重新创建实例会浪费这些工作。 + +## 使用 `Type.struct` 定义 Schema + +最常用的方式是定义并注册 Schema。 + +```ts +import Fory, { Type } from "@apache-fory/core"; + +const accountType = Type.struct( + { typeName: "example.account" }, + { + id: Type.int64(), + owner: Type.string(), + active: Type.bool(), + nickname: Type.string().setNullable(true), + }, +); + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(accountType); +``` + +## 序列化和反序列化 + +```ts +const bytes = serialize({ + id: 42n, + owner: "Alice", + active: true, + nickname: null, +}); + +const value = deserialize(bytes); +console.log(value); +// { id: 42n, owner: 'Alice', active: true, nickname: null } +``` + +返回的 `bytes` 值是 `Uint8Array`/平台缓冲区，可以通过网络发送或写入存储。 + +## 根级动态序列化 + +`Fory` 也可以直接序列化动态根值，无需预先绑定特定 Schema 的序列化器。 + +```ts +const fory = new Fory(); + +const bytes = fory.serialize( + new Map([ + ["name", "Alice"], + ["age", 30], + ]), +); + +const value = fory.deserialize(bytes); +``` + +这对于动态载荷很方便，但对于稳定接口和跨语言契约，显式 Schema 通常更合适。 + +## 原始值 + +```ts +const fory = new Fory(); + +fory.deserialize(fory.serialize(true)); +// true + +fory.deserialize(fory.serialize("hello")); +// 'hello' + +fory.deserialize(fory.serialize(123)); +// 123 + +fory.deserialize(fory.serialize(123n)); +// 123n + +fory.deserialize(fory.serialize(new Date("2021-10-20T09:13:00Z"))); +// Date +``` + +### Number 和 `bigint` + +JavaScript `number` 是 64 位浮点数，无法精确表示所有 64 位整数。对于跨语言契约或任何需要精确整数宽度的场景，请在 Schema 中使用显式字段类型： + +- `Type.int32()` — 32 位整数；使用 JavaScript `number` +- `Type.int64()` — 64 位整数；使用 JavaScript `bigint` +- `Type.float32()` / `Type.float64()` — 浮点数 + +动态根值序列化（不提供 Schema 而调用 `fory.serialize(someNumber)`）会推断类型，但 API 不保证推断出的类型。任何稳定契约都应使用 Schema。 + +## Array、Map 和 Set + +```ts +const inventoryType = Type.struct("example.inventory", { + tags: Type.list(Type.string()), + counts: Type.map(Type.string(), Type.int32()), + labels: Type.set(Type.string()), +}); + +const fory = new Fory({ ref: true }); +const { serialize, deserialize } = fory.register(inventoryType); + +const bytes = serialize({ + tags: ["hot", "new"], + counts: new Map([ + ["apple", 3], + ["pear", 8], + ]), + labels: new Set(["featured", "seasonal"]), +}); + +const value = deserialize(bytes); +``` + +## 嵌套 Struct + +```ts +const addressType = Type.struct("example.address", { + city: Type.string(), + country: Type.string(), +}); + +const userType = Type.struct("example.user", { + name: Type.string(), + address: Type.struct("example.address", { + city: Type.string(), + country: Type.string(), + }), +}); + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(userType); + +const bytes = serialize({ + name: "Alice", + address: { city: "Hangzhou", country: "CN" }, +}); + +const user = deserialize(bytes); +``` + +如果嵌套值可能缺失，请将其标记为可空： + +```ts +const wrapperType = Type.struct("example.wrapper", { + child: Type.struct("example.child", { + name: Type.string(), + }).setNullable(true), +}); +``` + +## 基于装饰器的注册 + +也支持 TypeScript 装饰器。 + +```ts +import Fory, { Type } from "@apache-fory/core"; + +@Type.struct("example.user") +class User { + @Type.int64() + id!: bigint; + + @Type.string() + name!: string; +} + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(User); + +const user = new User(); +user.id = 1n; +user.name = "Alice"; + +const copy = deserialize(serialize(user)); +console.log(copy instanceof User); // true +``` + +## 可空性 + +基于 Schema 的 struct 会显式声明字段可空性。 + +```ts +const nullableType = Type.struct("example.optional_user", { + name: Type.string(), + email: Type.string().setNullable(true), +}); +``` + +如果字段未标记为可空却尝试写入 `null`，序列化会抛出异常。 + +## 调试生成代码 + +可以使用 `hooks.afterCodeGenerated` 检查生成的序列化器代码。 + +```ts +const fory = new Fory({ + hooks: { + afterCodeGenerated(code) { + console.log(code); + return code; + }, + }, +}); +``` + +这有助于调试 Schema 行为、字段顺序或生成的快速路径。 + +## 跨语言互操作 {#cross-language-interoperability} + +以下内容说明默认 xlang 格式的跨语言类型映射、类型标识和互操作要求。 + Fory JavaScript 序列化为与 Java、Python、C++、Go、Rust、C#、Swift、Dart、Scala 和 Kotlin Fory 实现相同的二进制格式。无需转换层，即可在 JavaScript 中写入消息并在 Java 中读取，也支持任意其他方向。 注意事项： @@ -26,7 +248,7 @@ Fory JavaScript 序列化为与 Java、Python、C++、Go、Rust、C#、Swift、D - Fory JavaScript 只读写跨语言载荷，不支持任何原生模式格式。 - JavaScript 不支持带外模式。 -## 成功往返的要求 +### 成功往返的要求 要让消息在 JavaScript 和另一种语言之间成功往返： @@ -36,7 +258,7 @@ Fory JavaScript 序列化为与 Java、Python、C++、Go、Rust、C#、Swift、D 4. 双方都使用兼容 Schema 演进。JavaScript 默认启用此模式。 5. 如果数据包含共享引用或循环引用，使用**相同引用跟踪配置**。 -## 分步操作：从 JavaScript 到其他通信方 +### 分步操作：从 JavaScript 到其他通信方 1. 使用与通信方相同的类型名称或数字 ID 定义 JavaScript Schema。 2. 在双方注册 Schema。 @@ -72,13 +294,13 @@ const bytes = serialize({ - [Go 指南](../go/index.md) - [Rust 指南](../rust/index.md) -## 字段命名 +### 字段命名 Fory 按名称匹配字段。在多种语言中定义模型时，请保持字段名称一致；至少应使用能够跨语言明确映射的命名方案，例如全部使用 `snake_case`。 默认兼容 Schema 演进可以容忍字段顺序差异，但名称本身仍必须匹配。 -## 数值类型 +### 数值类型 JavaScript `number` 是 64 位浮点数，无法清晰地映射到其他语言中的每种整数类型。请使用显式 Schema 类型： @@ -86,7 +308,7 @@ JavaScript `number` 是 64 位浮点数，无法清晰地映射到其他语言 - `Type.int64()` 配合 `bigint` 值用于 64 位整数（Java `long`、Go `int64`） - 浮点值使用 `Type.float32()` 或 `Type.float64()` -## 列表和密集数组 +### 列表和密集数组 使用 `Type.list(T)` 表示普通 JavaScript `Array` 值和 Fory `list` Schema。密集 bool/数值向量使用下面列出的显式数组构建器。 @@ -107,13 +329,13 @@ JavaScript `number` 是 64 位浮点数，无法清晰地映射到其他语言 | `array` | `Type.float32Array()` | | `array` | `Type.float64Array()` | -## 日期和时间 +### 日期和时间 - `Type.timestamp()` — 时间点；往返后为 JavaScript `Date` - `Type.date()` — 不含时间的日期；反序列化为 `Date` - `Type.duration()` — 在 JavaScript 中公开为数值毫秒值 -## 多态字段 +### 多态字段 `Type.any()` 允许字段容纳不同具体类型，但更难以跨语言保持一致。请尽可能优先使用显式字段 Schema。 @@ -126,7 +348,7 @@ const wrapperType = Type.struct( ); ``` -## Enum +### Enum Enum 成员的**顺序**必须跨语言匹配。Fory 按序号位置而不是按值编码 enum。 @@ -138,17 +360,17 @@ fory.register(Type.enum({ typeId: 210 }, Color)); 每个通信方使用相同类型 ID 或类型名称。 -## 安全限制 +### 安全限制 `maxDepth` 选项限制嵌套载荷。它不会改变二进制格式，只控制本地 `Fory` 实例接受的内容。 -## 相关主题 +### 相关指南 - [支持的类型](supported-types.md) - [Schema 演进](schema-evolution.md) - [跨语言序列化规范](../../specification/xlang_serialization_spec.md) -## 内置值 +### 内置值 ```javascript import Fory from "@apache-fory/core"; @@ -159,7 +381,7 @@ const result = fory.deserialize(input); console.log(result); ``` -## 自定义值 +### 自定义值 ```javascript import Fory, { Type } from "@apache-fory/core"; @@ -178,7 +400,7 @@ const result = deserialize(input); console.log(result); ``` -## 共享引用与循环引用 +### 共享引用与循环引用 ```javascript import Fory, { Type } from "@apache-fory/core"; @@ -198,3 +420,9 @@ const input = serialize(data); const result = deserialize(input); console.log(result.bar.foo === result.foo); ``` + +## 相关主题 + +- [类型注册](type-registration.md) +- [支持的类型](supported-types.md) +- [引用](references.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/configuration.md index 70a32fa180c..6b3326bc8f7 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/configuration.md @@ -120,19 +120,11 @@ const fory = new Fory({ hps }); ## 安全 -安全相关配置： - -- 反序列化不可信载荷前，只注册预期 Schema。 -- 将 `maxDepth` 设置为服务允许的最大嵌套深度。 -- 将 `maxGraphMemoryBytes` 设为包含大量 collection、map、array、struct 和 object 的载荷的近似限制。它并非精确的堆上限；叶子值受剩余输入字节限制。 -- 保持 `maxTypeFields` 和 `maxTypeMetaBytes` 的默认值，除非数据可信且可信通信方会发送更大的远程元数据。 -- 保持 `maxSchemaVersionsPerType` 和 `maxAverageSchemaVersionsPerType` 的默认值，除非数据可信且可信通信方会发送大量远程 Schema 版本。 -- 对不可信输入，优先使用显式 `Type.struct(...)` Schema，而不是 `Type.any()`。 -- 只传入来自与 Fory 一起部署的官方包版本的 `hps`。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [JavaScript/TypeScript 安全](security.md)。 ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [Schema 元数据](schema-metadata.md) - [Schema 演进](schema-evolution.md) - [引用](references.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/core-api.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/core-api.md deleted file mode 100644 index 23ed4e1bbd4..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/core-api.md +++ /dev/null @@ -1,244 +0,0 @@ ---- -title: 基本序列化 -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本指南介绍 Apache Fory JavaScript 的核心序列化 API。 - -## 创建 `Fory` 实例 - -```ts -import Fory from "@apache-fory/core"; - -const fory = new Fory(); -``` - -创建一个实例，注册 Schema 后复用它。Fory 会在首次调用 `register` 后缓存生成的序列化器，因此每次请求都重新创建实例会浪费这些工作。 - -## 使用 `Type.struct` 定义 Schema - -最常用的方式是定义并注册 Schema。 - -```ts -import Fory, { Type } from "@apache-fory/core"; - -const accountType = Type.struct( - { typeName: "example.account" }, - { - id: Type.int64(), - owner: Type.string(), - active: Type.bool(), - nickname: Type.string().setNullable(true), - }, -); - -const fory = new Fory(); -const { serialize, deserialize } = fory.register(accountType); -``` - -## 序列化和反序列化 - -```ts -const bytes = serialize({ - id: 42n, - owner: "Alice", - active: true, - nickname: null, -}); - -const value = deserialize(bytes); -console.log(value); -// { id: 42n, owner: 'Alice', active: true, nickname: null } -``` - -返回的 `bytes` 值是 `Uint8Array`/平台缓冲区，可以通过网络发送或写入存储。 - -## 根级动态序列化 - -`Fory` 也可以直接序列化动态根值，无需预先绑定特定 Schema 的序列化器。 - -```ts -const fory = new Fory(); - -const bytes = fory.serialize( - new Map([ - ["name", "Alice"], - ["age", 30], - ]), -); - -const value = fory.deserialize(bytes); -``` - -这对于动态载荷很方便，但对于稳定接口和跨语言契约，显式 Schema 通常更合适。 - -## 原始值 - -```ts -const fory = new Fory(); - -fory.deserialize(fory.serialize(true)); -// true - -fory.deserialize(fory.serialize("hello")); -// 'hello' - -fory.deserialize(fory.serialize(123)); -// 123 - -fory.deserialize(fory.serialize(123n)); -// 123n - -fory.deserialize(fory.serialize(new Date("2021-10-20T09:13:00Z"))); -// Date -``` - -### Number 和 `bigint` - -JavaScript `number` 是 64 位浮点数，无法精确表示所有 64 位整数。对于跨语言契约或任何需要精确整数宽度的场景，请在 Schema 中使用显式字段类型： - -- `Type.int32()` — 32 位整数；使用 JavaScript `number` -- `Type.int64()` — 64 位整数；使用 JavaScript `bigint` -- `Type.float32()` / `Type.float64()` — 浮点数 - -动态根值序列化（不提供 Schema 而调用 `fory.serialize(someNumber)`）会推断类型，但 API 不保证推断出的类型。任何稳定契约都应使用 Schema。 - -## Array、Map 和 Set - -```ts -const inventoryType = Type.struct("example.inventory", { - tags: Type.list(Type.string()), - counts: Type.map(Type.string(), Type.int32()), - labels: Type.set(Type.string()), -}); - -const fory = new Fory({ ref: true }); -const { serialize, deserialize } = fory.register(inventoryType); - -const bytes = serialize({ - tags: ["hot", "new"], - counts: new Map([ - ["apple", 3], - ["pear", 8], - ]), - labels: new Set(["featured", "seasonal"]), -}); - -const value = deserialize(bytes); -``` - -## 嵌套 Struct - -```ts -const addressType = Type.struct("example.address", { - city: Type.string(), - country: Type.string(), -}); - -const userType = Type.struct("example.user", { - name: Type.string(), - address: Type.struct("example.address", { - city: Type.string(), - country: Type.string(), - }), -}); - -const fory = new Fory(); -const { serialize, deserialize } = fory.register(userType); - -const bytes = serialize({ - name: "Alice", - address: { city: "Hangzhou", country: "CN" }, -}); - -const user = deserialize(bytes); -``` - -如果嵌套值可能缺失，请将其标记为可空： - -```ts -const wrapperType = Type.struct("example.wrapper", { - child: Type.struct("example.child", { - name: Type.string(), - }).setNullable(true), -}); -``` - -## 基于装饰器的注册 - -也支持 TypeScript 装饰器。 - -```ts -import Fory, { Type } from "@apache-fory/core"; - -@Type.struct("example.user") -class User { - @Type.int64() - id!: bigint; - - @Type.string() - name!: string; -} - -const fory = new Fory(); -const { serialize, deserialize } = fory.register(User); - -const user = new User(); -user.id = 1n; -user.name = "Alice"; - -const copy = deserialize(serialize(user)); -console.log(copy instanceof User); // true -``` - -## 可空性 - -基于 Schema 的 struct 会显式声明字段可空性。 - -```ts -const nullableType = Type.struct("example.optional_user", { - name: Type.string(), - email: Type.string().setNullable(true), -}); -``` - -如果字段未标记为可空却尝试写入 `null`，序列化会抛出异常。 - -## 调试生成代码 - -可以使用 `hooks.afterCodeGenerated` 检查生成的序列化器代码。 - -```ts -const fory = new Fory({ - hooks: { - afterCodeGenerated(code) { - console.log(code); - return code; - }, - }, -}); -``` - -这有助于调试 Schema 行为、字段顺序或生成的快速路径。 - -## 相关主题 - -- [类型注册](type-registration.md) -- [支持的类型](supported-types.md) -- [引用](references.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/index.md index 129497dc240..393185bf80e 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/index.md @@ -150,14 +150,13 @@ Fory JavaScript 仅支持 xlang。`new Fory()` 默认使用兼容 Schema 演进 | 主题 | 说明 | | ------------------------------------------ | ----------------------------------------- | -| [基本序列化](core-api.md) | 核心 API 和日常用法 | +| [基本序列化](basic-serialization.md) | 核心 API 和日常用法 | | [配置](configuration.md) | Fory 选项、兼容模式、限制和 HPS | | [类型注册](type-registration.md) | 数字 ID、名称、装饰器和 Schema 注册 | | [Schema 元数据](schema-metadata.md) | 类型构建器、字段选项和装饰器 | | [支持的类型](supported-types.md) | 原始类型、集合、时间、enum 和 struct 映射 | | [引用](references.md) | 共享引用和循环对象图 | | [Schema 演进](schema-evolution.md) | 兼容模式和可演进 struct | -| [跨语言序列化](xlang.md) | 互操作指南和映射规则 | | [Fory IDL 编译器](../../compiler/index.md) | 从 `.fdl` Schema 生成 TypeScript 模型 | | [gRPC 支持](../../grpc/javascript.md) | Node.js gRPC 和浏览器 gRPC-Web 生成客户端 | | [故障排查](troubleshooting.md) | 常见问题、限制和调试技巧 | @@ -166,3 +165,5 @@ Fory JavaScript 仅支持 xlang。`new Fory()` 默认使用兼容 Schema 演进 - [跨语言序列化规范](../../specification/xlang_serialization_spec.md) - [跨语言类型映射](../../specification/xlang_type_mapping.md) + +解码来自应用信任边界之外的字节之前，请阅读 [JavaScript/TypeScript 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/references.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/references.md index 1a36747b244..a1b68a1be6e 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/references.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/references.md @@ -106,6 +106,6 @@ console.log(copy.left === copy.right); // true ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [Schema 演进](schema-evolution.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/schema-evolution.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/schema-evolution.md index 034a506946f..e134109292a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/schema-evolution.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/schema-evolution.md @@ -94,9 +94,9 @@ const fixedType = Type.struct( ## 跨语言要求 -兼容模式只能处理类型*字段*之间的 Schema 差异。各方仍需使用相同的类型标识（相同数字 ID 或相同 `typeName`）。参见[跨语言序列化](xlang.md)。 +兼容模式只能处理类型*字段*之间的 Schema 差异。各方仍需使用相同的类型标识（相同数字 ID 或相同 `typeName`）。参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ## 相关主题 - [类型注册](type-registration.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/security.md new file mode 100644 index 00000000000..09ba7195f55 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/security.md @@ -0,0 +1,52 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 JavaScript/TypeScript 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +安全相关配置： + +- 反序列化不可信载荷前，只注册预期 Schema。 +- 将 `maxDepth` 设置为服务允许的最大嵌套深度。 +- 将 `maxGraphMemoryBytes` 设为包含大量 collection、map、array、struct 和 object 的载荷的近似限制。它并非精确的堆上限；叶子值受剩余输入字节限制。 +- 保持 `maxTypeFields` 和 `maxTypeMetaBytes` 的默认值，除非数据可信且可信通信方会发送更大的远程元数据。 +- 保持 `maxSchemaVersionsPerType` 和 `maxAverageSchemaVersionsPerType` 的默认值，除非数据可信且可信通信方会发送大量远程 Schema 版本。 +- 对不可信输入，优先使用显式 `Type.struct(...)` Schema，而不是 `Type.any()`。 +- 只传入来自与 Fory 一起部署的官方包版本的 `hps`。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[类型注册](type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/supported-types.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/supported-types.md index a6f38c6dafc..7c2ca911aa7 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/supported-types.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/supported-types.md @@ -172,6 +172,6 @@ Type.struct("example.node").setTrackingRef(true).setNullable(true); ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [引用](references.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/troubleshooting.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/troubleshooting.md index ff5f6e92f2f..2203294138f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/troubleshooting.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/troubleshooting.md @@ -84,6 +84,6 @@ const fory = new Fory({ ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [引用](references.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/type-registration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/type-registration.md index 2b3c880ec20..17a359a5f4c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/type-registration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/javascript/type-registration.md @@ -152,11 +152,11 @@ const order = deserialize(bytes); ## 跨语言 -为了让消息在 JavaScript 和另一种语言之间往返，双方必须为给定类型使用相同标识：相同数字 ID 或相同 `typeName`。使用 `.` 在 `typeName` 中添加命名空间前缀。参见[跨语言序列化](xlang.md)。 +为了让消息在 JavaScript 和另一种语言之间往返，双方必须为给定类型使用相同标识：相同数字 ID 或相同 `typeName`。使用 `.` 在 `typeName` 中添加命名空间前缀。参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ## 相关主题 -- [基本序列化](core-api.md) +- [基本序列化](basic-serialization.md) - [Schema 元数据](schema-metadata.md) - [Schema 演进](schema-evolution.md) -- [跨语言序列化](xlang.md) +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/basic-serialization.md similarity index 84% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/xlang.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/basic-serialization.md index eb1907057b9..82674fad1e1 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/xlang.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Kotlin Xlang 序列化 +title: 基础序列化 sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,6 +19,12 @@ license: | limitations under the License. --- +Xlang 是 Fory Kotlin 的默认序列化模式。本页介绍该默认模式的基础序列化 API 和互操作规则。 + +## 跨语言互操作 {#cross-language-interoperability} + +以下内容介绍默认 xlang 模式的模型生成、注册和跨语言往返。 + Kotlin xlang 序列化通过 `ForyKotlin` 使用 JVM Fory 实现。当 Kotlin 载荷需要由其他受支持 的 Fory 运行时读取时，请使用该模式。在所有对端使用相同身份和字段 Schema 注册可移植 模型类型。 @@ -26,7 +32,7 @@ Kotlin xlang 序列化通过 `ForyKotlin` 使用 JVM Fory 实现。当 Kotlin Kotlin 数据类、枚举和密封类模型会在适用时使用 Kotlin 集成与生成的序列化器。确切的 可移植载体映射仍由 [xlang 类型映射](../../specification/xlang_type_mapping.md)定义。 -## 创建 xlang 实例 +### 创建 Fory 实例 ```kotlin import org.apache.fory.kotlin.ForyKotlin @@ -36,7 +42,7 @@ val fory = ForyKotlin.builder() .build() ``` -## 第一次往返处理 +### 第一次往返处理 ```kotlin import org.apache.fory.ThreadSafeFory diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/configuration.md index aa0acf7d2ac..034f3b3fb8c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/configuration.md @@ -142,29 +142,6 @@ Java builder 在 xlang 和原生模式下都默认启用兼容模式。当模型 语言使用相同 Schema 后，或者原生类型由 Fory Schema IDL 生成时，才调用 `withCompatible(false)`。 -## 安全性 +## 安全 -Kotlin 使用 Java 配置接口。在生产环境以及处理任何不可信载荷来源时，请保持启用类注册： - -```kotlin -val fory = ForyKotlin.builder() - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .withMaxTypeFields(512) - .withMaxTypeMetaBytes(4096) - .build() -``` - -与安全相关的配置： - -- 保持 `requireClassRegistration(true)`，并注册应用类或生成的模块。 -- 使用 `withMaxDepth(...)` 拒绝深度异常的对象图。 -- 使用 `withMaxGraphMemoryBytes(...)` 为包含大量集合、map、数组、结构体和对象的载荷 - 设置近似限制。它不是精确的堆上限；叶子值受剩余输入字节限制。 -- 除非可信的紧凑 codec 需要更大的根操作余量，否则将 - `withMaxUnbackedContainerItems(...)` 保持为 `8192`。零会拒绝每一个无输入支撑的条目。 -- 除非数据没有恶意，且可信对端会发送更大的元数据或许多 Schema 版本，否则请将 - `withMaxTypeFields(...)`、`withMaxTypeMetaBytes(...)` 和远端 Schema 版本限制保留为默认值。 -- 白名单和未知类控制请遵循[对象序列化安全](../security.md)。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [Kotlin 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/index.md index 6a6a7854655..313cbeb1da9 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/index.md @@ -117,7 +117,7 @@ Kotlin builder 设置请参阅[配置](configuration.md)，完整的 JVM 原生 Fory Kotlin 基于 Fory Java 构建。Fory Java 的大多数配置选项、功能和概念都直接适用于 Kotlin。以下内容请参阅 Java 文档： - [配置](../java/configuration.md) - 所有 ForyBuilder 选项 -- [基本序列化](../java/core-api.md) - 序列化模式和 API +- [基本序列化](../java/basic-serialization.md) - 序列化模式和 API - [类型注册](../java/type-registration.md) - 类注册与安全性 - [Schema 演进](../java/schema-evolution.md) - 向前/向后兼容性 - [自定义序列化器](../java/custom-serializers.md) - 实现自定义序列化器 @@ -126,7 +126,7 @@ Fory Kotlin 基于 Fory Java 构建。Fory Java 的大多数配置选项、功 ## Kotlin 专属文档 -- [Xlang 序列化](xlang.md) - 用于直接跨语言对象序列化的 Kotlin API +- [基础序列化](basic-serialization.md) - 默认 xlang 模式的模型、API 和跨语言互操作 - [配置](configuration.md) - Kotlin 特有的 Fory 设置要求 - [原生序列化](native.md) - 在 JVM 原生模式下序列化 Kotlin 类型 - [Schema 元数据](schema-metadata.md) - Kotlin 注解、可空性、引用和整数元数据 @@ -134,3 +134,5 @@ Fory Kotlin 基于 Fory Java 构建。Fory Java 的大多数配置选项、功 - [静态生成的序列化器](static-generated-serializers.md) - 通过 KSP 生成 xlang/Schema 序列化器 - [Kotlin gRPC 支持](../../grpc/kotlin.md) - Fory IDL 服务的协程桩和服务基类 - [Android 支持](../java/android.md) - Android 设置、R8 行为和发布构建验证 + +解码来自应用信任边界之外的字节之前，请阅读 [Kotlin 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/native.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/native.md index c50947b3966..0cd50c1f047 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/native.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/native.md @@ -20,7 +20,7 @@ license: | --- 本页介绍如何在原生模式下序列化 Kotlin 特有的 JVM 类型。跨语言 Kotlin 模型请使用 -[Kotlin Xlang 序列化](xlang.md)。 +[Kotlin Xlang 序列化](basic-serialization.md#cross-language-interoperability)。 启用兼容模式后，Kotlin 读取端会针对部分标量字段类型变更使用 JVM 兼容读取规则。当转换后 的值具有相同逻辑值时，匹配字段可以在 `Boolean`、`String`、数值标量和 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/security.md new file mode 100644 index 00000000000..854edf8c970 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/kotlin/security.md @@ -0,0 +1,67 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 Kotlin 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +Kotlin 使用 Java 配置接口。在生产环境以及处理任何不可信载荷来源时，请保持启用类注册： + +```kotlin +val fory = ForyKotlin.builder() + .requireClassRegistration(true) + .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .withMaxUnbackedContainerItems(8192) + .withMaxTypeFields(512) + .withMaxTypeMetaBytes(4096) + .build() +``` + +与安全相关的配置： + +- 保持 `requireClassRegistration(true)`，并注册应用类或生成的模块。 +- 使用 `withMaxDepth(...)` 拒绝深度异常的对象图。 +- 使用 `withMaxGraphMemoryBytes(...)` 为包含大量集合、map、数组、结构体和对象的载荷 + 设置近似限制。它不是精确的堆上限；叶子值受剩余输入字节限制。 +- 除非可信的紧凑 codec 需要更大的根操作余量，否则将 + `withMaxUnbackedContainerItems(...)` 保持为 `8192`。零会拒绝每一个无输入支撑的条目。 +- 除非数据没有恶意，且可信对端会发送更大的元数据或许多 Schema 版本，否则请将 + `withMaxTypeFields(...)`、`withMaxTypeMetaBytes(...)` 和远端 Schema 版本限制保留为默认值。 +- 白名单和未知类控制请遵循 [Java 安全](../java/security.md)。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[Java 类型注册](../java/type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/native.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/native.md index 5a443771a16..f62a7e2fb0c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/native.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/native.md @@ -1,6 +1,6 @@ --- title: 原生序列化 -sidebar_position: 2 +sidebar_position: 3 id: native license: | Licensed to the Apache Software Foundation (ASF) under one or more @@ -25,7 +25,7 @@ license: | 当同一运行时之间的通信需要语言特有对象形状、从宿主序列化器迁移，或者需要不受 xlang 类型映射约束且更小、更快的格式时，请使用原生模式。只要需要由不同运行时读取字节， -就应使用 [xlang 模式](xlang/index.md)。 +就应使用 [xlang 模式](xlang.md)。 ## 支持的运行时系列 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/basic-serialization.md similarity index 77% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/xlang.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/basic-serialization.md index 4f8bce38722..228a810d536 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/xlang.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: 跨语言序列化 +title: 基础序列化 sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,9 +19,96 @@ license: | limitations under the License. --- +本页介绍 Python 跨语言模式快速入门。`pyfory.Fory()` 默认使用支持兼容 Schema 演进的跨语言模式；示例会显式设置 `xlang=True`，以清楚表达模式选择。 + +## 基础对象序列化 + +使用简单的 API 序列化和反序列化 Python 对象： + +```python +import pyfory + +fory = pyfory.Fory(xlang=True) + +# Serialize xlang-compatible values +data = fory.dumps({"name": "Alice", "age": 30, "scores": [95, 87, 92]}) + +# Deserialize back to Python object +obj = fory.loads(data) +print(obj) # {'name': 'Alice', 'age': 30, 'scores': [95, 87, 92]} +``` + +**注意**：`dumps()`/`loads()` 是 `serialize()`/`deserialize()` 的别名。两组 API 完全相同，可选择更直观的一组。 + +## 自定义类序列化 + +使用 dataclass 和类型注解构建稳定的跨语言载荷： + +```python +import pyfory +from dataclasses import dataclass +from typing import List, Dict + +@dataclass +class Person: + name: str + age: pyfory.Int32 + scores: List[pyfory.Int32] + metadata: Dict[str, str] + +fory = pyfory.Fory(xlang=True, ref=True) +fory.register(Person, name="example.Person") +person = Person("Bob", 25, [88, 92, 85], {"team": "engineering"}) +data = fory.serialize(person) +result = fory.deserialize(data) +print(result) # Person(name='Bob', age=25, ...) +``` + +## 引用跟踪与循环引用 + +载荷使用跨语言兼容类型时，可以安全处理重复引用： + +```python +import pyfory + +f = pyfory.Fory(xlang=True, ref=True) + +shared = ["shared"] +value = [shared, shared] + +data = f.serialize(value) +result = f.deserialize(data) +assert result[0] is result[1] +``` + +任意 Python 对象图、本地类、函数和方法应使用[原生序列化](native.md)。 + +## 性能技巧 + +1. **不需要时禁用 `ref=True`**：引用跟踪存在开销 +2. **使用 type_id 而非 name**：整数 ID 比字符串名称更快 +3. **复用 Fory 实例**：创建一次，多次使用 +4. **启用 Cython**：确保设置 `ENABLE_FORY_CYTHON_SERIALIZATION=1` + +```python +# Good: Reuse instance +fory = pyfory.Fory(xlang=True) +for obj in objects: + data = fory.dumps(obj) + +# Bad: Create new instance each time +for obj in objects: + fory = pyfory.Fory(xlang=True) # Wasteful! + data = fory.dumps(obj) +``` + +## 跨语言互操作 {#cross-language-interoperability} + +以下内容说明默认 xlang 格式的跨语言类型映射、类型标识和互操作要求。 + `pyfory` 支持跨语言对象图序列化，可以在 Python 中序列化数据，再由 Java、C++、Go、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala、Kotlin 或其他受支持语言进行反序列化。 -## 创建跨语言 Fory 实例 +### Xlang 配置 Python 默认使用支持兼容 Schema 演进的跨语言模式。跨语言示例会显式设置该模式： @@ -30,9 +117,9 @@ import pyfory fory = pyfory.Fory(xlang=True, ref=False, strict=True) ``` -## 跨语言示例 +### 跨语言示例 -### Python（序列化端） +#### Python（序列化端） ```python import pyfory @@ -53,7 +140,7 @@ binary_data = f.serialize(person) # binary_data can now be sent to Java, Go, etc. ``` -### Java（反序列化端） +#### Java（反序列化端） ```java import org.apache.fory.*; @@ -72,7 +159,7 @@ fory.register(Person.class, "example.Person"); Person person = (Person) fory.deserialize(binaryData); ``` -### Rust（反序列化端） +#### Rust（反序列化端） ```rust use fory::Fory; @@ -90,7 +177,7 @@ fory.register_by_name::("example.Person"); let person: Person = fory.deserialize(&binary_data)?; ``` -## 跨语言类型注解 +### 跨语言类型注解 使用 pyfory 类型注解显式指定跨语言类型映射： @@ -112,13 +199,13 @@ class TypedData: 嵌套集合注解是字段 Schema 的一部分。兼容模式读取会根据远程 Schema 元数据消费字节，并且只有在解码值能够安全满足本地 Schema 时才进行赋值。 -## 低精度类型 +### 低精度类型 `pyfory.Float16` 和 `pyfory.BFloat16` 是为跨语言低精度字段保留的注解标记，并非值包装类；标量值会反序列化为原生 Python `float`。 稠密低精度数组使用具有类列表序列行为的公共稠密包装器。通过 `pyfory.Float16Array.from_values([...])` 或 `pyfory.BFloat16Array.from_values([...])` 从 Python 数字值构造。仅在使用 `from_buffer(...)` 和 `to_buffer()` 处理已经需要的打包小端序 `uint16` 存储，并希望走原始缓冲区快速路径时采用该方式。 -## 类型映射 +### 类型映射 | Python 标记/载体 | Java | Rust | Go | | ---------------------- | -------------- | --------------- | --------------------- | @@ -135,7 +222,7 @@ class TypedData: | `list` | `List` | `Vec` | `[]T` | | `dict` | `Map` | `HashMap` | `map[K]V` | -### 列表与稠密数组 +#### 列表与稠密数组 Python `List[T]` 映射到 Fory `list`。仅当需要使用 `pyfory.Array[T]`、`pyfory.NDArray[T]` 或 `pyfory.PyArray[T]` 表示稠密一维 `array` Schema 时，才采用这些注解。 @@ -170,24 +257,24 @@ Python `List[T]` 映射到 Fory `list`。仅当需要使用 `pyfory.Array[T]` PyArrow 是独立的行式/列式格式能力，不是 `pyfory.PyArray` 载体。使用 `pyfory.format.from_arrow_schema(...)` 和 `pyfory.format.to_arrow_schema(...)` 在 PyArrow Schema 与 Fory Row Format Schema 之间转换。 -## 与 Python 原生模式的区别 +### 与 Python 原生模式的区别 二进制协议和 API 与 `pyfory` 的 Python 原生模式相似，但 Python 原生模式可以序列化任意 Python 对象，包括全局函数、本地函数、lambda、本地类，以及使用 `__getstate__/__reduce__/__reduce_ex__` 自定义序列化的类型；这些值在跨语言模式中**不允许使用**。 -## 另请参阅 +### 规范与参考 - [跨语言序列化规范](../../specification/xlang_serialization_spec.md) - [类型映射参考](../../specification/xlang_type_mapping.md) -- [Java 跨语言序列化指南](../java/xlang.md) -- [Rust 跨语言序列化指南](../rust/xlang.md) +- [Java 跨语言序列化指南](../java/basic-serialization.md#cross-language-interoperability) +- [Rust 跨语言序列化指南](../rust/basic-serialization.md#cross-language-interoperability) -## 相关主题 +### 相关指南 - [配置](configuration.md) - 跨语言模式设置 - [Schema 演进](schema-evolution.md) - 兼容模式 - [类型注册](type-registration.md) - 注册模式 -## 读取 Java 文件示例 +### 读取 Java 文件示例 ```python import pyfory @@ -209,7 +296,7 @@ print(f"Name: {person.name}, Age: {person.age}") # Output: Name: Alice, Age: 30 ``` -## 内置值 +### 内置值 ```python import pyfory @@ -228,7 +315,7 @@ new_map = fory.deserialize(data) print(new_map) ``` -## 自定义值 +### 自定义值 ```python from dataclasses import dataclass @@ -285,7 +372,7 @@ if __name__ == "__main__": print(f.deserialize(data)) ``` -## 共享引用与循环引用 +### 共享引用与循环引用 ```python from typing import Dict @@ -305,3 +392,10 @@ data = fory.serialize(obj) # bytes can be deserialized by other languages print(fory.deserialize(data)) ``` + +## 相关主题 + +- [配置](configuration.md) - Fory 参数 +- [类型注册](type-registration.md) - 注册模式 +- [原生序列化](native.md) - 函数与 lambda +- [带外序列化](out-of-band.md) - 缓冲区回调 API diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/configuration.md index 34385968d73..049656d6a98 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/configuration.md @@ -173,110 +173,10 @@ fory = pyfory.Fory( ## 安全 -应像对待不可信 pickle 字节一样对待来自不可信来源的原生模式字节。设置 `strict=False` 时，原生模式可以重建 Python 对象、导入模块、调用归约钩子以及重建动态类或函数。 - -### 生产配置 - -生产载荷应保持 `strict=True`，除非整个数据来源都可信，并由 `DeserializationPolicy` 负责其余信任决策： - -```python -import pyfory - -fory = pyfory.Fory( - xlang=True, - ref=False, - strict=True, - max_depth=50, - max_type_fields=512, - max_type_meta_bytes=4096, - max_schema_versions_per_type=10, - max_average_schema_versions_per_type=3, - max_graph_memory_bytes=128 * 1024 * 1024, -) - -fory.register(UserModel, name="example.User") -fory.register(OrderModel, name="example.Order") -``` - -仅对可信的仅限 Python 载荷使用动态原生模式反序列化（`strict=False`）： - -```python -import pyfory - -fory = pyfory.Fory( - xlang=False, - ref=True, - strict=False, - max_depth=100, -) -``` - -接收的远程元数据也受到限制： - -- `max_type_fields` 限制单个已接收结构体元数据正文可接受的字段数。 -- `max_type_meta_bytes` 限制单个已接收 TypeDef 正文可接受的编码正文字节数。 -- `max_schema_versions_per_type` 限制每个逻辑类型可接受的远程元数据版本。 -- `max_average_schema_versions_per_type` 限制所有已接受远程类型的平均值。 -- `max_graph_memory_bytes` 为单次根反序列化期间实例化的对象图内存设置近似门限。估算主要覆盖列表、元组、集合、字典、对象数组、结构体和 Python 对象。它会跳过字符串、二进制数据、原始标量和稠密原始类型数组等叶子值，因此实际进程内存可能高于该值。叶子值仍受字节可用性检查保护：如果未读取的输入没有足够字节，Fory 不会读取或创建该叶子值。所有根输入形式的默认值固定为 `128 MiB`。可信载荷确实需要更大或更小门限时，请设置正数字节值。 -- `max_unbacked_container_items` 限制单次根反序列化中重复读取正文没有消耗相应输入的集合元素和映射条目。默认值为 `8192`；零表示严格限制。 - -这些限制不会改变 `strict`、`policy`、动态加载、未知类处理或 Schema 演进语义。 - -### DeserializationPolicy - -必须使用 `strict=False` 时，请使用 `DeserializationPolicy` 限制反序列化期间接受的动态类型和钩子： - -```python -import pyfory -from pyfory import DeserializationPolicy - -dangerous_modules = {"subprocess", "os", "__builtin__"} - -class SafeDeserializationPolicy(DeserializationPolicy): - def validate_class(self, cls, is_local, **kwargs): - if cls.__module__ in dangerous_modules: - raise ValueError(f"Blocked dangerous class: {cls.__module__}.{cls.__name__}") - - def intercept_reduce_call(self, callable_obj, args, **kwargs): - if getattr(callable_obj, "__name__", "") == "Popen": - raise ValueError("Blocked attempt to invoke subprocess.Popen") - return None - - def intercept_setstate(self, obj, state, **kwargs): - if isinstance(state, dict) and "password" in state: - state["password"] = "***REDACTED***" - return None - -policy = SafeDeserializationPolicy() -fory = pyfory.Fory(xlang=False, ref=True, strict=False, policy=policy) -``` - -可用的策略钩子包括： - -引用验证钩子通过抛出异常拒绝输入；不拒绝时，保持反序列化后的引用不变。 - -| 钩子 | 说明 | -| -------------------------------------------- | -------------------------------------- | -| `validate_class(cls, is_local)` | 验证或阻止类类型 | -| `validate_module(module_name, is_local)` | 验证或阻止模块导入 | -| `validate_function(func, is_local)` | 验证或阻止函数引用 | -| `validate_method(method, is_local)` | 验证或阻止方法引用 | -| `intercept_reduce_call(callable_obj, args)` | 拦截 `__reduce__` 调用 | -| `inspect_reduced_object(obj)` | 检查或替换通过 `__reduce__` 创建的对象 | -| `intercept_setstate(obj, state)` | 在 `__setstate__` 前清理状态 | -| `authorize_instantiation(cls, args, kwargs)` | 控制类实例化 | - -### 安全检查清单 - -- 对不可信数据保持 `strict=True`。 -- 反序列化前注册所有预期的应用类型。 -- 使用 `DeserializationPolicy` 约束必须设置 `strict=False` 的场景。 -- 将 `max_depth` 保持在足以拒绝异常深度载荷的较低值。 -- 对多数输入，将 `max_graph_memory_bytes` 保持为固定的 `128 MiB` 默认值；对于合法集合/映射/结构体大小不同的可信工作负载，则设置显式正数门限。 -- 不要将跨语言/原生模式选择视为安全控制。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [Python 安全](security.md)。 ## 相关主题 -- [基础序列化](core-api.md) - 使用配置后的 Fory +- [基础序列化](basic-serialization.md) - 使用配置后的 Fory - [类型注册](type-registration.md) - 注册模式 - [原生序列化](native.md) - 仅限 Python 的对象序列化 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/core-api.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/core-api.md deleted file mode 100644 index 61ff964154e..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/core-api.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -title: 基础序列化 -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本页介绍 Python 跨语言模式快速入门。`pyfory.Fory()` 默认使用支持兼容 Schema 演进的跨语言模式；示例会显式设置 `xlang=True`，以清楚表达模式选择。 - -## 基础对象序列化 - -使用简单的 API 序列化和反序列化 Python 对象： - -```python -import pyfory - -fory = pyfory.Fory(xlang=True) - -# Serialize xlang-compatible values -data = fory.dumps({"name": "Alice", "age": 30, "scores": [95, 87, 92]}) - -# Deserialize back to Python object -obj = fory.loads(data) -print(obj) # {'name': 'Alice', 'age': 30, 'scores': [95, 87, 92]} -``` - -**注意**：`dumps()`/`loads()` 是 `serialize()`/`deserialize()` 的别名。两组 API 完全相同，可选择更直观的一组。 - -## 自定义类序列化 - -使用 dataclass 和类型注解构建稳定的跨语言载荷： - -```python -import pyfory -from dataclasses import dataclass -from typing import List, Dict - -@dataclass -class Person: - name: str - age: pyfory.Int32 - scores: List[pyfory.Int32] - metadata: Dict[str, str] - -fory = pyfory.Fory(xlang=True, ref=True) -fory.register(Person, name="example.Person") -person = Person("Bob", 25, [88, 92, 85], {"team": "engineering"}) -data = fory.serialize(person) -result = fory.deserialize(data) -print(result) # Person(name='Bob', age=25, ...) -``` - -## 引用跟踪与循环引用 - -载荷使用跨语言兼容类型时，可以安全处理重复引用： - -```python -import pyfory - -f = pyfory.Fory(xlang=True, ref=True) - -shared = ["shared"] -value = [shared, shared] - -data = f.serialize(value) -result = f.deserialize(data) -assert result[0] is result[1] -``` - -任意 Python 对象图、本地类、函数和方法应使用[原生序列化](native.md)。 - -## 性能技巧 - -1. **不需要时禁用 `ref=True`**：引用跟踪存在开销 -2. **使用 type_id 而非 name**：整数 ID 比字符串名称更快 -3. **复用 Fory 实例**：创建一次，多次使用 -4. **启用 Cython**：确保设置 `ENABLE_FORY_CYTHON_SERIALIZATION=1` - -```python -# Good: Reuse instance -fory = pyfory.Fory(xlang=True) -for obj in objects: - data = fory.dumps(obj) - -# Bad: Create new instance each time -for obj in objects: - fory = pyfory.Fory(xlang=True) # Wasteful! - data = fory.dumps(obj) -``` - -## 相关主题 - -- [配置](configuration.md) - Fory 参数 -- [类型注册](type-registration.md) - 注册模式 -- [原生序列化](native.md) - 函数与 lambda -- [带外序列化](out-of-band.md) - 缓冲区回调 API diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/custom-serializers.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/custom-serializers.md index 5265d069d0d..996e8caa761 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/custom-serializers.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/custom-serializers.md @@ -135,4 +135,4 @@ fory.register(MyClass, name="com.example.MyClass", serializer=MySerializer(fory. - [类型注册](type-registration.md) - 注册模式 - [配置](configuration.md) - Fory 参数 -- [跨语言序列化](xlang.md) - 跨语言模式的类型注册和 Schema 规则 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言模式的类型注册和 Schema 规则 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/index.md index ed60f7f10b1..a0d01576d96 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/index.md @@ -145,12 +145,11 @@ print(result) # Person(name='Alice', age=30) 仅限 Python 的流量使用原生模式。通过 `xlang=False` 选择原生模式；该模式负责函数、lambda、类、方法、`__reduce__`、`__getstate__` 以及 pickle 协议 5 带外缓冲区等 pickle/cloudpickle 风格行为。它针对 Python 类型系统优化，支持比跨语言模式更广泛的 Python 对象，因此适合替代 pickle 或 cloudpickle。兼容模式默认启用。仅当每个读取端和写入端都使用相同的 Python 类 Schema，并且希望获得更快速度和更小体积时，才设置 `compatible=False`。 -仅限 Python 的序列化详情参见[原生序列化](native.md)，Python 跨语言注册与互操作规则参见[跨语言序列化](xlang.md)。 +仅限 Python 的序列化详情参见[原生序列化](native.md)，Python 跨语言注册与互操作规则参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ## 后续阅读 -- [基础序列化](core-api.md) - 基本使用模式 -- [跨语言序列化](xlang.md) - 跨语言模式 +- [基础序列化](basic-serialization.md) - 基本使用模式 - [原生序列化](native.md) - 仅限 Python 的序列化 - [配置](configuration.md) - Fory 参数、模式与安全 - [类型注册](type-registration.md) - 用户定义类型注册 @@ -165,3 +164,5 @@ print(result) # Person(name='Alice', age=30) - **PyPI**: https://pypi.org/project/pyfory/ - **问题跟踪**：https://github.com/apache/fory/issues - **Slack**: https://join.slack.com/t/fory-project/shared_invite/zt-36g0qouzm-kcQSvV_dtfbtBKHRwT5gsw + +解码来自应用信任边界之外的字节之前，请阅读 [Python 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/native.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/native.md index dd72affb68e..817e3b53c75 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/native.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/native.md @@ -21,7 +21,7 @@ license: | Python 原生序列化是通过 `xlang=False` 选择的仅限 Python 编码模式。当所有写入端和读取端都是 Python，并且载荷应遵循 Python 对象模型而非可移植的跨语言类型系统时，请使用该模式。 -如果字节必须由 Java、C++、Go、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala、Kotlin 或其他非 Python Fory 实现读取，请使用 Python 默认的[跨语言序列化](xlang.md)模式。 +如果字节必须由 Java、C++、Go、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala、Kotlin 或其他非 Python Fory 实现读取，请使用 Python 默认的[跨语言序列化](basic-serialization.md#cross-language-interoperability)模式。 ## 何时使用原生序列化 @@ -167,7 +167,7 @@ print(f"Pickle: {timeit.timeit(lambda: pickle.dumps(obj), number=1000):.3f}s") ## 相关主题 -- [跨语言序列化](xlang.md) - 跨语言 Python 载荷 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言 Python 载荷 - [配置](configuration.md) - Python `Fory` 选项 - [带外序列化](out-of-band.md) - 零拷贝缓冲区支持 -- [对象序列化安全](../security.md) - 反序列化策略 +- [Python 安全](security.md) - 反序列化策略 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/numpy-integration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/numpy-integration.md index 4f87d7df9b0..42c22ff1a21 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/numpy-integration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/numpy-integration.md @@ -100,4 +100,4 @@ assert np.array_equal(array, result) ## 相关主题 - [带外序列化](out-of-band.md) - 零拷贝缓冲区 -- [基础序列化](core-api.md) - 标准用法 +- [基础序列化](basic-serialization.md) - 标准用法 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/out-of-band.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/out-of-band.md index f98984aaaa2..0e382e07401 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/out-of-band.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/out-of-band.md @@ -175,5 +175,5 @@ for buffer_obj in buffer_objects: ## 相关主题 - [NumPy 集成](numpy-integration.md) - NumPy 数组序列化 -- [基础序列化](core-api.md) - 标准序列化 +- [基础序列化](basic-serialization.md) - 标准序列化 - [配置](configuration.md) - Fory 参数 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/schema-evolution.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/schema-evolution.md index 2ef0394a20e..3fae39758bd 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/schema-evolution.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/schema-evolution.md @@ -119,5 +119,5 @@ class SameSchemaMessage: ## 相关主题 - [配置](configuration.md) - 兼容模式设置 -- [跨语言序列化](xlang.md) - 跨语言 Schema 演进 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言 Schema 演进 - [类型注册](type-registration.md) - 注册模式 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/schema-metadata.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/schema-metadata.md index 4850c2e6d5e..e786c9d9a5c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/schema-metadata.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/schema-metadata.md @@ -513,6 +513,6 @@ class User: ## 相关主题 -- [基础序列化](core-api.md) - Fory 序列化入门 +- [基础序列化](basic-serialization.md) - Fory 序列化入门 - [Schema 演进](schema-evolution.md) - 兼容模式与 Schema 演进 -- [跨语言序列化](xlang.md) - 与 Java、Rust、C++、Go 互操作 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 与 Java、Rust、C++、Go 互操作 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/security.md new file mode 100644 index 00000000000..a88fc17f5fe --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/security.md @@ -0,0 +1,144 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 Python 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +应像对待不可信 pickle 字节一样对待来自不可信来源的原生模式字节。设置 `strict=False` 时，原生模式可以重建 Python 对象、导入模块、调用归约钩子以及重建动态类或函数。 + +### 生产配置 + +生产载荷应保持 `strict=True`，除非整个数据来源都可信，并由 `DeserializationPolicy` 负责其余信任决策： + +```python +import pyfory + +fory = pyfory.Fory( + xlang=True, + ref=False, + strict=True, + max_depth=50, + max_type_fields=512, + max_type_meta_bytes=4096, + max_schema_versions_per_type=10, + max_average_schema_versions_per_type=3, + max_graph_memory_bytes=128 * 1024 * 1024, +) + +fory.register(UserModel, name="example.User") +fory.register(OrderModel, name="example.Order") +``` + +仅对可信的仅限 Python 载荷使用动态原生模式反序列化（`strict=False`）： + +```python +import pyfory + +fory = pyfory.Fory( + xlang=False, + ref=True, + strict=False, + max_depth=100, +) +``` + +接收的远程元数据也受到限制： + +- `max_type_fields` 限制单个已接收结构体元数据正文可接受的字段数。 +- `max_type_meta_bytes` 限制单个已接收 TypeDef 正文可接受的编码正文字节数。 +- `max_schema_versions_per_type` 限制每个逻辑类型可接受的远程元数据版本。 +- `max_average_schema_versions_per_type` 限制所有已接受远程类型的平均值。 +- `max_graph_memory_bytes` 为单次根反序列化期间实例化的对象图内存设置近似门限。估算主要覆盖列表、元组、集合、字典、对象数组、结构体和 Python 对象。它会跳过字符串、二进制数据、原始标量和稠密原始类型数组等叶子值，因此实际进程内存可能高于该值。叶子值仍受字节可用性检查保护：如果未读取的输入没有足够字节，Fory 不会读取或创建该叶子值。所有根输入形式的默认值固定为 `128 MiB`。可信载荷确实需要更大或更小门限时，请设置正数字节值。 +- `max_unbacked_container_items` 限制单次根反序列化中重复读取正文没有消耗相应输入的集合元素和映射条目。默认值为 `8192`；零表示严格限制。 + +这些限制不会改变 `strict`、`policy`、动态加载、未知类处理或 Schema 演进语义。 + +### DeserializationPolicy + +必须使用 `strict=False` 时，请使用 `DeserializationPolicy` 限制反序列化期间接受的动态类型和钩子： + +```python +import pyfory +from pyfory import DeserializationPolicy + +dangerous_modules = {"subprocess", "os", "__builtin__"} + +class SafeDeserializationPolicy(DeserializationPolicy): + def validate_class(self, cls, is_local, **kwargs): + if cls.__module__ in dangerous_modules: + raise ValueError(f"Blocked dangerous class: {cls.__module__}.{cls.__name__}") + + def intercept_reduce_call(self, callable_obj, args, **kwargs): + if getattr(callable_obj, "__name__", "") == "Popen": + raise ValueError("Blocked attempt to invoke subprocess.Popen") + return None + + def intercept_setstate(self, obj, state, **kwargs): + if isinstance(state, dict) and "password" in state: + state["password"] = "***REDACTED***" + return None + +policy = SafeDeserializationPolicy() +fory = pyfory.Fory(xlang=False, ref=True, strict=False, policy=policy) +``` + +可用的策略钩子包括： + +引用验证钩子通过抛出异常拒绝输入；不拒绝时，保持反序列化后的引用不变。 + +| 钩子 | 说明 | +| -------------------------------------------- | -------------------------------------- | +| `validate_class(cls, is_local)` | 验证或阻止类类型 | +| `validate_module(module_name, is_local)` | 验证或阻止模块导入 | +| `validate_function(func, is_local)` | 验证或阻止函数引用 | +| `validate_method(method, is_local)` | 验证或阻止方法引用 | +| `intercept_reduce_call(callable_obj, args)` | 拦截 `__reduce__` 调用 | +| `inspect_reduced_object(obj)` | 检查或替换通过 `__reduce__` 创建的对象 | +| `intercept_setstate(obj, state)` | 在 `__setstate__` 前清理状态 | +| `authorize_instantiation(cls, args, kwargs)` | 控制类实例化 | + +### 安全检查清单 + +- 对不可信数据保持 `strict=True`。 +- 反序列化前注册所有预期的应用类型。 +- 使用 `DeserializationPolicy` 约束必须设置 `strict=False` 的场景。 +- 将 `max_depth` 保持在足以拒绝异常深度载荷的较低值。 +- 对多数输入，将 `max_graph_memory_bytes` 保持为固定的 `128 MiB` 默认值；对于合法集合/映射/结构体大小不同的可信工作负载，则设置显式正数门限。 +- 不要将跨语言/原生模式选择视为安全控制。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[类型注册](type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/troubleshooting.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/troubleshooting.md index 746cb5dcdd9..eb0f2e21184 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/troubleshooting.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/troubleshooting.md @@ -191,4 +191,4 @@ ruff check --fix . - [配置](configuration.md) - Fory 参数 - [类型注册](type-registration.md) - 注册最佳实践 -- [对象序列化安全](../security.md) - 安全配置 +- [Python 安全](security.md) - 安全配置 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/type-registration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/type-registration.md index 512fc751058..4335e27e4bd 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/type-registration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/python/type-registration.md @@ -19,7 +19,8 @@ license: | limitations under the License. --- -本页介绍 Python 类型注册 API。严格模式策略、最大深度限制和可信数据指南参见[对象序列化安全](../security.md)。 +本页介绍 Python 类型注册 API。严格模式策略、最大深度限制和可信数据指南参见 +[Python 安全](security.md)。 ## 类型注册 @@ -78,5 +79,5 @@ for model_class in [User, Order, Product, Invoice]: ## 相关主题 - [配置](configuration.md) - Fory 参数 -- [对象序列化安全](../security.md) - 严格模式、反序列化策略和最大读取深度 +- [Python 安全](security.md) - 严格模式、反序列化策略和最大读取深度 - [自定义序列化器](custom-serializers.md) - 自定义序列化 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/runtimes.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/runtimes.md deleted file mode 100644 index bdb850cfa79..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/runtimes.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -title: 对象序列化运行时 -sidebar_position: 3 -id: runtimes -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -请选择应用所使用 API 对应的运行时。运行时文档仅涵盖二进制对象序列化；Row Format、 -Fory JSON、编译器和 gRPC 页面仍归属于各自的产品功能。 - -| 运行时 | 模式 | 文档 | -| --------------------- | ----------------- | ----------------------------------------------------- | -| Java | xlang 和原生 | [Java 运行时](./java/index.md) | -| Python | xlang 和原生 | [Python 运行时](./python/index.md) | -| C++ | xlang 和原生 | [C++ 运行时](./cpp/index.md) | -| Go | xlang 和原生 | [Go 运行时](./go/index.md) | -| Rust | xlang 和原生 | [Rust 运行时](./rust/index.md) | -| JavaScript/TypeScript | xlang | [JavaScript/TypeScript 运行时](./javascript/index.md) | -| C# | xlang | [C# 运行时](./csharp/index.md) | -| Swift | xlang | [Swift 运行时](./swift/index.md) | -| Dart | xlang | [Dart 运行时](./dart/index.md) | -| Scala | xlang 和 JVM 原生 | [Scala 运行时](./scala/index.md) | -| Kotlin | xlang 和 JVM 原生 | [Kotlin 运行时](./kotlin/index.md) | diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/basic-serialization.md similarity index 55% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/xlang.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/basic-serialization.md index efb76ffa7d6..df6be5f49ff 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/xlang.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: 跨语言序列化 +title: 基础序列化 sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,9 +19,197 @@ license: | limitations under the License. --- +本文介绍 Fory Rust 默认 xlang 模式下的基本对象图序列化和支持的类型。 + +## 对象图序列化 + +Apache Fory™ 可自动序列化复杂对象图，并保留对象之间的结构和关系。`#[derive(ForyStruct)]` 宏在编译期生成高效序列化代码，从而消除反射开销。 + +**主要能力：** + +- 任意深度的嵌套结构体序列化 +- 集合类型（Vec、HashMap、HashSet、BTreeMap） +- 使用 `Option` 的可选字段 +- 自动处理原始类型和字符串 +- 使用变长整数的高效二进制编码 + +```rust +use fory::{Fory, Error}; +use fory::ForyStruct; +use std::collections::HashMap; + +#[derive(ForyStruct, Debug, PartialEq)] +struct Person { + name: String, + age: i32, + address: Address, + hobbies: Vec, + metadata: HashMap, +} + +#[derive(ForyStruct, Debug, PartialEq)] +struct Address { + street: String, + city: String, + country: String, +} + +let mut fory = Fory::builder().xlang(true).build(); +fory.register_by_name::
("example.Address").unwrap(); +fory.register_by_name::("example.Person").unwrap(); + +let person = Person { + name: "John Doe".to_string(), + age: 30, + address: Address { + street: "123 Main St".to_string(), + city: "New York".to_string(), + country: "USA".to_string(), + }, + hobbies: vec!["reading".to_string(), "coding".to_string()], + metadata: HashMap::from([ + ("role".to_string(), "developer".to_string()), + ]), +}; + +let bytes = fory.serialize(&person).unwrap(); +let decoded: Person = fory.deserialize(&bytes)?; +assert_eq!(person, decoded); +``` + +## 支持的类型 + +### 原始类型 + +| Rust 类型 | 说明 | +| ------------------------- | ------------- | +| `bool` | 布尔值 | +| `i8`, `i16`, `i32`, `i64` | 有符号整数 | +| `f32`, `f64` | 浮点数 | +| `BFloat16` | 16 位脑浮点数 | +| `String` | UTF-8 字符串 | + +### 集合 + +| Rust 类型 | 说明 | +| ---------------- | -------- | +| `Vec` | 动态数组 | +| `VecDeque` | 双端队列 | +| `LinkedList` | 双向链表 | +| `HashMap` | 哈希 map | +| `BTreeMap` | 有序 map | +| `HashSet` | 哈希 set | +| `BTreeSet` | 有序 set | +| `BinaryHeap` | 二叉堆 | +| `Option` | 可选值 | + +`Vec` 是 Schema 为 `array` 时的稠密载体。 + +### 智能指针 + +| Rust 类型 | 说明 | +| ------------ | -------------------------------------- | +| `Box` | 堆分配 | +| `Rc` | 引用计数（跟踪共享引用） | +| `Arc` | 线程安全的引用计数（跟踪共享引用） | +| `RcWeak` | 指向 `Rc` 的弱引用（打破循环引用） | +| `ArcWeak` | 指向 `Arc` 的弱引用（打破循环引用） | +| `RefCell` | 内部可变性（运行时借用检查） | +| `Mutex` | 线程安全的内部可变性 | + +### 日期和时间 + +| Rust 类型 | 说明 | +| ----------- | --------------------------------------- | +| `Date` | 不含时区的日期，存储为相对 epoch 的天数 | +| `Timestamp` | 时间点，存储为 epoch 秒数和纳秒 | +| `Duration` | 有符号时间长度，存储为秒数和规范化纳秒 | + +内置载体提供无依赖的构造函数、访问器、转换和经过检查的算术运算： + +```rust +use fory::{Date, Duration, Timestamp}; + +let date = Date::from_epoch_days(19_782); +assert_eq!(date.checked_add_days(1)?.epoch_days(), 19_783); + +let timestamp = Timestamp::from_epoch_millis(-1); +assert_eq!(timestamp.to_epoch_millis()?, -1); + +let duration = Duration::from_parts(1, 1_500_000_000)?; +assert_eq!(duration.to_millis()?, 2_500); +let later = timestamp.checked_add_duration(duration)?; +``` + +Rust 启用 `chrono::NaiveDate`、`chrono::NaiveDateTime` 和 `chrono::Duration` 所需的 `chrono` 功能时，即可支持这些类型： + +```toml +[dependencies] +fory = { version = "1.5.0", features = ["chrono"] } +``` + +### 自定义类型 + +对象图序列化使用 `#[derive(ForyStruct)]`。独立的 [Rust 行格式指南](../../row-format/rust.md)记录 `#[derive(ForyRow)]` 及其支持的类型集合。 + +## 序列化 API + +```rust +use fory::{Fory, Reader}; + +let mut fory = Fory::builder().xlang(true).build(); +fory.register::(1)?; + +let obj = MyStruct { /* ... */ }; + +// Basic serialize/deserialize +let bytes = fory.serialize(&obj)?; +let decoded: MyStruct = fory.deserialize(&bytes)?; + +// Serialize to existing buffer +let mut buf: Vec = vec![]; +fory.serialize_to(&mut buf, &obj)?; + +// Deserialize from reader +let mut reader = Reader::new(&buf); +let decoded: MyStruct = fory.deserialize_from(&mut reader)?; +``` + +Rust 值类型使用外部结构化序列化器或自定义序列化器时，请在根上显式选择： + +```rust +let bytes = fory.serialize_with::(&user)?; +let decoded: third_party::User = + fory.deserialize_with::(&bytes)?; +``` + +载体序列化器为根容器组合相同的选择： + +```rust +use fory::VecSerializer; + +let bytes = + fory.serialize_with::>(&users)?; +let decoded: Vec = + fory.deserialize_with::>(&bytes)?; +``` + +字段注解、所有支持的载体和注册方式参见[外部类型序列化](external-types.md)。 + +## 性能技巧 + +- **预分配缓冲区**：尽量减少序列化期间的内存分配 +- **紧凑编码**：使用变长编码提高空间效率 +- **小端序**：针对现代 CPU 架构优化 +- **引用去重**：共享对象只序列化一次 + +## 跨语言互操作 {#cross-language-interoperability} + +以下内容说明默认 xlang 格式的跨语言类型映射、类型标识和互操作要求。 + Apache Fory™ 支持 Java、Python、C++、Go、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala 和 Kotlin 之间的无缝数据交换。 -## 创建跨语言 Fory 实例 +### Xlang 配置 Rust 默认使用带兼容 Schema 演进的跨语言模式。跨语言示例中应显式设置模式： @@ -38,9 +226,9 @@ fory.register::(100)?; // fory.register_by_name::("com.example.MyStruct")?; ``` -## 跨语言类型注册 +### 跨语言类型注册 -### 按 ID 注册 +#### 按 ID 注册 为获得快速、紧凑的序列化，请在各语言中使用一致的 ID： @@ -50,7 +238,7 @@ let mut fory = Fory::builder().xlang(true).build(); fory.register::(100)?; // Same ID in Java, Python, etc. ``` -### 按名称注册 +#### 按名称注册 用于更灵活的类型命名： @@ -58,9 +246,9 @@ fory.register::(100)?; // Same ID in Java, Python, etc. fory.register_by_name::("com.example.User")?; ``` -## 跨语言示例 +### 跨语言示例 -### Rust（序列化器） +#### Rust（序列化器） ```rust use fory::Fory; @@ -85,7 +273,7 @@ let bytes = fory.serialize(&person)?; // bytes can be deserialized by Java, Python, etc. ``` -### 第三方 Rust 类型 +#### 第三方 Rust 类型 外部结构化序列化器为第三方 Rust 类型提供与等价本地派生相同的跨语言 Schema： @@ -114,11 +302,11 @@ let bytes = 只接受可用跨语言模式表示的 Schema。带多个元组字段或命名字段的原生 Rust 枚举变体可使用 `xlang(false)`，但其序列化器注册会在跨语言模式中被拒绝。参见[外部类型序列化](external-types.md)。 -### 动态 Rust 载体 +#### 动态 Rust 载体 当每个选定具体目标都有兼容跨语言模式的结构化或 EXT 标识时，可以在跨语言模式中使用 `Box`、`Rc`、`Arc` 和应用 `dyn Trait` 载体。Fory 写入具体的已注册目标标识；Rust 特征或擦除载体标识不会出现在编码格式中。 -### Java（反序列化器） +#### Java（反序列化器） ```java import org.apache.fory.*; @@ -139,7 +327,7 @@ fory.register(Person.class, 100); // Same ID as Rust Person person = (Person) fory.deserialize(bytesFromRust); ``` -### Python（反序列化器） +#### Python（反序列化器） ```python import pyfory @@ -156,11 +344,11 @@ fory.register_type(Person, type_id=100) # Same ID as Rust person = fory.deserialize(bytes_from_rust) ``` -## 类型映射 +### 类型映射 完整的跨语言类型映射参见 [xlang_type_mapping.md](../../specification/xlang_type_mapping.md)。 -### 常见类型映射 +#### 常见类型映射 | Rust | Java | Python | | --------------- | -------------- | --------------- | @@ -179,7 +367,7 @@ person = fory.deserialize(bytes_from_rust) | `HashMap` | `Map` | `Dict[K,V]` | | `Option` | nullable `T` | `Optional[T]` | -### 列表与稠密数组 +#### 列表与稠密数组 对于手写结构体，Rust `Vec` 默认映射到 Fory `list`。Schema 为稠密 `array` 时，请使用显式数组字段属性。 @@ -200,28 +388,28 @@ person = fory.deserialize(bytes_from_rust) | `array` | `#[fory(array)] Vec` | | `array` | `#[fory(array)] Vec` | -## 最佳实践 +### 互操作最佳实践 1. 在所有语言中**使用一致的类型 ID** 2. 为 Schema 演进**保持兼容模式** 3. 在序列化前**注册所有类型** 4. 开发期间**测试跨语言兼容性** -## 另请参阅 +### 规范与参考 - [Xlang 序列化规范](../../specification/xlang_serialization_spec.md) - [类型映射参考](../../specification/xlang_type_mapping.md) -- [Java Xlang 序列化指南](../java/xlang.md) -- [Python Xlang 序列化指南](../python/xlang.md) +- [Java Xlang 序列化指南](../java/basic-serialization.md#cross-language-interoperability) +- [Python Xlang 序列化指南](../python/basic-serialization.md#cross-language-interoperability) -## 相关主题 +### 相关指南 - [配置](configuration.md) - xlang 模式配置 - [Schema 演进](schema-evolution.md) - 兼容模式 - [类型注册](type-registration.md) - 注册方法 - [外部类型序列化](external-types.md) - xlang 模式中的第三方值 -## 内置值 +### 内置值 ```rust use fory::Fory; @@ -234,7 +422,7 @@ fn run() { } ``` -## 自定义值 +### 自定义值 ```rust use chrono::{NaiveDate, NaiveDateTime}; @@ -298,6 +486,14 @@ fn complex_struct() { } ``` -## 共享引用与循环引用 +### 共享引用与循环引用 由于所有权限制，Rust 无法实现循环引用。 + +## 相关主题 + +- [类型注册](type-registration.md) - 注册类型 +- [引用](references.md) - 共享引用与循环引用 +- [自定义序列化器](custom-serializers.md) - 自定义序列化 +- [外部类型序列化](external-types.md) - 第三方值和载体根值 +- [Row Format](../../row-format/rust.md) - Standard Row Format 和零拷贝借用视图 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/configuration.md index 2765222dc02..05fcf6ac75d 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/configuration.md @@ -181,17 +181,10 @@ let fory = Fory::builder() ## 安全 -安全相关配置： - -- 反序列化不可信载荷前，注册应用结构体和特征对象实现。 -- 使用 `max_dyn_depth(...)` 拒绝意外过深的动态对象图。 -- 对大多数输入，将 `max_graph_memory_bytes(...)` 保持为固定默认值 `128 MiB`；只有可信工作负载具有不同的合理集合、映射或结构体大小时才设置正字节限制。 -- 除非可信紧凑编解码器需要更大的根限额，否则将 `max_unbacked_container_items(...)` 保持为 `8192`。零会拒绝每个无输入支撑的条目。 -- 除非数据确定无恶意且可信对等端会发送更大的元数据或大量 Schema 版本，否则请保留远端 Schema 元数据限制的默认值。 -- 对不可信输入，优先使用具体类型字段，而非 `dyn Any` 或宽泛的特征对象字段。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [Rust 安全](security.md)。 ## 相关主题 -- [基本序列化](core-api.md) - 使用已配置的 Fory +- [基本序列化](basic-serialization.md) - 使用已配置的 Fory - [Schema 演进](schema-evolution.md) - 兼容模式详情 -- [跨语言序列化](xlang.md) - xlang 模式 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - xlang 模式 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/core-api.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/core-api.md deleted file mode 100644 index c277418067c..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/core-api.md +++ /dev/null @@ -1,212 +0,0 @@ ---- -title: 基本序列化 -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本文介绍基本对象图序列化和支持的类型。 - -## 对象图序列化 - -Apache Fory™ 可自动序列化复杂对象图，并保留对象之间的结构和关系。`#[derive(ForyStruct)]` 宏在编译期生成高效序列化代码，从而消除反射开销。 - -**主要能力：** - -- 任意深度的嵌套结构体序列化 -- 集合类型（Vec、HashMap、HashSet、BTreeMap） -- 使用 `Option` 的可选字段 -- 自动处理原始类型和字符串 -- 使用变长整数的高效二进制编码 - -```rust -use fory::{Fory, Error}; -use fory::ForyStruct; -use std::collections::HashMap; - -#[derive(ForyStruct, Debug, PartialEq)] -struct Person { - name: String, - age: i32, - address: Address, - hobbies: Vec, - metadata: HashMap, -} - -#[derive(ForyStruct, Debug, PartialEq)] -struct Address { - street: String, - city: String, - country: String, -} - -let mut fory = Fory::builder().xlang(true).build(); -fory.register_by_name::
("example.Address").unwrap(); -fory.register_by_name::("example.Person").unwrap(); - -let person = Person { - name: "John Doe".to_string(), - age: 30, - address: Address { - street: "123 Main St".to_string(), - city: "New York".to_string(), - country: "USA".to_string(), - }, - hobbies: vec!["reading".to_string(), "coding".to_string()], - metadata: HashMap::from([ - ("role".to_string(), "developer".to_string()), - ]), -}; - -let bytes = fory.serialize(&person).unwrap(); -let decoded: Person = fory.deserialize(&bytes)?; -assert_eq!(person, decoded); -``` - -## 支持的类型 - -### 原始类型 - -| Rust 类型 | 说明 | -| ------------------------- | ------------- | -| `bool` | 布尔值 | -| `i8`, `i16`, `i32`, `i64` | 有符号整数 | -| `f32`, `f64` | 浮点数 | -| `BFloat16` | 16 位脑浮点数 | -| `String` | UTF-8 字符串 | - -### 集合 - -| Rust 类型 | 说明 | -| ---------------- | -------- | -| `Vec` | 动态数组 | -| `VecDeque` | 双端队列 | -| `LinkedList` | 双向链表 | -| `HashMap` | 哈希 map | -| `BTreeMap` | 有序 map | -| `HashSet` | 哈希 set | -| `BTreeSet` | 有序 set | -| `BinaryHeap` | 二叉堆 | -| `Option` | 可选值 | - -`Vec` 是 Schema 为 `array` 时的稠密载体。 - -### 智能指针 - -| Rust 类型 | 说明 | -| ------------ | -------------------------------------- | -| `Box` | 堆分配 | -| `Rc` | 引用计数（跟踪共享引用） | -| `Arc` | 线程安全的引用计数（跟踪共享引用） | -| `RcWeak` | 指向 `Rc` 的弱引用（打破循环引用） | -| `ArcWeak` | 指向 `Arc` 的弱引用（打破循环引用） | -| `RefCell` | 内部可变性（运行时借用检查） | -| `Mutex` | 线程安全的内部可变性 | - -### 日期和时间 - -| Rust 类型 | 说明 | -| ----------- | --------------------------------------- | -| `Date` | 不含时区的日期，存储为相对 epoch 的天数 | -| `Timestamp` | 时间点，存储为 epoch 秒数和纳秒 | -| `Duration` | 有符号时间长度，存储为秒数和规范化纳秒 | - -内置载体提供无依赖的构造函数、访问器、转换和经过检查的算术运算： - -```rust -use fory::{Date, Duration, Timestamp}; - -let date = Date::from_epoch_days(19_782); -assert_eq!(date.checked_add_days(1)?.epoch_days(), 19_783); - -let timestamp = Timestamp::from_epoch_millis(-1); -assert_eq!(timestamp.to_epoch_millis()?, -1); - -let duration = Duration::from_parts(1, 1_500_000_000)?; -assert_eq!(duration.to_millis()?, 2_500); -let later = timestamp.checked_add_duration(duration)?; -``` - -Rust 启用 `chrono::NaiveDate`、`chrono::NaiveDateTime` 和 `chrono::Duration` 所需的 `chrono` 功能时，即可支持这些类型： - -```toml -[dependencies] -fory = { version = "1.5.0", features = ["chrono"] } -``` - -### 自定义类型 - -对象图序列化使用 `#[derive(ForyStruct)]`。独立的 [Rust 行格式指南](../../row-format/rust.md)记录 `#[derive(ForyRow)]` 及其支持的类型集合。 - -## 序列化 API - -```rust -use fory::{Fory, Reader}; - -let mut fory = Fory::builder().xlang(true).build(); -fory.register::(1)?; - -let obj = MyStruct { /* ... */ }; - -// Basic serialize/deserialize -let bytes = fory.serialize(&obj)?; -let decoded: MyStruct = fory.deserialize(&bytes)?; - -// Serialize to existing buffer -let mut buf: Vec = vec![]; -fory.serialize_to(&mut buf, &obj)?; - -// Deserialize from reader -let mut reader = Reader::new(&buf); -let decoded: MyStruct = fory.deserialize_from(&mut reader)?; -``` - -Rust 值类型使用外部结构化序列化器或自定义序列化器时，请在根上显式选择： - -```rust -let bytes = fory.serialize_with::(&user)?; -let decoded: third_party::User = - fory.deserialize_with::(&bytes)?; -``` - -载体序列化器为根容器组合相同的选择： - -```rust -use fory::VecSerializer; - -let bytes = - fory.serialize_with::>(&users)?; -let decoded: Vec = - fory.deserialize_with::>(&bytes)?; -``` - -字段注解、所有支持的载体和注册方式参见[外部类型序列化](external-types.md)。 - -## 性能技巧 - -- **预分配缓冲区**：尽量减少序列化期间的内存分配 -- **紧凑编码**：使用变长编码提高空间效率 -- **小端序**：针对现代 CPU 架构优化 -- **引用去重**：共享对象只序列化一次 - -## 相关主题 - -- [类型注册](type-registration.md) - 注册类型 -- [引用](references.md) - 共享引用与循环引用 -- [自定义序列化器](custom-serializers.md) - 自定义序列化 -- [外部类型序列化](external-types.md) - 第三方值和载体根值 -- [Row Format](../../row-format/rust.md) - Standard Row Format 和零拷贝借用视图 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/index.md index 80fc51add77..539923f3fb3 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/index.md @@ -97,7 +97,7 @@ fn main() -> Result<(), Error> { 仅限 Rust 的通信应使用原生模式。通过 `.xlang(false)` 选择原生模式，它会让 Rust 对象序列化保持 Rust 原生形式。该模式支持仅限原生模式的具体目标，以及没有跨语言表示的数据枚举形态。当每个选定具体目标都兼容跨语言模式时，也可以在跨语言模式中使用动态 `Any`、应用特征和共享引用载体。兼容模式默认启用。只有每个读取端和写入端都使用相同 Rust Schema，并且希望获得更快序列化和更小体积时，才设置 `.compatible(false)`。 -Rust 跨语言注册和互操作规则参见[跨语言序列化](xlang.md)，仅限 Rust 的载荷参见[原生序列化](native.md)。 +Rust 跨语言注册和互操作规则参见[跨语言序列化](basic-serialization.md#cross-language-interoperability)，仅限 Rust 的载荷参见[原生序列化](native.md)。 ## 线程安全 @@ -154,8 +154,7 @@ fn main() -> Result<(), Error> { ## 后续步骤 - [配置](configuration.md) - Fory 构建器选项和模式 -- [基本序列化](core-api.md) - 对象图序列化 -- [跨语言序列化](xlang.md) - 跨语言模式 +- [基本序列化](basic-serialization.md) - 对象图序列化 - [原生序列化](native.md) - 仅限 Rust 的序列化 - [引用](references.md) - 共享引用与循环引用 - [多态](polymorphism.md) - 特征对象序列化 @@ -163,3 +162,5 @@ fn main() -> Result<(), Error> { - [外部类型序列化](external-types.md) - 外部结构化序列化器、自定义序列化器和载体组合 - [Row Format](../../row-format/rust.md) - 支持借用视图的标准 Row Format - [gRPC 支持](../../grpc/rust.md) - 通过 tonic 传输 Fory 载荷 + +解码来自应用信任边界之外的字节之前，请阅读 [Rust 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/native.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/native.md index ce392c15cb0..a9eea80c249 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/native.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/native.md @@ -21,7 +21,7 @@ license: | Rust 原生序列化是通过 `.xlang(false)` 选择、仅限 Rust 的编码模式。当每个写入端和读取端都是 Rust，并且载荷应保留 Rust 对象图行为而非可移植的跨语言类型系统时，请使用该模式。 -如果字节需要由 Java、Python、C++、Go、JavaScript/TypeScript、C#、Swift、Dart、Scala、Kotlin 或其他非 Rust Fory 实现读取，请使用 Rust 默认模式[跨语言序列化](xlang.md)。 +如果字节需要由 Java、Python、C++、Go、JavaScript/TypeScript、C#、Swift、Dart、Scala、Kotlin 或其他非 Rust Fory 实现读取，请使用 Rust 默认模式[跨语言序列化](basic-serialization.md#cross-language-interoperability)。 ## 何时使用原生序列化 @@ -99,7 +99,7 @@ fory.register::(100)?; `Arc` 进行运行时类型分派。请将容器包装在已注册的结构体、枚举或 union 中；当不透明的 EXT/NAMED_EXT 表示合适时，也可以注册精确目标类型的自定义序列化器。 - 日期和时间载体，包括可选的 `chrono` 支持。 -相关示例参见[基本序列化](core-api.md)、[引用](references.md)和[特征对象序列化](polymorphism.md)。 +相关示例参见[基本序列化](basic-serialization.md)、[引用](references.md)和[特征对象序列化](polymorphism.md)。 ## 共享引用与循环引用 @@ -234,9 +234,9 @@ fory.register::(101)?; ## 相关主题 -- [跨语言序列化](xlang.md) - 跨语言 Rust 载荷 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言 Rust 载荷 - [配置](configuration.md) - 构建器选项 -- [基本序列化](core-api.md) - 对象图序列化 +- [基本序列化](basic-serialization.md) - 对象图序列化 - [共享引用与循环引用](references.md) - `Rc`、`Arc` 和弱指针 - [特征对象序列化](polymorphism.md) - 特征对象和动态分派 - [Schema 演进](schema-evolution.md) - 兼容模式 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/references.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/references.md index 317f24aeff0..2936588b8cc 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/references.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/references.md @@ -208,6 +208,6 @@ for child in &decoded.lock().unwrap().children { ## 相关主题 -- [基本序列化](core-api.md) - 支持的类型 +- [基本序列化](basic-serialization.md) - 支持的类型 - [多态](polymorphism.md) - 使用 Rc/Arc 的特征对象 - [配置](configuration.md) - 引用跟踪选项 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/schema-evolution.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/schema-evolution.md index 34f8a642fd5..e89c172e99f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/schema-evolution.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/schema-evolution.md @@ -257,5 +257,5 @@ assert_eq!(data, decoded); - [配置](configuration.md) - 兼容模式设置 - [多态](polymorphism.md) - 支持 Schema 演进的特征对象 -- [跨语言序列化](xlang.md) - 跨语言 Schema 演进 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 跨语言 Schema 演进 - [外部类型序列化](external-types.md) - 第三方值的兼容 Schema diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/schema-metadata.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/schema-metadata.md index d067b6a2261..72c06cb76a7 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/schema-metadata.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/schema-metadata.md @@ -485,7 +485,7 @@ struct User { ## 相关主题 -- [基本序列化](core-api.md) - 开始使用 Fory 序列化 +- [基本序列化](basic-serialization.md) - 开始使用 Fory 序列化 - [Schema 演进](schema-evolution.md) - 兼容模式和 Schema 演进 -- [跨语言序列化](xlang.md) - 与 Java、C++、Go、Python 互操作 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - 与 Java、C++、Go、Python 互操作 - [外部类型序列化](external-types.md) - 为第三方字段选择序列化器 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/security.md new file mode 100644 index 00000000000..126799f446c --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/security.md @@ -0,0 +1,51 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 Rust 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +安全相关配置： + +- 反序列化不可信载荷前，注册应用结构体和特征对象实现。 +- 使用 `max_dyn_depth(...)` 拒绝意外过深的动态对象图。 +- 对大多数输入，将 `max_graph_memory_bytes(...)` 保持为固定默认值 `128 MiB`；只有可信工作负载具有不同的合理集合、映射或结构体大小时才设置正字节限制。 +- 除非可信紧凑编解码器需要更大的根限额，否则将 `max_unbacked_container_items(...)` 保持为 `8192`。零会拒绝每个无输入支撑的条目。 +- 除非数据确定无恶意且可信对等端会发送更大的元数据或大量 Schema 版本，否则请保留远端 Schema 元数据限制的默认值。 +- 对不可信输入，优先使用具体类型字段，而非 `dyn Any` 或宽泛的特征对象字段。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[类型注册](type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/type-registration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/type-registration.md index 44ad072ca14..08b1203145a 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/type-registration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/rust/type-registration.md @@ -128,6 +128,6 @@ let handles: Vec<_> = (0..4) ## 相关主题 - [配置](configuration.md) - Fory 构建器选项 -- [跨语言序列化](xlang.md) - xlang 模式注册 +- [跨语言序列化](basic-serialization.md#cross-language-interoperability) - xlang 模式注册 - [自定义序列化器](custom-serializers.md) - 自定义序列化 - [外部类型序列化](external-types.md) - 第三方目标和载体根值 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/basic-serialization.md similarity index 94% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/xlang.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/basic-serialization.md index 462c614e314..649012fd038 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/xlang.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Scala Xlang 序列化 +title: 基础序列化 sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,11 +19,17 @@ license: | limitations under the License. --- +Xlang 是 Fory Scala 的默认序列化模式。本页介绍该默认模式的基础序列化 API 和互操作规则。 + +## 跨语言互操作 {#cross-language-interoperability} + +以下内容介绍默认 xlang 模式的模型生成、注册和跨语言往返。 + Fory Schema IDL 的 Scala 目标会为 xlang 载荷生成 Scala 3 源代码。Fory Scala 产物仍然 针对 Scala 2.13 和 Scala 3 进行交叉构建；只有 Schema IDL 输出和 quoted 宏派生需要 Scala 3。 -## 设置 +### 设置 生成的 Scala 代码使用 `org.apache.fory.scala` 中的公共宏 API，以及 `org.apache.fory.annotation` 中的共享 JVM 注解。宏内部实现在 @@ -51,7 +57,7 @@ val fory = ForyScala.builder() 临时序列化器或 Scala 特有注册状态。枚举和联合会直接与其序列化器一起注册，因为派生 序列化器负责 case 分派。 -## 生成的消息 +### 生成的消息 无环消息会生成 case class： @@ -97,7 +103,7 @@ Fory 的 xlang 序列化器也可以重建 `scala.collection.Seq` 和 `scala.collection.Map` 等受支持的可变集合接口，但除非显式生成，否则具体可变集合类 不属于 Schema IDL 接口范围。 -## 生成的枚举 +### 生成的枚举 IDL 枚举仅生成 Scala 3 枚举。编译器不会生成 Java 枚举文件。 @@ -116,7 +122,7 @@ enum Status { 生成的注册代码使用 `ScalaSerializers.registerEnum(...)`，因此 xlang 模式会使用来自 case 级 `@ForyEnumId` 元数据的稳定 Fory 枚举 ID。 -## 生成的联合 +### 生成的联合 IDL 联合会生成带宏派生序列化器的 Scala 3 ADT 枚举： @@ -152,7 +158,7 @@ Scala 未知 case 载体由 `@ForyUnknownCase` 选择，而不是由 Schema case 宏会直接写入现有的 xlang 联合信封，不会分配临时 Java `Union` 载体。 -## 手动 Scala 3 派生 +### 手动 Scala 3 派生 手写 Scala 3 模型可以派生相同的序列化器 typeclass： @@ -173,7 +179,7 @@ Schema IDL 为构造环使用的普通类形式。如果复制从参与环的不 例如 Scala 枚举 case 或 case class，序列化器会给出明确错误，因为在构造完成前无法发布 被复制对象的身份。 -## 第一次往返处理 +### 第一次往返处理 ```scala import org.apache.fory.Fory diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/configuration.md index cfa8a1f8721..b6c50c005b2 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/configuration.md @@ -183,29 +183,6 @@ Java builder 在 xlang 和原生模式下都默认启用兼容模式。当模型 语言使用相同 Schema 后，或者原生类型由 Fory Schema IDL 生成时，才调用 `withCompatible(false)`。 -## 安全性 +## 安全 -Scala 使用 Java 配置接口。在生产环境以及处理任何不可信载荷来源时，请保持启用类注册： - -```scala -val fory = ForyScala.builder() - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .withMaxTypeFields(512) - .withMaxTypeMetaBytes(4096) - .build() -``` - -与安全相关的配置： - -- 保持 `requireClassRegistration(true)`，并注册应用类或生成的模块。 -- 使用 `withMaxDepth(...)` 拒绝深度异常的对象图。 -- 使用 `withMaxGraphMemoryBytes(...)` 为包含大量集合、map、数组、结构体和对象的载荷 - 设置近似限制。它不是精确的堆上限；叶子值受剩余输入字节限制。 -- 除非可信的紧凑 codec 需要更大的根操作余量，否则将 - `withMaxUnbackedContainerItems(...)` 保持为 `8192`。零会拒绝每一个无输入支撑的条目。 -- 除非数据没有恶意，且可信对端会发送更大的元数据或许多 Schema 版本，否则请将 - `withMaxTypeFields(...)`、`withMaxTypeMetaBytes(...)` 和远端 Schema 版本限制保留为默认值。 -- 白名单和未知类控制请遵循[对象序列化安全](../security.md)。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [Scala 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/index.md index f5db3f9fabe..ab090662ace 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/index.md @@ -105,7 +105,7 @@ Scala builder 设置请参阅[配置](configuration.md)，完整的 JVM 原生 Fory Scala 基于 Fory Java 构建。Fory Java 的大多数配置选项、功能和概念都直接适用于 Scala。以下内容请参阅 Java 文档： - [配置](../java/configuration.md) - 所有 ForyBuilder 选项 -- [基本序列化](../java/core-api.md) - 序列化模式和 API +- [基本序列化](../java/basic-serialization.md) - 序列化模式和 API - [类型注册](../java/type-registration.md) - 类注册与安全性 - [Schema 演进](../java/schema-evolution.md) - 向前/向后兼容性 - [自定义序列化器](../java/custom-serializers.md) - 实现自定义序列化器 @@ -118,5 +118,7 @@ Fory Scala 基于 Fory Java 构建。Fory Java 的大多数配置选项、功能 - [原生序列化](native.md) - 在 JVM 原生模式下序列化 Scala 类型 - [Schema 元数据](schema-metadata.md) - Scala 注解、引用、枚举 ID 和联合元数据 - [默认值](default-values.md) - Scala 类默认值支持 -- [Xlang 序列化](xlang.md) - Scala 3 生成的模型和宏派生 xlang 序列化器 +- [基础序列化](basic-serialization.md) - 默认 xlang 模式的模型、API 和跨语言互操作 - [gRPC 支持](../../grpc/scala.md) - Scala 3 生成的 gRPC 服务配套代码 + +解码来自应用信任边界之外的字节之前，请阅读 [Scala 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/native.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/native.md index a0988950523..cbd1639db68 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/native.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/native.md @@ -20,7 +20,7 @@ license: | --- 本页介绍如何在原生模式下序列化 Scala 特有的 JVM 类型。跨语言 Scala 模型请使用 -[Xlang 序列化](xlang.md)中介绍的 xlang 路径。 +[跨语言互操作](basic-serialization.md#cross-language-interoperability)中介绍的 xlang 路径。 启用兼容模式后，Scala 读取端会针对部分标量字段类型变更使用 JVM 兼容读取规则。当转换后 的值具有相同逻辑值时，匹配字段可以在 `Boolean`、`String`、数值标量和 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/schema-metadata.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/schema-metadata.md index 623e5f237ac..1d85ea555d7 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/schema-metadata.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/schema-metadata.md @@ -114,6 +114,6 @@ Scala 宏根据 Scala 编译期类型构建描述符元数据，包括嵌套泛 ## 相关主题 -- [Xlang 序列化](xlang.md) +- [跨语言互操作](basic-serialization.md#cross-language-interoperability) - [配置](configuration.md) - [默认值](default-values.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/security.md new file mode 100644 index 00000000000..9fc85fbed54 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/scala/security.md @@ -0,0 +1,67 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 Scala 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +Scala 使用 Java 配置接口。在生产环境以及处理任何不可信载荷来源时，请保持启用类注册： + +```scala +val fory = ForyScala.builder() + .requireClassRegistration(true) + .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .withMaxUnbackedContainerItems(8192) + .withMaxTypeFields(512) + .withMaxTypeMetaBytes(4096) + .build() +``` + +与安全相关的配置： + +- 保持 `requireClassRegistration(true)`，并注册应用类或生成的模块。 +- 使用 `withMaxDepth(...)` 拒绝深度异常的对象图。 +- 使用 `withMaxGraphMemoryBytes(...)` 为包含大量集合、map、数组、结构体和对象的载荷 + 设置近似限制。它不是精确的堆上限；叶子值受剩余输入字节限制。 +- 除非可信的紧凑 codec 需要更大的根操作余量，否则将 + `withMaxUnbackedContainerItems(...)` 保持为 `8192`。零会拒绝每一个无输入支撑的条目。 +- 除非数据没有恶意，且可信对端会发送更大的元数据或许多 Schema 版本，否则请将 + `withMaxTypeFields(...)`、`withMaxTypeMetaBytes(...)` 和远端 Schema 版本限制保留为默认值。 +- 白名单和未知类控制请遵循 [Java 安全](../java/security.md)。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[Java 类型注册](../java/type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/security.md deleted file mode 100644 index d1ab7a39dd1..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/security.md +++ /dev/null @@ -1,166 +0,0 @@ ---- -title: 对象序列化安全性 -sidebar_position: 4 -id: security -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本指南定义 Fory 二进制对象序列化在 xlang 和原生模式下的信任边界与安全运行模型。面向贡献者的分类规则位于[反序列化安全模型](deserialization-security-model.md)。 - -Fory 是进程内序列化库。应用将 Fory 链接到自身进程中，配置序列化器和类型策略，并调用 -Fory API 序列化应用拥有的对象或反序列化已编码的 Fory 数据。Fory 不提供独立的网络 -服务、守护进程、身份验证系统或传输协议。 - -## 信任边界 - -Fory 的主要安全边界是从不可信或部分可信来源传给反序列化 API 的编码字节或流。嵌入 -Fory 的应用负责这些字节的来源，以及读取它们时使用的 Fory 配置、已注册类型、Schema -和策略。 - -不可信反序列化的对手模型是：发送方可以构造提交给 Fory 读取 API 的编码字节或流行为。 -除非应用本身公开了相关控制，否则不假设发送方能够更改嵌入应用的 Fory 配置、已注册类型 -集合、`TypeChecker` 或等价白名单策略、Schema 定义、classloader 或其他活动策略对象。 - -Fory 安全边界包括： - -- 运行时安全，包括避免崩溃、panic、未定义行为和越界内存访问。 -- 资源所有权，包括内存、CPU 进度、流缓冲区、原生内存分配、回调和保留的读取端状态。 -- 显式 Fory 策略检查，例如限制可实体化内容的类、类型、函数、方法、注册或反序列化策略。 -- 清理边界，即失败的根操作中创建的状态不得泄漏到后续操作。 - -运行时序列化器代码生成和 JIT 编译不是执行编码输入的路径。它们在当前注册检查、 -`TypeChecker`、Schema 检查或策略检查接受类型范围后，对类型和 Schema 进行操作。禁用 -类注册时，`TypeChecker` 或等价白名单策略就是相关关卡。生成的序列化器代码派生自经过 -检查的类型描述符，而不是攻击者控制的字节内容。 - -[反序列化安全模型](deserialization-security-model.md)定义了如何为不可信反序列化路径 -分类这些边界。 - -## 非目标 - -Fory 不提供： - -- 编码数据的真实性、完整性、机密性、签名、MAC 或加密。 -- 传输安全，也不保护字节在 Fory 外部存储或移动时的安全。 -- 对成功反序列化值的业务含义进行应用级授权或验证。 -- 为用户注册的类、函数、构造函数、setter、finalizer 或其他应用拥有的逻辑提供沙箱。 - -从不可信来源接收 Fory 数据的应用，如果需要保证真实性或防篡改，应在将字节传给 Fory -之前进行身份验证或完整性检查。 - -## 下游责任 - -应用负责： - -- 判断字节来源对于所配置的反序列化模式是否足够可信。 -- 对不可信数据保持启用类或类型注册，除非另一个显式 Fory 策略负责可接受类型范围。 -- 仅注册对应用信任边界安全的类型和序列化器。 -- 根据应用准备接受的最大数据形状配置深度和资源限制。 -- 将跨语言对端和 Schema 视为应用信任关系的一部分。 - -对可信数据禁用注册或使用动态反序列化属于配置选择。对于不可信数据，绕过显式 Fory -策略、崩溃、泄漏资源、保留攻击者控制的状态或不成比例地分配资源，仍然属于 -[反序列化安全模型](deserialization-security-model.md)所述的安全问题。 - -## 资源限制 - -### 深度限制 - -将运行时深度限制设置为应用有意接受的最深对象图。某些运行时将此限制应用于每个嵌套值； -其他运行时则应用单独的动态对象深度限制。确切范围和默认值请参阅所选运行时的配置页面。 -深度限制可防止过度嵌套的输入导致无界递归，但它不是字节或内存配额。 - -### 对象图内存限制 - -`maxGraphMemoryBytes` 或运行时等价选项，是对一次根反序列化操作所实体化对象图所有者的 -近似限制。固定默认值为 128 MiB，显式值必须为正数。每次根操作都以完整配置预算开始， -包括读取失败后的下一次根操作。 - -该预算根据每个实现的存储模型，涵盖运行时拥有的集合、map、数组、结构体和对象。它不是 -精确的堆内存核算、输入大小限制，也不能替代可读字节检查。实际进程内存可能更高。请在 -接收字节的边界上保留外部请求体或文件大小限制。 - -### 远端 Schema 元数据限制 - -兼容模式可能会为读取端尚未知的类型接收远端元数据（`TypeDef` 或 `TypeMeta`）。Fory 会 -限制可接受的不同远端元数据版本数量，也会限制每个接收元数据主体的大小： - -- `maxSchemaVersionsPerType`：一个逻辑类型可接受的最大远端元数据版本数。默认值为 `10`。 -- `maxAverageSchemaVersionsPerType`：所有已接受远端类型中，每个类型可接受的平均远端 - 元数据版本数。默认值为 `3`；有效全局下限为 `8192` 个元数据条目。 -- `maxTypeFields`：一个接收的结构体元数据主体可声明的最大字段数。默认值为 `512`。 -- `maxTypeMetaBytes`：一个接收的 TypeDef 或 TypeMeta 主体中，编码元数据主体的最大字节数， - 不包括 8 字节头部和任何扩展大小 varint。默认值为 `4096`。 - -这些限制用于保护资源。它们不会改变编码格式、注册要求、动态类型加载、未知类型处理或 -Schema 演进兼容性。 - -只有在已知对端有意发送更大的元数据或许多 Schema 版本时，才提高这些值。 - -### 由计数驱动的容器工作量限制 - -每个运行时都会限制重复读取主体未消耗相应输入的集合元素和 map 条目。默认根操作余量为 -`8192`。零表示严格限制，负值会被拒绝。只有可信载荷有意使用紧凑的零字节元素 codec -或空 Struct 主体时，才提高此限制。这是读取端资源限制，不会改变编码格式或写入端行为。 - -## 配置运行时 - -对于不可信输入，请保持启用注册，显式选择编码模式，并根据端点接受的模型设置限制值。 -最小 Java 边界如下： - -```java -Fory fory = - Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .build(); -``` - -仅注册端点接受的应用类型。如果禁用了注册，请在读取外部数据之前配置运行时的显式类型 -检查器或白名单。 - -确切的选项名称、默认值和模式特有行为请参阅各运行时配置指南： - -| 运行时 | 配置 | -| --------------------- | ---------------------------------------------- | -| Java | [Java 配置](java/configuration.md) | -| Python | [Python 配置](python/configuration.md) | -| C++ | [C++ 配置](cpp/configuration.md) | -| Go | [Go 配置](go/configuration.md) | -| Rust | [Rust 配置](rust/configuration.md) | -| JavaScript/TypeScript | [JavaScript 配置](javascript/configuration.md) | -| C# | [C# 配置](csharp/configuration.md) | -| Swift | [Swift 配置](swift/configuration.md) | -| Dart | [Dart 配置](dart/configuration.md) | -| Scala | [Scala 配置](scala/configuration.md) | -| Kotlin | [Kotlin 配置](kotlin/configuration.md) | - -## 验证边界 - -在正常往返测试之外添加负向测试。验证配置的读取端会拒绝： - -- 未注册或不允许的应用类型； -- 深度超过可接受模型的对象图； -- 超过所配置内存预算的对象图； -- 兼容 xlang 模式下过多的远端 Schema 版本或过大的元数据； -- 过多由计数驱动的容器工作量； -- 同一个可复用运行时中格式错误的根之后紧跟的有效根。 - -还应独立于 Fory 验证应用的外部身份验证、完整性、请求大小、超时和领域验证控制。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/basic-serialization.md similarity index 56% rename from i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/xlang.md rename to i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/basic-serialization.md index 90cd2d8eb90..8305dcc3675 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/xlang.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang 序列化 +title: 基础序列化 sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,17 +19,171 @@ license: | limitations under the License. --- +本页介绍 Fory Swift 默认 xlang 模式下的对象图序列化和核心 API 用法。 + +## 对象图序列化 + +使用 `@ForyStruct`、`@ForyEnum` 或 `@ForyUnion`，注册类型，然后进行序列化和反序列化。 + +```swift +import Foundation +import Fory + +@ForyStruct +struct Address: Equatable { + var street: String = "" + var zip: Int32 = 0 +} + +@ForyStruct +struct Person: Equatable { + var id: Int64 = 0 + var name: String = "" + var nickname: String? = nil + var tags: Set = [] + var scores: [Int32] = [] + var addresses: [Address] = [] + var metadata: [Int8: Int32?] = [:] +} + +let fory = Fory() +try fory.register(Address.self, id: 100) +try fory.register(Person.self, id: 101) + +let person = Person( + id: 42, + name: "Alice", + nickname: nil, + tags: ["swift", "xlang"], + scores: [10, 20, 30], + addresses: [Address(street: "Main", zip: 94107)], + metadata: [1: 100, 2: nil] +) + +let data = try fory.serialize(person) +let decoded: Person = try fory.deserialize(data) +assert(decoded == person) +``` + +## 使用现有缓冲区 + +将序列化字节追加到现有 `Data`，并从 `ByteBuffer` 反序列化。 + +```swift +var output = Data() +try fory.serialize(person, to: &output) + +let inputBuffer = ByteBuffer(data: output) +let fromBuffer: Person = try fory.deserialize(from: inputBuffer) +assert(fromBuffer == person) +``` + +## 选择序列化器 + +实现 `Serializer` 且 `Target == Self` 的类型会选择自身： + +```swift +let data = try fory.serialize(person) +let decoded: Person = try fory.deserialize(data) +``` + +这种隐式选择可以通过生成字段和普通 optional、array、set、dictionary 组合。当应用有意 +让外部类型追溯遵循以自身为目标的协议时，也适用此规则。 + +当单独的序列化器以该值为目标时，使用 `with` 选择它： + +```swift +try fory.register(UserSerializer.self, id: 200) + +let data = try fory.serialize( + externalUser, + with: UserSerializer.self +) +let decoded = try fory.deserialize( + data, + with: UserSerializer.self +) +``` + +同样的选择也适用于现有缓冲区： + +```swift +var output = Data() +try fory.serialize( + externalUser, + with: UserSerializer.self, + to: &output +) + +let input = ByteBuffer(data: output) +let decoded = try fory.deserialize( + from: input, + with: UserSerializer.self +) +``` + +外部结构化序列化器和递归载体根值请参阅[外部类型序列化](external-types.md)。由类型直接实现 +的序列化器、追溯遵循和单独的自定义序列化器请参阅 +[自定义序列化器](custom-serializers.md)。 + +## 内置支持的类型 + +### 基本类型和标量 + +- `Bool` +- `Int8`, `Int16`, `Int32`, `Int64`, `Int` +- `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt` +- `Float`, `Double` +- `String` +- `Data` + +### 日期和时间 + +- `Date` +- `LocalDate` +- `Duration` + +时间戳值使用 `Date`，只包含日期的值使用 `LocalDate`。`LocalDate` 支持纪元日转换和 `Date` +转换，可使用 `fromEpochDay(_:)`、`toEpochDay()`、`init(utcDate:)` 和 `toUTCDate()`。 + +### 集合 + +- 值直接实现 `Serializer` 的 optional 和 array +- 元素直接实现 `Serializer` 且为 `Hashable` 的 set +- 键和值直接实现 `Serializer`，且键为 `Hashable` 的 dictionary + +使用单独序列化器的子项通过以下类型组合： + +- `OptionalSerializer` +- `ArraySerializer` +- `SetSerializer` +- `DictionarySerializer` + +### 动态类型 + +- `Any` 和 `AnyObject` +- `AnyHashable` +- 任意应用 protocol 值 +- 受支持的异构 array 和 dictionary + +`Any` 和 `AnyObject` 根值使用直接根值 API。任意应用 protocol 根值和嵌套在载体中的动态值 +使用显式 `with:` 选择。请参阅[多态和动态类型](polymorphism.md)。 + +## 跨语言互操作 {#cross-language-interoperability} + +以下内容说明默认 xlang 格式的跨语言类型映射、类型标识和互操作要求。 + Fory Swift 可以使用 xlang 协议与其他 Fory 实现交换载荷。 -## 推荐的 Xlang 配置 +### 推荐的 Xlang 配置 ```swift let fory = Fory() ``` -## 使用共享标识注册类型 +### 使用共享标识注册类型 -### 基于 ID 的注册 +#### 基于 ID 的注册 ```swift @ForyStruct @@ -42,13 +196,13 @@ let fory = Fory() try fory.register(Order.self, id: 100) ``` -### 基于名称的注册 +#### 基于名称的注册 ```swift try fory.register(Order.self, name: "com.example.Order") ``` -## Xlang 规则 +### Xlang 规则 - 在不同语言间保持类型注册映射一致 - 独立演进 Schema 时保持启用兼容模式。Swift 默认启用兼容模式。 @@ -56,7 +210,7 @@ try fory.register(Order.self, name: "com.example.Order") - 对其他模块拥有的类型，使用外部结构化序列化器、单独的自定义序列化器，或一个有意添加的 追溯自目标遵循 -## List 和密集数组 +### List 和密集数组 Swift `Array` 字段映射为 Fory `list`，除非字段元数据显式请求密集 `array`。 `array` 只用于一维 bool 或数值数据。 @@ -81,7 +235,7 @@ Swift `Array` 字段映射为 Fory `list`，除非字段元数据显式请 使用单独元素序列化器的 array 仍使用普通 list 编码。只对受支持的密集 bool 或数值数组使用 `@ArrayField`。 -## 外部目标 +### 外部目标 外部结构化序列化器生成的 xlang STRUCT、ENUM 或 UNION Schema 和值字节，与等效普通 Swift 模型相同： @@ -105,7 +259,7 @@ try fory.register(OrderSerializer.self, id: 100) Swift 没有原生序列化模式。已知 `@ForyUnion` case 包含零个或一个关联值。如果 union 候选项 包含多个逻辑字段，请使用 struct 载荷。 -## Swift IDL 工作流 +### Swift IDL 工作流 直接从 Fory IDL/Proto/FBS 输入生成 Swift 模型： @@ -130,7 +284,7 @@ let payload = try fory.serialize(book) let decoded: Addressbook.AddressBook = try fory.deserialize(payload) ``` -### 运行 Swift IDL 集成测试 +#### 运行 Swift IDL 集成测试 ```bash cd integration_tests/idl_tests @@ -139,7 +293,7 @@ cd integration_tests/idl_tests 该命令运行 Swift 往返矩阵测试和 Java 对等端往返检查（`IDL_PEER_LANG=swift`）。 -## 调试 Xlang 测试 +### 调试 Xlang 测试 运行 xlang 测试时启用调试输出： @@ -147,7 +301,7 @@ cd integration_tests/idl_tests ENABLE_FORY_DEBUG_OUTPUT=1 FORY_SWIFT_JAVA_CI=1 mvn -T16 test -Dtest=org.apache.fory.xlang.SwiftXlangTest ``` -## 首次往返 +### 首次往返 ```swift import Fory @@ -170,7 +324,7 @@ print("\(result.name) \(result.age)") 更多跨语言规则和示例请参阅： -- [跨语言序列化指南](../xlang/index.md) +- [跨语言互操作](../xlang.md) - [Java 指南](../java/index.md) - [Python 指南](../python/index.md) - [Dart 指南](../dart/index.md) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/configuration.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/configuration.md index 05647b4f029..db55cba6e49 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/configuration.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/configuration.md @@ -152,12 +152,4 @@ let fory = Fory(compatible: false) ## 安全 -安全相关配置： - -- 反序列化不可信载荷前，只注册预期的生成模型。 -- 对有意采用相同 Schema 的载荷，将 `checkClassVersion` 与 `compatible: false` 配合使用。 -- 将 `maxDepth` 设置为服务允许的最大动态 `Any` 嵌套深度。 -- 将 `maxGraphMemoryBytes` 设置为 collection、map、array、struct、class 和对象密集型载荷的 - 近似限制。它不是精确的堆上限；叶子值受剩余输入字节限制。 -- 除非数据确定无恶意，且可信对等端会发送更大的元数据或大量 Schema 版本，否则保留远程 - Schema 元数据限制的默认值。 +有关信任边界、安全的读取端配置和验证方法，请参阅 [Swift 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/core-api.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/core-api.md deleted file mode 100644 index 9b74ba716b5..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/core-api.md +++ /dev/null @@ -1,170 +0,0 @@ ---- -title: 基本序列化 -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本页介绍 Swift 中的对象图序列化和核心 API 用法。 - -## 对象图序列化 - -使用 `@ForyStruct`、`@ForyEnum` 或 `@ForyUnion`，注册类型，然后进行序列化和反序列化。 - -```swift -import Foundation -import Fory - -@ForyStruct -struct Address: Equatable { - var street: String = "" - var zip: Int32 = 0 -} - -@ForyStruct -struct Person: Equatable { - var id: Int64 = 0 - var name: String = "" - var nickname: String? = nil - var tags: Set = [] - var scores: [Int32] = [] - var addresses: [Address] = [] - var metadata: [Int8: Int32?] = [:] -} - -let fory = Fory() -try fory.register(Address.self, id: 100) -try fory.register(Person.self, id: 101) - -let person = Person( - id: 42, - name: "Alice", - nickname: nil, - tags: ["swift", "xlang"], - scores: [10, 20, 30], - addresses: [Address(street: "Main", zip: 94107)], - metadata: [1: 100, 2: nil] -) - -let data = try fory.serialize(person) -let decoded: Person = try fory.deserialize(data) -assert(decoded == person) -``` - -## 使用现有缓冲区 - -将序列化字节追加到现有 `Data`，并从 `ByteBuffer` 反序列化。 - -```swift -var output = Data() -try fory.serialize(person, to: &output) - -let inputBuffer = ByteBuffer(data: output) -let fromBuffer: Person = try fory.deserialize(from: inputBuffer) -assert(fromBuffer == person) -``` - -## 选择序列化器 - -实现 `Serializer` 且 `Target == Self` 的类型会选择自身： - -```swift -let data = try fory.serialize(person) -let decoded: Person = try fory.deserialize(data) -``` - -这种隐式选择可以通过生成字段和普通 optional、array、set、dictionary 组合。当应用有意 -让外部类型追溯遵循以自身为目标的协议时，也适用此规则。 - -当单独的序列化器以该值为目标时，使用 `with` 选择它： - -```swift -try fory.register(UserSerializer.self, id: 200) - -let data = try fory.serialize( - externalUser, - with: UserSerializer.self -) -let decoded = try fory.deserialize( - data, - with: UserSerializer.self -) -``` - -同样的选择也适用于现有缓冲区： - -```swift -var output = Data() -try fory.serialize( - externalUser, - with: UserSerializer.self, - to: &output -) - -let input = ByteBuffer(data: output) -let decoded = try fory.deserialize( - from: input, - with: UserSerializer.self -) -``` - -外部结构化序列化器和递归载体根值请参阅[外部类型序列化](external-types.md)。由类型直接实现 -的序列化器、追溯遵循和单独的自定义序列化器请参阅 -[自定义序列化器](custom-serializers.md)。 - -## 内置支持的类型 - -### 基本类型和标量 - -- `Bool` -- `Int8`, `Int16`, `Int32`, `Int64`, `Int` -- `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt` -- `Float`, `Double` -- `String` -- `Data` - -### 日期和时间 - -- `Date` -- `LocalDate` -- `Duration` - -时间戳值使用 `Date`，只包含日期的值使用 `LocalDate`。`LocalDate` 支持纪元日转换和 `Date` -转换，可使用 `fromEpochDay(_:)`、`toEpochDay()`、`init(utcDate:)` 和 `toUTCDate()`。 - -### 集合 - -- 值直接实现 `Serializer` 的 optional 和 array -- 元素直接实现 `Serializer` 且为 `Hashable` 的 set -- 键和值直接实现 `Serializer`，且键为 `Hashable` 的 dictionary - -使用单独序列化器的子项通过以下类型组合： - -- `OptionalSerializer` -- `ArraySerializer` -- `SetSerializer` -- `DictionarySerializer` - -### 动态类型 - -- `Any` 和 `AnyObject` -- `AnyHashable` -- 任意应用 protocol 值 -- 受支持的异构 array 和 dictionary - -`Any` 和 `AnyObject` 根值使用直接根值 API。任意应用 protocol 根值和嵌套在载体中的动态值 -使用显式 `with:` 选择。请参阅[多态和动态类型](polymorphism.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/index.md index a8549b0980e..713df5f7adf 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/index.md @@ -52,8 +52,7 @@ targets: [ ## 指南目录 - [配置](configuration.md) -- [基本序列化](core-api.md) -- [Xlang 序列化](xlang.md) +- [基本序列化](basic-serialization.md) - [Schema 元数据](schema-metadata.md) - [类型注册](type-registration.md) - [外部类型序列化](external-types.md) @@ -83,3 +82,5 @@ let output: User = try fory.deserialize(data) assert(input == output) ``` + +解码来自应用信任边界之外的字节之前，请阅读 [Swift 安全](security.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/security.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/security.md new file mode 100644 index 00000000000..0322a98d392 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/swift/security.md @@ -0,0 +1,52 @@ +--- +title: 安全 +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +当 Swift 读取端接收来自应用信任边界之外的字节时，请遵循本页说明。Fory 会重建应用值， +但不会验证发送者身份、保护传输完整性，也不会判断一个合法值是否获准用于某项业务操作。 + +## 应用边界 + +反序列化之前： + +- 在传输层或存储层验证发送者身份并保护消息完整性。 +- 在 Fory 外部限制请求或文件大小、超时时间和并发量。 +- 只注册端点允许的应用类型，并在第一次根操作之前完成读取端配置。 +- 使用反序列化结果之前，根据应用的授权规则和领域规则验证该值。 + +## 运行时保护 + +安全相关配置： + +- 反序列化不可信载荷前，只注册预期的生成模型。 +- 对有意采用相同 Schema 的载荷，将 `checkClassVersion` 与 `compatible: false` 配合使用。 +- 将 `maxDepth` 设置为服务允许的最大动态 `Any` 嵌套深度。 +- 将 `maxGraphMemoryBytes` 设置为 collection、map、array、struct、class 和对象密集型载荷的 + 近似限制。它不是精确的堆上限；叶子值受剩余输入字节限制。 +- 除非数据确定无恶意，且可信对等端会发送更大的元数据或大量 Schema 版本，否则保留远程 + Schema 元数据限制的默认值。 + +## 验证 + +除正常往返测试外，还应为边界添加负向测试。确认配置后的读取端会拒绝意外的应用类型、 +过深的嵌套、超出资源限制的输入以及格式错误的输入。一次读取失败后，还应确认可复用运行时 +仍能正确读取下一个合法根值。 + +完整选项请参阅[配置](configuration.md)，运行时的注册 API 请参阅[类型注册](type-registration.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang.md new file mode 100644 index 00000000000..06401304edc --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang.md @@ -0,0 +1,351 @@ +--- +title: Xlang 序列化 +sidebar_position: 2 +id: xlang +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Xlang 是 Fory 默认的对象序列化模式。Java、Python、C++、Go、Rust、 +JavaScript/TypeScript、C#、Swift、Dart、Scala 和 Kotlin 共用同一种可移植二进制格式。 +各运行时的[基础序列化](#runtime-guides)页面负责介绍 API 和模型示例；本页说明所有通信方必须 +共同遵循的规则。 + +请先阅读[核心概念](core-concepts.md)，了解 xlang 和原生模式共用的对象图、Schema、引用和多态概念。 + +## 概述 {#overview} + +当字节需要跨运行时传输时，应使用 xlang 序列化，例如多语言服务、数据管道以及前后端通信。 +它提供以下能力： + +- 直接序列化各语言的原生模型，无需预先定义 IDL。 +- 通过协调一致的数字 ID 或名称标识应用类型。 +- 使用兼容模式支持独立部署的通信方演进 Schema。 +- 按需保留共享引用和循环引用。 +- 当每个具体类型都有可移植映射时支持多态值。 +- 在运行时支持的情况下，通过带外缓冲区传输大型二进制和数值数据。 + +如果写入端和读取端始终使用同一种受支持的运行时，并且对象图需要 Java 序列化钩子、Python +pickle 兼容对象等语言特有行为，请改用[原生序列化](native.md)。 + +### 支持的运行时 {#supported-runtimes} + +| 运行时 | 包或目标 | 模式 | +| --------------------- | ----------------------------------------- | ------------ | +| Java | `org.apache.fory:fory-core` | xlang/native | +| Python | `pyfory` | xlang/native | +| C++ | Fory C++ CMake 或 Bazel 目标 | xlang/native | +| Go | `github.com/apache/fory/go/fory` | xlang/native | +| Rust | `fory` crate | xlang/native | +| JavaScript/TypeScript | `@apache-fory/core` | xlang | +| C# | `Apache.Fory` | xlang | +| Swift | `Fory` Swift Package Manager 目标 | xlang | +| Dart | `fory` package | xlang | +| Scala | `org.apache.fory:fory-scala` | xlang/native | +| Kotlin | `org.apache.fory:fory-kotlin` 和 Java API | xlang/native | + +### 第一次跨语言往返 {#first-cross-language-round-trip} + +每个通信方都必须为同一个逻辑类型注册相同的类型标识和兼容字段。下面的示例使用共享类型名称。 + +Rust 写入端： + +```rust +use fory::{Fory, ForyStruct}; + +#[derive(ForyStruct)] +struct Person { + name: String, + age: i32, +} + +let mut fory = Fory::builder().xlang(true).build(); +fory.register_by_name::("example.Person").unwrap(); + +let person = Person { + name: "Alice".to_string(), + age: 30, +}; +let bytes = fory.serialize(&person).unwrap(); +``` + +Python 读取端： + +```python +from dataclasses import dataclass +import pyfory + +@dataclass +class Person: + name: str + age: pyfory.Int32 + +fory = pyfory.Fory(xlang=True) +fory.register_type(Person, name="example.Person") +person = fory.deserialize(bytes_from_rust) +``` + +虽然 xlang 是默认模式，示例通常仍会显式选择它，以便在应用代码中明确表达传输约定。 + +### 原生模型还是 Fory IDL {#native-models-or-fory-idl} + +| 方式 | 适用场景 | +| -------------------------------- | ------------------------------------------ | +| 各语言的原生模型 | 契约较小，希望无需编译步骤即可开始 | +| [Fory IDL](../compiler/index.md) | 多个消息或团队需要统一 Schema 和生成的模型 | +| 先用原生模型，再迁移到 IDL | 小型契约逐渐成为长期维护、跨团队的服务边界 | + +一个最小的 Fory IDL 消息如下： + +```protobuf +package example; + +message Person { + string name = 1; + int32 age = 2; + optional string email = 3; +} +``` + +使用 `foryc` 生成所需的运行时目标；生成的模型在这些目标之间使用一致的字段和类型元数据。 + +## 类型系统和类型标识 {#type-system-and-type-identity} + +### 内置类型和自定义类型 {#built-in-and-custom-types} + +原始数值、字符串、二进制、时间类型、列表、集合、映射、稠密数值数组、枚举、结构体和联合类型 +都有共享的 xlang Schema。内置值无需用户注册；应用结构体、枚举、联合和扩展类型则需要协调一致的 +类型标识。 + +规范性的[类型映射](../specification/xlang_type_mapping.md)定义了每个 xlang 类型在各宿主语言中的 +准确载体。需要特别注意： + +- 当 Python 原生类型无法表达所需位宽时，使用 `pyfory.Int32`、`pyfory.Float16`、 + `pyfory.BFloat16` 等标记。 +- 当一个宿主类型能够表示多个 xlang 类型时，Java、Dart 等运行时使用注解或 Schema 元数据区分。 +- `float16`、`bfloat16` 及其稠密数组使用运行时特有的载体。 +- `list` 和稠密 `array` 是不同的 Schema。在兼容模式下，如果元素域兼容，直接结构体字段 + 可以在列表和稠密布尔值/数值数组之间适配，但实际列表不能包含目标数组无法表示的 null 或 + 引用跟踪元素。 + +不要根据宿主语言中相似的类型名称推断兼容性，应以类型映射规范为准。 + +### 协调类型标识 {#coordinate-type-identity} + +每个通信方都必须使用相同的数字 ID，或者相同的命名空间和类型名称注册自定义类型。数字 ID 的 +元数据更小；名称则更适合由不同团队独立维护的服务。不要让同一契约的一端按 ID 注册、另一端按 +名称注册。 + +注册必须在第一次根序列化或反序列化操作之前完成。当多个团队共同维护通信方时，应维护一个小型 +契约注册表，或者使用 Fory IDL 生成的模块。 + +### 静态字段和动态字段 {#static-and-dynamic-fields} + +静态已知字段直接使用声明类型的序列化器，不写入具体运行时类型；动态字段则携带足够的类型信息， +用于选择已注册的具体类型。接口、抽象类型、trait object 和其他多态位置需要动态元数据；原始类型 +和准确的 final 类型不需要。 + +| 运行时 | 动态字段模型 | +| ------ | ---------------------------------------------------- | +| Java | `@ForyField(dynamic = ...)` 控制自动或强制写入元数据 | +| Python | `pyfory.field(dynamic=...)` 控制对象字段元数据 | +| C++ | `fory::F(...).dynamic(...)` 覆盖自动检测 | +| Go | interface 字段表达动态值 | +| Rust | trait object 载体表达动态值 | + +写入动态元数据会增加空间和类型解析开销。只有在字段绝不可能包含其他具体类型时才能禁用它。准确的 +注解和注册示例请参阅各运行时的 Schema 元数据、类型注册和多态文档。 + +## 可空性和引用跟踪 {#nullability-and-reference-tracking} + +可空性和引用跟踪解决的是不同问题： + +| 关注点 | 作用 | +| -------- | ---------------------------- | +| 可空性 | 允许字段或值位置不包含值 | +| 引用跟踪 | 保留重复对象身份并支持对象环 | + +线格式帧由 [xlang 序列化规范](../specification/xlang_serialization_spec.md)定义。应用应通过运行时 API +配置语义行为，不要依赖具体的标志值。 + +### 可空性 {#nullability} + +Xlang 结构体字段默认不可为空。不可空字段体积更小，也能明确表达必填数据。常见的可空或可选载体 +包括 Java 装箱类型或注解字段、Python `Optional[T]`、C++ `std::optional`、Go 指针、 +Rust `Option` 和 Scala `Option[T]`。 + +对应字段在各通信方之间应保持一致的可空性。在同 Schema 模式下，修改可空性会改变 Schema， +因此不兼容。兼容模式支持文档中定义的可空和标量适配，但如果本地类型没有有效的 null 或缺失值 +行为，远端 null 仍然无法物化。 + +### 共享引用和循环引用 {#shared-and-circular-references} + +当对象图中同一个对象出现多次或包含环时，应启用引用跟踪。对于值形数据应保持禁用，以避免身份表 +开销。 + +```java +import org.apache.fory.Fory; +import org.apache.fory.annotation.Ref; + +public class Node { + public String value; + @Ref public Node next; +} + +Node first = new Node(); +Node second = new Node(); +first.next = second; +second.next = first; + +Fory fory = Fory.builder() + .withXlang(true) + .withRefTracking(true) + .build(); +``` + +全局引用跟踪启用运行时机制，字段元数据决定哪些位置参与跟踪。常见的字段级控制包括 Java 和 +Scala `@Ref`、Go `fory:"ref"` 标签、Rust `#[fory(ref = true)]`，以及 C++ 智能指针或 +`fory::F().ref()` 元数据。不同语言和载体的默认行为不同，请参阅对应运行时指南。 + +引用支持还受宿主语言所有权模型约束。例如 Rust 可以保留受支持的共享引用载体，但循环引用必须 +使用可表达的所有权和弱引用形状。 + +## 多态 {#polymorphism} + +当字段、集合元素或根值声明为更宽泛的类型时，xlang 多态会保留值的具体已注册类型。每个读取端都 +必须： + +1. 注册相同的具体类型标识。 +2. 为该具体类型提供兼容字段 Schema。 +3. 当运行时无法推断时，将该位置标记或建模为动态。 +4. 使用具有可移植 xlang 映射的具体类型。 + +仅有宿主语言继承关系并不能让类型自动变得可移植。如果某种形状没有 xlang 映射，请为同语言流量 +使用原生模式，或者定义可移植模型。接口、trait object、联合和生成代码的语法请参阅各运行时的 +多态文档。 + +## Schema 演进 {#schema-evolution} + +兼容模式是 xlang 的默认模式。它携带 Schema 元数据，使独立部署的读取端能够容忍受支持的字段 +新增、删除、重排和文档中定义的兼容类型适配。 + +当通信方可能独立部署时，应保留兼容模式。只有在每个读写端都使用以下完全相同的契约时，才能选择 +同 Schema 模式： + +- 类型标识以及字段 ID 或名称。 +- 字段类型和嵌套泛型形状。 +- 可空性和引用元数据。 +- 多态候选类型。 + +同 Schema 模式减少元数据和载荷体积，但任何差异都可能产生 Schema hash 或类型错误。当所有 +通信方一起发布时，Fory IDL 生成的模型更容易保证准确协调。规范性的兼容行为请参阅 +[xlang 序列化规范](../specification/xlang_serialization_spec.md)。 + +## 零拷贝序列化 {#zero-copy-serialization} + +部分运行时可以把大型二进制或数值缓冲区移出主序列化字节流，从而避免将这些缓冲区复制到一个连续 +载荷中。 + +传输流程如下： + +1. 序列化对象图，并通过回调收集选中的缓冲区对象。 +2. 分别发送主元数据字节和收集到的缓冲区。 +3. 反序列化时按照相同顺序提供缓冲区。 + +Java： + +```java +Collection objects = new ArrayList<>(); +byte[] metadata = fory.serialize(value, object -> !objects.add(object)); +List buffers = objects.stream() + .map(BufferObject::toBuffer) + .toList(); +Object decoded = fory.deserialize(metadata, buffers); +``` + +Python： + +```python +objects = [] +metadata = fory.serialize(value, buffer_callback=objects.append) +buffers = [obj.to_buffer() for obj in objects] +decoded = fory.deserialize(metadata, buffers=buffers) +``` + +Go 通过其序列化和缓冲区 API 提供等价的回调缓冲区流程。当前方法名和支持的缓冲区载体请以运行时 +文档为准。 + +当缓冲区很大并且传输层能够避免额外复制时，带外序列化收益明显。对于小数组，回调和多缓冲区传输 +的开销可能高于复制。应用负责缓冲区顺序、生命周期和传输帧。Python 和 NumPy 的详细用法请参阅 +[Python 带外序列化](python/out-of-band.md)。 + +## 故障排除 {#troubleshooting} + +| 现象 | 可能原因 | 解决方法 | +| ---------------------- | ------------------------------------- | ------------------------------------------------- | +| 类型未注册 | 缺少注册或注册过晚 | 在第一次根操作前注册所有自定义类型 | +| 类型 ID 或名称不匹配 | 通信方使用了不同标识 | 使用相同数字 ID，或相同命名空间和类型名称 | +| 整数溢出或浮点精度损失 | 宿主载体使用了不同数值位宽 | 遵循类型映射并使用显式位宽元数据 | +| 字段解码错误 | 字段 ID、名称或类型不同 | 对齐字段元数据，或从同一份 IDL 重新生成所有通信方 | +| 循环对象图出现栈溢出 | 未启用引用跟踪 | 启用全局和字段级引用跟踪 | +| 共享对象被复制 | 值位置不跟踪引用 | 为对应载体或字段启用引用跟踪 | +| 宿主类型不受支持 | 类型没有可移植的 xlang 表示 | 改用可移植模型，或为同语言流量使用原生模式 | +| Schema/hash 不匹配 | 同 Schema 通信方使用了不同 Schema | 对齐所有通信方，或恢复兼容模式 | +| 升级后失败 | 通信方运行了不兼容的协议版本 | 对齐受支持的 Fory 版本并查看发布说明 | +| 载荷立即被拒绝 | 一端写入原生字节，另一端按 xlang 读取 | 跨语言契约的所有通信方都应使用 xlang | + +### 诊断清单 {#diagnostic-checklist} + +1. 确认每个通信方都使用 xlang 模式和相互支持的 Fory 版本。 +2. 比较注册的类型标识、字段 ID 或名称、数值位宽、可空性和引用元数据。 +3. 在测试跨运行时方向前，先复现同运行时往返。 +4. 对生产环境使用的每一种语言组合测试双向传输。 +5. 将值缩减为一个类型和一个字段，然后逐步恢复字段，直到差异再次出现。 +6. 对于生成代码、平台或 API 错误，请查看对应运行时的故障排除页面。 + +诊断二进制布局时，应使用规范和运行时调试工具。不要把十六进制转储或内部标志值当作稳定的应用 +API。 + +## 运行时指南 {#runtime-guides} + +- [Java](java/basic-serialization.md#cross-language-interoperability) +- [Python](python/basic-serialization.md#cross-language-interoperability) +- [C++](cpp/basic-serialization.md#cross-language-interoperability) +- [Go](go/basic-serialization.md#cross-language-interoperability) +- [Rust](rust/basic-serialization.md#cross-language-interoperability) +- [JavaScript/TypeScript](javascript/basic-serialization.md#cross-language-interoperability) +- [C#](csharp/basic-serialization.md#cross-language-interoperability) +- [Swift](swift/basic-serialization.md#cross-language-interoperability) +- [Dart](dart/basic-serialization.md#cross-language-interoperability) +- [Scala](scala/basic-serialization.md#cross-language-interoperability) +- [Kotlin](kotlin/basic-serialization.md#cross-language-interoperability) + +## 相关文档 {#related-documentation} + +- [Xlang 序列化格式](../specification/xlang_serialization_spec.md) — 规范性的线格式 +- [Xlang 类型映射](../specification/xlang_type_mapping.md) — 各运行时准确的载体映射 +- [Fory IDL 和编译器](../compiler/index.md) — Schema 优先的模型和代码生成 +- [快速开始](../start/index.md) — 各运行时的安装和第一次序列化 +- [行格式](../row-format/index.md) — 用于可信分析数据的随机访问行 + +## 运维最佳实践 {#operational-best-practices} + +1. 在所有通信方之间协调统一的类型标识和字段契约。 +2. 除非所有读写端总是同时部署相同 Schema，否则保留兼容模式。 +3. 只为具有对象身份或包含环的对象图启用引用跟踪。 +4. 复用已配置的 Fory 实例，不要为每次操作重新创建。 +5. 部署前，双向验证生产环境使用的每一种语言组合。 +6. 当契约包含大量消息、服务或由不同团队维护时，优先采用 Fory IDL。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/_category_.json b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/_category_.json deleted file mode 100644 index d2418b5e5e0..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/_category_.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "position": 1, - "label": "Xlang", - "collapsible": true, - "collapsed": true -} diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/index.md deleted file mode 100644 index 345ae38ffc0..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/index.md +++ /dev/null @@ -1,185 +0,0 @@ ---- -title: 跨语言序列化指南 -sidebar_position: 0 -id: index -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Apache Fory™ 跨语言序列化是跨语言载荷的默认编码格式。数据可以在一种语言中序列化，并在另一种语言中反序列化，无需手动转换。对于小型契约，可以直接使用语言模型类型；更适合 Schema 优先工作流时，也可以使用 Fory IDL 和代码生成。 - -## 功能特性 - -- **无需 IDL**：直接使用语言模型类型序列化对象。 -- **多语言支持**：Java、Python、C++、Go、Rust、JavaScript/TypeScript、C#、Swift、Dart、Scala 和 Kotlin 可通过相同的跨语言格式互操作。 -- **引用支持**：每个对等端启用引用跟踪后，共享引用和循环引用可以跨语言工作。 -- **Schema 演进**：兼容模式是跨语言序列化的默认模式，因此读取端可以容忍字段新增、删除或重排。 -- **带外缓冲区**：各语言实现可以为大型二进制数据提供零拷贝缓冲区路径。 -- **高性能**：Fory 实现在可用时使用生成的序列化器、JIT 序列化器或优化代码路径。 - -## 支持的语言 - -| 语言 | 状态 | 软件包或构建目标 | -| --------------------- | ---- | -------------------------------- | -| Java | 支持 | `org.apache.fory:fory-core` | -| Python | 支持 | `pyfory` | -| C++ | 支持 | Bazel/CMake 构建 | -| Go | 支持 | `github.com/apache/fory/go/fory` | -| Rust | 支持 | `fory` crate | -| JavaScript/TypeScript | 支持 | `@apache-fory/core` | -| C# | 支持 | `Apache.Fory` | -| Swift | 支持 | Swift Package Manager 目标 | -| Dart | 支持 | `fory` package | -| Scala | 支持 | `org.apache.fory:fory-scala` | -| Kotlin | 支持 | `org.apache.fory:fory-kotlin` | - -## 何时使用跨语言模式 - -以下场景使用跨语言模式： - -- 构建多语言微服务 -- 创建多语言数据流水线 -- 在前端 JavaScript/TypeScript 与 Java、Python、Go、C#、Scala 或 Kotlin 等后端服务之间共享数据 - -对于 Java、Scala、Kotlin、Python、C++、Go 或 Rust 的同语言通信，请使用原生模式： - -- 所有序列化和反序列化都在同一种语言中完成 -- 需要 Python pickle 风格对象或 Java 序列化钩子等语言专属功能 -- 希望同语言服务使用原生模式载荷 - -## 快速示例 - -### Java（生产端） - -```java -import org.apache.fory.*; -import org.apache.fory.config.*; - -public class Person { - public String name; - public int age; -} - -Fory fory = Fory.builder().withXlang(true).build(); -fory.register(Person.class, "example.Person"); - -Person person = new Person(); -person.name = "Alice"; -person.age = 30; -byte[] bytes = fory.serialize(person); -// Send bytes to Python, Go, Rust, etc. -``` - -### Python（消费端） - -```python -import pyfory -from dataclasses import dataclass - -@dataclass -class Person: - name: str - age: pyfory.Int32 - -fory = pyfory.Fory(xlang=True) -fory.register_type(Person, name="example.Person") - -# Receive bytes from Java -person = fory.deserialize(bytes_from_java) -print(f"{person.name}, {person.age}") # Alice, 30 -``` - -## Fory IDL - -对于 Schema 优先的项目，Fory 还提供 **Fory IDL** 和代码生成。 - -- 编译器文档：[Fory IDL 概述](../../compiler/index.md) -- 最适合大型多语言消息契约和长期维护的 Schema - -### 最小 IDL 示例 - -创建 `person.fdl`： - -```protobuf -package example; - -message Person { - string name = 1; - int32 age = 2; - optional string email = 3; -} -``` - -生成代码： - -```bash -foryc person.fdl --lang java,python,cpp,go,rust,javascript,csharp,swift,dart,scala,kotlin --output ./generated -``` - -这会为所有目标生成字段和类型映射一致的原生语言类型。 - -## 何时使用 Fory IDL - -| 选项 | 适用场景 | 原因 | -| ---------------------------------- | -------------------------------------------- | ---------------------------------------------------- | -| 原生跨语言类型（无 IDL） | 只有少量消息类型，并希望快速推进 | 避免引入和运行编译器的集成与设置成本 | -| Fory IDL（Schema 优先 + 代码生成） | 多种语言、团队或服务之间存在大量消息 | 提供统一契约、更强的一致性，并使长期演进更容易 | -| 混合模式（先原生，后迁移到 IDL） | 项目初期较小，但消息数量和跨团队依赖不断增长 | 保持早期开发速度，并在 Schema 复杂度提高后实现标准化 | - -## 文档 - -| 主题 | 说明 | -| ----------------------------------------------------- | ------------------------------------ | -| [快速入门](../../start/index.md) | 所有语言的安装和基本设置 | -| [类型映射](../../specification/xlang_type_mapping.md) | 跨语言类型映射参考 | -| [类型系统](type-system.md) | 内置类型和跨运行时类型行为 | -| [类型标识](type-identity.md) | 在对等端之间协调名称和数字 ID | -| [可空性](nullability.md) | 可空字段的行为和配置 | -| [引用](references.md) | 共享引用和循环对象引用 | -| [多态](polymorphism.md) | 运行时类型选择和注册 | -| [Schema 演进](schema-evolution.md) | 兼容模式和相同 Schema 模式的选择 | -| [零拷贝](zero-copy.md) | 大型数据的带外序列化 | -| [行格式](../../row-format/index.md) | 支持随机访问且对缓存友好的二进制格式 | -| [故障排查](troubleshooting.md) | 常见问题和解决方案 | - -## 各语言指南 - -有关各语言的详细信息和 API 参考，请参阅： - -- [Java 跨语言序列化指南](../java/xlang.md) -- [Python 跨语言序列化指南](../python/xlang.md) -- [C++ 跨语言序列化指南](../cpp/xlang.md) -- [Go 跨语言序列化指南](../go/xlang.md) -- [Rust 跨语言序列化指南](../rust/xlang.md) -- [JavaScript/TypeScript 跨语言序列化指南](../javascript/xlang.md) -- [C# 跨语言序列化指南](../csharp/xlang.md) -- [Swift 跨语言序列化指南](../swift/xlang.md) -- [Dart 跨语言序列化指南](../dart/xlang.md) -- [Scala 跨语言序列化指南](../scala/xlang.md) -- [Kotlin 跨语言序列化指南](../kotlin/xlang.md) - -## 规范 - -- [跨语言序列化规范](../../specification/xlang_serialization_spec.md) - 二进制协议详情 -- [类型映射规范](../../specification/xlang_type_mapping.md) - 完整的类型映射参考 - -## 运行最佳实践 - -1. **使用一致的类型名称**：确保所有语言使用相同的类型名称或 ID -2. **启用引用跟踪**：数据包含循环引用或共享引用时启用 -3. **复用 Fory 实例**：创建 Fory 的成本较高，应复用实例 -4. **使用类型注解**：在 Python 中使用 `pyfory.Int32` 等标记实现精确类型映射 -5. **测试跨语言互操作**：验证序列化可在所有目标语言之间正常工作 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/nullability.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/nullability.md deleted file mode 100644 index 211da14ed7b..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/nullability.md +++ /dev/null @@ -1,267 +0,0 @@ ---- -title: 字段可空性 -sidebar_position: 4 -id: nullability -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本文说明 Fory 在跨语言序列化模式中如何处理字段可空性。 - -## 默认行为 - -在跨语言模式中，**字段默认不可空**。这意味着： - -- 值必须始终存在（非 null） -- 不会为字段写入 null 标志字节 -- 序列化结果更紧凑 - -以下类型默认可空： - -- `Optional` (Java, C++) -- Java 装箱类型（`Integer`、`Long`、`Double` 等） -- Go 指针类型（`*int32`、`*string` 等） -- Rust `Option` -- Python `Optional[T]` -- Scala `Option[T]` - -| 字段类型 | 默认可空 | 写入 null 标志 | -| ------------------------------------- | -------- | -------------- | -| 原始类型（`int`、`bool`、`float` 等） | 否 | 否 | -| `String` | 否 | 否 | -| `List`、`Map`、`Set` | 否 | 否 | -| 自定义结构体 | 否 | 否 | -| 枚举 | 否 | 否 | -| Java 装箱类型（`Integer`、`Long` 等） | 是 | 是 | -| Go 指针类型（`*int32`、`*string`） | 是 | 是 | -| `Optional` / `Option` | 是 | 是 | - -## 编码格式 - -可空标志控制是否在字段值之前写入 **null 标志字节**： - -``` -Non-nullable field: [value data] -Nullable field: [null_flag] [value data if not null] -``` - -其中 `null_flag` 为： - -- `-1` (NULL_FLAG)：值为 null -- `-2` (NOT_NULL_VALUE_FLAG)：值存在 - -## 可空性与引用跟踪 - -二者相关，但属于不同概念： - -| 概念 | 用途 | 标志值 | -| ------------ | -------------------- | ---------------------------------------------- | -| **可空** | 允许字段使用 null 值 | `-1`（null）、`-2`（非 null） | -| **引用跟踪** | 去重共享对象引用 | `-1`（null）、`-2`（非 null）、`≥0`（引用 ID） | - -主要区别： - -- **仅可空**：写入 `-1` 或 `-2` 标志，不进行引用去重 -- **引用跟踪**：使用引用 ID（`≥0`）扩展可空语义，以表示之前出现过的对象 -- 两者使用相同的标志字节位置——引用跟踪是可空语义的超集 - -当 `refTracking=true` 时，null 标志字节同时用作引用标志： - -``` -ref_flag = -1 → null value -ref_flag = -2 → new object (first occurrence) -ref_flag >= 0 → reference to object at index ref_flag -``` - -引用跟踪的详细行为参见[引用跟踪](references.md)。 - -## 各语言示例 - -### Java - -```java -public class Person { - // Non-nullable by default in xlang mode - String name; // Must not be null - int age; // Primitive, always non-nullable - List tags; // Must not be null - - // Explicitly nullable - @Nullable - String nickname; // Can be null - - // Optional wrapper - nullable by default - Optional bio; // Can be empty/null -} - -Fory fory = Fory.builder() - .withXlang(true) - .build(); -fory.register(Person.class, "example.Person"); -``` - -### Python - -```python -from dataclasses import dataclass -from typing import Optional, List -import pyfory - -@dataclass -class Person: - # Non-nullable by default - name: str # Must have a value - age: pyfory.Int32 # Primitive - tags: List[str] # Must not be None - - # Optional makes it nullable - nickname: Optional[str] = None # Can be None - bio: Optional[str] = None # Can be None - -fory = pyfory.Fory(xlang=True) -fory.register_type(Person, name="example.Person") -``` - -### Rust - -```rust -use fory::{Fory, ForyStruct}; - -#[derive(ForyStruct)] -struct Person { - // Non-nullable by default - name: String, - age: i32, - tags: Vec, - - // Option is nullable - nickname: Option, // Can be None - bio: Option, // Can be None -} -``` - -### Go - -```go -type Person struct { - // Non-nullable by default - Name string - Age int32 - Tags []string - - // Pointer types for nullable fields - Nickname *string // Can be nil - Bio *string // Can be nil -} - -fory := forygo.NewFory(forygo.WithXlang(true)) -fory.RegisterStructByName(Person{}, "example.Person") -``` - -### C++ - -```cpp -struct Person { - // Non-nullable by default - std::string name; - int32_t age; - std::vector tags; - - // std::optional for nullable - std::optional nickname; - std::optional bio; -}; -FORY_STRUCT(Person, name, age, tags, nickname, bio); -``` - -## 自定义可空性 - -### Java：@Nullable 注解 - -```java -public class Config { - @Nullable - String optionalSetting; // Explicitly nullable - - String requiredSetting; // Explicitly non-nullable (default) -} -``` - -### C++：FORY_STRUCT 字段配置 - -```cpp -struct Config { - std::optional optional_setting; - std::string required_setting; -}; - -FORY_STRUCT(Config, - (optional_setting, fory::F(1)), - (required_setting, fory::F(2)) -); -``` - -对于可空指针载体，使用 `.nullable()` 显式启用： - -```cpp -struct ConfigRef { - std::shared_ptr optional_setting; - std::shared_ptr required_setting; -}; - -FORY_STRUCT(ConfigRef, - (optional_setting, fory::F(1).nullable()), - (required_setting, fory::F(2)) -); -``` - -## null 值处理 - -不可空字段收到 null 值时： - -| 语言 | 行为 | -| ------ | ---------------------------------------- | -| Java | 抛出 `NullPointerException` 或序列化错误 | -| Python | 抛出 `TypeError` 或序列化错误 | -| Rust | 编译期错误（非 Option 类型不能为 None） | -| Go | 使用零值（空字符串、0 等） | -| C++ | 使用默认构造值或产生未定义行为 | - -## Schema 兼容性 - -可空标志是结构体 Schema 指纹的一部分。禁用兼容模式时，更改字段可空性属于**破坏性变更**，会导致 Schema 版本不匹配错误。 - -``` -Schema A: { name: String (non-nullable) } -Schema B: { name: String (nullable) } -// These have different fingerprints when compatible mode is disabled -``` - -在兼容模式中，如果顶层标量字段的标量类型在其他方面兼容，即使可空性或可选包装器不同，仍可匹配。存在的值通过兼容标量转换读取，并且必须满足常规的无损转换检查。远端 null 值遵循本地字段的兼容读取 null/默认值行为。 - -## 最佳实践 - -1. **默认使用不可空字段**：仅当 null 是有效的语义值时才将字段设为可空 -2. **使用 Optional/Option 包装器**：避免使用带可空注解的原始类型 -3. **保持跨语言一致**：对应字段使用相同的可空性 -4. **记录可空字段**：在 API 中明确说明哪些字段可以为 null - -## 另请参阅 - -- [引用跟踪](references.md) - 共享引用和循环引用处理 -- [跨语言概述](index.md) - 跨语言序列化工作流和运行时指南 -- [类型映射](../../specification/xlang_type_mapping.md) - 跨语言类型映射参考 -- [跨语言规范](../../specification/xlang_serialization_spec.md) - 二进制协议详情 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/polymorphism.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/polymorphism.md deleted file mode 100644 index ab3dc2880a9..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/polymorphism.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: 多态 -sidebar_position: 6 -id: polymorphism -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -当字段、元素或根值声明为更宽泛的类型时，跨语言多态会保留值的具体已注册类型。每个对等端都必须就具体类型标识和兼容字段 Schema 达成一致。 - -## 运行时规则 - -确切语法参见所选运行时的 Schema 元数据、注册和多态页面。[跨语言序列化规范](../../specification/xlang_serialization_spec.md)定义共享行为和限制。 - -不要仅根据宿主语言的继承关系推断跨语言支持。具体子类型必须具有可移植的跨语言映射，并在每个可能接收它的对等端协调注册。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/references.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/references.md deleted file mode 100644 index 04ca09d7f3d..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/references.md +++ /dev/null @@ -1,300 +0,0 @@ ---- -title: 引用跟踪 -sidebar_position: 5 -id: references -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本文说明 Fory 在跨语言序列化中如何对共享引用和循环引用进行引用跟踪。 - -## 概述 - -引用跟踪支持： - -- **共享引用**：多次引用的同一对象只序列化一次 -- **循环引用**：支持引用自身或形成环的对象 -- **内存效率**：重复对象不会产生重复数据 - -## 启用引用跟踪 - -### Java - -```java -Fory fory = Fory.builder() - .withXlang(true) - .withRefTracking(true) - .build(); -``` - -### Python - -```python -fory = pyfory.Fory(xlang=True, ref=True) -``` - -### Go - -```go -fory := forygo.NewFory( - forygo.WithXlang(true), - forygo.WithTrackRef(true), -) -``` - -### C++ - -```cpp -auto fory = fory::serialization::Fory::builder().xlang(true).track_ref(true).build(); -``` - -### Rust - -```rust -let fory = Fory::builder() - .xlang(true) - .track_ref(true).build(); -``` - -### Scala - -```scala -import org.apache.fory.scala.ForyScala - -val fory = ForyScala.builder() - .withXlang(true) - .withRefTracking(true) - .build() -``` - -## 编码格式 - -启用引用跟踪后，可空字段会在值之前写入一个**引用标志字节**： - -``` -[ref_flag] [value data if not null/ref] -``` - -其中 `ref_flag` 为： - -| 值 | 含义 | -| -------------------------- | ----------------------------- | -| `-1` (NULL_FLAG) | 值为 null | -| `-2` (NOT_NULL_VALUE_FLAG) | 值存在，且是首次出现 | -| `≥0` | 指向之前已序列化对象的引用 ID | - -## 引用跟踪与可空性 - -二者是**相互独立**的概念： - -| 概念 | 用途 | 控制方式 | -| ------------ | -------------------- | ------------------------------- | -| **可空性** | 字段能否保存 null 值 | 字段类型（`Optional`）或注解 | -| **引用跟踪** | 是否对重复对象去重 | 全局 `refTracking` 选项 | - -关键行为： - -- 引用标志字节**只为可空字段写入** -- 即使设置 `refTracking=true`，不可空字段也完全跳过引用标志 -- 引用去重只适用于多次出现的对象 - -```java -// Reference tracking enabled, but non-nullable fields still skip ref flags -Fory fory = Fory.builder() - .withXlang(true) - .withRefTracking(true) - .build(); -``` - -## 按字段进行引用跟踪 - -默认情况下，即使全局设置了 `refTracking=true`，**大多数字段也不会跟踪引用**。只有特定的指针或智能指针类型默认跟踪引用。 - -### 各语言的默认行为 - -| 语言 | 默认引用跟踪 | 默认跟踪引用的类型 | -| ------ | ------------ | ---------------------------------------------------------- | -| Java | 否 | 无（使用注解启用） | -| Python | 否 | 无（使用注解启用） | -| Go | 否 | 无（使用 `fory:"ref"` 启用） | -| C++ | 是 | `std::shared_ptr`、`fory::serialization::SharedWeak` | -| Rust | 否 | `Rc`、`Arc`、`Weak` | -| Scala | 否 | 无（使用 `@Ref` 启用） | - -### 自定义按字段引用跟踪 - -#### Java：@Ref 注解 - -```java -public class Document { - // Default: no ref tracking - String title; - - // Enable ref tracking for this field - @Ref - Author author; - - // Shared across documents, track refs to avoid duplicates - List<@Ref Tag> tags; -} -``` - -#### C++：FORY_STRUCT 字段配置 - -```cpp -struct Document { - std::string title; - - // shared_ptr/SharedWeak track refs by default - std::shared_ptr author; - fory::serialization::SharedWeak data; - - std::shared_ptr tag_owner; -}; -FORY_STRUCT(Document, - title, - author, - data, - (tag_owner, fory::F().ref()) -); -``` - -要完全禁用 C++ 引用跟踪，请在序列化器上设置 -`Fory::builder().xlang(true).track_ref(false).build()`。 - -#### Rust：字段属性 - -```rust -use fory::ForyStruct; -use std::rc::Rc; - -#[derive(ForyStruct)] -struct Document { - title: String, - - // Rc/Arc track refs by default - author: Rc, - - // Explicitly enable ref tracking - #[fory(ref = true)] - tags: Vec, -} -``` - -#### Scala：@Ref 注解 - -Scala Schema IDL 和 Scala 3 宏派生使用同一个共享 JVM `@Ref` 注解： - -```scala -import org.apache.fory.annotation.{ForyField, ForyStruct, Ref} -import org.apache.fory.scala.ForySerializer - -@ForyStruct -final class Node() derives ForySerializer { - @ForyField(id = 1) - var children: List[Node @Ref] = List.empty - - @Ref - @ForyField(id = 2) - var parent: Option[Node] = None -} -``` - -在 Scala 中，顶层字段的引用跟踪由字段或构造函数参数上的 `@Ref` 控制。类型使用位置的 `T @Ref` 用于嵌套元素、值或载荷引用，例如 `List[Node @Ref]`。 - -#### Go：结构体标签 - -```go -type Document struct { - Title string - - // Enable ref tracking for pointer to struct - Author *Author `fory:"ref"` - - // Enable ref tracking for slice - Tags []Tag `fory:"ref"` -} -``` - -### 何时启用按字段引用跟踪 - -对符合以下条件的字段启用引用跟踪： - -- 可能多次包含同一个对象实例 -- 属于循环引用链的一部分 -- 保存可能被共享的大型对象 - -对符合以下条件的字段禁用引用跟踪（或保留默认设置）： - -- 始终包含唯一值 -- 属于原始类型或简单值类型 -- 不参与对象共享 - -## 示例：共享引用 - -```java -public class Container { - List data; - List sameData; // Points to same list -} - -Container obj = new Container(); -obj.data = Arrays.asList("a", "b", "c"); -obj.sameData = obj.data; // Shared reference - -// With refTracking=true: data serialized once, sameData stores reference ID -// With refTracking=false: data serialized twice (duplicate) -``` - -## 示例：循环引用 - -```java -public class Node { - String value; - Node next; -} - -Node a = new Node("A"); -Node b = new Node("B"); -a.next = b; -b.next = a; // Circular reference - -// With refTracking=true: works correctly -// With refTracking=false: infinite recursion error -``` - -## 语言支持 - -| 语言 | 共享引用 | 循环引用 | -| ---------- | -------- | ---------------- | -| Java | 是 | 是 | -| Python | 是 | 是 | -| Go | 是 | 是 | -| C++ | 是 | 是 | -| JavaScript | 是 | 是 | -| Rust | 是 | 否（所有权规则） | - -## 性能注意事项 - -- **开销**：引用跟踪会为每个对象增加一次哈希表查找 -- **何时启用**：数据包含共享引用或循环引用时启用 -- **何时禁用**：用于不含共享关系的简单数据结构时禁用 - -## 另请参阅 - -- [字段可空性](nullability.md) - 可空性如何影响序列化 -- [跨语言概述](index.md) - 跨语言序列化工作流和运行时指南 -- [跨语言规范](../../specification/xlang_serialization_spec.md) - 二进制协议详情 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/schema-evolution.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/schema-evolution.md deleted file mode 100644 index 76b37ef956f..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/schema-evolution.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: Schema 演进 -sidebar_position: 3 -id: schema-evolution -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -跨语言兼容模式携带 Schema 元数据，使读取端可以容忍受支持的字段新增、删除和重排。只有每个读取端和写入端都使用相同 Schema 时，相同 Schema 模式才能减少元数据。 - -## 选择兼容模式或相同 Schema 模式 - -独立部署的对等端应保持兼容模式。只有确认每个运行时都使用相同的字段标识、可空性、引用元数据和类型后，才使用相同 Schema 模式。 - -运行时 API 和示例位于各运行时的 `schema-evolution.md` 页面。规范性 Schema 元数据和兼容行为由[跨语言序列化规范](../../specification/xlang_serialization_spec.md)定义。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/troubleshooting.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/troubleshooting.md deleted file mode 100644 index 808c3e3304b..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/troubleshooting.md +++ /dev/null @@ -1,313 +0,0 @@ ---- -title: 故障排查 -sidebar_position: 90 -id: troubleshooting -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -本文介绍使用跨语言序列化时的常见问题及解决方案。 - -## 类型注册错误 - -### “Type not registered”错误 - -**现象：** - -``` -Error: Type 'example.Person' is not registered -``` - -**原因：**反序列化前未注册类型，或类型名称不匹配。 - -**解决方案：** - -1. 确保两端使用相同名称注册类型： - - ```java - // Java - fory.register(Person.class, "example.Person"); - ``` - - ```python - # Python - fory.register_type(Person, name="example.Person") - ``` - -2. 检查类型名称是否存在拼写错误或大小写差异 - -3. 在任何序列化或反序列化调用之前注册类型 - -### “Type ID mismatch”错误 - -**现象：** - -``` -Error: Expected type ID 100, got 101 -``` - -**原因：**不同语言使用了不同的类型 ID。 - -**解决方案：**使用一致的类型 ID： - -```java -// Java -fory.register(Person.class, 100); -fory.register(Address.class, 101); -``` - -```python -# Python -fory.register_type(Person, type_id=100) -fory.register_type(Address, type_id=101) -``` - -## 类型映射问题 - -### 整数溢出 - -**现象：**值被意外截断或回绕。 - -**原因：**不同语言使用了不同宽度的整数。 - -**解决方案：** - -1. 在 Python 中使用显式类型注解： - - ```python - @dataclass - class Data: - value: pyfory.Int32 # Not just 'int' - ``` - -2. 确保整数范围兼容： - - `int8`: -128 to 127 - - `int16`: -32,768 to 32,767 - - `int32`: -2,147,483,648 to 2,147,483,647 - -### 浮点精度损失 - -**现象：**浮点值的精度与预期不符。 - -**原因：**混用了 `float32` 和 `float64` 类型。 - -**解决方案：** - -1. 使用一致的浮点类型： - - ```python - @dataclass - class Data: - value: pyfory.Float32 # Explicit 32-bit float - ``` - -2. 注意 Python 的 `float` 默认映射为 `float64` - -### 字符串编码错误 - -**现象：** - -``` -Error: Invalid UTF-8 sequence -``` - -**原因：**字符串未使用 UTF-8 编码。 - -**解决方案：** - -1. 确保所有字符串都是有效的 UTF-8 -2. 在 Python 中，序列化前先解码字节： - - ```python - text = raw_bytes.decode('utf-8') - ``` - -## 字段顺序问题 - -### “Field mismatch”错误 - -**现象：**反序列化对象的字段值错误。 - -**原因：**不同语言的字段顺序不同。 - -**解决方案：**Fory 按字段的 snake_case 名称排序。请确保字段名称一致： - -```java -// Java - fields will be sorted: age, email, name -public class Person { - public String name; - public int age; - public String email; -} -``` - -```python -# Python - same field order -@dataclass -class Person: - name: str - age: pyfory.Int32 - email: str -``` - -## 引用跟踪问题 - -### 循环引用导致栈溢出 - -**现象：** - -``` -StackOverflowError or RecursionError -``` - -**原因：**引用跟踪已禁用，但数据包含循环引用。 - -**解决方案：**启用引用跟踪： - -```java -// Java -Fory fory = Fory.builder() - .withRefTracking(true) - .build(); -``` - -```python -# Python -fory = pyfory.Fory(ref=True) -``` - -### 对象重复 - -**现象：**反序列化后，共享对象变成了多个副本。 - -**原因：**引用跟踪已禁用。 - -**解决方案：**对象图中存在共享对象时启用引用跟踪。 - -## 跨语言类型问题 - -### 跨语言模式中的不兼容类型 - -**现象：** - -``` -Error: Type 'Optional' is not supported in xlang mode -``` - -**原因：**使用了没有跨语言对应类型的 Java 专属类型。 - -**解决方案：**使用兼容类型： - -```java -// Instead of Optional -public String email; // nullable - -// Instead of BigDecimal -public double amount; - -// Instead of EnumSet -public Set statuses; -``` - -## 版本兼容性 - -### Schema 哈希不匹配 - -**现象：**反序列化失败，并出现 `class version hash mismatch`、`schema version mismatch`、`struct version mismatch` 或 `hash mismatch` 等错误。 - -**原因：**写入端和读取端禁用了兼容模式，但它们的结构体/类 Schema 不同。在跨语言模式下，即使每种语言都进行了合理的本地更改，也可能出现这种情况，因为字段名称、类型注解、字段 ID、可空性和生成的 Schema 元数据仍必须完全一致。 - -**解决方案：** - -1. 仔细对齐每个服务和语言中的 Schema：字段名称或字段 ID、字段顺序、类型注解、可空性以及类型注册 ID/名称。 -2. 当前实现的跨语言模式默认使用兼容模式。如果某个对等端显式选择了 `compatible=false`，请移除该覆盖设置，或在每个对等端启用兼容模式。兼容模式会写入额外的 Schema 元数据，因此载荷更大，但建议可能独立演进的跨语言服务使用该模式。 -3. 只有每个读取端和写入端始终使用相同 Schema 时，才设置 `compatible=false`。对于跨语言载荷，只有确认每种语言都使用该 Schema，或原生类型由 Fory Schema IDL 生成时才这样做。 - -### 序列化格式已更改 - -**现象：**升级 Fory 后反序列化失败。 - -**原因：**序列化格式发生了破坏性变更。 - -**解决方案：** - -1. 确保所有服务使用兼容的 Fory 版本 -2. 查看发行说明中的破坏性变更 -3. 考虑使用 Schema 演进（兼容模式）进行渐进式升级 - -## 调试技巧 - -### 启用调试日志 - -**Java:** - -```java -// Add to JVM options --Dfory.debug=true -``` - -**Python:** - -```python -import logging -logging.getLogger('pyfory').setLevel(logging.DEBUG) -``` - -### 检查序列化数据 - -使用十六进制转储检查二进制格式： - -```python -data = fory.serialize(obj) -print(data.hex()) -``` - -### 测试往返序列化 - -始终在每种语言中测试往返序列化： - -```java -byte[] bytes = fory.serialize(obj); -Object result = fory.deserialize(bytes); -assert obj.equals(result); -``` - -### 跨语言测试 - -部署前测试所有目标语言之间的序列化： - -```bash -# Serialize in Java -java -jar serializer.jar > data.bin - -# Deserialize in Python -python deserializer.py data.bin -``` - -## 常见错误 - -1. **未注册类型**：始终在使用前注册自定义类型 -2. **类型名称/ID 不一致**：所有语言使用相同的名称/ID -3. **混用跨语言和原生载荷**：确保每个对等端都使用跨语言编码格式 -4. **类型注解错误**：在 Python 中使用 `pyfory.Int32` 等标记 -5. **忽略引用跟踪**：存在循环引用或共享引用时启用 - -## 另请参阅 - -- [类型映射](../../specification/xlang_type_mapping.md) - 跨语言类型映射参考 -- [快速入门](../../start/index.md) - 运行时设置指南 -- [Java 故障排查](../java/troubleshooting.md) - Java 专属问题 -- [Python 故障排查](../python/troubleshooting.md) - Python 专属问题 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/type-identity.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/type-identity.md deleted file mode 100644 index 33068117613..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/type-identity.md +++ /dev/null @@ -1,295 +0,0 @@ ---- -title: 类型标识与动态字段 -sidebar_position: 2 -id: type-identity -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -字段类型元信息配置控制序列化结构体字段时是否写入类型信息。当实际具体类型可能不同于声明的字段类型时，这是支持多态的关键。 - -## 概述 - -序列化结构体字段时，Fory 需要确定是否写入类型元数据： - -- **静态类型**：直接使用声明字段类型的序列化器（不写入类型信息） -- **动态类型**：写入类型信息以支持子类型 - -## 何时需要类型元信息 - -以下情况需要类型元数据： - -1. **接口/抽象字段**：声明类型是抽象类型，因此必须记录具体类型 -2. **多态字段**：具体类型可能是声明类型的子类 -3. **跨语言兼容性**：接收端需要类型信息才能正确反序列化 - -以下情况不需要类型元数据： - -1. **final/具体类型**：声明类型是 final/sealed，无法被继承 -2. **原始类型**：类型在编译期已知 -3. **性能优化**：确定具体类型始终与声明类型一致 - -## 各语言配置 - -### Java - -Java 需要显式配置，因为具体类除非标记为 `final`，否则仍可被继承。 - -使用 `@ForyField` 注解的 `dynamic` 参数： - -```java -import org.apache.fory.annotation.ForyField; -import org.apache.fory.annotation.ForyField.Dynamic; - -public class Container { - // AUTO (default): Interface types write type info, concrete types don't - @ForyField(id = 0) - private Shape shape; // Interface - type info written - - // FALSE: Never write type info (use declared type's serializer) - @ForyField(id = 1, dynamic = Dynamic.FALSE) - private Circle circle; // Always treated as Circle - - // TRUE: Always write type info (support subtypes) - @ForyField(id = 2, dynamic = Dynamic.TRUE) - private Shape concreteShape; // Type info written even if concrete -} -``` - -**动态选项**： - -| 值 | 行为 | -| ------- | ---------------------------------------- | -| `AUTO` | 接口/抽象类型为动态类型，具体类型不是 | -| `FALSE` | 从不写入类型信息，使用声明类型的序列化器 | -| `TRUE` | 始终写入类型信息以支持子类型 | - -**使用场景**： - -- `AUTO`：默认行为，适用于大多数场景 -- `FALSE`：已知确切类型时用于性能优化 -- `TRUE`：具体字段可能保存子类实例时使用 - -### C++ - -C++ 使用 `.dynamic(bool)` 构建器方法，并将其放在 `FORY_STRUCT` 中： - -```cpp -#include "fory/serialization/fory.h" - -// Abstract base class with pure virtual methods -struct Animal { - virtual ~Animal() = default; - virtual std::string speak() const = 0; -}; - -struct Zoo { - // Auto: type info written because Animal is polymorphic (std::is_polymorphic) - std::shared_ptr animal; - - // Force non-dynamic: skip type info even though Animal is polymorphic - std::shared_ptr fixed_animal; - - // Force dynamic: write type info even for non-polymorphic types - std::shared_ptr polymorphic_data; -}; -FORY_STRUCT(Zoo, - (animal, fory::F(0).nullable()), // Auto-detect polymorphism - (fixed_animal, fory::F(1).nullable().dynamic(false)), // Skip type info - (polymorphic_data, fory::F(2).dynamic(true)) // Force type info -); -``` - -**默认行为**：Fory 通过 `std::is_polymorphic` 自动检测多态。带纯虚方法的类型默认视为动态类型。 - -### Go 和 Rust - -Go 和 Rust **不需要**显式动态配置，因为： - -- **Go**：接口类型本身就是动态类型，Fory 可以根据类型判断它是否为接口 -- **Rust**：特征对象（`dyn Trait`）在类型系统中有显式标记 - -这些语言的类型系统已经能够表明字段是否为多态字段： - -```go -// Go: interface types are automatically dynamic -type Container struct { - Shape Shape // Interface - type info written automatically - Circle Circle // Concrete struct - no type info needed -} -``` - -```rust -// Rust: trait objects are explicitly marked -struct Container { - shape: Box, // Trait object - type info written automatically - circle: Circle, // Concrete type - no type info needed -} -``` - -### Python - -使用 `pyfory.field()` 的 `dynamic` 参数： - -```python -from dataclasses import dataclass -from abc import ABC, abstractmethod -import pyfory - -class Shape(ABC): - @abstractmethod - def area(self) -> float: - pass - -@dataclass -class Circle(Shape): - radius: float = 0.0 - - def area(self) -> float: - return 3.14159 * self.radius * self.radius - -@dataclass -class Container: - # Abstract class: dynamic is always True (type info written) - shape: Shape = pyfory.field(id=0) - - # Concrete type with explicit dynamic=True (force type info) - circle: Circle = pyfory.field(id=1, dynamic=True) - - # Concrete type with explicit dynamic=False (skip type info) - fixed_circle: Circle = pyfory.field(id=2, dynamic=False) -``` - -**默认行为**： - -| 模式 | 抽象类 | 具体对象类型 | 数值/str/time 类型 | -| ---------- | ------ | ------------ | ------------------ | -| 原生模式 | `True` | `True` | `False` | -| 跨语言模式 | `True` | `False` | `False` | - -- **抽象类**：`dynamic` 始终为 `True`（必须写入类型信息） -- **原生模式**：对象类型的 `dynamic` 默认为 `True`，数值/str/time 类型默认为 `False` -- **跨语言模式**：具体类型的 `dynamic` 默认为 `False` - -## 默认行为 - -| 语言 | 接口/抽象类型 | 具体类型 | -| ------ | ----------------- | -------------- | -| Java | 动态（写入类型） | 静态（无类型） | -| C++ | 动态（virtual） | 静态 | -| Go | 动态（interface） | 静态（struct） | -| Rust | 动态（dyn Trait） | 静态 | -| Python | 动态（所有对象） | 动态 | - -## 性能注意事项 - -写入类型元数据会产生开销： - -- **空间**：类型信息会增加序列化输出的字节数 -- **时间**：序列化和反序列化期间需要解析类型 - -以下情况使用 `dynamic = FALSE`（Java）或 `dynamic(false)`（C++）： - -- 确定具体类型与声明类型一致 -- 性能至关重要且不需要多态 -- 字段类型实际上是 final - -## 跨语言兼容性 - -为跨语言消费序列化数据时： - -1. **使用一致的类型注册**：各语言使用相同的 ID 注册类型 -2. **优先显式配置**：不确定接收端预期时使用 `dynamic = TRUE` -3. **记录多态字段**：明确说明哪些字段可能包含子类型 - -## 示例：多态容器 - -### Java - -```java -public interface Animal { - String speak(); -} - -public class Dog implements Animal { - private String name; - - @Override - public String speak() { return "Woof!"; } -} - -public class Cat implements Animal { - private String name; - - @Override - public String speak() { return "Meow!"; } -} - -public class Zoo { - // Type info written because Animal is an interface - @ForyField(id = 0) - private Animal animal; - - // Force type info for concrete type that may hold subtypes - @ForyField(id = 1, dynamic = Dynamic.TRUE) - private Dog maybeMixedBreed; -} -``` - -### C++ - -```cpp -// Abstract base class with pure virtual methods -class Animal { -public: - virtual std::string speak() const = 0; - virtual ~Animal() = default; -}; - -class Dog : public Animal { -public: - std::string name; - std::string speak() const override { return "Woof!"; } -}; - -struct Zoo { - std::shared_ptr animal; - std::shared_ptr maybe_mixed_breed; -}; - -FORY_STRUCT(Zoo, - (animal, fory::F(0).nullable()), // Auto-detect (Animal is polymorphic) - (maybe_mixed_breed, fory::F(1).dynamic(true)) // Force dynamic for concrete type -); -``` - -## 相关主题 - -- [字段可空性](nullability.md) - 控制字段的 null 处理 -- [引用跟踪](references.md) - 管理共享引用和循环引用 -- [类型映射](../../specification/xlang_type_mapping.md) - 跨语言类型兼容性 - -## 在对等端之间协调类型标识 - -每个对等端都必须使用相同的数字 ID，或相同的命名空间和类型名称注册自定义类型。数字 ID 生成的元数据更小；名称则更容易在独立部署的服务之间协调。不要在一个对等端使用 ID 注册，而在另一个对等端使用名称注册。 - -确切的注册 API 请参阅所选运行时指南： -[Java](../java/xlang.md)、[Python](../python/xlang.md)、 -[C++](../cpp/xlang.md)、[Go](../go/xlang.md)、 -[Rust](../rust/xlang.md)、[JavaScript](../javascript/xlang.md)、 -[C#](../csharp/xlang.md)、[Swift](../swift/xlang.md)、 -[Dart](../dart/xlang.md)、[Scala](../scala/xlang.md) 和 -[Kotlin](../kotlin/xlang.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/type-system.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/type-system.md deleted file mode 100644 index 642f5d22a62..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/type-system.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: 类型系统 -sidebar_position: 1 -id: type-system -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -## 序列化内置类型 - -常见类型无需注册即可自动序列化，包括原始数字类型、字符串、二进制、数组、列表、映射等。 - -低精度浮点值也是内置跨语言类型系统的一部分： - -- `float16` 和 `array` -- `bfloat16` 和 `array` - -请使用类型映射参考中记录的语言专用载体类型。Python 仅将 `pyfory.Float16` 和 `pyfory.BFloat16` 用作注解标记；标量值是原生 Python `float`，稠密低精度数组使用 `pyfory.Float16Array` 和 `pyfory.BFloat16Array`。Go 使用 `float16` 和 `bfloat16` 包提供标量、切片和数组载体；JavaScript 使用 `number` 表示标量 `float16` 和 `bfloat16`，并使用稠密数组载体 `BoolArray`、`Float16Array` 和 `BFloat16Array` 表示相应的 `array` Schema。Dart 使用 `double` 配合 `Float16Type` 或 `Bfloat16Type` 元数据表示标量字段，稠密数组则使用 `Float16List` / `Bfloat16List`。Java 在受支持的低精度载体上使用 `@ArrayType` 表示 `array` / `array` Schema，而普通对象数组仍走 `list` 路径；C++、Rust 和 C# 提供各自专用的标量与数组载体。 - -设置 `compatible=true` 后，直接结构体/类字段可以在 `list` 和 `array` 之间演进，其中 `T` 为稠密布尔/数字类型。具有相同符号性和宽度范围的整数列表元素编码与相应的稠密数组元素范围匹配。该规则仅适用于直接匹配的字段 Schema，不适用于嵌套集合、映射、数组、联合或泛型位置。当实际载荷没有 null 元素时，可以将对等端的 `list` Schema 读取到本地 `array` 字段。如果载荷包含 null 元素或启用引用跟踪的元素编码，读取到本地 `array` 字段会引发兼容读取错误。 - -## 序列化自定义类型 - -用户定义类型必须使用注册 API 进行注册，以建立不同语言类型之间的映射关系。所有语言都应使用一致的类型名称。 - -## 精确映射 - -规范性[跨语言类型映射](../../specification/xlang_type_mapping.md)定义每个运行时的精确载体映射。运行时页面展示该映射的 API 语法和示例。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/zero-copy.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/zero-copy.md deleted file mode 100644 index 54a8d901dbd..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/object-serialization/xlang/zero-copy.md +++ /dev/null @@ -1,203 +0,0 @@ ---- -title: 零拷贝序列化 -sidebar_position: 7 -id: zero-copy -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -零拷贝序列化允许将大型二进制数据（字节数组、数字数组）带外序列化，从而避免内存复制并降低序列化开销。 - -## 何时使用零拷贝 - -以下情况使用零拷贝序列化： - -- 序列化大型字节数组或二进制块 -- 处理数字数组（int[]、double[] 等） -- 通过高性能网络传输数据 -- 对内存效率要求很高 - -## 工作原理 - -1. **序列化**：提取大型缓冲区，并通过回调单独返回 -2. **传输**：分别传输主序列化数据和缓冲区对象 -3. **反序列化**：重新提供缓冲区以重建原始对象 - -这样可以避免将大型数据复制到主序列化缓冲区。 - -## Java - -```java -import org.apache.fory.*; -import org.apache.fory.config.*; -import org.apache.fory.serializer.BufferObject; -import org.apache.fory.memory.MemoryBuffer; - -import java.util.*; -import java.util.stream.Collectors; - -public class ZeroCopyExample { - public static void main(String[] args) { - Fory fory = Fory.builder().withXlang(true).build(); - - // Data with large arrays - List list = List.of( - "str", - new byte[1000], // Large byte array - new int[100], // Large int array - new double[100] // Large double array - ); - - // Collect buffer objects during serialization - Collection bufferObjects = new ArrayList<>(); - byte[] bytes = fory.serialize(list, e -> !bufferObjects.add(e)); - - // Convert to buffers for transport - List buffers = bufferObjects.stream() - .map(BufferObject::toBuffer) - .collect(Collectors.toList()); - - // Deserialize with buffers - Object result = fory.deserialize(bytes, buffers); - System.out.println(result); - } -} -``` - -## Python - -```python -import array -import pyfory -import numpy as np - -fory = pyfory.Fory(xlang=True) - -# Data with large arrays -data = [ - "str", - bytes(bytearray(1000)), # Large byte array - array.array("i", range(100)), # Large int array - np.full(100, 0.0, dtype=np.double) # Large numpy array -] - -# Collect buffer objects during serialization -serialized_objects = [] -serialized_data = fory.serialize(data, buffer_callback=serialized_objects.append) - -# Convert to buffers for transport -buffers = [obj.to_buffer() for obj in serialized_objects] - -# Deserialize with buffers -result = fory.deserialize(serialized_data, buffers=buffers) -print(result) -``` - -## Go - -```go -package main - -import forygo "github.com/apache/fory/go/fory" -import "fmt" - -func main() { - serializer := forygo.NewFory(forygo.WithXlang(true)) - - // Data with large arrays - list := []any{ - "str", - make([]byte, 1000), // Large byte array - } - - buf := forygo.NewByteBuffer(nil) - var bufferObjects []forygo.BufferObject - - // Collect buffer objects during serialization - if err := serializer.SerializeWithCallback(buf, list, func(o forygo.BufferObject) bool { - bufferObjects = append(bufferObjects, o) - return false - }); err != nil { - panic(err) - } - - // Convert to buffers for transport - var buffers []*forygo.ByteBuffer - for _, o := range bufferObjects { - buffers = append(buffers, o.ToBuffer()) - } - - // Deserialize with buffers - var newList []any - if err := serializer.DeserializeWithCallbackBuffers(buf, &newList, buffers); err != nil { - panic(err) - } - fmt.Println(newList) -} -``` - -## 使用场景 - -### 高性能数据传输 - -通过网络发送大型数据集时： - -```java -// Sender -Collection buffers = new ArrayList<>(); -byte[] metadata = fory.serialize(dataObject, e -> !buffers.add(e)); - -// Send metadata and buffers separately -network.sendMetadata(metadata); -for (BufferObject buf : buffers) { - network.sendBuffer(buf.toBuffer()); -} - -// Receiver -byte[] metadata = network.receiveMetadata(); -List buffers = network.receiveBuffers(); -Object data = fory.deserialize(metadata, buffers); -``` - -### 内存映射文件 - -零拷贝非常适合内存映射文件： - -```java -// Write -Collection buffers = new ArrayList<>(); -byte[] data = fory.serialize(largeObject, e -> !buffers.add(e)); -writeToFile("data.bin", data); -for (int i = 0; i < buffers.size(); i++) { - writeToFile("buffer" + i + ".bin", buffers.get(i).toBuffer()); -} - -// Read -byte[] data = readFromFile("data.bin"); -List buffers = readBufferFiles(); -Object result = fory.deserialize(data, buffers); -``` - -## 性能注意事项 - -1. **阈值**：由于回调开销，小型数组可能无法从零拷贝中受益 -2. **网络**：缓冲区可以无复制发送时，零拷贝的收益最大 -3. **内存**：避免缓冲区复制，从而降低峰值内存用量 - -## 另请参阅 - -- [跨语言概述](index.md) - 标准序列化工作流和运行时指南 -- [Python 带外指南](../python/out-of-band.md) - Python 专用零拷贝详情 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/compact.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/compact.md deleted file mode 100644 index 98bfcffb602..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/compact.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -title: Compact Row -sidebar_position: 2 -id: compact -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Compact Row 是仅支持 Java 的 Row 编码，可减少固定槽位和 null bitmap 开销。它与 Standard Row -的编码格式不兼容。 - -## 创建 Compact Encoder - -```java -RowEncoder encoder = - Encoders.buildBeanCodec(MyBean.class) - .compactEncoding() - .build() - .get(); - -BinaryRow row = encoder.toRow(value); -MyBean decoded = encoder.fromRow(row); -``` - -在单个线程中复用 encoder。并发线程应分别创建 encoder。 - -## 布局权衡 - -- 固定大小字段使用自然宽度，而不是 Standard Row 的八字节槽位。 -- 字段按对齐要求排序，以减少 padding。 -- 没有可空字段时省略 null bitmap。 -- 固定大小的嵌套 struct 可以内联存储。 - -仅当所有 reader 都使用 Java，且节省的空间足以抵消 Java 特有布局的限制时，才应选择 Compact Row。 -Java/Python/C++/Rust 互操作请使用 [Standard Row](standard.md)。 - -精确二进制布局请参阅 [Row Format 规范](../specification/row_format_spec.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/cpp.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/cpp.md index 11b955355c4..28812940ff8 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/cpp.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/cpp.md @@ -504,6 +504,6 @@ int32_t id = row.get_int32(0); ## 相关主题 - [C++ 行格式示例](https://github.com/apache/fory/tree/main/examples/cpp/hello_row) - 完整的可运行示例 -- [基本序列化](../object-serialization/cpp/core-api.md) - 对象图序列化 +- [基本序列化](../object-serialization/cpp/basic-serialization.md) - 对象图序列化 - [C++ 对象序列化配置](../object-serialization/cpp/configuration.md) - 构建器选项 - [C++ 对象序列化支持的类型](../object-serialization/cpp/supported-types.md) - 对象序列化类型 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/index.md index 542e9cfb4dd..c3db15c2c7e 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/index.md @@ -1,5 +1,5 @@ --- -title: Row Format +title: 介绍 sidebar_position: 0 id: index license: | @@ -19,24 +19,77 @@ license: | limitations under the License. --- -Row Format 使用缓存友好的二进制布局存储有类型的值，无需重建完整对象图即可随机或部分访问。 -它适用于分析和内存数据处理。 +Row Format 是一种缓存友好的二进制格式，可实现高效的随机访问和部分反序列化。与对象图 +序列化不同，它允许读取方无需重建完整对象即可访问单个字段。 -## 选择 Row 系列 +Row Format 仅用于可信的分析数据，包括内存映射数据、选择性字段访问和数据管道。如果应用 +需要通用对象图、共享或循环引用，或者以完整对象重建为主要访问方式，请使用 +[对象序列化](../object-serialization/index.md)。 -| 系列 | 运行时支持 | 兼容性 | -| --------------------------- | ----------------------- | --------------------------- | -| [Standard Row](standard.md) | Java、Python、C++、Rust | 共享的 Standard Row 布局 | -| [Compact Row](compact.md) | Java | 仅 Java、面向空间优化的布局 | +## 选择布局 -如果目标是完整对象重建、引用处理或通用应用消息传递，请使用二进制对象序列化。如果工作负载需要 -直接从编码数据中读取选定字段、嵌套数组或 Map，请使用 Row Format。 +| 布局 | 运行时支持 | 兼容性 | +| ------------ | ----------------------- | --------------------------- | +| Standard Row | Java、Python、C++、Rust | 共享的 Standard Row 布局 | +| Compact Row | Java | 仅 Java、面向空间优化的布局 | -## 运行时指南 +## Standard Row -- [Java](java.md) -- [Python](python.md) -- [C++](cpp.md) -- [Rust](rust.md) +Standard Row 是 Java、Python、C++ 和 Rust 之间可互操作的布局。 -规范性 [Row Format 规范](../specification/row_format_spec.md)定义了 Standard 和 Compact 布局。 +### 功能特性 + +- **零拷贝随机访问**：直接从编码数据中读取选定字段。 +- **部分反序列化**：只重建应用所需的值。 +- **跨语言兼容**：在 Java、Python、C++ 和 Rust 之间共享 Standard Row 字节。 +- **Apache Arrow 集成**：在 Java 和 Python 中将 Row 转换为 Arrow 数据。 + +### 布局 + +Standard Row 内联存储固定宽度值，并通过偏移量和大小存储变长值。Row、数组和 Map 使用 +Schema 解析字段位置和元素类型。规范性的字节布局、对齐规则、类型表和字节序由 +[Row Format 规范](../specification/row_format_spec.md)定义。 + +### 运行时支持 + +| 运行时 | Standard Row 兼容性 | 运行时指南 | 其他集成 | +| ------ | ------------------- | ------------------- | ------------------------------ | +| Java | 兼容 | [Java](java.md) | Arrow 转换；接口和扩展类型映射 | +| Python | 兼容 | [Python](python.md) | PyArrow Schema 和 table 转换 | +| C++ | 兼容 | [C++](cpp.md) | 原生 Row 读取器和写入器 | +| Rust | 兼容 | [Rust](rust.md) | 借用式结构体、数组和 Map 视图 | + +安装、Schema 构建、编码、随机访问、部分读取和特定语言集成请参阅各运行时指南。 + +## Compact Row + +Compact Row 是仅支持 Java 的 Row 编码，可减少固定槽位和 null bitmap 开销。它与 Standard Row +的编码格式不兼容。 + +### 创建 Compact Encoder + +```java +RowEncoder encoder = + Encoders.buildBeanCodec(MyBean.class) + .compactEncoding() + .build() + .get(); + +BinaryRow row = encoder.toRow(value); +MyBean decoded = encoder.fromRow(row); +``` + +在单个线程中复用 encoder。并发线程应分别创建 encoder。 + +### 布局权衡 + +- 固定大小字段使用自然宽度，而不是 Standard Row 的八字节槽位。 +- 字段按对齐要求排序，以减少 padding。 +- 没有可空字段时省略 null bitmap。 +- 固定大小的嵌套 struct 可以内联存储。 + +仅当所有 reader 都使用 Java，且节省的空间足以抵消 Java 特有布局的限制时，才应选择 +Compact Row。Java、Python、C++ 和 Rust 互操作请使用 Standard Row。 + +精确的 Standard 和 Compact 二进制布局请参阅 +[Row Format 规范](../specification/row_format_spec.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/java.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/java.md index 5608e2b7385..64c14228f60 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/java.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/java.md @@ -269,6 +269,6 @@ Parent decoded = encoder.fromRow(row); ## 相关主题 -- [Xlang 序列化](../object-serialization/java/xlang.md) - xlang 模式 +- [跨语言互操作](../object-serialization/java/basic-serialization.md#cross-language-interoperability) - xlang 模式 - [Java 高级功能](../object-serialization/java/advanced-features.md) - 零拷贝对象序列化 - [行格式规范](https://fory.apache.org/docs/specification/row_format_spec) - 协议详情 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/python.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/python.md index d7c8379e35c..70c23bff308 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/python.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/python.md @@ -191,6 +191,6 @@ pip install pyfory[format] ## 相关主题 -- [Xlang 序列化](../object-serialization/python/xlang.md) - xlang 模式 -- [基本序列化](../object-serialization/python/core-api.md) - 对象序列化 +- [跨语言互操作](../object-serialization/python/basic-serialization.md#cross-language-interoperability) - xlang 模式 +- [基本序列化](../object-serialization/python/basic-serialization.md) - 对象序列化 - [行格式规范](https://fory.apache.org/docs/specification/row_format_spec) - 协议详情 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/rust.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/rust.md index fbc27f99186..ff23094b31e 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/rust.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/rust.md @@ -184,6 +184,6 @@ assert!(scores.get(scores.len()).is_err()); ## 相关主题 -- [基本序列化](../object-serialization/rust/core-api.md) - 对象图序列化 -- [标准行格式](standard.md) - Java、Python、C++ 和 Rust 的共享布局 +- [基本序列化](../object-serialization/rust/basic-serialization.md) - 对象图序列化 +- [标准行格式](index.md#standard-row) - Java、Python、C++ 和 Rust 的共享布局 - [行格式规范](../specification/row_format_spec.md) - 协议详情 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/standard.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/standard.md deleted file mode 100644 index 4ac61321e66..00000000000 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/row-format/standard.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: Standard Row Format -sidebar_position: 1 -id: standard -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Fory Row Format 是一种缓存友好的二进制格式，可实现高效的随机访问和部分反序列化。 -与对象图序列化不同，它允许读取方无需重建完整对象即可访问单个字段。 - -## 功能特性 - -- **零拷贝随机访问**：直接从编码数据中读取选定字段。 -- **部分反序列化**：只重建应用所需的值。 -- **跨语言兼容**：在 Java、Python、C++ 和 Rust 之间共享 Standard Row 字节。 -- **Apache Arrow 集成**：在 Java 和 Python 中将 Row 转换为 Arrow 数据。 - -## 格式边界 - -Standard Row 内联存储固定宽度值，并通过偏移量和大小存储变长值。Row、数组和 Map 使用 -Schema 解析字段位置和元素类型。规范性的字节布局、对齐规则、类型表和字节序由 -[Row Format 规范](../specification/row_format_spec.md)中定义。 - -Row Format 仅用于可信的分析数据，适合分析、内存映射数据、选择性字段访问和数据管道。 -如果应用需要通用对象图、共享或循环引用，或者以完整对象重建为主要访问方式，请使用 -[对象序列化](../object-serialization/index.md)。 - -## 实现 - -| 运行时 | Standard Row 兼容性 | 运行时指南 | 其他集成 | -| ------ | ------------------- | ------------------- | ------------------------------ | -| Java | 兼容 | [Java](java.md) | Arrow 转换；接口和扩展类型映射 | -| Python | 兼容 | [Python](python.md) | PyArrow Schema 和 table 转换 | -| C++ | 兼容 | [C++](cpp.md) | 原生 Row 读取器和写入器 | -| Rust | 兼容 | [Rust](rust.md) | 借用式结构体、数组和 Map 视图 | - -安装、Schema 构建、编码、随机访问、部分读取和特定语言集成请参阅各运行时指南。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/cpp.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/cpp.md index 7f4f7dd6549..70ddab914db 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/cpp.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/cpp.md @@ -90,7 +90,7 @@ cmake --build build --parallel ./build/fory_example ``` -Bazel、Windows、错误处理和线程安全实例请参阅 [C++ 对象序列化](../object-serialization/cpp/index.md)；然后继续阅读 [xlang](../object-serialization/cpp/xlang.md)或 [native 模式](../object-serialization/cpp/native.md)。 +Bazel、Windows、错误处理和线程安全实例请参阅 [C++ 对象序列化](../object-serialization/cpp/index.md)；然后继续阅读 [xlang](../object-serialization/cpp/basic-serialization.md#cross-language-interoperability)或 [native 模式](../object-serialization/cpp/native.md)。 ## 其他能力 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/csharp.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/csharp.md index 1f5c9c03ce2..926ca4069fa 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/csharp.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/csharp.md @@ -67,7 +67,7 @@ public static class Program dotnet run ``` -C# 使用 xlang 模式。接下来可阅读 [C# 对象序列化](../object-serialization/csharp/index.md)、[xlang 类型](../object-serialization/csharp/xlang.md)、[配置](../object-serialization/csharp/configuration.md)和 [Schema 演进](../object-serialization/csharp/schema-evolution.md)。 +C# 使用 xlang 模式。接下来可阅读 [C# 对象序列化](../object-serialization/csharp/index.md)、[xlang 类型](../object-serialization/csharp/basic-serialization.md#cross-language-interoperability)、[配置](../object-serialization/csharp/configuration.md)和 [Schema 演进](../object-serialization/csharp/schema-evolution.md)。 ## 其他能力 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/go.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/go.md index a07aca9dc82..89c3ebbca10 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/go.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/go.md @@ -78,7 +78,7 @@ func main() { go run . ``` -跨语言数据请使用 [xlang 模式](../object-serialization/go/xlang.md)，仅供 Go 使用的数据请使用 [native 模式](../object-serialization/go/native.md)。接下来可阅读 [Go 对象序列化](../object-serialization/go/index.md)、[配置](../object-serialization/go/configuration.md)和 [Schema 演进](../object-serialization/go/schema-evolution.md)。 +跨语言数据请使用 [xlang 模式](../object-serialization/go/basic-serialization.md#cross-language-interoperability)，仅供 Go 使用的数据请使用 [native 模式](../object-serialization/go/native.md)。接下来可阅读 [Go 对象序列化](../object-serialization/go/index.md)、[配置](../object-serialization/go/configuration.md)和 [Schema 演进](../object-serialization/go/schema-evolution.md)。 ## 其他能力 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/index.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/index.md index 62737cd51ba..ae2c7747fff 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/index.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/index.md @@ -1,5 +1,5 @@ --- -title: 快速入门 +title: 概述 sidebar_position: 0 id: index license: | @@ -51,12 +51,12 @@ Apache Fory™ 以源代码制品和各语言专用软件包的形式发布。 ## 可以构建什么 -| 能力 | 用途 | 可用运行时 | 详细指南 | -| ---------- | ----------------------------------------------------------- | ------------------------------------------------------------------ | -------------------------------------------------------- | -| 对象序列化 | 重建对象图，包括共享引用和 Schema 变更 | 所有运行时 | [对象序列化](../object-serialization/index.md) | -| Row Format | 对可信分析数据进行零拷贝、随机或部分字段访问 | Java、Python、C++、Rust | [Row Format](../row-format/index.md) | -| Fory JSON | 高性能标准 JSON 映射 | Java | [Fory JSON](../json/index.md) | -| Fory IDL | 从 Fory、protobuf 或 FlatBuffers IDL 生成原生模型和序列化器 | 所有运行时 | [Fory IDL 与编译器](../compiler/index.md) | -| Fory gRPC | 通过常规 gRPC 传输使用生成的模型和 Fory 编码的消息 | Java、Python、C++、Go、Rust、JavaScript、C#、Dart、Scala、Kotlin | [Fory gRPC](../grpc/index.md) | +| 能力 | 用途 | 可用运行时 | 详细指南 | +| ---------- | ----------------------------------------------------------- | ---------------------------------------------------------------- | ---------------------------------------------- | +| 对象序列化 | 重建对象图，包括共享引用和 Schema 变更 | 所有运行时 | [对象序列化](../object-serialization/index.md) | +| Row Format | 对可信分析数据进行零拷贝、随机或部分字段访问 | Java、Python、C++、Rust | [Row Format](../row-format/index.md) | +| Fory JSON | 高性能标准 JSON 映射 | Java | [Fory JSON](../json/index.md) | +| Fory IDL | 从 Fory、protobuf 或 FlatBuffers IDL 生成原生模型和序列化器 | 所有运行时 | [Fory IDL 与编译器](../compiler/index.md) | +| Fory gRPC | 通过常规 gRPC 传输使用生成的模型和 Fory 编码的消息 | Java、Python、C++、Go、Rust、JavaScript、C#、Dart、Scala、Kotlin | [Fory gRPC](../grpc/index.md) | -对象序列化使用 xlang 模式生成可移植的跨语言数据。Java、Python、C++、Go、Rust、Scala 和 Kotlin 还提供 native 模式，用于同一运行时内的数据。如果尚未明确产品应选择哪种格式，请参阅[选择格式](../introduction/choose-a-format.md)。 +对象序列化使用 xlang 模式生成可移植的跨语言数据。Java、Python、C++、Go、Rust、Scala 和 Kotlin 还提供 native 模式，用于同一运行时内的数据。如果尚未确定使用哪种格式，请参阅[选择格式](../introduction/choose-a-format.md)。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/java.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/java.md index ab9881b7d85..dbbe4224bcf 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/java.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/java.md @@ -78,7 +78,7 @@ public final class ForyExample { } ``` -请在单个线程内复用 `Fory` 实例，不要为每个值重新创建实例。`Fory` 不是线程安全的；如需多个线程共享并发访问，请使用 `ThreadSafeFory`。接下来可阅读 [Java 对象序列化](../object-serialization/java/index.md)、[xlang 模式](../object-serialization/java/xlang.md)、[native 模式](../object-serialization/java/native.md)或[配置](../object-serialization/java/configuration.md)。 +请在单个线程内复用 `Fory` 实例，不要为每个值重新创建实例。`Fory` 不是线程安全的；如需多个线程共享并发访问，请使用 `ThreadSafeFory`。接下来可阅读 [Java 对象序列化](../object-serialization/java/index.md)、[xlang 模式](../object-serialization/java/basic-serialization.md#cross-language-interoperability)、[native 模式](../object-serialization/java/native.md)或[配置](../object-serialization/java/configuration.md)。 ## Fory JSON diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/javascript.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/javascript.md index 42609111589..d794e2ad7b3 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/javascript.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/javascript.md @@ -60,7 +60,7 @@ console.log(deserialize(bytes)); node example.cjs ``` -JavaScript 使用 xlang 模式。接下来可阅读 [JavaScript/TypeScript 对象序列化](../object-serialization/javascript/index.md)、[xlang 类型](../object-serialization/javascript/xlang.md)、[配置](../object-serialization/javascript/configuration.md)和 [Schema 演进](../object-serialization/javascript/schema-evolution.md)。 +JavaScript 使用 xlang 模式。接下来可阅读 [JavaScript/TypeScript 对象序列化](../object-serialization/javascript/index.md)、[xlang 类型](../object-serialization/javascript/basic-serialization.md#cross-language-interoperability)、[配置](../object-serialization/javascript/configuration.md)和 [Schema 演进](../object-serialization/javascript/schema-evolution.md)。 若要使用可选的 Node.js 字符串快速路径，请安装版本匹配的软件包： diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/kotlin.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/kotlin.md index a823b4be5f8..1594a0cf53b 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/kotlin.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/kotlin.md @@ -66,7 +66,7 @@ fun main() { ./gradlew run ``` -与其他 Fory 运行时共享的数据使用 xlang 模式，仅供 Kotlin/JVM 使用的数据使用 native 模式。接下来可阅读 [Kotlin 对象序列化](../object-serialization/kotlin/index.md)、[xlang](../object-serialization/kotlin/xlang.md)或 [native 模式](../object-serialization/kotlin/native.md)。 +与其他 Fory 运行时共享的数据使用 xlang 模式，仅供 Kotlin/JVM 使用的数据使用 native 模式。接下来可阅读 [Kotlin 对象序列化](../object-serialization/kotlin/index.md)、[xlang](../object-serialization/kotlin/basic-serialization.md#cross-language-interoperability)或 [native 模式](../object-serialization/kotlin/native.md)。 ## 其他能力 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/python.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/python.md index 5d16c6b6fb3..5900533d9c7 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/python.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/python.md @@ -58,7 +58,7 @@ decoded = fory.deserialize(data) print(decoded) ``` -跨语言数据请使用 [xlang 模式](../object-serialization/python/xlang.md)。仅供 Python 使用的对象（包括 Python 可调用对象和序列化钩子）请使用 [native 模式](../object-serialization/python/native.md)。接下来可阅读 [Python 指南](../object-serialization/python/index.md)、[配置](../object-serialization/python/configuration.md)和[类型注册](../object-serialization/python/type-registration.md)。 +跨语言数据请使用 [xlang 模式](../object-serialization/python/basic-serialization.md#cross-language-interoperability)。仅供 Python 使用的对象（包括 Python 可调用对象和序列化钩子）请使用 [native 模式](../object-serialization/python/native.md)。接下来可阅读 [Python 指南](../object-serialization/python/index.md)、[配置](../object-serialization/python/configuration.md)和[类型注册](../object-serialization/python/type-registration.md)。 ## 其他能力 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/rust.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/rust.md index 8378febbfc3..e9a0222b76f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/rust.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/rust.md @@ -61,7 +61,7 @@ fn main() -> Result<(), Error> { } ``` -跨语言数据请使用 [xlang 模式](../object-serialization/rust/xlang.md)，仅供 Rust 使用的数据请使用 [native 模式](../object-serialization/rust/native.md)。接下来可阅读 [Rust 对象序列化](../object-serialization/rust/index.md)、[配置](../object-serialization/rust/configuration.md)和[类型注册](../object-serialization/rust/type-registration.md)。 +跨语言数据请使用 [xlang 模式](../object-serialization/rust/basic-serialization.md#cross-language-interoperability)，仅供 Rust 使用的数据请使用 [native 模式](../object-serialization/rust/native.md)。接下来可阅读 [Rust 对象序列化](../object-serialization/rust/index.md)、[配置](../object-serialization/rust/configuration.md)和[类型注册](../object-serialization/rust/type-registration.md)。 ## 其他能力 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/scala.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/scala.md index c1067aa8238..856295ceec5 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/scala.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/scala.md @@ -64,7 +64,7 @@ object ScalaExample { sbt run ``` -与其他 Fory 运行时共享的数据使用 xlang 模式，仅供 Scala/JVM 使用的数据使用 native 模式。接下来可阅读 [Scala 对象序列化](../object-serialization/scala/index.md)、[xlang](../object-serialization/scala/xlang.md)或 [native 模式](../object-serialization/scala/native.md)。 +与其他 Fory 运行时共享的数据使用 xlang 模式，仅供 Scala/JVM 使用的数据使用 native 模式。接下来可阅读 [Scala 对象序列化](../object-serialization/scala/index.md)、[xlang](../object-serialization/scala/basic-serialization.md#cross-language-interoperability)或 [native 模式](../object-serialization/scala/native.md)。 ## 其他能力 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/swift.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/swift.md index 96653fe49eb..85bbb44b57f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/swift.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/start/swift.md @@ -35,7 +35,7 @@ swift --version swift package init --type executable --name ForyExample ``` -在生成的 `Package.swift` 中添加已发布的软件包和 `Fory` 产品： +在生成的 `Package.swift` 中添加已发布的软件包，并依赖其中的 `Fory` 库： ```swift title="Package.swift" dependencies: [ @@ -73,7 +73,7 @@ assert(input == decoded) swift run ``` -Swift 使用 xlang 模式。接下来可阅读 [Swift 对象序列化](../object-serialization/swift/index.md)、[xlang 类型](../object-serialization/swift/xlang.md)、[配置](../object-serialization/swift/configuration.md)和 [Schema 演进](../object-serialization/swift/schema-evolution.md)。 +Swift 使用 xlang 模式。接下来可阅读 [Swift 对象序列化](../object-serialization/swift/index.md)、[xlang 类型](../object-serialization/swift/basic-serialization.md#cross-language-interoperability)、[配置](../object-serialization/swift/configuration.md)和 [Schema 演进](../object-serialization/swift/schema-evolution.md)。 ## 其他能力 diff --git a/scripts/unified-docs/script.py b/scripts/unified-docs/script.py index a62d8b16cae..8761deaebf5 100644 --- a/scripts/unified-docs/script.py +++ b/scripts/unified-docs/script.py @@ -29,10 +29,17 @@ def copy_markdown_file(src_file, dst_file): print(f"Skipped {dst_file} (already exists)") -def copy_markdown_files(src_folder, dst_folder): +def copy_markdown_files(src_folder, dst_folder, excluded_top_level=None): tasks = [] + excluded_top_level = excluded_top_level or set() - for root, _, files in os.walk(src_folder): + for root, directories, files in os.walk(src_folder): + if os.path.abspath(root) == os.path.abspath(src_folder): + directories[:] = [ + directory + for directory in directories + if directory not in excluded_top_level + ] for file in files: if file.endswith(".md"): src_file = os.path.join(root, file) @@ -51,7 +58,7 @@ def execute(): base_blog_folder = "../../blog/" zh_cn_blog_dst = "../../i18n/zh-CN/docusaurus-plugin-content-blog/" - copy_markdown_files(base_src_folder, zh_cn_docs_dst) + copy_markdown_files(base_src_folder, zh_cn_docs_dst, {"security"}) copy_markdown_files(base_blog_folder, zh_cn_blog_dst)