[Jakfut]

Reduces world size by fixing compression of unused buffer space, sharing repeated metadata for empty cubes, and adding Zstd as the default compression. GZIP, LZ4 and uncompressed storage remain available.

Also adds automatic region compaction during server/world startup to reclaim unused space, with a loading progress bar in singleplayer and progress logs on servers. It does not run during gameplay or shutdown.

Existing compression settings are preserved, and existing worlds keep their storage format. Compact empty-cube storage is the default for new worlds.

The total save size was reduced by around 30-50%.

[DarkShadow44]

Mind splitting that up a bit more? I feel like ZStd, new format and compaction could be their own commits at least, would be easier to review.

[Jakfut]

dict is per dim, trained, then shipped with the mod
I dont think they are worth it with Zstd, so I did not add that

Codec	MiB	Encode	Decode
LZ4 fast	51.29	188 ms	75 ms
LZ4 fast + dict	45.95	210 ms	78 ms
LZ4 HC3 + dict	35.69	1514 ms	61 ms
Zstd 1	31.15	459 ms	226 ms
Zstd 1 + dict	30.03	463 ms	197 ms
Zstd 3 + dict	29.35	598 ms	198 ms
GZIP 6	29.71	3242 ms	575 ms

[Jakfut]

its now 3 separate commits, for easier review

[DarkShadow44]

I did some testing with a ~1GB GTNH world (128 chunks pregen'd with DH)

While your compact approach works, I have an idea that might be simpler and more effective: We could skip cubes completely if we can easily regen them.
Most sectionless cubes carry almost no information: no blocks, entities, tile entities or ticks, and their only real state is a few flags plus the heightmap snapshot.
In this world that is 61% of all cubes. Instead of writing them, we store one state byte per cube Y in the column (region2d) and rebuild the cube directly on load, without running the generator. The byte holds:

• skipped yes/no
• population (0 or 255)
• initLightDone, isSurfaceTracked
• whether the saved heightmap is entirely below or entirely above the cube (cubes whose heightmap crosses them are written normally)

Conceptionally it's very similar to your idea, but it seems a lot simpler than the template approach.

Not sure how far you want to go, but there's also other things we can improve - if we make a new format we might as well make it as effective as we can rather than have multiple format changes in succession:

1. Clip:
   Every cube stores LastHeightMap as 256 absolute Y values, but only their position relative to the cube matters. Clamping them to [cubeBaseY, cubeBaseY + 17] gives identical results, stays compatible both ways, and saves ~4.6% here.

2. Grouping:
   The idea is to group cubes in 2x4x2 or 1x16x1 sections - currently our cubes are very small and we miss out on a lot of compression potential. Of course, this might have overhead for loading/saving.
   Each record occupies whole 512 B sectors, and neighbouring cubes share most of their content, which per-record compression can't see. One Zstd stream per group of 16 cubes recovers nearly all of that. Saving a cube becomes read group, replace cube, recompress, write though.

3. Palette:
   Per section, a palette of full block states:
   (id = Blocks | Add<<8 | BlocksB2Hi<<12, meta = Data | Data1High<<4 | Data2<<8, so it covers the ChunkAPI arrays)
   plus one byte per voxel, stored columnar inside the group.
   GTNH sections average 14 states and none had more than 256. Zstd doesn't seem to do this on its own since the data is spread out too much.

4. Could also group columns, saves 38.9 -> 13.8 MB here

If you want I could PR some changes unto your PR?

Size comparison using zstd level1:
Same world, written to real region files by a test command that re-encodes the stored cubes/columns (region2d + region3d):

variant	total MB	vs today
today on disk (gzip, with holes)	1255.5	1.00x
today, rewritten without holes (gzip)	1160.2	1.08x
compact (your table format)	873.1	1.44x
proposed skipping approach	844.5	1.49x
1x16x1 groups	645.3	1.95x
skipping + 1x16x1	618.5	2.03x
skip + 1x16x1 + clip	617.2	2.03x
skip + palette (no groups)	619.7	2.03x
2x4x2 + palette + clip	365.5	3.44x
skip + 2x4x2 + palette	359.4	3.49x
skip + 2x4x2 + palette + clip	342.9	3.66x

[Note: "compact" was written by my test exporter using your eligibility rules and .cce format, not by your code, so your real numbers may differ slightly. It's 30% smaller than today's files, 25% smaller than today's format without holes.]

[Jakfut]

I tested 3. a bunch
doing a local palette almost doubles cube read/write time... but when using Java 21 SMID you can get that down to an only a 10% increase
I also tested a bunch of other palette options, like 0-12 bit variable or an 8bit palette per dim, all ended up with roughly the same size savings but 8bit local seems like the cleanest option and works well with SMID

[Jakfut]

for the grouping I would prefer 2x2x2 over the other options
a bit worse compression for less view distance boundary overhead and also less overhead when a miner/other machine only has to touch a single cube
also worth looking if it makes sense to do the palette over the group instead of over per cube at that point, but I have not tried anything with groupings yet

[RecursivePineapple]

I'm fine with adding SIMD classes, but we'll have to be careful to use factory methods to provide the compressor because MRJARs don't work with incubating APIs. It would probably be worth it to follow the same pattern as compression, by switching between palettized and non-palettized at runtime by checking the header (or some other variant field, I used headers because it was backwards-compatible with GZIP). I'm sure there's a way to check if the vector API is enabled, allowing us to disable the SIMD palette (de)compressor at runtime.

[Jakfut]

So I implemented most of your proposed changes @DarkShadow44

• Clip: added

• Empty cubes: I kept the templates, with clipping they are smaller than the state byte in practice, because templates can cover more cases, like ones with pending light checks, or almost all cubes from converted worlds, which have no lighting info
  Templates also keep that state in the cube storage, so there are no potential problems with column/cube save ordering

• Bundles: (my name for grouping) They are 2x2x2. With client (blocks, light, biome) and server (everything else) parts compressed separately. This allows the server to send the client part directly to the client, without having to decode/encode the data again, in a future network protocol change

• Palette: one palette of block states (ID and metadata, including the EndlessIDs arrays) per bundle, with 8-bit indices up to 256 states and 16-bit above (in practice none of the bundles in my UHV world had more than 256 states)

• Columns: grouped in a 2x2 grid, aligned with the 2x2x2 cube bundles

• Save cache: Rebuild cube io cache #154 added the new cache, here it keeps the written bundle records and saves of unloaded cubes up to 10s (bundleWriteWaitSeconds) for the rest of its bundle, so a bundle is usually written once instead of once per cube. That roughly halves the number of records written.

Existing worlds are converted gradually as their cubes are saved, unless storageFormat is set to an older format, or completely at the next start with convertStorageOnWorldLoad=true. This branch includes #154 by merge, so it works whichever of the two is merged first.

Overworld region data (region2d, region3d and the compact tables) of the same content

World	Before	Startup compaction only	Zstd, templates, clip	With bundles and column groups
fresh world, 93k cubes	84.5 MB	76.3 MB	44.0 MB	18.6 MB (22%)
played GTNH world, 1.27M cubes	1,291 MB	1,018 MB	722 MB	262 MB (20%)
converted UHV world, 1.05M cubes	1,130 MB	1,065 MB	690 MB	373 MB (33%)

[Jakfut]

Fixed a few storage bugs in CC, not introduced in this PR
A failed cube save no longer leaves the cube half unloaded, and unreadable region record no longer gets regenerated and overwritten, and forEachCube/forEachColumn no longer list positions that were never saved

[Jakfut]

The compact storage held one lock for everything, this caused the server thread cube reads to wait for the entire save to finish. Reads now no longer wait for saves, in a 2.5 min test this reduced the server thread's lock waiting from 2.4s to 15ms

[DarkShadow44]

• Empty cubes: I kept the templates, with clipping they are smaller than the state byte in practice, because templates can cover more cases, like ones with pending light checks, or almost all cubes from converted worlds, which have no lighting info
  Templates also keep that state in the cube storage, so there are no potential problems with column/cube save ordering

I measured both on three worlds (settled GTNH, fresh, converted vanilla), same settings.
The default approach is was 13% worse on the converted world, which confirms your coverage point. But this is easily fixed by adding a no-lighting case, then size is a tie.
The byte is smaller per cube, so templates only win through coverage (pending light checks stay uncovered).

Agreed, state byte couples us to the column, but it's a lot easier than the sync code you need for templates. In my prototype it's about half the code, so that seems worth it.

Also, why keep the other format (COMPACT_EMPTY) in the PR?

[Jakfut]

changed dependencies to align with #97
this caused 0.1.29 to not include #139 silently

[Jakfut]

It's now 2 bytes per empty cube, a flags byte and the full population status, since 0.5–5% of the empty cubes in my worlds have population 1, which a 0/255 bit can't hold. With your no-lighting case it comes out 0.4–2.6% smaller than the templates on all three worlds.

The states live in the column group records and are handled entirely by the storage. A cube's state and its record change in the same write batch. The state is written before the record is removed, and a full record always wins over a state, so there's no column/cube save ordering problem.

COMPACT_EMPTY and the template code are gone. Worlds from v0.1.27-alpha-pre get a clear error message instead of the NPE in 0.1.29

[DarkShadow44]

Will take a look the coming days