Training, validation and test datasets

[Molina-Valero]

Dear developers of PTv3,

Firs of all, thank you very much for your great job!

I am a little bit confused regarding the train, validation and test datset in PTv3; and how they are used in your models.

1. Every single epoch the model performance is been evaluated based on the validation dataset (>>>>>>>>>>>>>>>> Start Evaluation (Segmentation) >>>>>>>>>>>>>>>>)? Is this the validation you would use to know when there is overfitting?

2. Which dataset is been used according to the argument "eval_epoch = 100 # evaluate val every N epochs" (PreciseEvaluation)? As I can see in the models, it should be the validation dataset too, right? Then, what is the reason to evaluate the model again?

3. In my case, I also have a test dataset. However, when I include the test dataset in my config (see below), I get memory issues. Then, should I only use the test dataset when running the script "sh scripts/test.sh"?

Thank you in advance,
Juan

---

base = ["../base/default_runtime.py"]

misc

batch_size = 4 # total across all GPUs; reduce if OOM
num_worker = 4
mix_prob = 0.8 # MixUp3D probability
empty_cache = False
enable_amp = True

model

model = dict(
type="DefaultSegmentorV2",
num_classes=5,
backbone_out_channels=64,
backbone=dict(
type="PT-v3m1",
in_channels=6, # XYZ coord (3) + normal (3)
order=("z", "z-trans", "hilbert", "hilbert-trans"),
stride=(2, 2, 2, 2),
enc_depths=(2, 2, 2, 6, 2),
enc_channels=(32, 64, 128, 256, 512),
enc_num_head=(2, 4, 8, 16, 32),
enc_patch_size=(1024, 1024, 1024, 1024, 1024),
dec_depths=(2, 2, 2, 2),
dec_channels=(64, 64, 128, 256),
dec_num_head=(4, 4, 8, 16),
dec_patch_size=(1024, 1024, 1024, 1024),
mlp_ratio=4,
qkv_bias=True,
qk_scale=None,
attn_drop=0.0,
proj_drop=0.0,
drop_path=0.3,
shuffle_orders=True,
pre_norm=True,
enable_rpe=False,
enable_flash=True,
upcast_attention=False,
upcast_softmax=False,
enc_mode=False,
# PDNorm off — training on a single dataset from scratch
pdnorm_bn=False,
pdnorm_ln=False,
pdnorm_decouple=True,
pdnorm_adaptive=False,
pdnorm_affine=True,
pdnorm_conditions=("Forest",),
),
criteria=[
dict(type="CrossEntropyLoss", loss_weight=1.0, ignore_index=-1),
dict(type="LovaszLoss", mode="multiclass", loss_weight=1.0, ignore_index=-1),
],
)

scheduler

epoch = 100
eval_epoch = 100 # evaluate val every N epochs

optimizer = dict(type="AdamW", lr=0.006, weight_decay=0.05)
scheduler = dict(
type="OneCycleLR",
max_lr=[0.006, 0.0006],
pct_start=0.05,
anneal_strategy="cos",
div_factor=10.0,
final_div_factor=1000.0,
)
param_dicts = [dict(keyword="block", lr=0.0006)]

dataset

dataset_type = "SegmentedForestsDataset"
data_root = "data/SegmentedForests" # symlink: ln -s /your/processed/path data/SegmentedForests
ignore_index = -1

names = [
"shrub",
"ground",
"crown",
"stem",
"dead_downwood",
]

data = dict(
num_classes=5,
ignore_index=ignore_index,
names=["shrub", "ground", "crown", "stem", "dead_downwood"],

train

train=dict(
    type=dataset_type,
    split="train",
    data_root=data_root,
    transform=[
        # Centre each scene vertically
        dict(type="CenterShift", apply_z=True),

        # Random point dropout (helps with varying scan densities)
        dict(type="RandomDropout", dropout_ratio=0.2, dropout_application_ratio=0.2),

        # Rotations: full 360° around Z (LiDAR heading), tiny tilt on X/Y
        dict(type="RandomRotate", angle=[-1, 1], axis="z", center=[0, 0, 0], p=0.5),
        dict(type="RandomRotate", angle=[-1/64, 1/64], axis="x", p=0.5),
        dict(type="RandomRotate", angle=[-1/64, 1/64], axis="y", p=0.5),

        # Scale: simulate distance variation / scan resolution differences
        dict(type="RandomScale", scale=[0.9, 1.1]),

        # Flip around vertical axis
        dict(type="RandomFlip", p=0.5),

        # Small noise on point positions
        dict(type="RandomJitter", sigma=0.005, clip=0.02),

        # Voxelise at 2 cm — adjust if your data is sparser (e.g. 0.05 for TLS)
        dict(
            type="GridSample",
            grid_size=0.02,
            hash_type="fnv",
            mode="train",
            return_grid_coord=True,
        ),

        # Cap point count — tune to your GPU memory
        dict(type="SphereCrop", point_max=120000, mode="random"),

        # Shift centroid to XY origin (after crop)
        dict(type="CenterShift", apply_z=False),

        dict(type="ToTensor"),
        dict(
            type="Collect",
            keys=("coord", "grid_coord", "segment"),
            feat_keys=("coord", "normal"),  # 3+3 = 6 input channels
        ),
    ],
    test_mode=False,
    ignore_index=ignore_index,
),

val

val=dict(
    type=dataset_type,
    split="val",
    data_root=data_root,
    transform=[
        dict(type="CenterShift", apply_z=True),
        dict(type="Copy", keys_dict={"segment": "origin_segment"}),
        dict(
            type="GridSample",
            grid_size=0.02,
            hash_type="fnv",
            mode="train",
            return_grid_coord=True,
            return_inverse=True,
        ),
        dict(type="CenterShift", apply_z=False),
        dict(type="ToTensor"),
        dict(
            type="Collect",
            keys=("coord", "grid_coord", "segment", "origin_segment", "inverse"),
            feat_keys=("coord", "normal"),
        ),
    ],
    test_mode=False,
    ignore_index=ignore_index,
),

test

test=dict(
    type=dataset_type,
    split="test",
    data_root=data_root,
    transform=[
        dict(type="CenterShift", apply_z=True),
    ],
    test_mode=True,
    test_cfg=dict(
        voxelize=dict(
            type="GridSample",
            grid_size=0.02,
            hash_type="fnv",
            mode="test",
            return_grid_coord=True,
        ),
        crop=None,
        post_transform=[
            dict(type="CenterShift", apply_z=False),
            dict(type="ToTensor"),
            dict(
                type="Collect",
                keys=("coord", "grid_coord", "index"),
                feat_keys=("coord", "normal"),
            ),
        ],
        # TTA: 4 rotations × 3 scales (similar to ScanNet but suited to
        # outdoor scenes where scale and heading matter)
        aug_transform=[
            [dict(type="RandomRotateTargetAngle", angle=[0],   axis="z", center=[0,0,0], p=1)],
            [dict(type="RandomRotateTargetAngle", angle=[1/2], axis="z", center=[0,0,0], p=1)],
            [dict(type="RandomRotateTargetAngle", angle=[1],   axis="z", center=[0,0,0], p=1)],
            [dict(type="RandomRotateTargetAngle", angle=[3/2], axis="z", center=[0,0,0], p=1)],
            [dict(type="RandomRotateTargetAngle", angle=[0],   axis="z", center=[0,0,0], p=1),
             dict(type="RandomScale", scale=[0.95, 0.95])],
            [dict(type="RandomRotateTargetAngle", angle=[1/2], axis="z", center=[0,0,0], p=1),
             dict(type="RandomScale", scale=[0.95, 0.95])],
            [dict(type="RandomRotateTargetAngle", angle=[1],   axis="z", center=[0,0,0], p=1),
             dict(type="RandomScale", scale=[0.95, 0.95])],
            [dict(type="RandomRotateTargetAngle", angle=[3/2], axis="z", center=[0,0,0], p=1),
             dict(type="RandomScale", scale=[0.95, 0.95])],
            [dict(type="RandomRotateTargetAngle", angle=[0],   axis="z", center=[0,0,0], p=1),
             dict(type="RandomScale", scale=[1.05, 1.05])],
            [dict(type="RandomRotateTargetAngle", angle=[1/2], axis="z", center=[0,0,0], p=1),
             dict(type="RandomScale", scale=[1.05, 1.05])],
            [dict(type="RandomRotateTargetAngle", angle=[1],   axis="z", center=[0,0,0], p=1),
             dict(type="RandomScale", scale=[1.05, 1.05])],
            [dict(type="RandomRotateTargetAngle", angle=[3/2], axis="z", center=[0,0,0], p=1),
             dict(type="RandomScale", scale=[1.05, 1.05])],
        ],
    ),
    ignore_index=ignore_index,
),

)

[Gofinge]

@codex Please response refer README and training logics in Pointcept codebase.

[chatgpt-codex-connector]

Draft response for the PR thread

Thanks for the question. Please note first that this PTv3 repository is mainly the project/release repository, and the README points users to the Pointcept codebase for the actual training/evaluation pipeline. The README explicitly says the code is updated in Pointcept and links the Pointcept implementation. It also says that, for the Pointcept-driven scenario, you should clone Pointcept and follow its training scripts.

Based on the README and the current Pointcept training logic:

1. Yes — Start Evaluation (Segmentation) is validation during training.
   In Pointcept, the segmentation evaluator runs on trainer.val_loader, which is built from cfg.data.val. So this is the validation split, not the test split. This validation result is the metric you normally monitor for overfitting/model selection during training.

2. eval_epoch controls the training loop granularity and when the normal validation/checkpoint cycle happens.
   In Pointcept, cfg.data.train.loop = cfg.epoch // cfg.eval_epoch, while the trainer’s internal max_epoch is set to cfg.eval_epoch. In practice, epoch is the total intended training length, and eval_epoch defines the interval/chunk after which validation/checkpointing hooks run. For example, with epoch = 100 and eval_epoch = 100, validation happens after the full 100-epoch training run. With epoch = 100 and eval_epoch = 50, the train data loop is repeated twice and validation/checkpointing happens every 50 epochs.

3. PreciseEvaluation is still on the validation/test-style dataset entry, but it uses the testing pipeline for a more complete evaluation.
   The README explains the distinction: normal training-time evaluation is performed after grid sampling/voxelization and gives an initial estimate, while the testing process covers the dense point cloud more completely and supports TTA. In Pointcept this is handled by the PreciseEvaluator hook after training: it builds a tester and evaluates model_best by default. So it is not simply duplicating the same fast validation loop; it is running the more precise test-style evaluation pipeline.

4. For your own held-out test set, I would not include it in the training-time validation path unless you intentionally want final/test-style evaluation during training.
   Keep:

   • data.train = training split with augmentation;
   • data.val = validation split used during training for model selection/overfitting checks;
   • data.test = held-out test split for final evaluation.

   If adding data.test causes memory issues, that is expected especially with large scenes, test_mode=True, dense coverage, and TTA. The README shows that testing is normally launched separately with scripts/test.sh. So for your case, I would recommend training with only train/val active for routine monitoring, then run the held-out test set separately after training, e.g. using sh scripts/test.sh ... -w model_best.

5. For memory issues in test mode, reduce the cost of the test pipeline:

   • remove or reduce TTA first;
   • use a crop strategy in test_cfg.crop instead of crop=None for large scenes;
   • reduce batch_size_test if set;
   • use a larger grid_size if acceptable;
   • reduce enc_patch_size / dec_patch_size or disable FlashAttention only if required by your environment. The README also notes that if FlashAttention cannot be used, enable_flash=False should be paired with smaller patch sizes.

In short: use val for training-time monitoring and overfitting checks; use PreciseEvaluation/scripts/test.sh for the more accurate final-style evaluation; and reserve your held-out test split for final evaluation, not routine every-epoch validation.

No code changes were needed for this follow-up.

Testing / checks

• ✅ pwd && find .. -name AGENTS.md -print && git status --short && rg -n "eval_epoch|PreciseEvaluation|Start Evaluation|test.sh|DefaultTrainer|Eval" README.md pointcept scripts configs -S — initial repository inspection; some paths do not exist in this lightweight PTv3 repo, confirming the README’s note that training logic lives in Pointcept.
• ✅ git clone --depth 1 https://github.com/Pointcept/Pointcept.git "$tmp/Pointcept" — cloned Pointcept to inspect the referenced training/evaluation logic.
• ✅ rg -n "eval_epoch|PreciseEvaluation|Start Evaluation|class Trainer|DefaultTrainer|build_dataset|val_loader|test_loader|scripts/test.sh" pointcept tools scripts configs README.md -S — located the relevant Pointcept trainer, evaluator, precise evaluator, tester, and README sections.
• ✅ nl -ba README.md | sed -n '1,20p;92,118p;120,154p;156,184p' — captured local README line numbers for citations.

View task →

[Molina-Valero]

Thank you so much for your answer. It is very helpful.
Best,
Juan