SeaLion: Semantic Part-Aware Latent Point Diffusion Models for 3D Generation

CVPR 2025

Dekai Zhu   Yan Di   Stefan Gavranovic   Slobodan Ilic

Paper   Project Page

Abstract

Denoising diffusion probabilistic models have achieved significant success in point cloud generation, enabling numerous downstream applications, such as generative data augmentation and 3D model editing. However, little attention has been given to generating point clouds with point-wise segmentation labels, as well as to developing evaluation metrics for this task. Therefore, in this paper, we present SeaLion, a novel diffusion model designed to generate high-quality and diverse point cloud with fine-grained segmentation labels. Specifically, we introduce the semantic part-aware latent point diffusion technique, which leverages the intermediate features of the generative models to jointly predict the noise for perturbed latent points and associated part segmentation labels during the denoising process, and subsequently decodes the latent points to point clouds conditioned on part segmentation labels. To effectively evaluate the quality of generated point clouds, we introduce a novel point cloud pairwise distance calculation method named part-aware Chamfer distance (p-CD). This method enables existing metrics, such as 1-NNA, to measure both the local structural quality and inter-part coherence of generated point clouds. Experiments on the large-scale synthetic dataset ShapeNet and real-world medical dataset IntrA, demonstrate that SeaLion achieves remarkable performance in generation quality and diversity, outperforming the existing state-of-the-art model, DiffFacto, by 13.33% and 6.52% on 1-NNA (p-CD) across the two datasets. Experimental analysis shows that SeaLion can be trained semi-supervised, thereby reducing the demand for labeling efforts. Lastly, we validate the applicability of SeaLion in generative data augmentation for training segmentation models and the capability of SeaLion to serve as a tool for part-aware 3D shape editing.

Install

• Dependencies:

  • CUDA 11.6

• Setup the environment Install from conda file

      conda env create --name lion_env --file=env.yaml 
      conda activate lion_env
  

  Tested with conda version 22.9.0

Training

data

• You can download the full ShapeNetPart dataset from this link and manually split it following the official train/val/test split. Alternatively, you can download an example of the car category here.
• Place the downloaded data into ${ShapeNetPart_dir} following the structure below:

    ${ShapeNetPart_dir}/
    ├── 02958343/
    │   ├── train/
    │   ├── val/
    │   └── test/
    ├── 02691156/
    │   ├── train/
    │   ├── val/
    │   └── test/
    └── other categories.../
        ├── train/
        ├── val/
        └── test/

train VAE

• Set the data path "data.data_dir" in the config file for VAE training, e.g. config/sealion_shapenet/cfg_vae_car.yml.
• Run the following command to start training:

python train_dist.py  --num_process_per_node ${NUM_GPU}  --config ${VAE_CONFIG_FILE}  --exp_root ${EXP_ROOT}  --exp_name ${EXP_NAME}

e.g. python train_dist.py  --num_process_per_node 1  --config config/sealion_shapenet/cfg_vae_car.yml  --exp_root /data/dekai/sealion/ShapeNetPart/  --exp_name car_vae

• If you want to inspect the training progress, the logging folder is located at ${EXP_ROOT}/${category}/. SeaLion follows the latent diffusion framework introduced in Lion. According to the suggestions from the Lion authors, the VAE training time depends on the trade-off between reconstruction accuracy and the smoothness of the latent feature distribution.

train diffusion prior

• Set the data path "data.data_dir" in the config file for DDPM training, e.g. config/sealion_shapenet/cfg_ddpm_car.yml.
• A trained VAE checkpoint is required. Please set the "sde.vae_checkpoint" field in the DDPM config file accordingly. You can download a pretrained VAE model for the car category here.
• Run the following command to start training:

python train_dist.py  --num_process_per_node ${NUM_GPU}  --config ${DDPM_CONFIG_FILE}  --exp_root ${EXP_ROOT}  --exp_name ${EXP_NAME}

e.g. python train_dist.py  --num_process_per_node 1  --config config/sealion_shapenet/cfg_ddpm_car.yml  --exp_root /data/dekai/sealion/ShapeNetPart/  --exp_name car_ddpm

evaluate a trained prior

• Set DDPM_PRETRAINED_WEIGHT to the path of the trained DDPM weights. Alternatively, you can download the pretrained weights from here and unzip them in the current directory.
• Run the following command to start sampling and evaluation:

python train_dist.py  --skip_nll 1  --eval_generation  --pretrained $DDPM_PRETRAINED_WEIGHT

e.g. python train_dist.py  --skip_nll 1  --eval_generation  --pretrained sealion_car_trained_weights/ddpm/checkpoints/snapshot

Acknowledgment

Our implementation builds upon the codebase of Lion, and is released as a forked branch of it.

Citation

@inproceedings{zhu2025sealion,
    title={SeaLion: Semantic Part-Aware Latent Point Diffusion Models for 3D Generation},
    author={Zhu, Dekai and Di, Yan and Gavranovic, Stefan and Ilic, Slobodan},
    booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
    pages={11789--11798},
    year={2025}
}