Something odd in the training loop

[cem-sirin]

Hi

I am training a custom DBNet and I have a problem. The predicted probability maps during the training is completely different (almost random noise other than the padding) compared to initializing the dataset, normalizing the image and forwarding in a notebook. See the images below.

The datasets and its pipelines are exactly the same. I have built the same TextDetDataPreprocessor. For some reason during training, the image is processed differently. Does someone have any idea?

These are the maps produced in a notebook:

These are the maps produced during training. It's directly the first batch before taking any steps:

Note: I have converted/implemented a model from PPOCR, that's why it is line prediction. I can share it, if it indeed works. I am using a Macbook M3. I just made a small cheat instance_data.py by adding [not ideal i know :( ]:

...
elif get_device() == 'mps':
    BoolTypeTensor = Union[torch.BoolTensor, torch.mps.Tensor]
    LongTypeTensor = Union[torch.LongTensor, torch.mps.Tensor]
else:
    BoolTypeTensor = Union[torch.BoolTensor, torch.cuda.BoolTensor]
    LongTypeTensor = Union[torch.LongTensor, torch.cuda.LongTensor]

Here are the system details:

System environment:
    sys.platform: darwin
    Python: 3.8.20 (default, Oct  3 2024, 10:22:23) [Clang 14.0.6 ]
    CUDA available: False
    MUSA available: False
    numpy_random_seed: 42
    GCC: Apple clang version 15.0.0 (clang-1500.3.9.4)
    PyTorch: 2.4.1
    PyTorch compiling details: PyTorch built with:
  - GCC 4.2
  - C++ Version: 201703
  - clang 14.0.3
  - OpenMP 201811
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: NO AVX
  - Build settings: BLAS_INFO=accelerate, BUILD_TYPE=Release, CXX_COMPILER=/Applications/Xcode_14.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_PYTORCH_QNNPACK -DAT_BUILD_ARM_VEC256_WITH_SLEEF -DUSE_XNNPACK -DUSE_PYTORCH_METAL_EXPORT -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DUSE_COREML_DELEGATE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=braced-scalar-init -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wvla-extension -Wsuggest-override -Wnewline-eof -Winconsistent-missing-override -Winconsistent-missing-destructor-override -Wno-pass-failed -Wno-error=pedantic -Wno-error=old-style-cast -Wno-error=inconsistent-missing-override -Wno-error=inconsistent-missing-destructor-override -Wconstant-conversion -Wno-invalid-partial-specialization -Wno-missing-braces -Qunused-arguments -fcolor-diagnostics -faligned-new -Wno-unused-but-set-variable -fno-math-errno -fno-trapping-math -Werror=format -DUSE_MPS -Wno-unused-private-field -Wno-missing-braces, LAPACK_INFO=accelerate, TORCH_VERSION=2.4.1, USE_CUDA=0, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

    TorchVision: 0.20.0
    OpenCV: 4.10.0
    MMEngine: 0.10.5

Runtime environment:
    cudnn_benchmark: False
    mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
    dist_cfg: {'backend': 'nccl'}
    seed: 42
    Distributed launcher: none
    Distributed training: False
    GPU number: 1

[cem-sirin]

Oops, the weights weren't loading, I thought setting the Weights, in the metafile.yml did the job, since it worked for inference and testing. I am closing the discussion :)