Problem reading 4 channels .png image using ImageDeserializer

[ChunHsinWang]

Hi,

I am having trouble read in 4 channels .png images with ImageDeserializer. What I am trying to do is read the 4 channel images, and split the alpha channel as my groundtruth label. Something like:

image_height = 224
image_width  = 224
num_channels = 4
def create_reader(training_path, is_training, num_label_classes, num_samples):
    transforms = []
    transforms += [xforms.crop(crop_type='randomside', side_ratio=(0.3,0.5), jitter_type='uniratio')]
    transforms += [xforms.scale(width=image_width, height=image_height, channels=num_channels, 
    interpolations='linear')] 
    train_source = C.io.ImageDeserializer(training_path, C.io.StreamDefs(
        input_var  = C.io.StreamDef(field='image', transforms=transforms),
        labels_viz = C.io.StreamDef(field='label', shape=num_label_classes)
    ))
    
    return C.io.MinibatchSource([train_source], randomize=True,
            max_sweeps = num_samples if is_training else 1,
            multithreaded_deserializer=True)

input_4ch = C.input_variable((num_channels, image_height, image_width), np.float32)
target    = input_4ch[3,:,:]
input_3ch = input_4ch[:3,:,:]
input_map = {
    input_4ch : minibatch_source.streams.input_var
}

but the image does not seem to be loading correctly. From my understanding, flag cv::IMREAD_COLOR in ImageDataDeserializer.cpp line 298 will only read the color image as 3 channels, to make it support 4 channels, will it be better if the flag is changed to cv::IMREAD_UNCHANGED (sorry I have trouble compiling the source, can not try it myself)?

[cha-zhang]

Are you working on segmentation? Can you use composite reader to load two images?

[ChunHsinWang]

Yes, I'm trying to do prediction on the alpha channel. I was actually using composite reader like you suggested at first, but there is a bug where both readers will not get the same random transform for minibatch size > 1, according to issue #2309. So I was hoping I could just load the image as 4 channels as a workaround. Will there be any problem if the flag is set to cv::IMREAD_UNCHANGED?

[cha-zhang]

I see. There should not be an issue to load with cv::IMREAD_UNCHANGED.

Can you tell me what's your difficulty compiling the source?

[ChunHsinWang]

I'm not quite sure what the problems are, I have followed the development environment setup described here: https://docs.microsoft.com/en-us/cognitive-toolkit/setup-cntk-with-script-on-windows. But still got a lot of errors as following:

Severity Code Description Project File Line Suppression State
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' V2LibraryTests (Tests\UnitTests\V2LibraryTests) D:\CNTK\Tests\UnitTests\V2LibraryTests\LINK 1
Error CS0246 The type or namespace name 'IEvaluateModelManagedF' could not be found (are you missing a using directive or an assembly reference?) CSEvalClientTest D:\CNTK\Examples\Evaluation\LegacyEvalDll\CSEvalClient\ModelEvaluator.cs 34 Active
Error CS0246 The type or namespace name 'CNTKException' could not be found (are you missing a using directive or an assembly reference?) CSEvalClientTest D:\CNTK\Examples\Evaluation\LegacyEvalDll\CSEvalClient\Program.cs 147 Active
Error CS0246 The type or namespace name 'IEvaluateModelManagedF' could not be found (are you missing a using directive or an assembly reference?) ManagedEvalTests D:\CNTK\Tests\UnitTests\ManagedEvalTests\EvalManagedTests.cs 466 Active
Error MSB8013 This project doesn't contain the Configuration and Platform combination of Release|x64. Multiverso C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppBuild.targets 349
Error C2220 warning treated as error - no 'object' file generated Common C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include\cuda_runtime.h 1
Error LNK1181 cannot open input file 'Multiverso.lib' MultiversoTests D:\CNTK\Source\Multiverso\Test\unittests\LINK 1
Error C2220 warning treated as error - no 'object' file generated PerformanceProfilerDll C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include\cuda_runtime_api.h 1946
Error LNK1181 cannot open input file 'Cntk.Common-2.1.lib' Math D:\CNTK\Source\Math\LINK 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' UCIFastReader D:\CNTK\Source\Readers\UCIFastReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.PerformanceProfiler-2.1.lib' CompositeDataReader D:\CNTK\Source\Readers\CompositeDataReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' BinaryReader D:\CNTK\Source\Readers\BinaryReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.PerformanceProfiler-2.1.lib' CNTKTextFormatReader D:\CNTK\Source\Readers\CNTKTextFormatReader\LINK 1
Error C2220 warning treated as error - no 'object' file generated CNTKv2LibraryDll C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include\cuda_runtime.h 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' DSSMReader D:\CNTK\Source\Readers\DSSMReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' LibSVMBinaryReader D:\CNTK\Source\Readers\LibSVMBinaryReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' HTKMLFReader D:\CNTK\Source\Readers\HTKMLFReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.PerformanceProfiler-2.1.lib' CNTKBinaryReader D:\CNTK\Source\Readers\CNTKBinaryReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.PerformanceProfiler-2.1.lib' ImageReader D:\CNTK\Source\Readers\ImageReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' MathPerformanceTests D:\CNTK\Tests\UnitTests\MathPerformanceTests\LINK 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' SparsePCReader D:\CNTK\Source\Readers\SparsePCReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' CommandEval D:\CNTK\Tests\UnitTests\CommandEval\LINK 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' LMSequenceReader D:\CNTK\Source\Readers\LMSequenceReader\LINK 1
Error LNK1181 cannot open input file 'Cntk.Common-2.1.lib' MathTests D:\CNTK\Tests\UnitTests\MathTests\LINK 1
Error LNK1181 cannot open input file 'Cntk.Math-2.1.lib' LUSequenceReader D:\CNTK\Source\Readers\LUSequenceReader\LINK 1
Error C2220 warning treated as error - no 'object' file generated HTKDeserializers (Reader Plugins\HTKDeserializers) c:\local\boost_1_60_0-msvc-14.0\boost\spirit\home\support\char_encoding\iso8859_1.hpp 1
Error C2220 warning treated as error - no 'object' file generated CNTKv2LibraryDll C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include\cuda_runtime.h 1
Error LNK1181 cannot open input file 'Cntk.Common-2.1.lib' BrainScriptTests D:\CNTK\Tests\UnitTests\BrainScriptTests\LINK 1
Error C2220 warning treated as error - no 'object' file generated CNTKv2LibraryDll C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include\cuda_runtime.h 1
Error LNK1181 cannot open input file 'Cntk.PerformanceProfiler-2.1.lib' ReaderTests D:\CNTK\Tests\UnitTests\ReaderTests\LINK 1
Error C2220 warning treated as error - no 'object' file generated CNTKv2LibraryDll C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\include\cuda_runtime.h 1
Error C2220 warning treated as error - no 'object' file generated CNTKv2LibraryDll c:\local\Opencv3.1.0\build\include\opencv2\core\mat.hpp 1965
Error C2220 warning treated as error - no 'object' file generated CNTKv2LibraryDll c:\local\Opencv3.1.0\build\include\opencv2\core\mat.hpp 1965
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' CPPExtensibilityExamplesLibrary D:\CNTK\Examples\Extensibility\CPPLib\LINK 1
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' CPPExtensibilityExamples D:\CNTK\Examples\Extensibility\CPP\LINK 1
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' BinaryConvolutionLib D:\CNTK\Examples\Extensibility\BinaryConvolution\BinaryConvolutionLib\LINK 1
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' ProposalLayerLib D:\CNTK\Examples\Extensibility\ProposalLayer\ProposalLayerLib\LINK 1
Error MSB3073 The command ".\vsbuild.bat "D:\CNTK\x64\Release" "false" "true" "2.1" "c:\local\swigwin-3.0.10" "" "" "" "c:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntkdev-py35" """ exited with code 1. PythonBindings C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets 48
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' CNTKLibraryJavaBinding D:\CNTK\bindings\java\Swig\LINK 1
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' CNTKLibraryCPPEvalExamplesTest D:\CNTK\Tests\EndToEndTests\EvalClientTests\CNTKLibraryCPPEvalExamplesTest\LINK 1
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' CNTKLibraryCSBinding D:\CNTK\bindings\csharp\Swig\LINK 1
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' V2LibraryEndToEndTests D:\CNTK\Tests\EndToEndTests\CNTKv2Library\EndToEndTests\LINK 1
Error LNK1181 cannot open input file 'Cntk.Common-2.1.lib' EvalDll D:\CNTK\Source\EvalDll\LINK 1
Error LNK1181 cannot open input file 'Cntk.Eval-2.1.lib' CPPEvalClientTest D:\CNTK\Tests\EndToEndTests\EvalClientTests\CPPEvalClientTest\LINK 1
Error LNK1181 cannot open input file 'Cntk.Eval-2.1.lib' CPPEvalExtendedClientTest D:\CNTK\Tests\EndToEndTests\EvalClientTests\CPPEvalExtendedClientTest\LINK 1
Error LNK1181 cannot open input file 'Cntk.Eval-2.1.lib' EvalTests D:\CNTK\Tests\UnitTests\EvalTests\LINK 1
Error LNK1181 cannot open input file 'Cntk.Eval-2.1.lib' EvalWrapper D:\CNTK\Source\Extensibility\EvalWrapper\LINK 1
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' CNTK D:\CNTK\Source\CNTK\LINK 1
Error LNK1181 cannot open input file 'Cntk.Core-2.1.lib' NetworkTests D:\CNTK\Tests\UnitTests\NetworkTests\LINK 1

[ChunHsinWang]

I have manged to compile the image deserializer by turning off some projects' treat warning as errors. I will close this for now.