Enabling Inference using OpenVINO and GPU

[syafie-nzm]

Hi, I have question is it possible to use OpenVINO as inference engine and run the inference on purely GPU/iGPU?
Right now I managed to run segmentation inference example with this script (not specify any inference device):

import fast

image_name = "/path/images/CMU-1.svs"

importer = fast.WholeSlideImageImporter.create(image_name)

tissueSegmentation = fast.TissueSegmentation.create()\
    .connect(importer)

generator = fast.PatchGenerator.create(
        256, 256,
        magnification=20,
        overlapPercent=0,
    ).connect(importer)\
    .connect(1, tissueSegmentation)

segmentation = fast.SegmentationNetwork.create(
        '/path/model/high_res_nuclei_unet.onnx',
        scaleFactor=1./255.,
        inferenceEngine='OpenVINO',
    ).connect(generator)

stitcher = fast.PatchStitcher.create()\
    .connect(segmentation)

finished = fast.RunUntilFinished.create()\
    .connect(stitcher)

renderer = fast.ImagePyramidRenderer.create()\
    .connect(importer)

segment = fast.SegmentationRenderer.create(
        borderOpacity=0.5
        ).connect(stitcher)

fast.SimpleWindow2D.create()\
    .connect(renderer)\
    .connect(segment)\
    .run() 

And here is the snapshot of the CPU/GPU utilization, I don't know and not sure if this already consider utilize the GPU or not.

I think this can be optimize by running the inference purely on GPU since OpenVINO support inferencing on multi device (CPU/GPU/NPU). To add I am testing on Intel Core Ultra 7 155H it has iGPU and NPU, wanted to test the inference performance on both seperately.

I did try to add .setDevice('GPU') on the SegmentationNetwork (blindly try, sorry for this silly approach)

segmentation = fast.SegmentationNetwork.create(
        '/home/user/work/fast/high_res_nuclei_unet.onnx',
        scaleFactor=1./255.,
        inferenceEngine='OpenVINO',
    ).setDevice('GPU').connect(generator)

I got this error:
TypeError: ProcessObject.setDevice() missing 1 required positional argument: 'device'

This is so far I can get, now I know it needs two argument deviceNumber and device. I don't know what the right 'variable' to this argument. Hope this approach is close to the solution.
May anyone leads me to the right path/answer @smistad @andreped. Thank you
Thank you too for this amazing work!

[syafie-nzm]

hi, to update this discussion.

I did take sometime to read FastPathology paper. it did answer my curiosity! Fast can run specifically on OpenVINO GPU.
The question right now, I don't know how to do that :(

Since from the paper it has benchmark data, I also try to find the benchmark script that the authors used, so that I can reproduce or learn how to enable it. I cannot find it (I might not looking deep enough).

Again I am tagging @smistad and @andreped, in case you two missed my discussion.
Thank you in advance for the time entertaining this humble question/discussion.

[andreped]

Hello, @syafie-nzm :]

Since from the paper it has benchmark data, I also try to find the benchmark script that the authors used, so that I can reproduce or learn how to enable it. I cannot find it (I might not looking deep enough).

Here is the original C++ code used for the benchmark:
https://github.com/AICAN-Research/FAST-Pathology/blob/master/misc%2Finference_study%2FmeasurePipelinePerformance.cpp

But this runs a really old FAST version, would think a lot of the code here is no longer relevant, but could give you some hints.

If that does not work, I can start digging into the source code to find how to do this, but try this first :]

[smistad]

Hi

By default FAST's OpenVINO engine automatically selects which device to use. Not sure what it uses to determine which device is best.

To see what it uses you can add the following line above your segmentation = fast.SegmentationNetwork.create line:
fast.Reporter.setGlobalReportMethod(fast.Reporter.COUT)

You should then see an output like so:

INFO [140202277758784] Inference engine OpenVINO selected
INFO [140202277758784] OpenVINO found device: CPU
INFO [140202277758784] OpenVINO found device: GNA
INFO [140202277758784] OpenVINO found device: GPU
INFO [140202277758784] OpenVINO loaded model
INFO [140202277758784] Found input node input_1:0 with shape [?,256,256,1] and index 0
INFO [140202277758784] Guessed image ordering to be channel last as shape was -1 256 256 1 
INFO [140202277758784] Found output node conv2d_26/truediv:0 with shape [?,256,256,3]
INFO [140202277758784] OpenVINO successfully compiled model
INFO [140202277758784] OpenVINO is running network on CPU

To my surprise I notice that OpenVINO now automatically selects the CPU for this specific model, and not the GPU. Maybe OpenVINO believes this model will run faster on the CPU. Not sure.

Unfortunately, I don't think there is a way of selecting which device to use from Python right now. This is something we should add though.

[syafie-nzm]

@andreped thank you for sharing the benchmark script, now i can see how to implement it :)

@smistad thank you for this, yeah i did check on my side also it runs on CPU. So it is confirm it can be implemented on C++, I will try to migrate the python pipeline to C++ and will share here for anyone references. Bear with me my C++ is rusty haha :D

[syafie-nzm]

#include <FAST/Algorithms/NeuralNetwork/InferenceEngineManager.hpp>
#include <FAST/Visualization/SimpleWindow.hpp>
#include <FAST/Visualization/SegmentationRenderer/SegmentationRenderer.hpp>
#include <FAST/Importers/WholeSlideImageImporter.hpp>
#include <FAST/Visualization/ImagePyramidRenderer/ImagePyramidRenderer.hpp>
#include <FAST/Algorithms/ImagePatch/PatchGenerator.hpp>
#include <FAST/Algorithms/ImagePatch/PatchStitcher.hpp>
#include <FAST/Algorithms/NeuralNetwork/SegmentationNetwork.hpp>
#include <FAST/Algorithms/TissueSegmentation/TissueSegmentation.hpp>
#include <FAST/Reporter.hpp>

using namespace fast;

int main(int argc, char** argv) {

        auto importer = WholeSlideImageImporter::create(Config::getTestDataPath() + "/WSI/CMU-1.svs");
        
        auto tissueSegmentation = TissueSegmentation::create()->connect(importer);
        
        auto generator = PatchGenerator::create(256, 256, 1, 0)
            ->connect(importer)
            ->connect(1, tissueSegmentation);

        Reporter::setGlobalReportMethod({} , Reporter::COUT);

        auto network = SegmentationNetwork::create(
                Config::getTestDataPath() + "NeuralNetworkModels/high_res_nuclei_unet.onnx",
                {}, {}, "OpenVINO")
        ->connect(generator);
        network->setScaleFactor(1.0f / 255.0f);
        network->getInferenceEngine()->setDeviceType(InferenceDeviceType::GPU);

        auto stitcher = PatchStitcher::create()
            ->connect(network);
            
        auto renderer = ImagePyramidRenderer::create()
            ->connect(importer);
            
        auto segmentationRenderer = SegmentationRenderer::create()
            ->connect(stitcher);
        segmentationRenderer->setBorderOpacity(0.5);

        // auto heatmapRenderer = HeatmapRenderer::create(true, false, 0.5, 0.3)
        //     ->connect(stitcher);
        
        auto window = SimpleWindow2D::create()
            ->connect({renderer, segmentationRenderer});
        window->run();
}

Here is the C++ implementation, i also added the Reporter::setGlobalReportMethod({} , Reporter::COUT) to monitor which device it use. Can someone check if my implementation is correct or not? I followed @andreped benchmark script to set the inference device.

Unfortunately, it is still running on CPU, this is the log from the reporter:

NFO [137143437754048] Inference engine OpenVINO selected
INFO [137143437754048] OpenVINO found device: CPU
INFO [137143437754048] OpenVINO found device: GNA
INFO [137143437754048] OpenVINO found device: GPU
INFO [137143437754048] OpenVINO loaded model
INFO [137143437754048] Found input node input_1:0 with shape [?,256,256,3] and index 0
INFO [137143437754048] Guessed image ordering to be channel last as shape was -1 256 256 3 
INFO [137143437754048] Found output node conv2d_26/truediv:0 with shape [?,256,256,2]
INFO [137143437754048] OpenVINO successfully compiled model
INFO [137143437754048] OpenVINO is running network on CPU

Double confirmed with CPU and GPU utilization still the same as previously shared. Can someone reproduce this and see whether it behaves the same?

I tried with other model NeuralNetworkModels/wsi_classification.onnx. Still running on CPU.

Thank you guys

[andreped]

To my surprise I notice that OpenVINO now automatically selects the CPU for this specific model, and not the GPU. Maybe OpenVINO believes this model will run faster on the CPU. Not sure.

@smistad made this comment about this before. But I believe we were able to set device from C++ before. Maybe that is no longer the case?

@syafie-nzm Hmm, this pattern looked a bit different than what I did way back in 2020. I see that you did:

auto network = SegmentationNetwork::create(
                Config::getTestDataPath() + "NeuralNetworkModels/high_res_nuclei_unet.onnx",
                {}, {}, "OpenVINO")
network->getInferenceEngine()->setDeviceType(InferenceDeviceType::GPU);

While I did (with some modifications):

auto network = SegmentationNetwork::create();
network->setInferenceEngine("OpenVINO");
network->getInferenceEngine()->setDeviceType(InferenceDeviceType::GPU);
network->load(path/to/model.onnx);

I would expect both of these implementations to work, but perhaps that is not the case?

Unfortunately, I do not have a device available with Intel CPU/GPU atm, nor have time to run tests as I am preparing for my PhD defence, but perhaps @smistad does?

[smistad]

This API needs improvement, there is a mix of old (pre v4) API and the latest which is more python friendly.

I have been thinking about this and I think a better and more clean solution to this will be to drop the "string" argument of inference engine and instead have something like this:

import fast
engine = fast.InferenceEngine.get('OpenVINO')
engine.setDeviceType(fast.InferenceDeviceType_GPU)
network = fast.SegmentationNetwork.create('model.onnx', inferenceEngine=engine)

Should be quite straightforward to implement as well.