_image.data is undefined

[kkmehta03]

Hello!
I'm trying to write a segmentation drawing extension tool over OHIF Viewer.
I'm referring to this example, to get the Part10 dcmjs representation of the segmentation.
In the example this image promise returns the image without any issues and creates the segmentation, as shown below:

But in the viewer, the same image Promise returns an image without the property "data" in it, which is causing the mentioned error. Shown below:


I wasn't sure if the question is more suited under cornerstone tools repository, so I just put it up here because the error is coming under dcmjs script.
Can anyone help me resolve this issue?

[pieper]

Hi @KhyatiMehta3 - it's hard to know from your screenshots. Are you planning to make your implementation available? It would be great if you could do that, and start by making a pull request so that others can try your code and maybe help debug it.

It can be tricky to work on features like this that require possible changes to both dcmjs and ohif, but it's possible to set up an environment for testing (I think there are instructions in the ohif docs for that).

[kkmehta03]

Hi @pieper thank you for your response! I'll send a PR. and refer this issue there as well.

[helghast79]

Hello @pieper and @KhyatiMehta3, I'm having exactly same issue here. Could this be related to the type of loader being used when loading the dicom files? Because, if i use the cornerstone.loadImage() function with an imageId set for dicomWeb the data property is there in my viewer but if I use wadouri it's not.
Anyway, the generateSegmentation function in dcmjs expects image.data.byteArray.buffer. So, how can we generate this arraybuffer from the existing data? the getPixelData() function does not contain all the byteArray info and fails if used.
Any ideas?

[sinanmb]

Hi @pieper , @helghast79 , @KhyatiMehta3 , I'm facing the same problem, have you found a way to address it?

[pieper]

I'm not sure myself, that sounds like a cornerstone issue with the ways the segmentation tool is implemented.

Segmentation display is working in OHIF with cornerstone and vtk.js, so maybe looking at that implementation will help. Segmentation creation is not yet in OHIF, just in a few demos like the one in dcmjs.

https://viewer.ohif.org/viewer/1.2.826.0.13854362241694438965858641723883466450351448

[helghast79]

Hello @pieper and all,

I don't think it as anything to do with the segmentation tool since the data property exists in the images object even without the segmentation extension. If I use the imageIds from this example which have a "dicomweb" loader prefix then the data property is there in any version of Ohif viewer I have tried.
On the other hand, if I use other imageIds with "wadors" loader prefix, then the data property dosen't exist. So maybe it has something to do with the cornerstone-wado-image-loader. Hope it will make sense for someone ...

As to a possible workaround and with the aim of using DCMJS functions like the generateSegmentation which require a datatset object stored in the image data property, we can generate this dataset with dicom-parser library with the help of cornerstone, cornerstoneTools and OHIF.utils to generate the required data.

import cornerstone from 'cornerstone-core';
import cornerstoneTools from 'cornerstone-tools';
import OHIF from '@ohif/core';
import dcmjs from 'dcmjs';
import dicomParser from 'dicom-parser';

const { studyMetadataManager, DicomLoaderService } = OHIF.utils;
const enabledElements = cornerstone.getEnabledElements()
const element = enabledElements[0].element

const globalToolStateManager =
cornerstoneTools.globalImageIdSpecificToolStateManager;
const toolState = globalToolStateManager.saveToolState();

const stackToolState = cornerstoneTools.getToolState(element, "stack");
const imageIds = stackToolState.data[0].imageIds;

let imagePromises = [];
for (let i = 0; i < imageIds.length; i++) {
    imagePromises.push(cornerstone.loadImage(imageIds[i]));
}

const { getters } = cornerstoneTools.getModule('segmentation');
const { labelmaps3D } = getters.labelmaps3D(element);

Promise.all(imagePromises)
    .then(async images => {
      const studyInstanceUID = cornerstone.metaData.get('StudyInstanceUID', images[0].imageId)
      const studyMetadata = studyMetadataManager.get(studyInstanceUID)
      const displaySet = studyMetadata._displaySets.filter(ds => ds.images && ds.images.length)[0]
      const studies = [studyMetadata._data]
      //load the dicom raw data with DicomLoaderService
      const arrayBuffer = await DicomLoaderService.findDicomDataPromise(displaySet, studies);
      const byteArray = new Uint8Array(arrayBuffer);
      //use dicomParser to get a dataset object
      const dataset = dicomParser.parseDicom(byteArray, { untilTag: '' });
      
      //set the missing data property in all images
      images = images.map(image => ({ ...image, data: dataset }))

      //use dcmj segmentation class to generate segmentation in dicom-seg format
      const segBlob = dcmjs.adapters.Cornerstone.Segmentation.generateSegmentation(
        images,
        labelmaps3D
      );

      //now download the blob
      window.open(URL.createObjectURL(segBlob))
      //or send it to pacs server with axios for example
      //const response = await axios.post(....)
    })
    .catch(err => {
      console.log(err)
    });
}

Warning: labelmaps3D array will probably need to be adapted also because DCMJS expects the individual labelMap3d elements to have a metadata property with data that follows some schema, so in order to use the generateSegmentation function mentioned above we need to make sure it's there.

cheers

[sinanmb]

Hi @helghast79, thank you so much for your detailed answer! I'll try your approach soon.

[sinanmb]

Hi @helghast79, I have tried your solution and it allowed me to advance further however as you mentioned there was an issue with labelmaps3D. Should I use a similar logic to fill in labelmaps3D with the dataset?

[helghast79]

Hello Sinan,

I don't know exactly what's the problem with your labelMaps3D but probably is the lack of metadata associated with each individual labelMap3d. These should be set according to the specifics of each segment before running dcmjs generateSegmentation function. You can however, mock this data with something like:

labelMap3d.metadata[segIndex] = {
              RecommendedDisplayCIELabValue: dcmjs.data.Colors.rgb2DICOMLAB([
                1,
                0,
                0
              ]),
              SegmentedPropertyCategoryCodeSequence: {
                CodeValue: "T-D0050",
                CodingSchemeDesignator: "SRT",
                CodeMeaning: "Tissue"
              },
              SegmentNumber: segIndex.toString(),
              SegmentLabel: "Tissue " + segIndex.toString(),
              SegmentAlgorithmType: "SEMIAUTOMATIC",
              SegmentAlgorithmName: "Slicer Prototype",
              SegmentedPropertyTypeCodeSequence: {
                CodeValue: "T-D0050",
                CodingSchemeDesignator: "SRT",
                CodeMeaning: "Tissue"
              }
            }

(took it from generateMockMetadata function from here. All props but the RecommendedDisplayCIELabValue are required)

There is also a correction to the workaround I posted before. If you use that code you'll find all segmentations collapsed in the first image. That's because the dataset added to each image is the same for all images. While I couldn't find a way to generate an arrayBuffer that contains only data from each image I found that generating an instance of the image metadata with cornerstone.metaData.get('instance', imageId), outputs a dataset similar to the one dcmjs generates inside de generateSegmentation function with the image buffers. So, my current workaround is to tweak dcmj main library and add a method that accepts a dataset instead of images to generate the segmentation. The code can look like the following:

• dcmjs.js
  (add this function to the library and make sure it is exposed to the exported Segmentation class in Segmentation$3 and Segmentation$2)

function generateSegmentationWithDataset(dataset, inputLabelmaps3D) {
    const userOptions = arguments.length > 2 && arguments[2] !== undefined ? arguments[2] : {};
    const multiframe = Normalizer.normalizeToDataset(dataset);
    const segmentation = new Segmentation([multiframe], userOptions);
    return fillSegmentation(segmentation, inputLabelmaps3D, userOptions);
  }

(note: I prefer to make a copy of the modified dcmjs close to the script using it and then import it instead)

• yourScript.js

const isMultiframe = imageIds[0].includes("?frame")

let dataset = []

if (isMultiframe) {
      dataset.push( cornerstone.metaData.get('instance', imageId[0]) )

} else {
    imageIds.forEach(imageId => {
        let instance = cornerstone.metaData.get('instance', imageId)
        instance._meta = [] //array needs to be present
        dataset.push(instance)
    })
}

const segBlob = dcmjs.adapters.Cornerstone.Segmentation.generateSegmentationWithDataset(
    dataset,
    labelMaps3d
)
//now download or upload this file somewhere

(I didn't test with multiframe images but followed similar logic found in dcmjs)

Hope It's not too much confusing... cheers :)

[sinanmb]

Thank you very much @helghast79 for taking the time to provide detailed explanations!

[sinanmb]

Hi @helghast79, it seems there's an issue with the segmentations not being loaded on the right image. Here's what I did:

Created a segmentation in a series on the first and fifth images. Saved the segmentation thanks to your updated logic then imported the segmentations in Orthanc. After that I visited the series again in OHIF Viewer and I noticed the segmentation were displayed on the last and fifth before last images of the series, instead of the first and fifth images. It seems something has been reversed.

I've tried reversing the loop on imageIds in your previous answer but that does not help. Can you please point me towards the way to have the segmentation be loaded on the correct images?

[helghast79]

Hello @sinanmb, I could not replicate this issue on any of my images set. So I'm just guessing here. Maybe some particular issue between the Dicom loader ordering and the ordering mechanism dcmjs uses. Most certainly the problem should be on saving the segmentation file and not on loading it. Some more details could help to figure out the reason behind this behaviour. Also, the code you use could help to replicate and debug non obvious errors.

note: I've submitted a pull request to include the function that generates the segmentation from an array of datasets as discussed above. While it's not included yet, you could take a look at the code changes and check if the you've followed the same path

[helghast79]

finally, I could replicate your issue @sinanmb. While making the pull request I made changes to source code and build dcmjs plugin. The release file then contained a lot of changes from the dcmjs plugin included in Ohif viewer, which was the one I've modified with the GenerateSegmentationFromDatasets function. So, I can't really tell whats going on with this newer version of dcmjs but seems that some change must be the cause to some series (those that for some reason have inverted ordering) to become out of sync from labelmaps ordering. Is this what's happening to you?

[sinanmb]

Hi @helghast79, thank you for getting back to me! Yes that is the problem I was facing. The way I have fixed the issue was to add this line: ReferencedSeriesSequence.ReferencedInstanceSequence.reverse(); inside the '_addPerFrameFunctionalGroups' function, in the file src/derivations/Segmentation.js

I feel it's a bit hacky though.

[helghast79]

@sinanmb, you could do that but then the ReferencedInstanceSequence series would always be reversed even when it's not supposed to. For example, if you have 2 segments, the _addPerFrameFunctionalGroups function will run twice and then the first segment will be in correct order and the other will be inverted. But actually, your fix gave me the hint of where the problem might be, since my modified version worked and the compiled didn't, I took a deeper look at the changes in terms of the ReferencedInstanceSequence and concluded that the problem was actually coming from the generation of the multiframe with multiframe = Normalizer.normalizeToDataset(datasets). Specifically the ImageNormalizer class of the normalizers.js file. In this class, the convertToMultiframe function creates the ReferencedInstanceSequence by pushing SOPInstanceUID from a sorted dataset into it. The thing is, the order has to be the same as the labelMaps, so the source of this SOPInstanceUID cannot be sorted when it happens. Changing these lines:
distanceDatasetPairs.forEach(function(pair) {
const dataset = pair[1];
to
this.datasets.forEach(function(dataset) {
//const dataset = pair[1];
will fix this problem for reversed and normal series

cheers

[sinanmb]

Thank you @helghast79 for letting me know! I will try your solution soon.

[sinanmb]

Hi @helghast79, I am currently able to generate and save segmentations that contain 1 single segment. I would like to be able to create multiple segments. I have tried adding a new object inside the labelmap2D object but was not successful. If you know how to do it, can you please point me in the right direction?

[thewildnath]

I know that it's quite an old thread but this might help somebody. I did manage to find a 'hacky' solution for getting 'image.data' for every image in OHIF V2, without having to fork dcmjs.

Similarly to the solution in the comment above, you can actually call DicomLoaderService.findDicomDataPromise with a slightly changed displaySet parameter for every image, such that the value of displaySet.images[0] matches the imageId that you are currently calling the function for. I wrote a small function that loads every image in a displaySet, alongside their data:

import OHIF from '@ohif/core';

import cornerstone from 'cornerstone-core';
import dicomParser from 'dicom-parser';

// Returns a list of uncalled promises that load all images
export default displaySet => {
  const imageIds = displaySet.images.map(image => image.getImageId());

  const loadImage = async imageId => {
    return cornerstone.loadImage(imageId).then(async image => {
      // In some cases (?), the image loader doesn't provide the 'data' field
      // This is very hacky
      if (image.data === undefined) {
        const newDisplaySet = {
          ...displaySet,
          images: displaySet.images.filter(img => img.getImageId() === imageId),
        };

        const arrayBuffer = await OHIF.utils.DicomLoaderService.findDicomDataPromise(
          newDisplaySet
        );

        const byteArray = new Uint8Array(arrayBuffer);
        const dataset = dicomParser.parseDicom(byteArray, { untilTag: '' });

        image.data = dataset;
      }

      return image;
    });
  };

  return imageIds.map(imageId => () => {
    return loadImage(imageId);
  });
};

You can use it as:

const imagePromises = getLoadPromisesFromDisplaySet(displaySet).map(fn =>
    fn()
);

Promise.all(imagePromises)
  .then(images => {
    // Call dcmjs
  });

Again, it's quite hacky and I'm not sure that it will work in all cases, but at least with images loaded over from Orthanc (or any DicomWeb endpoint probably) it seems to work alright.