New Tool: DICOM Overlays (C.9 Overlays)

[kofifus]

Hi,

Does cornerstone supports displaying DICOM overlays ? (C.9 Overlays - http://dicom.nema.org/dicom/2013/output/chtml/part03/sect_C.9.html)

It doesn't show in my tests.

Thank you!

[kofifus]

bump

[swederik]

It doesn't display them automatically, no. You can extract them from the DICOM object with dicomParser (or from .data after loading via the WADO Image Loader) and then create a Layer to display them on top of the image.

I'd be happy to have a tool or example to do this, if you get it working!

[kofifus]

Thanks swederik !

Any more information on how to "create a Layer to display them on top of the image" ?

[khadakeb]

To display DICOM overlays, you can refer example provided at:
https://github.com/cornerstonejs/cornerstone/tree/master/example/layers
To check above example in action:
https://rawgit.com/cornerstonejs/cornerstone/master/example/layers/index.html

[jamesg1]

I've been interested in trying out cornerstone more for displaying one or more layers on top of images from the dicom web loader. Is it possible to have these overlay layer(s) constructed on the fly per index of the series (say from pixel/shape manipulation of another image) and shown as you scroll through the series? I would want to keep the overlay layers in sync with the base image (viewport). The examples I've seen so far have a specific overlay layer created first.

[jdnarvaez]

What I ended up doing is modifying my image loader to include the parsed dataset in the image object along with the pixel data, then added a render listener so that when the image is rendered I can extract the overlay data and render it on top.

[kofifus]

jdnarvaez - can you provide a sample please ? I am really trying to make progresss with this .. thx!

[jdnarvaez]

@kofifus this should get you started: https://gist.github.com/jdnarvaez/3368ac3e4f7f4eb14bc5f453f8a1ed1b

the imageRendered method is what would be called from an image_rendered listener added to the element

also I store the parsed data set on the image as $presentationStateDataSet so that it can be referenced and the data extracted

[dannyrb]

Thanks for the code, @jdnarvaez 👍
I think this is a good 3rd party tool candidate, or something you'd be more likely to see in an imageLoader, but probably isn't a good candidate for core.

I'm going to "transfer" this issue to tools.

[kofifus]

Thanks for the code, @jdnarvaez 👍 👍 very helpful

@dannyrb, IMO this is very core - DICOM overlays are not an added tool but an integral part of the image and cornerstone should be able to show them (ideally with toggle/opacity) even without the tools package ...

[dannyrb]

@kofifus, you may be right. I don't have the qualifications to determine that one way or another. From my perspective, the tech side, we do most of our tag parsing and usage in loaders/tools. I don't think core ever uses get on a metaDataProvider.

Are DICOM overlays "rendering", or do they impact rendering? Is there a consistent visual result we expect if DICOM overlays are implemented?

@swederik, care to weigh in?

[kofifus]

The overlays are described by the dicom standard here - http://dicom.nema.org/dicom/2013/output/chtml/part03/sect_C.9.html

"An Overlay Plane describes graphics or bit-mapped text that is associated with an Image. It may also describe a Region of Interest in an Image.
Each Overlay Plane is one bit deep. Sixteen separate Overlay Planes may be associated with an Image."

"How an Overlay Plane is rendered is undefined; specifically there is no mechanism to specify with what color or intensity an Overlay Plane is to be displayed, except when rendered under the control of a Softcopy Presentation State SOP Instance."

Usually you want to:

• toggle a particular layer on or off
• determine the opacity of the layer
• color them (less commonly useful I think)

So I think that core should have functions like - ToggleOverlay(index), SetOverlayOpacity(index), setOverlayColor(index)

Then a Tool can be made to use those functions in some graphical way, as there is usually only one layer there can maybe be a single toggle tick and opacity slider etc ...

(The standard also specify ROI overlays which are a bit of a different story ...)

But if its not desirable for core to read tags then maybe it does not belong in core - not sure ..

[kofifus]

Ok I came up with something

See PRs - cornerstonejs/cornerstoneWADOImageLoader#239
and - cornerstonejs/cornerstone#347

Basically wadouri metaDataProvider now accepts the type 'overlays' and reads and returns an array of overlays if existing in the dataset

then in core drawImageSync, if the image has overlays they are added to the canvas.

Each overlay has a 'fillStyle' and 'hidden' properties that can be changed to toggle overlays or color them.

I have tested with single overlays, I don't have images containing multiple overlays or ROI. Perhaps someone can donate some examples.

Much of the code is a rewrite of @jdnarvaez solution above thanks!

Let me know what you think ..

[dannyrb]

@kofifus, thanks for taking the time to PR these concepts! I think the cornerstoneWADOImageLoader enhancement makes sense, and is implemented well. It does make me wonder if we should pull out the metaDataProvider into it's own repository, but that's just a side thought.

If you have any helpful, anonymized, images to test this functionality with, could you pass them along? I'm curious to see what these overlays are used for in practice. I'm also curious how a viewer like Radiant or Osirix handles overlays.

I'm less sure about the cornerstone-core changes. I value your time, and was curious of the answer myself, so I reached out to @swederik and a few others outside of GitHub. The consensus seems to be that the rendering logic for overlays should live in a tool.

If we don't expose granular enough APIs to accomplish this, or if you need assistance creating a 3rd Party Tool, I would be more than happy to help. If you disagree, and it's okay if you do, we can continue to discuss.

[jdnarvaez]

@kofifus I have updated the gist with the tag file. as far as examples, I suggest using the IHE test case for overlays.

[jdnarvaez]

ovly_p01.zip

[jdnarvaez]

FWIW, that contains both the dicom file and the presentation state, as some overlays are read from the PR and one layer of the overlays is read from the embedded pixel data.

[kofifus]

sure @dannyrb

Can you point out an existing tool that this is similar to and I can use as a skeleton ? I may give it a go ..
If the maintainers think a tool is better then we should try that but its more involved so I need more assistance or maybe someone else can take it from here ?

On a side note, regarding your metadata repository comment, personally I don't like the multiple repositories trend, I feel Javascript offers more than enough mechanisms for encapsulation (ie modules) as is and multiple repositories just create an unnecessary overhead.

Also, we should probably look at adding an equivalent to the new wadouri processing of type 'overlayPlaneModule' to imageLoader\wadors\metaData\metaDataProvider.js, I am not sure what is the purpose of wadors and how to do that, maybe someone else can chip in ...

@jdnarvaez - in https://gist.github.com/jdnarvaez/3368ac3e4f7f4eb14bc5f453f8a1ed1b#file-overlays-js-L7

I don't follow that for loop .. the way I read the standard there can be up to 16 layers in even Group Numbers 6000-601E (not ee) ?

[kofifus]

I removed the cornerstone PR (cornerstonejs/cornerstone#347)

and created an overlays tool - #788

The tool is very simple having only enable and disable

Hope this helps :)

[dannyrb]

I've created a feature branch for us to iterate on the cornerstone-tools portion of this tool:

• feature/overlay-tool

When new code is merged, a release is cut and tagged on NPM for @cornerstonejs/tools@feature-overlay-tool

I have a CodeSandbox up that is using the cornerstoneWADOImageLoader and cornerstoneTools branches that pertain to this feature:

https://codesandbox.io/s/1qj7j8op0l

[dannyrb]

@kofifus, here's an update:

• The image loader now stores the information in overlayPlaneModule
• The tool retrieves the data from the overlayPlaneModule and finds 2 overlays in our test DICOM
• When disabled, the image shows 2-lines of text. When enabled, the image shows no new lines of text

Here is what I am able to see in a desktop viewer:

I haven't found a viewer yet that shows all 6-lines, but I only have a few installed locally.

[kofifus]

Can you send me a link to download the dicom image pls ?

[dannyrb]

It's a bit higher up in this thread. JD shared it: #780 (comment)

[dannyrb]

Updated Sandbox: https://codesandbox.io/s/1qj7j8op0l

This is the PR that fixed things up: #826

If I do a DICOM dump of the file, using dicom-parser, I don't see any other overlays hiding around:

I see a similar result in my local tag dumper. I'm wondering if the "6 lines of text" for the image isn't entirely accurate?

[jdnarvaez]

The code i provided should read all six layers as we render all six in our viewer. I’ll try to have a look at this this week

[kofifus]

Possibly the missing 2 layers are in the presentation state (see #780 (comment)) which cornerstone does not support ?

[jdnarvaez]

Yep, that’s the case.

[kofifus]

cool, I believe this can be closed :)

[dannyrb]

@all-contributors please add @jdnarvaez for sharing his code for Overlays

[allcontributors]

@dannyrb

I've put up a pull request to add @jdnarvaez! 🎉

[kofifus]

@jdnarvaez - in https://gist.github.com/jdnarvaez/3368ac3e4f7f4eb14bc5f453f8a1ed1b#file-graphicoverlays-js-L38 you again call setTransform after calling just above, can you pls explain what is the purpose of the second call ? Thanks!

[jdnarvaez]

@kofifus the transform is first set to the coordinates of the current canvas, then it it is saved, then it is set to 1:1 scaling, then reset to the previous scaling. This allows us to draw the overlays in a uniform coordinate system, so that any transformations or scaling by the user are also applied to the overlays

[luvargas1]

Should the cornestone magnify tool work on overlay data?