Enable rendering of images on GitHub and after kernel restart

[kolibril13]

I just wrote a small ImageWidget with anywidget and React.

Now, it would be amazing if it was possible to render that image on GitHub, and even after kernel restart! ✨

That is possible with from IPython.display import Image See notebook1.
But that does not work when I use from ipywidgets import Image See notebook2
Likewise, it does not work with anywidget. See notebook3
Here's an overview summary how it currently looks like:

Related issues:

jupyterlab/jupyterlab#15361

jupyter-widgets/ipywidgets#2280

[manzt]

But that does not work when I use from ipywidgets import Image

Yes, this is expected behavior for widgets. It seems you've found and linked the related issues, so I don't really know what to say other than to explain why this is happening.

When a Python object is the last element in a notebook cell, the ipython kernel looks for a _repr_mimbundle_ method and invokes that returns a dictionary keyed MIME type.

You can inspect the mimebundles by calling the _repr_mimebundle_() yourself:

import IPython.display

IPython.display.Image("./dog.jpg")._repr_mimebundle_()
# ({'image/jpeg': '/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAYEBQYFBAYGBQYHBwYIChAKCgkJChQODwwQFxQYGBcUFhYaHSUfGhsjHBYWICwgIyYnKSopG

import ipywidgets

with open("./dog.jpg", "rb") as f:
    data = f.read()
    
ipywidgets.Image(value=data)._repr_mimebundle_()
# {
#  'text/plain': "Image(value=b'\\xff\\xd8\\xff\\xe0\\x00\\x10JFIF\\x00\\x01\\x01\\x00\\x00\\x01\\x00\\x01\\x00\\x00\\xff\\xdb\\x00C\\x00\\x06\\x04\\x0…",
#  'application/vnd.jupyter.widget-view+json': {
#    'version_major': 2,
#    'version_minor': 0,
#    'model_id': '57bf31944747439a96a70e368d2ef891'
#   }
# }

This dictionary gets saved to the notebook (JSON) and then is interpreted later by whatever environment is rendering the JSON notebook.

As you can see, for the IPython.display the image data is embedded as a base64-encoded string, whereas the widget has a plain text repr as well as a custom application/vnd.jupyter.widget-view+json MIME type describing the widget. Rendering this MIME type requires executing JavaScript, so many static renderers (like GitHub) fallback to using the plain text repr. I don't anticipate that there will ever be support for executing JavaScript because it would be a security concern for GitHub.

It might be possible to mutate the notebook cell and save the image/jpeg yourself, but this is not what Widgets were designed for and would likely require some hacky work-arounds to get the data out of the front end into the cell.

For example you could make a notebook where the last cells runs over the notebook and mutates the notebook output cells embedding the most current Image.value (the image data).

import base64
import ipywidgets

with open("./dog.jpg", "rb") as f:
    data = f.read()

img = ipywidgets.Image(value=data, width="500px")
img # displays a widget

import json
import base64
import ipywidgets.widgets.widget

# read the current notebook file
with open("Untitled.ipynb", "r") as f:
    nb = json.load(f)
    # iterate over the notebook cells
    for cell in nb["cells"]:
        if cell["cell_type"] != "code":
            continue
        # iterate over the outputs for the cell
        for output in cell["outputs"]:
            # grab the model_id for the embeded widget output cell
            if model_id := output.get("data", {}).get('application/vnd.jupyter.widget-view+json', {}).get("model_id", None):
                # if there is a model_id see if we have the widget in our current session
                widget = ipywidgets.widgets.widget._instances.get(model_id)
                if isinstance(widget, ipywidgets.Image):
                    # add the image/jpeg data to the notebook cell
                    output["data"]["image/jpeg"] = base64.encodebytes(widget.value).decode("utf-8")
            
# write the new notebook to a different file
with open("with_embedded_images.ipynb", "w") as f:
    f.write(json.dumps(nb))

Alternatively, you could also just override the _repr_mimbundle_ with the metadata directly:

import ipywidgets
import base64

class MyImageWidget(ipywidgets.Image):
    def _repr_mimebundle_(self, *args, **kwargs):
        mimebundle = super()._repr_mimebundle_(*args, **kwargs)
        mimebundle["image/jpeg"] = base64.encodebytes(self.value).decode("utf-8")
        return mimebundle

with open("./dog.jpg", "rb") as f:
    data = f.read()

img = MyImageWidget(value=data, width="500px")
img # displays a widget and embds the image data in the notebook

However, in this case if value ever changes the value embedded in the notebook JSON will remain the original value.

Of course, both of these examples only account for image/jpeg and would need to be extended to embed other image MIME types. Also I'd avoid modifying the notebook in place or be very careful doing so. I made a mistake writing this code snippet and it crashed my Jupyter session. Again, this kind of in place modification is not something Jupyter Widgets were designed for.

[kolibril13]

Thank you so much for these insights, I really appreciate your time and effort.

I agree, modifying the notebook does not sound like best practice.

Your last example that overwrites the repr_mimbundle with mimebundle["image/jpeg"] = ... works great and renders great on GitHub as well -> notebook4

I've extended your example by metadata = {"image/jpeg": {"width": self.width}}, so that the width is also stored in the notebook metadata, here's the updated example:

import ipywidgets
import base64

class MyImageWidget(ipywidgets.Image):
    def _repr_mimebundle_(self, *args, **kwargs):
        mimebundle = super()._repr_mimebundle_(*args, **kwargs)
        encoded_image = base64.encodebytes(self.value).decode("utf-8")
        mimebundle["image/jpeg"] = encoded_image
        metadata = {"image/jpeg": {"width": self.width}}
        return mimebundle, metadata

with open("./cute_dog.jpg", "rb") as f:
    data = f.read()

img = MyImageWidget(value=data, width="300px")
img

Next, I want to explore how to embed an image into a notebook after some interactions in the widget happened.
I think this is a use case that a lot of notebook users might encounter, and therefore it would be great to have a button that converts a widget into an image of image/jpeg type.

I've just written a prototype here: https://github.com/kolibril13/anywidget-image/blob/main/js/widget_convert.jsx#L5-L19
which adds a convert button

The button does not yet have an effect.
Do you have a suggestion, how the mimetype conversion could happen here?
I've introduced the traitlet embed_image that changes to True when the button is pressed.
From the python side, it could look like this

    embed_image = Bool(False).tag(sync=True)

    @observe("embed_image")
    def _observe_embed_image(self, change):
        print("Converted the image to image/jpeg mime type")
        # now some conversion code here.

but is there maybe a way to make this conversion directly from the JavaScript/React side?