GUI snapshot

[chenghuzi]

I found a way to freeze and show GUI snapshots:

1. embed the app layout to html content (using embed_minimal_html from ipywidgets.embed)
2. save the embedded content to a tmp file
3. load the html content using iframe (using IFrame from IPython.display)

But here are some deficits in this approach, which I'm trying to address and would like to share to see you have any ideas:

1. This involves saving and loading files from the disk and we cannot bypass this through replacing the src attribute in the <iframe> to srcdoc as the latter does not support cross-domain requests, while the embedded html content requires JS scripts.
2. It seems that the embed_minimal_html is not good at capturing Output widget.
   • When we initialize the layout for a certain Output, embed_minimal_html won't store the style.
   • When contents are dynamically added under an Output widget, embed_minimal_html won't store the content.
   • However, if we repeat the same function again in the next cell (manually), embed_minimal_html correctly stores everything.

[chenghuzi]

It seems jupyter-widgets/ipywidgets#2617 is pretty messy...

[jasmainak]

I was thinking a screenshot-based approach might be more successful.

Option 1: html2canvas as in: https://github.com/msjgriffiths/jupyter-shot/blob/master/jupyter-shot.ipynb
Option 2: selenium as in: https://github.com/innovationOUtside/iframe-shot

Option 1 if you're more comfortable with JS ... you can start with my PR. I got it working once but can't reproduce LOL. Option 2 if you want to keep it pure Python.

iframes have always felt messy to me ... but I'm not a web-developer. As long as it works, and is fully automated I think it's fine.

[jasmainak]

here is the code I started by the way: https://github.com/jasmainak/hnn-core/blob/hnn_gui_nbconvert/examples/gui/evoked_tutorial.ipynb. In case you missed it

[jasmainak]

okay after playing around with html2canvas some more, I think that approach is hopeless. Let's see what you can get. Other options are here:

https://ipywidgets.readthedocs.io/en/latest/embedding.html#python-interface

[chenghuzi]

yeah I tried nearly a whole day but it doesn't work. I guess we may have to use selenium finally. But let me try just one more round as I'd rather not to touch headless browsers if there's any chances.

[jasmainak]

I feel something to do with the security settings of the browsers ... because I have no problem with the code cells. It only fails with the ipywidgets.

I wonder if we should try with a very simple button/slider widget this procedure before we completely write it off.

[jasmainak]

Not super elegant but another option on the table along these lines:

import time

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains

from PIL import Image

driver = webdriver.Firefox()
driver.get('http://localhost:8910/notebooks/evoked_tutorial.ipynb?token=e1b5af1b39680df1bcd5f887ae1626fd2282332c9929dd0c')

actions = ActionChains(driver)

time.sleep(5.)
actions.key_down(Keys.SHIFT).key_down(Keys.ENTER).key_up(
    Keys.SHIFT).key_up(Keys.ENTER).perform()

time.sleep(10.)

element = driver.find_element_by_xpath(
    '//div[@class="p-Widget p-Panel jupyter-widgets widget-container widget-gridbox"]')
location = element.location
size = element.size

actions.move_to_element(element).perform()

driver.save_screenshot("shot.png")

x = location['x']
y = location['y']
w = size['width']
h = size['height']
width = x + w
height = y + h

im = Image.open('shot.png')
im = im.crop((int(x), int(y), int(width), int(height)))
im.show()

[chenghuzi]

😂 this is crazy

[chenghuzi]

I figured out the reason why embed_minimal_html doesn't work. The frontend widget view (the visible part) is updated after receiving information about state changes from the kernel, which makes it impossible for the python code to know whether the actual rendering process is finished or not. This becomes more severe when we have a stack of Outputs to be updated as the actual UI updates take longer time.

Will time.sleep work if we put that into the next cell? Not really. As the sleep function block everything, including the saving process. I tried to use threading to separate the embedded HTML saving process but it failed too.

Basically we need something that can "pause" the execution of cells without blocking the kernel.

[jasmainak]

Just to be clear, is it so that putting render_iframe in a separate cell will solve the issue? If so, I think that is a temporary solution even if not perfect. We could have one function that renders the GUI and another to "capture". Let us discuss on Thursday ... I feel we might be close.