Cluster widget displaying strangely

[jacobtomlinson]

Was playing with the GPU instructions and the cluster widget displayed a bit funny. I've seen rich do this when you print a bunch of stdout and show a rich console at the same time.

In [6]: coiled.create_software_environment(
   ...:     name="rapids-23-04-nightly-conda",
   ...:     account="dask",
   ...:     gpu_enabled=True,  # sets CUDA version for Conda to ensure GPU version of packages get installed
   ...:     conda={
   ...:         "channels": ['rapidsai-nightly', 'conda-forge', 'nvidia'],
   ...:         "dependencies": ['rapids=23.04', 'python=3.10', 'cudatoolkit=11.5'],
   ...:     },
   ...: )
...
In [7]: cluster = coiled.Cluster(
   ...:     account="dask",
   ...:     software="rapids-23-04-nightly-conda",  # specify the software env you just created
   ...:     jupyter=True,  # run Jupyter server on scheduler
   ...:     scheduler_gpu=True,  # add GPU to scheduler
   ...:     n_workers=4,
   ...:     worker_gpu=1,  # single T4 per worker
   ...:     worker_class="dask_cuda.CUDAWorker",  # recommended
   ...: )
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                                                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮
│                  https://cloud.coiled.io/clusters/173861/overview?account=dask                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮
│                                                ││                                                │
│ Cluster Name: dask-2fb24011-4                  ││ Region: us-east-2                              │
│                                                ││                                                │
│ Scheduler Status: starting.                    ││ Scheduler Instance Type: g4dn.xlarge           │
│                                                ││                                                │
│ Dashboard Address: .                           ││ Worker Instance Type(s): g4dn.xlarge (4)       │
│                                                ││                                                │
│                                                ││ Workers Requested: 4                           │
│                                                ││                                                │
╰────────────────────────────────────────────────╯╰────────────────────────────────────────────────╯
╭────────────────────────── Dask Worker States (2023/03/22 11:05:03 GMT) ──────────────────────────╮
│                Provisioning            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0                │
│                Booting Instance        ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0                │
│                Downloading environment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4                │
│                Ready                   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0                │
│                Stopping                ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0                │
│                Stopped                 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0                │
│                                                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

[dchudz]

@jacobtomlinson Can we get your pip list? Especially the rich version.

[jacobtomlinson]

Package              Version
-------------------- -----------
aiobotocore          2.4.0
aiohttp              3.8.3
aioitertools         0.11.0
aiosignal            1.3.1
anyio                3.6.2
appnope              0.1.3
argon2-cffi          21.3.0
argon2-cffi-bindings 21.2.0
asttokens            2.1.0
async-timeout        4.0.2
attrs                22.1.0
backcall             0.2.0
backoff              1.11.1
beautifulsoup4       4.11.1
bleach               5.0.1
bokeh                3.0.2
boto3                1.24.59
botocore             1.27.59
certifi              2022.9.24
cffi                 1.15.1
charset-normalizer   2.1.1
click                8.1.3
cloudpickle          2.2.0
coiled               0.5.6
commonmark           0.9.1
contourpy            1.0.6
dask                 2022.11.0
debugpy              1.6.3
decorator            5.1.1
defusedxml           0.7.1
distributed          2022.11.0
entrypoints          0.4
executing            1.2.0
fastjsonschema       2.16.2
filelock             3.10.0
frozenlist           1.3.3
fsspec               2022.11.0
HeapDict             1.0.1
idna                 3.4
importlib-metadata   4.13.0
ipykernel            6.17.1
ipython              8.6.0
ipython-genutils     0.2.0
ipywidgets           7.7.2
jedi                 0.18.1
Jinja2               3.1.2
jmespath             1.0.1
jsondiff             2.0.0
jsonschema           4.17.0
jupyter-client       7.3.4
jupyter_core         5.0.0
jupyter-server       1.23.2
jupyterlab-pygments  0.2.2
jupyterlab-widgets   1.1.1
locket               1.0.0
MarkupSafe           2.1.1
matplotlib-inline    0.1.6
mistune              2.0.4
msgpack              1.0.4
multidict            6.0.2
nanoid               2.0.0
nbclassic            0.4.8
nbclient             0.7.0
nbconvert            7.2.5
nbformat             5.7.0
nest-asyncio         1.5.6
notebook             6.5.2
notebook_shim        0.2.2
numpy                1.23.4
packaging            21.3
pandas               1.5.1
pandocfilters        1.5.0
parso                0.8.3
partd                1.3.0
pexpect              4.8.0
pickleshare          0.7.5
Pillow               9.3.0
pip                  22.2.2
platformdirs         2.5.4
prometheus-client    0.15.0
prompt-toolkit       3.0.32
psutil               5.9.4
ptyprocess           0.7.0
pure-eval            0.2.2
pycparser            2.21
Pygments             2.13.0
pyparsing            3.0.9
pyrsistent           0.19.2
python-dateutil      2.8.2
pytz                 2022.6
PyYAML               6.0
pyzmq                24.0.1
rich                 12.6.0
s3fs                 2022.11.0
s3transfer           0.6.0
Send2Trash           1.8.0
setuptools           65.5.0
six                  1.16.0
sniffio              1.3.0
sortedcontainers     2.4.0
soupsieve            2.3.2.post1
stack-data           0.6.1
tblib                1.7.0
terminado            0.17.0
textual              0.4.0
tinycss2             1.2.1
toolz                0.12.0
tornado              6.1
traitlets            5.5.0
typing_extensions    4.4.0
urllib3              1.26.12
wcwidth              0.2.5
webencodings         0.5.1
websocket-client     1.4.2
wheel                0.37.1
widgetsnbextension   3.6.1
wrapt                1.14.1
xyzservices          2022.9.0
yarl                 1.8.1
zict                 2.2.0
zipp                 3.10.0

[jacobtomlinson]

I'm also seeing some errors about STS. I created the cluster from a machine that doesn't have the AWS CLI configured (that was set up on a different machine). So perhaps there is some error output related to that which rich is colliding with.

error sending AWS credentials to cluster: str expected, not NoneType

For info I recently was doing some rich work in dask-kubernetes and the cluster manager was emitting two warnings and the rich console had the same effect of repeating the first lines. The number of repeated rich lines was equal to the number of lines of warnings.

[dchudz]

@jacobtomlinson regarding the duplication issues, we've seen similar things that are fixed as of rich 13.3.2 (but still an issue as of 13.3.1).

[shughes-uk]

I think @jacobtomlinson is right here. When its just duplicating the header like that I've found its something writing to stdout in the past