diff --git a/notebooks/cnmfe.ipynb b/notebooks/cnmfe.ipynb index 5403444..f73ae44 100644 --- a/notebooks/cnmfe.ipynb +++ b/notebooks/cnmfe.ipynb @@ -12,37 +12,20 @@ }, "outputs": [], "source": [ - "from mesmerize_core import *\n", - "import numpy as np\n", "from copy import deepcopy\n", + "\n", + "import numpy as np\n", "import pandas as pd\n", "import tifffile\n", + "from ipywidgets import IntSlider, VBox\n", + "import fastplotlib as fpl\n", + "\n", "from caiman.motion_correction import high_pass_filter_space\n", - "from caiman.summary_images import correlation_pnr" - ] - }, - { - "cell_type": "markdown", - "id": "4198e3da-d8d1-4a26-84ce-9907f5af8870", - "metadata": {}, - "source": [ - "**You will need `fastplotlib` installed for the visualizations**" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "b1546b14-2939-4077-adef-e98179f33a1c", - "metadata": { - "pycharm": { - "name": "#%%\n" - }, - "tags": [] - }, - "outputs": [], - "source": [ - "from fastplotlib import ImageWidget\n", - "from ipywidgets import VBox, IntSlider, Layout" + "from caiman.summary_images import correlation_pnr\n", + "\n", + "import mesmerize_core as mc\n", + "from mesmerize_core.arrays import LazyTiff\n", + "from mesmerize_viz import *" ] }, { @@ -59,35 +42,42 @@ }, { "cell_type": "markdown", - "id": "3bac0da4-e3eb-4717-b5dc-1e34cfd79e74", + "id": "1cda36a4-704d-4762-a3c4-0e717ebc1f4d", "metadata": { "pycharm": { "name": "#%% md\n" - } + }, + "tags": [] }, "source": [ "# Paths\n", - "These are the only variables you will need to modify in this demo notebook. You will need to set the paths according to your own `caiman_data` dir path\n", - "\n", - "Explanation:\n", - "\n", - "`set_parent_raw_data_path()` - This function from `mesmerize_core` sets the **top level raw data directory**. It is generally the top level directory for your raw experimental data. This allows you to move your experiment directory structure between computers, as long as you keep everything under the parent path the same.\n", - "\n", - "For example,\n", - "\n", - "On Linux based systems if you have your experimental data in the following dir:\n", - "\n", - "`/data/my_name/exp_top_level/....`\n", "\n", - "You could set `/data/my_name` as the \"parent raw data path\", and you can then move `exp_top_level/...` between computers.\n", + "`mesmerize-core` helps manage the outputs of caiman algorithms and organizes \"parameter variants\" - the output of a given combination of input data and algorithm parameters. In order to run the algorithms you must tell `mesmerize-core` where your _input data_ are located and decide on a **top level raw data directory**. For example consider the following directory structure of experimental data (you may organize your raw data however you wish, this is just an example). We can see that all the experimental data lies under `/data/group_name/my_name/exp_data`. Therefore we can use this `exp_data` dir as a `parent raw data path`. `mesmerize-core` will then only store the _relative_ paths to the raw data files, this allows you to move datasets between computers and filesystems. `mesmerize-core` does not store any hard file paths, only relative paths.\n", "\n", - "On windows:\n", + "```\n", + "/data/group_name/my_name\n", + " └── exp_data\n", + " ├── axonal_imaging\n", + " │   ├── mouse_1\n", + " │   │   ├── exp_a.tiff\n", + " │   │   ├── exp_b.tiff\n", + " │   │   └── exp_c.tiff\n", + " │   ├── mouse_2\n", + " │   │   ├── exp_a.tiff\n", + " │   │   └── exp_b.tiff\n", + " │   └── mouse_3\n", + " └── hippocampus_imaging\n", + " ├── mouse_1\n", + " │   ├── exp_a.tiff\n", + " │   ├── exp_b.tiff\n", + " │   └── exp_c.tiff\n", + " ├── mouse_2\n", + " └── mouse_3\n", + "```\n", "\n", - "`D:/my_name/exp_top_level/...`\n", + "**For this demo set the `caiman_data` dir as the parent raw data path**\n", "\n", - "You could set `D:/my_name` as the parent raw data path, and you can then move `exp_top_level/...` between computers.\n", - "\n", - "**Even on windows just use `/`, you do not have to worry about the annoying issue of `\\\\` and `\\` on windows if you use `pathlib` :D**" + "Sidenote: We recommend using [pathlib](https://docs.python.org/3/library/pathlib.html) instead of manually managing paths as strings. `pathlib` is just a part of the Python standard library, it makes it much easier to deal with paths and saves a lot of time in the long-run! It also makes your paths compatible across operating systems. Therefore even if you are on Windows you can use the regular `/` for paths, you do not have to worry about the strangeness of `\\\\` and `\\`" ] }, { @@ -103,19 +93,17 @@ "outputs": [], "source": [ "# for this demo set this dir as the path to your `caiman_data` dir\n", - "set_parent_raw_data_path(\"/home/kushal/caiman_data/\")" + "mc.set_parent_raw_data_path(\"/home/kushal/caiman_data/\")" ] }, { "cell_type": "markdown", - "id": "ddde637b-598c-4d99-9c5e-38409d7634fc", + "id": "91abab59-07a1-4716-9935-0e1840619930", "metadata": {}, "source": [ "### Batch path, this is where caiman outputs will be organized\n", "\n", - "This can be anywhere, it does not need to be under the parent raw data path.\n", - "\n", - "**We recommend using [pathlib](https://docs.python.org/3/library/pathlib.html) instead of manually managing paths as strings. `pathlib` is just a part of the Python standard library, it makes it much easier to deal with paths and saves a lot of time in the long-run! It also makes your paths compatible across operating systems.**" + "This can be anywhere, it does not need to be under the parent raw data path." ] }, { @@ -130,12 +118,12 @@ }, "outputs": [], "source": [ - "batch_path = get_parent_raw_data_path().joinpath(\"mesmerize-cnmfe/batch.pickle\")" + "batch_path = mc.get_parent_raw_data_path().joinpath(\"mesmerize-cnmfe/batch.pickle\")" ] }, { "cell_type": "markdown", - "id": "4c777a10-64dd-4061-99e1-e9c9172d8bf6", + "id": "986a89a9-7245-4fe2-a2dc-2c1e97d5488a", "metadata": {}, "source": [ "# Create a new batch\n", @@ -158,19 +146,19 @@ "outputs": [], "source": [ "# create a new batch\n", - "df = create_batch(batch_path)\n", + "df = mc.create_batch(batch_path)\n", "# to load existing batches use `load_batch()`\n", "# df = load_batch(batch_path)" ] }, { "cell_type": "markdown", - "id": "50199499-c3f7-4aab-81ef-84913450d63c", + "id": "f117bc14-b228-4427-84a9-8dd915837fc4", "metadata": {}, "source": [ "# View the dataframe\n", "\n", - "It is empty with the appropriate columns for mesmerize" + "It is empty and has the required columns for mesmerize" ] }, { @@ -185,6 +173,19 @@ "df" ] }, + { + "cell_type": "code", + "execution_count": null, + "id": "25fb134e-2e2b-4ed1-8be9-ab0deb82c122", + "metadata": {}, + "outputs": [], + "source": [ + "# We'll use the data_endoscope movie from caiman\n", + "# download it if you don't have it\n", + "from caiman.utils.utils import download_demo\n", + "download_demo(\"data_endoscope.tif\")" + ] + }, { "cell_type": "code", "execution_count": null, @@ -194,7 +195,7 @@ }, "outputs": [], "source": [ - "movie_path = get_parent_raw_data_path().joinpath(\"example_movies/data_endoscope.tif\")" + "movie_path = mc.get_parent_raw_data_path().joinpath(\"example_movies/data_endoscope.tif\")" ] }, { @@ -255,9 +256,11 @@ "}\n", "\n", "# input movie will be shown on left, filtered on right\n", - "iw_gs = ImageWidget(\n", + "iw_gs = fpl.ImageWidget(\n", " data=[input_movie, input_movie.copy()],\n", " frame_apply=funcs,\n", + " names=[\"raw\", \"filtered\"],\n", + " grid_plot_kwargs={\"size\": (1200, 600)},\n", " cmap=\"gnuplot2\"\n", ")\n", "\n", @@ -366,11 +369,11 @@ "tags": [] }, "source": [ - "## We can now see that there is one item, a.k.a. row or pandas `Series`, in the batch dataframe, we can add another item with the same input movie but with different parameters.\n", + "We can now see that there is one item, a.k.a. row or pandas `Series`, in the batch dataframe, we can add another item with the same input movie but with different parameters.\n", "\n", - "### **When adding batch items with the same `input_movie_path` (i.e. same input movie but different parameters) it is useful to give them the same `item_name`.**\n", + "**When adding batch items with the same `input_movie_path` (i.e. same input movie but different parameters) it is useful to give them the same `item_name`.**\n", "\n", - "Let's just try one with different `gSig_filt`" + "Let's try one more with different `gSig_filt`" ] }, { @@ -409,9 +412,9 @@ "id": "9012a748-79d5-4e4a-887f-cc8f26b86cc6", "metadata": {}, "source": [ - "## We can see that there are two batch items for the same input movie.\n", + "# We can see that there are two batch items for the same input movie.\n", "\n", - "### We can also use a `for` loop to add multiple different parameter variants more efficiently." + "Use a `for` loop to add multiple different parameter variants more efficiently." ] }, { @@ -458,9 +461,7 @@ "tags": [] }, "source": [ - "## Now we can see that there are many parameter variants, but it is not easy to see the differences in parameters between the rows that have the same `item_name`.\n", - "\n", - "### We can use the `caiman.get_params_diffs()` extension to see the unique parameters between rows with the same `item_name`" + "We can use the `caiman.get_params_diffs()` extension to see the unique parameters between rows with the same `item_name`" ] }, { @@ -476,58 +477,14 @@ }, { "cell_type": "markdown", - "id": "7390441c-6244-4eac-93e6-cf688b6d76fb", - "metadata": { - "pycharm": { - "name": "#%% md\n" - } - }, - "source": [ - "# Indexing rows and running batch item(s)\n", - "#### You can run a single batch item by calling `caiman.run()` on a `Series` (row) of the DataFrame. One way to get the row is integer indexing using `df.iloc[index]`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3c11e0c4-8dd5-4c33-a5fa-4677a6f10bde", + "id": "4cd8cdde-6a83-4d63-808a-1a05628893ea", "metadata": {}, - "outputs": [], "source": [ - "# get the first batch item\n", - "row = df.iloc[0]" - ] - }, - { - "cell_type": "markdown", - "id": "7ead6a13-6531-4cf4-b82e-b3aef28e4fab", - "metadata": {}, - "source": [ - "### You can see how the various `pandas.Series` extensions are accessible at the level of dataframe rows.\n", - "\n", - "Move the cursor to the end of the following line and press `Tab` on your keyboard. You can select the `caiman.run()` function and press `Shift + Tab` to see the docstring. You can also instead refer to the API docs. https://mesmerize-core.readthedocs.io/en/latest/api/common.html#mesmerize_core.CaimanSeriesExtensions\n", + "# Run an item\n", "\n", - "Note tab completion doesn't work if you use `df.iloc[i].caiman.`, you need to apply the indexer to see the docstring." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3893e8e9-1130-4cb6-8434-5632afe0da41", - "metadata": {}, - "outputs": [], - "source": [ - "row.caiman." - ] - }, - { - "cell_type": "markdown", - "id": "eec440e0-7c95-463b-95bf-deab0c7d4e6f", - "metadata": {}, - "source": [ - "# Run a single batch item\n", + "There is only one item in this DataFrame and it is located at index `0`. You can run a row using `df.iloc[index].caiman.run()`\n", "\n", - "Run the row that we have selected above, on Linux & Mac it will run in subprocess but on Windows it will run in the local kernel. If using the subprocess backend you can use `run(wait=False)` to not block the current kernel when running." + "Technical notes: On Linux & Mac it will run in subprocess but on Windows it will run in the local kernel. If using the subprocess backend (only Linux & Mac) you can use `run(wait=False)` if you don't want to block the kernel while running. " ] }, { @@ -541,27 +498,35 @@ }, "outputs": [], "source": [ - "# run the first \"batch item\"\n", - "# this will run in a subprocess by default on Linux & Mac\n", - "# on windows it will run locally\n", - "process = row.caiman.run()\n", + "df.iloc[0].caiman.run()\n", "\n", "# reload dataframe from disk when done\n", "df = df.caiman.reload_from_disk()" ] }, + { + "cell_type": "markdown", + "id": "3e6f003d-2e34-42f5-9fc4-526c025b550c", + "metadata": {}, + "source": [ + "# Run multiple batch items.\n", + "\n", + "`df.iterrows()` iterates through rows and returns the numerical index and row for each iteration" + ] + }, { "cell_type": "code", "execution_count": null, - "id": "be469f61-b712-4832-b8be-5118cd59f5c6", + "id": "cb02c6d2-b302-487b-bcf6-81e8dc47006c", "metadata": { "tags": [] }, "outputs": [], "source": [ "for i, row in df.iterrows():\n", - " if not i > 0: # skip the first item since we've run it already\n", - " continue\n", + " if row[\"outputs\"] is not None: # item has already been run\n", + " continue # skip\n", + " \n", " process = row.caiman.run()\n", " \n", " # on Windows you MUST reload the batch dataframe after every iteration because it uses the `local` backend.\n", @@ -581,8 +546,9 @@ "tags": [] }, "source": [ - "# Reload the DataFrame to see the outputs information for the mcorr batch item\n", - "### It is necessary to ALWAYS use `df = df.caiman.reload_from_disk()` after running a single batch item or a loop of batch items. You must not add new batch items until you reload it if you have ran items!" + "# Reload the DataFrame\n", + "\n", + "It is necessary to use `df = df.caiman.reload_from_disk()` after running a single batch item or a loop of batch items. You must not add new batch items until you reload it if you have ran items." ] }, { @@ -619,183 +585,121 @@ }, { "cell_type": "markdown", - "id": "5eadeee7-61e1-474a-ad33-4aac34b7b4a1", - "metadata": {}, - "source": [ - "# Visualization using `fastplotlib`\n", - "You will need `fastplotlib` installed for this, see https://github.com/kushalkolar/fastplotlib" - ] - }, - { - "cell_type": "markdown", - "id": "9da1df35-f215-482b-a70b-5b44c50ee716", + "id": "5885441e-a049-4412-9306-fb716751d6dc", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ - "# Get the input movie and mcorr so we can visualize them\n", - "\n", - "Note that you DO NOT need to manually work with file paths. For tiff input files it returns it as a memmaped array (if possible) with lazy loading. It will try to use a mesmerize `LazyArray` if the file cannot be memmaped.\n", + "# Visualize results\n", "\n", - "For other file formats you can pass a function that returns a sliceable object, ideally you want to use lazy loading." + "We will use the high pass spatial filter to make it easier to perform visual quality control." ] }, { "cell_type": "code", "execution_count": null, - "id": "f8a7ff3f-27e7-4b59-9e84-5ea3db83a9a7", + "id": "18098d5e-d560-4a49-a400-63efff86d7ff", "metadata": {}, "outputs": [], "source": [ - "# you can change the index to look at the mcorr results of different batch items\n", - "index = 0\n", - "\n", - "# load input movie, this demo movie isn't memmapable and LazyTiff isn't fully read yet so we'll use tifffile.imread\n", - "input_movie = df.iloc[index].caiman.get_input_movie(tifffile.imread)\n", - "\n", - "# load mcorr output movie, this loads it as a memmap\n", - "mcorr_movie = df.iloc[index].mcorr.get_output()" + "a = LazyTiff(df.iloc[0].caiman.get_input_movie_path())\n", + "a.shape" ] }, { "cell_type": "markdown", - "id": "5885441e-a049-4412-9306-fb716751d6dc", - "metadata": { - "pycharm": { - "name": "#%% md\n" - } - }, + "id": "6b474cdd-6dcd-4cc5-98c4-50c99d23777b", + "metadata": {}, "source": [ - "# Visualize raw & MCorr movie side-by-side\n", - "\n", - "### fastplotlib `ImageWidget` to visualize raw & mcorr movie side by side\n", - "\n", - "`ImageWidget` assumes `\"txy\"` dimension order by default for 2D movies. You can set other orders using the `dims_order` kwarg\n", - "\n", - "### High pass filtering seems useful again to see movement more easily" + "**This tiff file doesn't work with `tiffile.memmap` and `LazyTiff` is unable to guess its shape. So we will use `LazyTiff` with a forced shape**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3d8acc70-2fdc-4bd7-a015-d252f78edef0", + "metadata": {}, + "outputs": [], + "source": [ + "def load_weird_movie(path):\n", + " return LazyTiff(path, shape=(1000, 128, 128))" ] }, { "cell_type": "code", "execution_count": null, - "id": "a687037f-9cad-41d1-aa63-7b34ce0b7950", + "id": "a4a1616b-e66e-470e-8413-a498ab8be82b", "metadata": {}, "outputs": [], "source": [ "# high pass filter the data to see shifts more easily\n", - "filt = lambda x: high_pass_filter_space(x, df.iloc[index][\"params\"][\"main\"][\"gSig_filt\"])\n", + "filt = lambda x: high_pass_filter_space(x, (3, 3))\n", "\n", "funcs = {\n", " 0: filt,\n", " 1: filt\n", "}\n", "\n", - "iw = ImageWidget(\n", - " [input_movie, mcorr_movie.astype(np.float32)],\n", - " frame_apply=funcs, # the filter func\n", - " names=[\"input\", \"mcorr\"],\n", - " cmap=\"gnuplot2\",\n", - ")\n", - "\n", - "iw.reset_vmin_vmax()\n", - "\n", - "iw.show()" + "viz_mcor = df.mcorr.viz(\n", + " input_movie_kwargs={\"reader\": load_weird_movie},\n", + " image_widget_kwargs={\"frame_apply\": funcs}\n", + ")" ] }, { - "cell_type": "markdown", - "id": "bc864ddd-cb70-4c3d-8a78-e0b50df0abf4", + "cell_type": "code", + "execution_count": null, + "id": "07414f76-5183-4e16-b05f-b2d81ba82084", "metadata": {}, + "outputs": [], "source": [ - "# Frame averaging with a rolling window using `ImageWidget` \"window functions\".\n", - "\n", - "## This makes it easier to visually inspect motion" + "viz_mcor.show()" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "0afe7eab-a35f-49f0-931d-4d39c7a3c2d7", + "cell_type": "markdown", + "id": "a72edc75-2a35-4a85-a9f5-c6ade16d5569", "metadata": {}, - "outputs": [], "source": [ - "iw.window_funcs = {\"t\": (np.mean, 11)}" + "# Customize the visualization" ] }, { "cell_type": "markdown", - "id": "95b8d84a-5dd0-47e3-870a-df2f6505ced2", + "id": "4f48ce6d-6dea-41dd-8d08-cd1317012df3", "metadata": {}, "source": [ - "## Close the widget to free up GPU processing time, not necessary if you have a powerful GPU" + "disable the spatial filter. Reset vmin vmax after removing it." ] }, { "cell_type": "code", "execution_count": null, - "id": "f3a0f83c-051a-4498-a578-fbefbaef3f77", - "metadata": { - "pycharm": { - "name": "#%%\n" - } - }, + "id": "333f29c6-ecf4-4d67-9a2c-12e184c80747", + "metadata": {}, "outputs": [], "source": [ - "iw.close()" + "viz_mcor.image_widget.frame_apply = dict() # empty dict" ] }, { "cell_type": "markdown", - "id": "70575247-f15a-4db8-99ac-b3119763689a", + "id": "1193b3de-e8b1-4239-b036-46812602d017", "metadata": {}, "source": [ - "## With `ImageWidget` you can view all 5 mcorr results simultaneously!\n", - "\n", - "### This depends on your hard drive's capabilities" + "Use the spatial filter again" ] }, { "cell_type": "code", "execution_count": null, - "id": "a3dabee5-d4cf-473d-8dc0-934da0f423f8", + "id": "102e923e-df4b-48f9-862b-63ef7a561bca", "metadata": {}, "outputs": [], "source": [ - "# first item is just the raw movie\n", - "movies = [df.iloc[0].caiman.get_input_movie(tifffile.imread)]\n", - "\n", - "# subplot titles\n", - "subplot_names = [\"raw\"]\n", - "\n", - "# we will use the mean images later\n", - "means = [df.iloc[0].caiman.get_projection(\"mean\")]\n", - "\n", - "# add all the mcorr outputs to the list\n", - "for i, row in df.iterrows():\n", - " # add to the list of movies to plot\n", - " movies.append(row.mcorr.get_output())\n", - " \n", - " # subplot title to show dataframe index\n", - " subplot_names.append(f\"ix: {i}\")\n", - "\n", - "# filter using the same sigma to make visualization easier\n", - "# same filter for all movies, this syntax is just dictionary comprehension\n", - "filt = {subplot_ix: lambda frame: high_pass_filter_space(frame, (3, 3)) for subplot_ix in range(len(movies))}\n", - "\n", - "# create the widget\n", - "mcorr_iw_multiple = ImageWidget(\n", - " data=movies, # list of movies\n", - " window_funcs={\"t\": (np.mean, 3)}, # window_funcs is also a kwarg\n", - " frame_apply=filt, # same func for all\n", - " names=subplot_names, # subplot names used for titles\n", - " cmap=\"gnuplot2\"\n", - ")\n", - "\n", - "mcorr_iw_multiple.reset_vmin_vmax()\n", - "\n", - "mcorr_iw_multiple.show()" + "viz_mcor.image_widget.frame_apply = funcs" ] }, { @@ -803,11 +707,21 @@ "id": "9ad69731-e057-4ad7-acf7-dd61269ecd72", "metadata": {}, "source": [ - "# All the movies here look pretty good so I'll continue with `index = 0`. You can cleanup the DataFrame and remove all other items.\n", + "# Optional, cleanup\n", + "\n", + "All the movies here look pretty good so I'll continue with `index = 0`. You can cleanup the DataFrame and remove all other items.\n", "\n", - "### You can remove batch items (i.e. rows) using `df.caiman.remove_item()`\n", + "Remove batch items (i.e. rows) using `df.caiman.remove_item()`" + ] + }, + { + "cell_type": "markdown", + "id": "b420e485-4f70-41c1-ae68-97c8c0f502b4", + "metadata": {}, + "source": [ + "**Note:** On windows calling `remove_item()` will raise a `PermissionError` if you have the memmap file open. The workaround is to shutdown the current kernel and then use `df.caiman.remove_item()`. For example, you can keep another notebook that you use just for cleaning unwanted mcorr items.\n", "\n", - "**Note that this also cleans up the output data in the batch directory!**" + "There is currently no way to close a `numpy.memmap`: https://github.com/numpy/numpy/issues/13510" ] }, { @@ -858,6 +772,9 @@ "metadata": {}, "outputs": [], "source": [ + "# get the motion corrected output, this is a memmap array\n", + "mcorr_movie = df.iloc[0].mcorr.get_output()\n", + "\n", "gSig = 3\n", "corr, pnr = correlation_pnr(mcorr_movie[::2], gSig=gSig, swap_dim=False)" ] @@ -870,7 +787,7 @@ "outputs": [], "source": [ "# to show the correlation and pnr images\n", - "iw_corr_pnr = ImageWidget(\n", + "iw_corr_pnr = fpl.ImageWidget(\n", " [corr, pnr], \n", " names=[\"corr\", \"pnr\"],\n", " grid_plot_kwargs={\"size\": (650, 300)},\n", @@ -886,9 +803,9 @@ " \"size\": (650, 600)\n", "}\n", "\n", - "iw_thres_movie = ImageWidget(\n", + "iw_thres_movie = fpl.ImageWidget(\n", " mcorr_vids, \n", - " names=[\"> corr\", \"> pnr\", \"< corr\", \"< pnr\"],\n", + " names=[\"over corr threshold\", \"over pnr threshold\", \"under corr threshold\", \"under pnr threshold\"],\n", " # sync this with the corr-pnr plot\n", " grid_plot_kwargs=threshold_grid_plot_kwargs,\n", " cmap=\"gnuplot2\"\n", @@ -948,7 +865,7 @@ "id": "d66b40c7-72f6-40d2-97e4-29811d31e532", "metadata": {}, "source": [ - "### corr and pnr values from the plot" + "# corr and pnr values from the plot" ] }, { @@ -1007,7 +924,7 @@ "id": "fe711552-fb1c-4f3d-8098-6967a1ca46cf", "metadata": {}, "source": [ - "### Add a single cnmf item to the batch\n", + "# Add a single cnmf item to the batch\n", "\n", "When you use `algo=\"cnmfe\"`, it basically forces the following parameters:\n", "```python\n", @@ -1041,7 +958,9 @@ "id": "102e295a-5543-4ad3-aaa5-fb76e819950a", "metadata": {}, "source": [ - "### Just like with motion correction, we can use loops to add multiple parameter variants. This is useful to perform a parameter search to find the params that work best for your dataset. Here I will use `itertools.product` which is better than deeply nested loops." + "# Parameter search\n", + "\n", + "Just like with motion correction, we can use loops to add multiple parameter variants. This is useful to perform a parameter search to find the params that work best for your dataset. Here I will use `itertools.product` which is better than deeply nested loops." ] }, { @@ -1088,7 +1007,7 @@ "id": "162090e3-5298-4843-a4cb-108d3ad19631", "metadata": {}, "source": [ - "### See that there are a lot of new cnmf batch items" + "See that there are a lot of new cnmf batch items" ] }, { @@ -1106,9 +1025,9 @@ "id": "aca2e87c-76ce-4fa3-93b4-146aad957aa9", "metadata": {}, "source": [ - "## Since it is difficult to see the different parameter variants above, we can just view the diffs\n", + "# Param diffs\n", "\n", - "### The index numbers on the diffs correspond to the indices in the parent DataFrame above" + "The index numbers on the diffs correspond to the indices in the parent DataFrame above" ] }, { @@ -1126,48 +1045,29 @@ "id": "4e26265c-5cb7-461f-8e89-5744e2f5e82d", "metadata": {}, "source": [ - "# Run the added CNMFE items\n", - "\n", - "### First, this is how you can filter a pandas DataFrame using multiple columns. This gives you the rows (batch items) using the \"cnmf\" `\"algo\"` and those that match a particular `\"item_name\"`" + "# Run items" ] }, { "cell_type": "code", "execution_count": null, - "id": "c11826e5-1409-4411-9617-84f6408b4bdd", + "id": "5694187a-48e9-4769-82d3-afd46dd19a00", "metadata": { "tags": [] }, "outputs": [], "source": [ - "df[\n", - " (df[\"algo\"] == \"cnmfe\") & # algo\n", - " (df[\"item_name\"] == df.iloc[0][\"item_name\"]) # item name\n", - "]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f8f67319-9343-41a6-a4f8-30adbdae032a", - "metadata": { - "scrolled": true, - "tags": [] - }, - "outputs": [], - "source": [ - "for i, row in df[\n", - " (df[\"algo\"] == \"cnmfe\") &\n", - " (df[\"item_name\"] == df.iloc[0][\"item_name\"])\n", - "].iterrows():\n", - " \n", + "for i, row in df.iterrows():\n", + " if row[\"outputs\"] is not None: # item has already been run\n", + " continue # skip\n", + " \n", " process = row.caiman.run()\n", " \n", " # on Windows you MUST reload the batch dataframe after every iteration because it uses the `local` backend.\n", " # this is unnecessary on Linux & Mac\n", " # \"DummyProcess\" is used for local backend so this is automatic\n", " if process.__class__.__name__ == \"DummyProcess\":\n", - " df = load_batch(df.paths.get_batch_path())" + " df = df.caiman.reload_from_disk()" ] }, { @@ -1180,7 +1080,7 @@ "tags": [] }, "source": [ - "### We now have CNMF outputs" + "# We now have CNMFE outputs :D " ] }, { @@ -1191,7 +1091,7 @@ "outputs": [], "source": [ "df = df.caiman.reload_from_disk()\n", - "df[df[\"algo\"] == \"cnmfe\"]" + "df" ] }, { @@ -1224,7 +1124,6 @@ "outputs": [], "source": [ "viz = df.cnmf.viz(\n", - " start_index=1, \n", " image_data_options=[\"input\", \"rcm\"], # cnmfe does not support rcb and residuals yet\n", ")" ] @@ -1291,7 +1190,6 @@ "outputs": [], "source": [ "funcs = {\n", - " # data_index: function\n", " 0: lambda frame: high_pass_filter_space(frame, (3, 3))\n", "}\n", "\n", @@ -1301,7 +1199,17 @@ { "cell_type": "code", "execution_count": null, - "id": "7f924002-88ba-4c23-ad6d-5479f515d3f9", + "id": "8190092a-50d0-4093-9dcc-61122580bdac", + "metadata": {}, + "outputs": [], + "source": [ + "viz.image_widget.cmap = \"gray\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c86a0ef4-3de3-4d4a-b9d1-279796ae9fc0", "metadata": {}, "outputs": [], "source": [] @@ -1323,7 +1231,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.2" + "version": "3.11.3" } }, "nbformat": 4,