Store sparse count data JSON in visualization

[fedarko]

This isn't a problem yet, but as we start trying larger datasets here this will become an issue.

Action items:

• Disable Altair's "MaxRows check" -- this is more of a temporary measure than anything, but it'll prevent Altair from giving up immediately as soon as more than 5k features or 5k samples are in a dataset.

• Store datasets in their own JSON files in the data directory. Our use of column mapping in the sample plot means that the costs of storing data in memory/etc aren't that severe, but it is still probably a good idea to separate datasets from the Vega-Lite JSON specs generated by Altair. This should also remove our need to disable the MaxRows check (if there are more than 5k features, would the column mapping stuff trigger a MaxRowsError? -- I should look into this, and consider storing that stuff in its own JSON file as well although that sounds a bit excessive).

  • NOTE that this will probably (?) require using cross-origin requests again, which will essentially nullify the benefits of Remove need for cross-origin requests #72 (i.e. users will again have to use view.qiime2.org or python3 -m http.server to view a visualization). It should be doable to make this an option—i.e. if you want to generate a "serverless" visualization then that should be a default, and if you want to make a viz that uses CORS but potentially can tolerate more data then you can specify a --use-external-data flag or something for the python script.

  • This is probably a pretty far off thing, but worth it to start thinking about it now.

• Reduce the number of samples shown to the user (idea c/o @mortonjt). IMO we should approach this carefully, giving as much control to the user as possible (e.g. making the user very aware of how many samples have been excluded).

[fedarko]

Also worth using a sparse DF for the table instead of dense DF, as is currently done. (Note that this would need to either 1) convert the BIOM DF to a sparse DF, since until biocore/biom-format#808 the default biom.Table.to_dataframe() produced a dense DF, or 2) would need to require an up-to-date version of biom.) Although I'm still having trouble with generating a sparse DF in a reasonable amount of time—maybe a scipy CSR matrix would be a better option, if we can make it work in the code. Update: Done as of beaf464

[fedarko]

Another thing: as mentioned in 041aaaf, it'd be cool if possible to load count data in a web worker or something to minimize strain on the browser. That's the main scaling problem IMO—it's basically just the contents of a BIOM table.

One silly option: could we just... not include samples that have 0 counts for a given feature? (or vice versa, doesn't really matter, just depends on the way the counts JSON is laid out.) As long as we know that we have a list of all sample IDs, we can infer in the JS code which samples have "0" for a given feature (or vice versa). This would basically let us "cheat" and store a sparse representation of the BIOM table, and since BIOM tables for microbiome studies are usually super duper sparse this could actually let us do silly crap like load a huge portion of the EMP in.

Another option would be to use https://github.com/molbiodiv/biojs-io-biom

[fedarko]

Another good idea: improving DataFrame iteration. I know iterrows() is used in process_input(), and there are probably at least a few other places where something faster (e.g. vectorized operations) could be done.

[fedarko]

Draft algorithm for how this would work:

# to be added to the end of gen_sample_plot()
counts_dict = table.to_dict()
sparse_counts_dict = {}
for feature_id, sample_counts in counts_dict.items():
    fdict = {}
    for sample_id, ct in sample_counts.items():
        if ct != 0:
            fdict[sample_id] = ct
    if len(fdict.keys()) > 0:
        sparse_counts_dict[feature_id] = fdict

Now, sparse_counts_dict only contains data for nonzero counts. So for sparse datasets, we'd need a lot less space for abundance data (i.e. much lower memory overhead).

In the JS, we'll need to somehow figure out a list of sample IDs (probs can just embed it in the sample plot JSON analogously to how feature lists are embedded in the rank plot JSON) independent of the count JSON.

Also, note how we might not even add fdict into sparse_counts_dict (this would happen if all samples have 0 for a given feature, which is ostensibly possible in Qurro at present). This would mean that that feature is effectively useless in the Qurro visualization. I think we can get around this by just only passing nonzero features to the visualization -- see #171 for how we'd address this.