Set metadata on a single individual or node

[hyanwong]

At the moment I think adding to the metadata arrays (e.g. on populations, individuals, or nodes) is quite intricate, and I have to look it up every time. I think it's something like this:

ts = msprime.sim_ancestry(10)
tables = ts.dump_tables()
pop_metadatas = [row.metadata for row in tables.populations]
pop_metadatas[0]['name'] += "(the only population)"
tables.populations.packset_metadata(
    [tables.populations.metadata_schema.validate_and_encode_row(r) for r in pop_metadatas])
new_ts = tables.tree_sequence()
print([p for p in new_ts.populations()])

(it looks me about 10 minutes to look up how to do that, and the information is scattered all over the docs - I don't think a beginner could do it TBH).

Should we have some convenience methods to help with this? I quite often want to edit metadata on just a single entry in a table: e.g. label a particular individual, flag up a single node, etc.

We should also give an example of doing this in the metadata tutorial

[jeromekelleher]

We don't generally have a way to set an individual row, which I think is a precondition for doing this. In general nobody should need to do the packset stuff for normal use. Here's what we'd like to do:

ts = msprime.sim_ancestry(10)
tables = ts.dump_tables()
metadata = tables.population[0].metadata
metadata["name"] = "the only population"
tables.populations[0] = tables.populations[0].replace(metadata=metadata)

I thought we had an issue open about setting rows, but I can't find it. It'll be fiddly to do and will probably require implementing in C. @benjeffery, do you know if we have an issue tracking this?

[jeromekelleher]

So, we'd have, e.g., individual_table_set_row( tsk_individual_table_t *self, tsk_id_t index, tsk_individual_t *row), which would do the ragged column fiddling required. Messy, but not too bad if we don't try to be too clever about it I think. (It's an efficient operation, so let's not worry too much about optimising it)

[hyanwong]

ts = msprime.sim_ancestry(10)
tables = ts.dump_tables()
metadata = tables.populations[0].metadata
metadata["name"] = "the only population"
tables.populations[0] = tables.populations[0].replace(metadata=metadata)

Yeah, this is nice, thanks. Which reminds me, is there any way to stop people assigning into the metadata directly:

tables.populations[0].metadata = {'name':"I think I've assigned this"}  # This is banned, hurrah!
tables.populations[0].metadata['name'] = "I think I've assigned this"  # is there a way to stop this?

[jeromekelleher]

Not sure there's much we can do about that.

[benjeffery]

We could make the codecs return some kind of frozen dict?

[benjeffery]

This is a similar problem for all the ragged arrays, but has come up less as changing the lengths of an individuals parents or location, a sites ancestral_state or a mutations derived_state are much less frequent use cases.

I think to do this properly and efficiently requires some support in the C API for modifying ragged arrays. As you have it written above you are decoding and then validating and encoding all rows just to change one.

I'm thinking along the lines of a C API method that takes row indexes and a ragged array, and updates the row indexes in the table's ragged array with the contents of the passed ragged array.

I'll promote this discussion to an issue and add it to the metadata project, even though the solution is for all ragged arrays.

[hyanwong]

In later versions of tskit, we can now do as Jerome suggested:

ts = msprime.sim_ancestry(10)
tables = ts.dump_tables()
metadata = tables.populations[0].metadata
metadata["name"] = "the only population"
tables.populations[0] = tables.populations[0].replace(metadata=metadata)