Add possibility to traverse inner nodes

[liamsi]

Adds an optional "NodeVisitor" function to the tree that will be called on each node of the tree.

This can be used to tackle #9, and can also be used in the IPLD-plugin. It might also be useful to inspect/debug the tree, or, to print-out the tree in different ways.

[codecov]

Codecov Report

Merging #16 (8340e76) into master (4b4c58a) will increase coverage by 0.16%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #16      +/-   ##
==========================================
+ Coverage   95.13%   95.30%   +0.16%     
==========================================
  Files           7        7              
  Lines         288      298      +10     
==========================================
+ Hits          274      284      +10     
  Misses         10       10              
  Partials        4        4              

Impacted Files	Coverage Δ
nmt.go	92.95% <100.00%> (+0.53%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b4c58a...8340e76. Read the comment docs.

[evan-forbes]

This is super neat! The implementation as an Option is perfect.

[liamsi]

Looks like this is all we need for the IPLD plugin.

[musalbas]

As NodeVisitorFn is stateless (it just takes in the hash of a node and its children), how would the function determine which nodes are part of which Merkle proof or is it not relevant?

[liamsi]

Good point. Currently, this is not relevant - at least for the first LL ipld plugin it isn't and I tried to keep the diff as small as possible. That said, there probably are use-cases where this will become relevant (e.g. when doing #15 this might need to change) 🤔 Did you have something specific in mind when you wondered about this? Maybe I'm missing something?

For now, I don't see any urgent need: nodes who generate proofs are expected to have all the data to generate these, and, they can always recompute the (inner) proof-nodes from scratch. The NodeVisitorFn is only called when computing the root (and simply visits all nodes).

[musalbas]

I assume this is used in the IPLD plugin in order to publish the intermediate nodes? If so this seems like a non-generalisable feature but I guess it's OK if it's implemented as an option rather than a mandatory constructor argument.

[liamsi]

Yes, that is exactly what it is used for. What kind of generalization do you have in mind though? For reading the nodes from storage?

If there is sth. obvious which do need soon, we can add this here too. Otherwise, let's track this in a separate issue.

[musalbas]

Yes, that is exactly what it is used for. What kind of generalization do you have in mind though? For reading the nodes from storage?

If there is sth. obvious which do need soon, we can add this here too. Otherwise, let's track this in a separate issue.

I suppose actually storing the tree itself (in memory) within NMT itself and letting people read from it.

[liamsi]

The tree exists in memory but proofs are currently recomputed on demand. That's isn't "reading from it" but should be quite efficient already, especially for a first version / mvp (the leaf-data and the leaf-hashes are also stored in memory and the former can be retrieved via the API).

If we want to avoid recomputing the inner proof nodes every time they are needed, we can actually store the inner nodes too. Not sure this would need to be exposed as an exported API or sth like this. Of course, if we want to store and then read the inner nodes from somewhere else, we'd need to change the API. We could also pass in an optional NodeGetterFn or SubTreeGetter maybe? For now I think recomputing the nodes is fine and will be fast enough, especially way faster than retrieving them from any kind of persistent storage (especially much faster thanreading them from sth like IPFS). Does that make sense?

[adlerjohn]

Hashing is pretty fast and involves no disk I/O. Storing inner nodes would mean multiple on-disk accesses along with more storage overhead. So it might not actually be so bad to just fetch the leaves all at once then recompute proofs as needed.