-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change from JSON encoding to using CBOR/BSON encoding #58
Comments
Currently we are using things like:
And more just to represent the type that actually that comes out of the DB after we submit JSON because of how buffers are encoded. This adds quite a bit of unnecessary noise. If the DB could support binary data, and support types that is native to JS like Non-native JS types like Other JS types that could be supported include things like |
This type would be particularly important:
These types all need to be supported, and other kinds of values can be added to the list. We could then create a |
This coincides with #3. |
Protobuf btw is not suitable for this. It must be schemaless. Other choices include messagepack too. |
BSON is old school and not suitable. Protobuf requires a schema. CBOR seems the best, but I think the libraries are sort of unmaintained. This seems suitable: https://github.com/kriszyp/cbor-x |
This might be a breaking change in relation to #3. However it will make js-db far more user friendly and reduce the amount of encoding/decoding steps in Polykey, especially as we store alot of binary data into js-db like IDs. All of those encoding/decoding procedures could then be entirely eliminated as CBOR takes over. |
Using the CBOR library could be shared with PK when it needs to use it for binary streaming for mixed messages or chunked processing. |
Specification
Currently the DB uses JSON encoding by default for storing structured data.
This encoding is lossy. Not all of JS types can be represented using JSON, and in other cases it can be quite fat when encoding binary data.
Sometimes we want to store structured data that may include binary data and other useful things like Dates.
Remember things like
undefined
gets turned intonull
when in arrays, so that can be surprising.Consider checking out CBOR (which seems an evolution from message pack and BSON).
Additional context
Tasks
undefined
ArrayBuffer
that is accepted by the NAPI into rocksdb.The text was updated successfully, but these errors were encountered: