-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use bufferWrap
to support ArrayBuffer
, Uint8Array
and Buffer
#3
Comments
Note that node module resolution would load locally first before loading from anywhere else. I'm not sure how this applies to non-node runtimes like in NativeScript. Recommend not doing anything here until we have understood the platform properly. The usage of |
Note that all Node If something supports We are also using alot of Therefore we can be satisfied with supporting |
The type we use in As for |
The only issue would be the |
No it doesn't because it uses However one could wrap it in our |
The function domainPath(levels: DBDomain, key: string | ArrayBuffer): string | Buffer {
// prefixer only takes string or Node Buffer
// convert key into key_ which wraps ArrayBuffer as Buffer
let key_: string | Buffer;
if (typeof key === 'string') {
key_ = key;
} else if (ArrayBuffer.isView(key)) {
key_ = Buffer.from(key.buffer, key.byteOffset, key.byteLength);
} else {
key_ = Buffer.from(key);
}
if (!levels.length) {
return key_;
}
let prefix = key_;
for (let i = levels.length - 1; i >= 0; i--) {
prefix = prefixer(levels[i], prefix);
}
return prefix;
} However this doesn't fix everything. We also use When using worker manager, we want to copy the But yes this issue looks like it is totally possible. |
Note that this would only present an |
The
|
Need to ensure that this doesn't result in a performance regression with respect to benchmarks. |
Level/subleveldown#111 - seems like we should be able to use |
This is somewhat more complicated now that we have alot of functions relying on the Applications that use this library right now has to use the feross buffer package if their runtime environment doesn't have the Having an Although the new abstract-level has support for Uint8Array, so that may be a better way to support it https://github.com/Level/browser-level/blob/98c13d4356fdb6ca5668ddd35859e6c9b326b5cb/UPGRADING.md#uint8array-support rather than directly supporting |
We are no longer attached to The One way to quickly solve this issue is to just take |
If we receive an If we use So let's just do that. Here's the /**
* Zero-copy wraps ArrayBuffer-like objects into Buffer
* This supports ArrayBuffer, TypedArrays and the NodeJS Buffer
*/
function bufferWrap(
array: BufferSource,
offset?: number,
length?: number,
): Buffer {
if (Buffer.isBuffer(array)) {
return array;
} else if (ArrayBuffer.isView(array)) {
return Buffer.from(
array.buffer,
offset ?? array.byteOffset,
length ?? array.byteLength,
);
} else {
return Buffer.from(array, offset, length);
}
} |
bufferWrap
to support ArrayBuffer
, Uint8Array
and Buffer
This means when we are passing Ids into the DB, we no longer have to do |
@amydevs Since in js-db we are planning to continue using Do this: #3 (comment) when you check |
Use the
Now the issue is that However this should only be done in the application context. So either the application remembers to bundle up with |
Actually for So that means if it is an |
Specification
The
ArrayBuffer
type is more generalised in the JS ecosystem. The DB right now focuses on taking Node Buffers.Node buffers can be converted to
ArrayBuffer
easily, andArrayBuffer
can be wrapped in Node buffers.According to this comment: Level/level-js#34 (comment) it is possible to use ArrayBuffers directly in leveldb, it just not documented.
It says that the
asBuffer
has to be turned false.I'm not sure how this would work with
keyEncoding
andvalueEncoding
that we have set to'binary'
.Our
raw
is also specifying that we would getBuffer
, would that mean we instead say that we returnArrayBuffer
instead, and with keys it would bestring | ArrayBuffer
?Furthermore this would impact js-encryptedfs, so it's worthwhile to explore the implications of this.
The primary reason to do this would be cross-platform compatibility for mobile platforms that may not have the Node buffer.
The alternative would be use https://github.com/feross/buffer everywhere as a dependency so that way everything just uses the
buffer
library. This will mean that any usage ofimport { Buffer } from 'buffer';
will be resolved by feross/buffer first though, so one should beware of that.Note that all Node
Buffer
isUint8Array
which isArrayBuffer
.If something supports
ArrayBuffer
, they would supportUint8Array
andBuffer
at the same time.One benefit would be integration of js-id MatrixAI/js-id#1 can be simplified since
Id
asUint8Array
can be stored directly in the DB without first wrapping them asBuffer
.Additional context
Tasks
ArrayBuffer
compatibility in leveldbArrayBuffer
in DB without losingBuffer
supportstring | Buffer
withstring | ArrayBuffer
for keys, but retainBuffer
on returned output.domainPath
with UsebufferWrap
to supportArrayBuffer
,Uint8Array
andBuffer
#3 (comment)this.crypto.key
to be assumed to beArrayBuffer
toArrayBuffer
to an explicit slice copy of all possible instances ofArrayBuffer
typefromArrayBuffer
is still useful since we returnBuffer
.serialize
anddeserialize
to useArrayBuffer
andTextEncoder
andTextDecoder
which can do theutf-8
encoding/decoding.deserialize
andserialize
might need to change.Uint8Array
andArrayBuffer
usage, and ensure that slice-copying is working.The text was updated successfully, but these errors were encountered: