Review js-id usage for NodeId, VaultId, NotificationId, PermId, ClaimId, Gestalts and GenericIdTypes.ts

[joshuakarp]

Specification

The current GenericIdTypes module is slightly out of place within the larger architecture of the js-polykey codebase. We should find some time to review this, and refactor its required implementation to more suitable areas of the codebase. It was originally introduced in order to facilitate usage of the js-id module within Polykey.

Additional context

• relevant comment on this here Introducing vault sharing and the restrictions of pulling and cloning with vault permissions #266 (comment) (more generally talking about js-id too) - discussion originally from here https://vimeo.com/manage/videos/652339635 5:00 - 17:30
• originally introduced here c5f22e5
• No longer preserve vault ID on cloneVault #253 - vault id usage in EFS and when cloning

Tasks

1. Add some of the encoding wrappers into the IdInternal class, this requires a new PR in js-id. Include toMultibase and toUUID and toBuffer as instance methods, and the fromString, fromMultibase, fromUUID, and fromBuffer as static methods (these represent alternatives of creating the IdInternal). Make sure that these functions return Id as the type and not IdInternal.
2. Create a new PR for addressing the NodeId changes that fixes Refactor NodeId as a proxy/singleton instance #254 and relates to Review js-id usage for NodeId, VaultId, NotificationId, PermId, ClaimId, Gestalts and GenericIdTypes.ts #299 (this requires more work to be done, not just node id).
3. This PR will focus on applying the new NodeId which is an opaque alias of Id (which itself is an alias of IdInternal & number).
4. All the areas which is currently using NodeId is expecting an encoded string. These will need to be changed to expect a sort of type NodeIdEncoded = string;, you still need a nodesUtils.encodeNodeId.
5. The encoding functions and decoding functions of NodeId should be inside nodes/utils.ts. This should be similar to how I'm doing it for VaultId. An alternative might be to create class extension or generics, but this seems complicated.
6. Raw usage of the NodeId will mostly occur inside the nodes domain, in particular the NodeGraph can use the Uint8Array directly. Note that the js-db currently doesn't yet allow direct usage of the Id, because it requires a Buffer type. However there is an issue Use bufferWrap to support ArrayBuffer, Uint8Array and Buffer js-db#3 to allow it to eventually take Id, so for now, just use idUtils.toBuffer and idUtils.fromBuffer when interacting with the DB.
7. Once this PR is merged into master, rebase Introducing vault sharing and the restrictions of pulling and cloning with vault permissions #266 and Extracting Node Connection Management out of NodeManager to NodeConnectionManager #310 on top to benefit from it.
8. Then the GenericIdTypes.ts can be removed once all other areas of code stop using it.

[CMCDragonkai]

I want to know if we actually need the makeVaultId as well. Originally the ACL could just take arbitrary strings that were casted as VaultId but these are now going through a function called makeVaultId.

Furthermore in other places we see literal strings like this:

          vaults: {
            vault1xxxxxxxxxx: {},
            vault4xxxxxxxxxx: { pull: null },
          },

I'm not sure what was the reason for doing this.

This seems overly complicated:

    vaultId1 = vaultsUtils.makeVaultId(idUtils.fromString('vault1xxxxxxxxxx'));
    vaultId2 = vaultsUtils.makeVaultId(idUtils.fromString('vault2xxxxxxxxxx'));
    vaultId3 = vaultsUtils.makeVaultId(idUtils.fromString('vault3xxxxxxxxxx'));
    vaultId4 = vaultsUtils.makeVaultId(idUtils.fromString('vault4xxxxxxxxxx'));

Which is currently in the tests/acl/ACL.test.ts.

Vault Id should be using IdRandom right? But then base encoded appropriately. The IdRandom is part of a common type called Id.

Anyway some of the function names I need to review and see if it's appropriate.

Other places where the same thing is happening:

• PermissionId used in acl
• VaultId used in vaults
• NodeId used in nodes and KeyManager - note that Public key by itself isn't the node ID, it has to be encoded appropriately somehow

[CMCDragonkai]

Made this a high level issue overall to examine all Id usage.

[CMCDragonkai]

Any usage of ES6 Maps does not require the conversion of Id into string before usage. In the vaults sharing PR I'm working on, this was done, possibly because the vault map was originally a POJO. But ES6 maps can be keyed by objects, so Id can be kept the same. I'm changing over to this in #266.

@tegefaulkes can you make sure that in your nodes refactoring, that if NodeId is being used as a key in any ES6 map, that there's no conversion to string, it's just unnecessary.

Note that if it is being put into a POJO, there are special provisions where Id will automatically be coerced into a primitive string. This should be sufficient and doesn't require explicit conversion to string. It's only when you iterate or acquire the raw key where you need to convert it back to an Id. Prototype with the js-id and review the the tests there to understand.

[CMCDragonkai]

The @matrixai/id got updated to 3.0.0. This is because it was not exporting the right Id type. It was exporting IdInternal class as Id. But Id is just a type alias of IdInternal & number. So now it exports the type Id and the class IdInternal. This actually solves a bunch of confusion in vaults domain #266 involved in the using of the VaultId.

The vault map (ES6 Map) can just use VaultId directly, no conversion required to string. furthermore VaultId can also be used in POJOs as well with no problems. This was all covered in js-id.

[CMCDragonkai]

In vaults/utils.ts I can see:

function isVaultId(arg: any) {
  return isId<VaultId>(arg);
}

/**
 * This will return arg as a valid VaultId or throw an error if it can't be converted.
 * This will take a multibase string of the ID or the raw Buffer of the ID.
 * @param arg - The variable we wish to convert
 * @throws vaultsErrors.ErrorInvalidVaultId  if the arg can't be converted into a VaultId
 * @returns VaultId
 */
function makeVaultId(arg: any): VaultId {
  return makeId<VaultId>(arg);
}

function isVaultIdPretty(arg: any): arg is VaultIdPretty {
  return isIdString<VaultIdPretty>(arg);
}

function makeVaultIdPretty(arg: any): VaultIdPretty {
  return makeIdString<VaultIdPretty>(arg);
}

Which are just thin wrappers around functions inside the src/GenericIdTypes.ts. After reading the GenericIdTypes.ts, I think @tegefaulkes you must have extracted them out of the vaults/utils.ts right?

So I think some less magic is appropriate here.

For encoded vault ids, it should be sufficient to just have 2 functions:

function encodeVaultId(vaultId: VaultId): string {
  return idUtils.toMultibase(vaultId, 'base58btc');
}

function decodeVaultId(vaultIdString: string): VaultId | undefined {
  return idUtils.fromMultibase(vaultIdString);
}

This is because we will always be using the multi-base encoding for any kind of string form. No need to also make ids from buffers.

[tegefaulkes]

They're typeGuards for easy conversion between types. the generic types were for the basic conversions while the vault ones were for specifically the vaultIds. this was to keep seperation between NodeIds and VaultIds as different opaque types while they were technically the same ID type underneath.

But yeah, the type functions could be tidied up.

[CMCDragonkai]

This reduces the list of vault ID utility functions to just:

const vaultIdGenerator = new IdRandom();

function generateVaultId(): VaultId {
  return vaultIdGenerator.get();
}

function encodeVaultId(vaultId: VaultId): string {
  return idUtils.toMultibase(vaultId, 'base58btc');
}

function decodeVaultId(vaultIdString: string): VaultId | undefined {
  return idUtils.fromMultibase(vaultIdString);
}

And that's all that is needed for vaults domain.

[CMCDragonkai]

They're typeGuards for easy conversion between types. the generic types were for the basic conversions while the vault ones were for specifically the vaultIds. this was to keep seperation between NodeIds and VaultIds as different opaque types while they were technically the same ID type underneath.

But yeah, the type functions could be tidied up.

Yes that's right, but I think simpler solutions can suffice. For example instead of returning string, I could also just do as VaultIdEncoded.

The generic Id utilities adds bit too much magic for my liking atm.

I also prefer most domain exceptions to be in the procedural areas of the code instead of the utility functions. Utility functions should be as simple as possible.

[CMCDragonkai]

Currently KeyManager.getNodeId() returns NodeId.

However NodeId is expected to be an opaque string.

I want to change this up a bit. I want NodeId to be an alias or opaque alias of Id.

This means where I want a string form of NodeId, we have to do an explicit encoding. Either this is done automatically as per IdInternal type. Or it's the multibase encoded form, where should be using utility functions like the above I did for vaultsUtils.

So right now the encoded string form of NodeId should be multibase encoded base32hex. And this is what I'm going to be using for the author name for the git commit log.

[CMCDragonkai]

The js-id can now be used for NodeIds properly. I've given an example of this here: #254 (comment). It will require changes to keys/utils.ts to support it.

[CMCDragonkai]

So it seems that what we should do is:

1. Add some of the encoding wrappers into the IdInternal class, this requires a new PR in js-id. Include toMultibase and toUUID and toBuffer as instance methods, and the fromString, fromMultibase, fromUUID, and fromBuffer as static methods (these represent alternatives of creating the IdInternal). Make sure that these functions return Id as the type and not IdInternal.
2. Create a new PR for addressing the NodeId changes that fixes Refactor NodeId as a proxy/singleton instance #254 and relates to Review js-id usage for NodeId, VaultId, NotificationId, PermId, ClaimId, Gestalts and GenericIdTypes.ts #299 (this requires more work to be done, not just node id).
3. This PR will focus on applying the new NodeId which is an opaque alias of Id (which itself is an alias of IdInternal & number).
4. All the areas which is currently using NodeId is expecting an encoded string. These will need to be changed to expect a sort of type NodeIdEncoded = string;, you still need a nodesUtils.encodeNodeId.
5. The encoding functions and decoding functions of NodeId should be inside nodes/utils.ts. This should be similar to how I'm doing it for VaultId. An alternative might be to create class extension or generics, but this seems complicated.
6. Raw usage of the NodeId will mostly occur inside the nodes domain, in particular the NodeGraph can use the Uint8Array directly. Note that the js-db currently doesn't yet allow direct usage of the Id, because it requires a Buffer type. However there is an issue Use bufferWrap to support ArrayBuffer, Uint8Array and Buffer js-db#3 to allow it to eventually take Id, so for now, just use idUtils.toBuffer and idUtils.fromBuffer when interacting with the DB.
7. Once this PR is merged into master, rebase Introducing vault sharing and the restrictions of pulling and cloning with vault permissions #266 and Extracting Node Connection Management out of NodeManager to NodeConnectionManager #310 on top to benefit from it.

Here's an example of extending the IdInternal to "specialise" our Ids for a given usecase.

type NodeId = NodeIdInternal & number;

class NodeIdInternal extends IdInternal {
  public encode() {
    return utils.toMultibase(this, 'base32hex');
  }
}

// The `create` static method always returns `Id`, it's compatible with `NodeId` type here
const nodeId = NodeIdInternal.create() as NodeId;

let x: NodeId = nodeId;

// this is not a type error
x.encode();

[CMCDragonkai]

@tegefaulkes can you link the new PRs to this issue as well.

[tegefaulkes]

• Add encoding and decoding methods to IdInternal js-id#11
• Change NodeId from encoded string to an opaque alias of Id which is a typed array from js-id #318