Vault Mutation and Reading

[tegefaulkes]

Specification

The usage of withF and withG presents us a new API for VaultInternal mutation and reading:

VaultInternal.readF
VaultInternal.readG
VaultInternal.writeF
VaultInternal.writeG

Make sure to check the remote before writing anything.

You also need to test what happens if the vault were to be checked out to a previous version, and is also considered a remote vault, does that mean pulling fails? If so, we can make it work, by temporarily checking out to the branch pointer, and completing the pull.

Make sure to use the dirty boolean to indicate dirty writing in the middle of writeFand writeG.

When performing operations with a vault, expect to pass a callback that receives the vault. I would want to align the API between NodeConnectionManager and VaultManager. This requires further specification.

As discussed HEAD and MASTER will separately point to the same commit. When we commit using the ref: 'HEAD the HEAD will be moved but the MASTER will remain in place. The MASTER will need to be updated using git.writeRef.

When handling a commit in from previous history we need to check if the HEAD points to the same commit as MASTER using git.resolveRef. If they are different commits then it is a branching commit from the history. In this case we do the commit and update master to the resulting ref using git.writeRef. The old commits between the branching point and the old master ref are deleted.

When we create a new vault we need to check for dirty state. If the dirty flag is set in the metadata we need to checkout the ref that master points to and run a global GC operation. We need to iterate over all commits and check if they are part of the master commit chain.

Additional context

• blocked by metadata changes in Redesign usage of CDSS with VaultInternal #305
• Fixed by Introducing vault sharing and the restrictions of pulling and cloning with vault permissions #266
• As for GCing the unreachable commits see Prevent checking out discarded vault commit IDs with version #252 commentary.

Tasks

• 1. Update VaultInternal to use readF/G and writeF/G.
• 2. Check that the vault is allowed to be written to by checking the remote field in the metadata.
• 3. When writing to the vault, update the boolean metadata field.
• 4. Handle 'branching' history commits by creating new branch, switch the branch names around and delete the old history.
• 5. Handle and recover from 'dirty' state.
• 6. Look into updating commit messages based on file changes.
• 7. Look into aligning API between NodeConnectionManager and VaultManager. seperate issue? this issue is VaultInternal scope and this task seems to be VaultManager scope.

[tegefaulkes]

From prototyping I've discovered some things.
Just a note, for the examples I'll be using the following functions as shorthand for some operations. Otherwise length of the examples blow out quickly and becomes hard to read.

CLICK ME

    dirMeta = {
      fs: fs,
      dir: dir,
    }
    commitMeta = {
      ...dirMeta,
      author: {
        name: 'Mr. Test',
        email: 'mrtest@example.com',
      },
      message: 'hello there!',
    };
  })

  const shortLog = async (stuff) => {
    const log = await git.log({
      ...dirMeta,
      ...stuff,
    });
    return log.map(entry => {
      return `${entry.commit.message}: ${entry.oid}`
    });
  }

  async function quickCommit(message: string, data: string, ref?: string) {
    await fs.promises.writeFile(file, data)
    await git.commit({
      ...commitMeta,
      message,
      ref,
    });
  }

  async function quickRename(oldref: string, ref: string, override?) {
    await git.renameBranch({
      ...dirMeta,
      checkout: true,
      oldref,
      ref,
      ...override
    })
  }

Normal committing

During normal committing where we're appending to the linear history we can maintain the proper HEAD by NOT specifying the ref when committing. It seems that if we specify a ref when committing we end up with either the branch or HEAD not getting updated properly.

// We commit by doing 
    await git.commit({
      ...commitMeta,
      message: 'test commit',
    });

Not providing the ref ensures that both the head and branch are updated. here is another example

    await git.init({
      ...dirMeta,
      defaultBranch: 'main',
    });
    await quickCommit('A', 'one');
    // {[A]}
    await quickCommit('B', 'two');
    // A -> {[B]}
    const log = await shortLog({});
    // [
    //   'B\n: b3850ed0d3f13cbd06c403531b24b2c78303d106',
    //   'A\n: 40dbe0c4a03c8adbc9ae562fc6646f741cc19f91'
    // ]
    console.log(await git.currentBranch(dirMeta));
    // main

branching from detached head state

If we make use of branches then branching from a detached head can be pretty simple. We can create a temp branch called newBranch and commit the new history there. Then we can switch the branch names around changing main->oldMain and newBranch->main. then we can delete the oldMain branch an do any GC of the un-needed refs at that point. if this fails to complete for whatever reason we should have clearly defined branches to recover from. An example of this whole process is shown in the summary block.

CLICK ME

    // main branch: []
    // temp branch: ()
    // HEAD: {} 
    await git.init({
      ...dirMeta,
      defaultBranch: 'main'
    });
    await quickCommit('S', 'one');
    // {[S]}
    const ref = (await git.log(dirMeta)).pop()
    await quickCommit('A', 'two');
    // S => {[A]}
    console.log('head', await shortLog({}));
    // head [
    //   'A\n: b13c65fdc1e600802106822d111cdc2998eab0ef',
    //   'S\n: 95c49e1126dafd347a51bc20d1af845d4f123210' <- saved ref here
    // ]
    console.log(await git.currentBranch(dirMeta));
    // main
    console.log(await git.listBranches(dirMeta));
    // ['main']
    // checkout previous commit
    await git.checkout({
      ...dirMeta,
      ref: ref!.oid,
    })
    // {S} -> [A]
    // creating a temp branch
    console.log('head', await shortLog({}));
    // head [ 'S\n: 95c49e1126dafd347a51bc20d1af845d4f123210' ]
    console.log(await git.currentBranch(dirMeta));
    // undefined
    await git.branch({
      ...dirMeta,
      ref: 'newBranch',
      checkout: true,
    })
    // {(S)} -> [A]
    // adding new history.
    await quickCommit('B', 'alternative');
    // S ->  [A]
    //  \ -> {(B)}
    console.log('head', await shortLog({}));
    // head [
    //    'B\n: 2f91269317ff35d7b56b3b3b0e285cf9b990b403',
    //    'S\n: 95c49e1126dafd347a51bc20d1af845d4f123210'
    //  ]
    console.log(await git.listBranches(dirMeta));
    // [ 'main', 'newBranch' ]
    console.log(await git.currentBranch(dirMeta));
    // newBranch
    // --------- switching over branches
    // The ole switch-a-roo
    // 1. rename main to oldMain
    // 2. rename newBranch to main
    // 3. delete oldMain
    await quickRename('main', 'oldMain');
    await quickRename('newBranch', 'main');
    // S ->  (A)
    //  \ -> {[B]}
    // await git.deleteBranch({
    //   ...dirMeta,
    //   ref: 'oldMain'
    // })
    // --------
    console.log('head', await shortLog({}));
    // head [
    //   'B\n: 2f91269317ff35d7b56b3b3b0e285cf9b990b403',
    //   'S\n: 95c49e1126dafd347a51bc20d1af845d4f123210'
    // ]
    console.log('main', await shortLog({ref: 'main'}));
    // main [
    //   'B\n: 2f91269317ff35d7b56b3b3b0e285cf9b990b403',
    //   'S\n: 95c49e1126dafd347a51bc20d1af845d4f123210'
    // ]
    console.log('oldMain', await shortLog({ref: 'oldMain'}));
    // oldMain [
    //   'A\n: b13c65fdc1e600802106822d111cdc2998eab0ef',
    //   'S\n: 95c49e1126dafd347a51bc20d1af845d4f123210'
    // ]
    console.log(await git.listBranches(dirMeta));
    // [ 'main', 'oldMain' ]
    console.log(await git.currentBranch(dirMeta));
    // main

[tegefaulkes]

I'll have to review how the commit messages are generated. I recall there being an issue with it. It's low priority for now.

[tegefaulkes]

The constant canonicalBranch comes from the vaultUtils. It seems to be a constant for use within VaultInternal so shouldn't it be constant or static within VaultInternal?

[CMCDragonkai]

I've kept other constants in the utils usually in all other domains, so I kept it consistent.

[tegefaulkes]

The GC operation seems trickier than I expected. Isogit doesn't provide any features for removing commits. This means we need to walk the git data structure and remove any object files we don't need from the FS directly. But I don't know if just deleting the objects is all you need to do. In any case this requires a bit of prototyping and testing.

As a first order solution we can just walk the commits and only remove the commit objects. this should remove the commits we don't need and prevent checking them out. However this doesn't remove the underlying files.

To do this properly I'll need to use a reach-ability algorithm to walk the the commits and underlying objects that are needed by existing branches and then remove any objects that are not in that set.

I'll look into prototyping this now.

[tegefaulkes]

Removing the commit object is sufficient to remove unwanted commits for now. I'll have to expand this to remove any unneeded objects in the git tree at some point.

[tegefaulkes]

Core of the implementation is done here. just need to;

• create global GC implementation. If we want to remove any unneeded trees and blobs then it will take a little prototyping and testing.
• fixing up test. most are working but a few cases need tests still such as handling dirty state and checking if writeG is committing properly.

[tegefaulkes]

Need to create a new issue for task 7.