ipfs:preload and ipfs:pin takes much time on ipfs.add

[MicrowaveDev]

• Version:

js-ipfs version: 0.41.0-
Repo version: 7
System version: x64/linux
Node.js version: v10.17.0

• Platform:

Linux ipfs-node-geesome 4.4.0-174-generic #204-Ubuntu SMP Wed Jan 29 06:41:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Type: Enhancement

Severity: Medium

Description: When I'm trying to save content through ipfs.add function - it takes around 4 seconds.

Content example: "1" with IPFS hash: "QmWYddCPs7uR9EvHNCZzpguVFVNfHc6aM3hPVzPdAEESMc"

Logs with DEBUG=ipfs*:

  before time 1582307929842
  ipfs:preload https://node1.preload.ipfs.io/api/v0/refs?r=true&arg=QmWYddCPs7uR9EvHNCZzpguVFVNfHc6aM3hPVzPdAEESMc +3s
  ipfs:preload successfully preloaded QmWYddCPs7uR9EvHNCZzpguVFVNfHc6aM3hPVzPdAEESMc in 1605ms +4s
  ipfs:pin Flushed pins with root: QmPUsuVotrcPccnQbCoWqwJaoQtN22QYj13WgqCCW9joMg +8s
after time 1582307933696

Diff between timestamps: 3.8 seconds

In the same time logs on the fresh node:

before time 1582307433778
  ipfs:preload https://node1.preload.ipfs.io/api/v0/refs?r=true&arg=QmWYddCPs7uR9EvHNCZzpguVFVNfHc6aM3hPVzPdAEESMc +0ms
  ipfs:pin Flushed pins with root: QmWqnLGtgUtE1y9s1zgfMAKzEj9z81Zqg1MrSJdZxV6j4G +446ms
after time 1582307433844

Diff between timestamps: 0.066 seconds

Steps to reproduce the error:

It occurs only on my old(by time, but not by version) node with bunch of saved content. When I'm trying to reproduce it in new node - it's saved instantly, less then 1 second.

Node code:

const IPFS = require('ipfs');
IPFS.create({
   pass: 'test test test test test test test test test test',
   EXPERIMENTAL: {
        pubsub: true,
        ipnsPubsub: true
    },
    config: {
      Addresses: {
        Swarm: [
          "/ip4/0.0.0.0/tcp/4002",
          "/ip4/127.0.0.1/tcp/4003/ws",
        ]
      }
    }
  }).then(async (ipfs) => {
    setInterval(async () => {
      console.log('before time', new Date().getTime());
      await itFirst(ipfs.add('1'));
      console.log('after time', new Date().getTime());
    }, 10 * 1000);
    return ipfs;
  });

[achingbrain]

This will be fixed by #2771, fingers crossed in the next release. Please subscribe to notifications on that PR for updates.

Essentially as currently implemented pinning performance gets progressively worse as the number of pins increases. The solution is to store the pins in a format that has better scalability..