Support Yarn v2

[eps1lon]

While yarn v2 is still in development alphas are already released and documented: https://yarnpkg.github.io/berry/

Since I'm a big fan of dependabot and yarn v2 I would like those two to get along better.

For the default configuration of yarn v2 dependabot already updates versions correctly. However, it does not update the PnP file of yarn. This means that in a fresh clone of a project that cannot enable zero-install you create a diff by simply running yarn because that will update the pnp.js. I don't know any dev history of dependabot with regards to yarn but it might make more sense for dependabot to "just" run yarn up which should cover package.json, yarn.lock and .pnp.js.

In addition to that a nice enhancement would be to run yarn cache clean for those who have the offline mirror checked into version control.

I solved both of these issues for me by letting a github action cleanup after dependabot by running yarn and yarn cache clean but that does mean dependabot can't make changes to the PR anymore.

A hardcoded yarn.lock filename might also be problematic in the future since the lockfile name is configurable in yarn v2.

[qd-qd]

Any updates ? 😄
Some projects start to migrate his yarn from v1 to v2, a good compatibility will be more and more requested.

[rebelagentm]

Hi! 👋 We're still pretty swamped integrating Dependabot into GitHub, so we haven't yet gotten to this.

[mormahr]

Now that yarn v2 is out, and pnp is enabled by default, maybe the prioritization of this issue should be revisited. We’re in the process of switching to yarn v2 / pnp, but will for the time being not include the pnp file in the repo, because it’ll cause problems with dependabot. We want to change this to use zero-installs, though.

[anthonator]

Agreed that prioritization on this should be revisited. Dependabot is pretty much broken for anyone using Yarn v2 / pnp.

[anthonator]

Is there anything the community can do to help push this along?

[eps1lon]

I push uncommited changes after yarn install on dependabot branches. This means that you have to recreate PRs instead of letting dependabot rebase them. But this isn't much of an issue for me personally.

Using azure pipelines:

trigger:
  - master

pool:
  name: 'Hosted Ubuntu 1604'
  vmImage: 'ubuntu-latest'

steps:
  - checkout: self
    clean: true
    persistCredentials: true

  - task: NodeTool@0
    inputs:
      versionSpec: '10.16.x'
    displayName: 'Install Node.js'

  - script: |
      yarn install
    displayName: 'Install packages'
  - script: |
      git config --global user.email "silbermann.sebastian@gmail.com"
      git config --global user.name "eps1lon[bot]"
      git add -A
      git status
      git diff-index --quiet HEAD || (git commit  --message 'yarn autofix' && git push -u origin HEAD:$(System.PullRequest.SourceBranch))
    # should test the actor but Build.RequestedFor does not point to dependabot but Microsoft.VisualStudio-something
    condition: and(succeeded(), startsWith(variables['System.PullRequest.SourceBranch'], 'dependabot/'))
    displayName: 'Autofix yarn for dependabot'

You could also include dependency deduplication or other autofixes in here.

[trulysinclair]

Same problem here. Working on Yarnberry Cookbook and dependabot breaks yarn.lock. Opening the generated PRs and running yarn throws YAMLException: end of the stream or a document separator is expected at ...:. So I'm guessing D'bot needs to know Yarn 2. As @eps1lon said,

A hardcoded yarn.lock filename might also be problematic in the future since the lockfile name is configurable in yarn v2.

My guess, maybe there should be a lockfile option or detect the configuration from .yarnrc.yml which might be smarter. Another issue is workspaces, it's also breaking my templates. I'm working on a similar approach as @eps1lon with Actions.

[koistya]

I see that Dependabot can resolve updates for Yarn v2 repo, but chokes on local dependencies such as "workspace:*".

See Node.js API Starter Kit (Yarn v2 based monorepo), kriasoft/graphql-starter-kit#215.

[AArnott]

Since Dependabot apparently doesn't work at all in our yarn v2 "Zero Install" repo, I wrote a GitHub Actions workflow that basically does what Dependabot did, but with just one PR for all updates:

name: Update dependencies

on:
  schedule:
  - cron: '0 2 * * *'
  workflow_dispatch:

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - run: yarn up '*'

    - name: Create pull request
      uses: peter-evans/create-pull-request@v3.1.0
      with:
        commit-message: Update all dependencies
        branch: automatic_dependency_updates
        title: Update all dependencies
        body: An updated update of all NPM dependencies.
        #labels: auto-merge
        #reviewers: # optional

[cometkim]

I'm using a similar as @AArnott does, with a simple plugin for using yarn up with more controls

https://github.com/cometkim/yarn-plugin-bump

[ylemkimon]

Similar to #1297 (comment), I'm using a GitHub Actions workflow to fix and update PRs created by Dependabot. This way we can still benefit from its version update logic and release notes.

name: Dependabot

on:
  push:
    branches: [ dependabot/npm_and_yarn/** ]

jobs:
  build:
    runs-on: ubuntu-latest
    if: github.actor == 'dependabot[bot]'

    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 2
        persist-credentials: false # minimize exposure
    - name: Use Node.js 12.x
      uses: actions/setup-node@v1
      with:
        node-version: '12'
    - name: Autofix lockfile
      run: |
        # change directory
        # assuming Angular commit style (build: bump XXX from AAA to BBB in YYY)
        # use $8 for default commit message style (Bump XXX from AAA to BBB in YYY)
        cd .`git log -1 --pretty=%s | awk '{ print $9 }'`

        # restore yarn.lock from the previous commit
        git checkout HEAD^ -- yarn.lock

        # install yarn-plugin-deduplicate
        yarn plugin import https://raw.githubusercontent.com/eps1lon/yarn-plugin-deduplicate/latest/bin/%40yarnpkg/plugin-deduplicate.js

        # if package.json was not updated, upgrade the dependency
        # assuming Angular commit style (build: bump XXX from ...)
        # use $2 for default commit message style (Bump XXX from ...)
        git diff --name-only HEAD^ HEAD | grep -q 'package.json' || yarn up `git log -1 --pretty=%s | awk '{ print $3 }'`

        # restore package.json from the last commit
        git checkout HEAD -- package.json

        yarn install

        # deduplicate lockfile
        yarn deduplicate
      env:
        YARN_ENABLE_SCRIPTS: 0 # disable postinstall scripts
    - name: Config Git
      run: |
        # use personal access token to allow triggering new workflow
        BASIC_AUTH=$(echo -n "x-access-token:${{ secrets.GH_TOKEN }}" | base64)
        echo "::add-mask::$BASIC_AUTH"
        git config --global user.name '${{ github.event.commits[0].author.name }}'
        git config --global user.email '${{ github.event.commits[0].author.email }}'
        git config --local http.$GITHUB_SERVER_URL/.extraheader "AUTHORIZATION: basic $BASIC_AUTH"
    - name: Commit changes
      run: |
        cd .`git log -1 --pretty=%s | awk '{ print $9 }'` # ditto
        git add yarn.lock .yarn/cache .pnp.* # only add yarn.lock if not using zero-installs
        git commit -m "Dependabot autofix"
        git push

EDIT (Aug. 13): Updated to support subdirectories. The test repo is available at https://github.com/ylemkimon/berry-dependabot-test.
EDIT (Aug. 19): Updated to use personal access token to allow triggering new workflow. Moved Git credentials setup after dependencies install to limit (not remove) exposure.

[paul-soporan]

Since Dependabot apparently doesn't work at all in our yarn v2 "Zero Install" repo, I wrote a GitHub Actions workflow that basically does what Dependabot did, but with just one PR for all updates:

name: Update dependencies

on:
  schedule:
  - cron: '0 2 * * *'
  workflow_dispatch:

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - run: yarn up '*'

    - name: Create pull request
      uses: peter-evans/create-pull-request@v3.1.0
      with:
        commit-message: Update all dependencies
        branch: automatic_dependency_updates
        title: Update all dependencies
        body: An updated update of all NPM dependencies.
        #labels: auto-merge
        #reviewers: # optional

FYI: I'd recommend not using this in production (at least for now). yarn up '*' doesn't really do what most people think it does. The way it currently works is that * matches all of the dependencies stored inside the project, and each one of them is then forwarded with range unknown to our suggestUtils.getSuggestedDescriptors function that resolves the unknown range into the latest npm version of that dependency. This means that yarn up '*' will turn all non-npm dependencies into npm dependencies, even when it shouldn't (just like yarn up utils will turn utils into an npm dependency, no matter if it is a git dependency / a portal dependency / whatever else).

Issue to track inside the Yarn 2 repo: yarnpkg/berry#1492.

[oliversalzburg]

@jurre That is working beautifully. Can't wait to have this enabled on all my berry repos ❤

Also, what is it with the spam from @wwahammy 😅

[kachkaev]

Thanks for your reply @jurre! I removed separate config for /cli in kachkaev/njt 🐸. Saw your PR a bit too late, sorry!

[wwahammy]

@jurre That is working beautifully. Can't wait to have this enabled on all my berry repos ❤

Also, what is it with the spam from @wwahammy 😅

Argh, sorry about that. I have a workflow to work around this bug. I was trying to explain why my dependabot PR's had extra commits for working around this. So I put the url to this issue in the commit message and forgot that it would also notify this issue.

I'll remove that url from the workflow first thing this morning.

Sorry all!

[jurre]

@jurre Just as a shot update, after the last run of dependabot about a week ago the status is still at "Checking now" at https://github.com/JabRef/JabRefOnline/network/updates. As I cannot see the logs, I don't know whats wrong but maybe you can spot the origin of the issue in the internal logs?

The last job that I can see on our end was a rebase of JabRef/JabRefOnline#1447 at 2022-10-02 23:39:14 UTC 🤔 I'll see if I can find why the UI shows otherwise

@tobiasdiez I can't find anything that explains why you're seeing that "Checking now" in the logs and am a bit puzzled. Do you mind if I manually start a new update job from our end?

[tobiasdiez]

Seems to be a Schrödinger bug: once you look at it, it disappeared. Now it displayed " Last checked 11 hours ago " (with an error symbol) and the log is

  proxy | time="2022-10-01T15:00:51Z" level=info msg="proxy starting" commit=e2b085ba431f2cff844f6583fd2c63a3a8e67053
  proxy | 2022/10/01 15:00:51 Listening (:1080)
updater | 2022-10-01T15:00:51.746011461 [anonymous-instance:main:WARN:src/firecracker/src/main.rs:370] You are using a deprecated parameter: --seccomp-level 2, that will be removed in a future version.
updater | 2022-10-01T15:00:51.769239401 [473805070:main:WARN:src/devices/src/legacy/serial.rs:432] Detached the serial input due to peer close/error.
updater | time="2022-10-01T15:00:53Z" level=info msg="guest starting" commit=cd770ebd405b5ea5902c9acaaf54f45f1eec960f
updater | time="2022-10-01T15:00:53Z" level=info msg="starting job..." fetcher_timeout=5m0s job_id=473805070 updater_timeout=45m0s updater_version=e96f472ac72427cda33462af129322553e9c10b9
updater | I, [2022-10-01T15:00:54.853799 #7]  INFO -- sentry: ** [Raven] Raven 3.1.2 ready to catch errors
updater | To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
updater | INFO <job_473805070> Starting job processing
  proxy | 2022/10/01 15:00:57 [002] GET https://api.github.com:443/repos/JabRef/JabRefOnline
  proxy | 2022/10/01 15:00:57 [002] * authenticating github api request
  proxy | 2022/10/01 15:00:57 [002] 200 https://api.github.com:443/repos/JabRef/JabRefOnline
  proxy | 2022/10/01 15:00:57 [004] GET https://api.github.com:443/repos/JabRef/JabRefOnline/git/refs/heads/main
  proxy | 2022/10/01 15:00:57 [004] * authenticating github api request
  proxy | 2022/10/01 15:00:57 [004] 200 https://api.github.com:443/repos/JabRef/JabRefOnline/git/refs/heads/main
  proxy | 2022/10/01 15:00:57 [006] GET https://github.com:443/JabRef/JabRefOnline/info/refs?service=git-upload-pack
  proxy | 2022/10/01 15:00:57 [006] * authenticating git server request (host: github.com)
  proxy | 2022/10/01 15:00:58 [006] 200 https://github.com:443/JabRef/JabRefOnline/info/refs?service=git-upload-pack
  proxy | 2022/10/01 15:00:58 [008] POST https://github.com:443/JabRef/JabRefOnline/git-upload-pack
  proxy | 2022/10/01 15:00:58 [008] * authenticating git server request (host: github.com)
  proxy | 2022/10/01 15:00:58 [008] 200 https://github.com:443/JabRef/JabRefOnline/git-upload-pack
  proxy | 2022/10/01 15:00:58 [010] POST https://github.com:443/JabRef/JabRefOnline/git-upload-pack
  proxy | 2022/10/01 15:00:58 [010] * authenticating git server request (host: github.com)
  proxy | 2022/10/01 15:00:58 [010] 200 https://github.com:443/JabRef/JabRefOnline/git-upload-pack
updater | INFO <job_473805070> Finished job processing
updater | time="2022-10-01T15:00:58Z" level=info msg="task complete" container_id=job-473805070-file-fetcher exit_code=0 job_id=473805070 step=fetcher
updater | I, [2022-10-01T15:00:59.898097 #8]  INFO -- sentry: ** [Raven] Raven 3.1.2 ready to catch errors
updater | To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
updater | INFO <job_473805070> Starting job processing
updater | INFO <job_473805070> Starting update job for JabRef/JabRefOnline
updater | INFO <job_473805070> Checking if @variantjs/core 0.0.79 needs updating
  proxy | 2022/10/01 15:01:07 [016] GET https://registry.npmjs.org:443/@variantjs%2Fcore
  proxy | 2022/10/01 15:01:07 [016] 200 https://registry.npmjs.org:443/@variantjs%2Fcore
  proxy | 2022/10/01 15:01:07 [018] GET https://registry.npmjs.org:443/@variantjs%2Fcore/0.0.90
  proxy | 2022/10/01 15:01:08 [018] 200 https://registry.npmjs.org:443/@variantjs%2Fcore/0.0.90
updater | INFO <job_473805070> Latest version is 0.0.90
  proxy | 2022/10/01 15:01:14 [020] GET https://registry.yarnpkg.com:443/@favware/skip-dependency/-/skip-dependency-1.2.0.tgz
  proxy | 2022/10/01 15:01:15 [020] 200 https://registry.yarnpkg.com:443/@favware/skip-dependency/-/skip-dependency-1.2.0.tgz
  proxy | 2022/10/01 15:01:15 [022] GET https://registry.yarnpkg.com:443/unist-builder
  proxy | 2022/10/01 15:01:15 [022] 200 https://registry.yarnpkg.com:443/unist-builder
  proxy | 2022/10/01 15:01:15 [024] GET https://registry.yarnpkg.com:443/@types%2funist
  proxy | 2022/10/01 15:01:15 [024] 200 https://registry.yarnpkg.com:443/@types%2funist
  proxy | 2022/10/01 15:01:15 [026] GET https://registry.yarnpkg.com:443/remark-parse
  proxy | 2022/10/01 15:01:15 [026] 200 https://registry.yarnpkg.com:443/remark-parse
  proxy | 2022/10/01 15:01:15 [030] GET https://registry.yarnpkg.com:443/@types%2fmdast
  proxy | 2022/10/01 15:01:15 [031] GET https://registry.yarnpkg.com:443/mdast-util-from-markdown
  proxy | 2022/10/01 15:01:15 [032] GET https://registry.yarnpkg.com:443/unified
  proxy | 2022/10/01 15:01:15 [032] 200 https://registry.yarnpkg.com:443/unified
  proxy | 2022/10/01 15:01:15 [030] 200 https://registry.yarnpkg.com:443/@types%2fmdast
  proxy | 2022/10/01 15:01:15 [039] GET https://registry.yarnpkg.com:443/bail
  proxy | 2022/10/01 15:01:15 [040] GET https://registry.yarnpkg.com:443/extend
  proxy | 2022/10/01 15:01:15 [041] GET https://registry.yarnpkg.com:443/is-buffer
  proxy | 2022/10/01 15:01:15 [042] GET https://registry.yarnpkg.com:443/vfile
  proxy | 2022/10/01 15:01:15 [043] GET https://registry.yarnpkg.com:443/is-plain-obj
  proxy | 2022/10/01 15:01:15 [044] GET https://registry.yarnpkg.com:443/trough
.... going on with a lot of proxy calls

  proxy | 2022/10/01 15:45:57 [144] 200 https://registry.yarnpkg.com:443/postcss-flexbugs-fixes
  proxy | 2022/10/01 15:45:57 [204] GET https://registry.yarnpkg.com:443/string.prototype.padstart
  proxy | 2022/10/01 15:45:57 [094] 200 https://registry.yarnpkg.com:443/@storybook%2fui
  proxy | 2022/10/01 15:45:57 [169] 200 https://registry.yarnpkg.com:443/has-glob
  proxy | 2022/10/01 15:45:57 [206] GET https://registry.yarnpkg.com:443/symbol.prototype.description
  proxy | 2022/10/01 15:45:57 [208] GET https://registry.yarnpkg.com:443/ts-pnp
updater | time="2022-10-01T15:45:59Z" level=info msg="task complete" container_id=job-473805070-updater exit_code=137 job_id=473805070 step=updater
  proxy | 2022/10/01 15:46:00 [196] 200 https://registry.yarnpkg.com:443/promise.allsettled
  proxy | 2022/10/01 15:46:00 [196] WARN: Cannot write TLS response body from mitm'd client: write tcp 192.168.1.1:1080->192.168.1.2:48570: write: broken pipe
  proxy | 2022/10/01 15:46:00 [202] 200 https://registry.yarnpkg.com:443/string.prototype.padend
  proxy | 2022/10/01 15:46:00 [195] 200 https://registry.yarnpkg.com:443/object.getownpropertydescriptors
  proxy | 2022/10/01 15:46:00 [202] WARN: Cannot write TLS response header from mitm'd client: write tcp 192.168.1.1:1080->192.168.1.2:48606: write: broken pipe
  proxy | 2022/10/01 15:46:00 [195] WARN: Cannot write TLS response body from mitm'd client: write tcp 192.168.1.1:1080->192.168.1.2:48554: write: broken pipe
  proxy | 2022/10/01 15:46:00 [204] 200 https://registry.yarnpkg.com:443/string.prototype.padstart
  proxy | 2022/10/01 15:46:00 [204] WARN: Cannot write TLS response header from mitm'd client: write tcp 192.168.1.1:1080->192.168.1.2:48622: write: broken pipe
  proxy | 2022/10/01 15:46:00 [206] 200 https://registry.yarnpkg.com:443/symbol.prototype.description
  proxy | 2022/10/01 15:46:00 [206] WARN: Cannot write TLS response header from mitm'd client: write tcp 192.168.1.1:1080->192.168.1.2:48628: write: broken pipe
  proxy | 2022/10/01 15:46:00 [208] 200 https://registry.yarnpkg.com:443/ts-pnp
  proxy | 2022/10/01 15:46:00 [200] 200 https://registry.yarnpkg.com:443/string.prototype.matchall
  proxy | 2022/10/01 15:46:00 [208] WARN: Cannot write TLS response header from mitm'd client: write tcp 192.168.1.1:1080->192.168.1.2:48630: write: broken pipe
  proxy | 2022/10/01 15:46:00 [200] WARN: Cannot write TLS response body from mitm'd client: write tcp 192.168.1.1:1080->192.168.1.2:48592: write: broken pipe
  proxy | 2022/10/01 15:46:00 [198] 200 https://registry.yarnpkg.com:443/promise.prototype.finally
  proxy | 2022/10/01 15:46:00 [198] WARN: Cannot write TLS response header from mitm'd client: write tcp 192.168.1.1:1080->192.168.1.2:48576: write: broken pipe
updater | time="2022-10-01T15:46:01Z" level=warning msg="timeout running job" error="waiting for updater: waiting for container: context deadline exceeded" job_id=473805070

I didn't start a new run so that you can have a look, but feel free to start a new update if you would like to.

[Kurt-von-Laven]

Quoting myself describing what may be the same issue for @jurre’s convenience in case they have circumstances in common:

I also noticed that Dependabot hung on the same repository, displaying "Checking now" for the npm ecosystem from approximately 01:02 to 04:30 UTC today after ScribeMD/docker-cache#180 was merged. A pull request was never opened even though one should have been, so I triggered it again manually once "Checking now" was no longer displayed. I did not trigger that scan manually in case that matters. Also, the average performance degraded dramatically relative to a few days ago. Dependabot used to reliably open a pull request within a minute of a run being triggered. Now it seems to take up to 30 minutes.

In the meantime, we remain satisfied with the original explanation that Dependabot was waiting in a queue.

[jurre]

I'm starting to lose track of what's happening in this thread, so asking folks to please open a new issue if you're running into things and tag me in it 🙇

[osdiab]

wondering when this will be generally available - thanks!

[rizqirizqi]

looks like the MR is already merged and it's just waiting for the next release?

[kachkaev]

Re calling yarn dedupe: I created an upstream issue (yarnpkg/berry#4976) – feedback welcome! 🙏

[jurre]

I'm happy to say that Yarn Berry support is now available on all repositories! 🎉 Thanks to everyone for providing valuable feedback and help with getting this shipped!

If there are any edge-cases that we've missed, please feel free to open up new issues for those and we'll try to fix them, but I'm going to close this issue out ❤️

[kylerjensen]

Thank you @jurre!

[lizthegrey]

Thank you so much @jurre. Confirmed working on my Yarn v4 repo (https://github.com/eve-val/eve-roster)

[viniciuspalma]

I believe that it's not working with Zero Installs feature together with --immutable option when running yarn.

Seems that the .yarn/cache isn't being included in the commits neither the pnp.cjs file what cause yarn install --immutable --immutable-cache to fail, since there's change on these folders/files. This command is run before each test job.

• To solve it, I still need to pull the branch and run locally to make the correct changes and then push back, so then the branch get green again.

[oliversalzburg]

@viniciuspalma Create a new issue. See #1297 (comment)

Please lock the issue before we get a dozen more "Thank you" responses and people attaching new problems to the ticket.

[jurre]

Seems that the .yarn/cache isn't being included in the commits

@viniciuspalma that should be supported, but please open up a new issue and describe the issue you're having. Links to repo's where it can be reproduced will make it much easier for us to resolve the issue.

I'm going to lock this issue as suggested, but please do open up new issues for any problems you run into.