test(docs): fix test timeout

[ST-DDT]

This is an attempt to fix the frequent test timeouts in verify-jsdoc-tags.

I was unable to identify the root cause of this beyond:

pnpm run test run test/scripts/apidocs/verify-jsdoc-tags.spec.ts

✓ test/scripts/apidocs/verify-jsdoc-tags.spec.ts (2410 tests) 34405ms

Test Files 1 passed (1)
Tests 2410 passed (2410)
Type Errors no errors
Start at 20:46:37
Duration 41.12s (transform 22.75s, setup 92ms, collect 5.58s, tests 34.41s, environment 0ms, prepare 391ms)

Some strange things I noticed:

• AFAICT Only happens on mac and windows (AFAICT)
  • I can reproduce it quite consistently on my system now, but I'm unable to debug it (or at least I don't know how)
• It only happens on the first run of pnpm run test, re-running the tests (without restarting node) doesn't repeat the error
• It always fails a random test, I was unable to identify which one exactly (first, last or any in between)
• It appears to be related to import(path) but not sure
• The duration of the "delay" seems to be very consistent (hence the 30s timeout)

[netlify]

✅ Deploy Preview for fakerjs canceled.

Name	Link
🔨 Latest commit	415f321
🔍 Latest deploy log	https://app.netlify.com/sites/fakerjs/deploys/66238a054e415d000890d50a

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.96%. Comparing base (016a5b6) to head (415f321).

Additional details and impacted files

@@            Coverage Diff             @@
##             next    #2782      +/-   ##
==========================================
- Coverage   99.96%   99.96%   -0.01%     
==========================================
  Files        2973     2973              
  Lines      212613   212613              
  Branches      601      950     +349     
==========================================
- Hits       212540   212529      -11     
- Misses         73       84      +11     

see 1 file with indirect coverage changes

[xDivisionByZerox]

Not sure if pumping up the timeout isn't only kicking the porplem down the road.

[Shinigami92]

Not sure if pumping up the timeout isn't only kicking the porplem down the road.

This is exactly the reason why I didn't approved this PR yet

[xDivisionByZerox]

I simply wanted to leave the comment to explain why I feel unsure about approving this (same as @Shinigami92 stated). That being said, I currently don't feel motivated to inspect this issue myself.

[ST-DDT]

Well, not doing anything to prevent the errors is also just "kicking the problem down the road" with extra work
(because the tests fail in CI and locally).
We could create an investigation issue to raise/retain awareness of this test quirk.

[Shinigami92]

Collecting some of the failing CIs

• https://github.com/faker-js/faker/actions/runs/8595755757/job/23551020812#step:7:352 (Build & Unit Test: node-18, windows-latest)

   ❯ test/scripts/apidocs/verify-jsdoc-tags.spec.ts  (2410 tests | 1 failed) 12514ms
   ❯ test/scripts/apidocs/verify-jsdoc-tags.spec.ts > verify JSDoc tags > utils > mergeLocales > 0 > verify @example tag
     → Test timed out in 5000ms.

• https://github.com/faker-js/faker/actions/runs/8598702878/job/23560053450#step:7:352 (Build & Unit Test: node-18, windows-latest)

  ❯ test/scripts/apidocs/verify-jsdoc-tags.spec.ts  (2410 tests | 1 failed) 12145ms
  ❯ test/scripts/apidocs/verify-jsdoc-tags.spec.ts > verify JSDoc tags > datatype > boolean > 0 > verify @deprecated tag
     → Test timed out in 5000ms.

• https://github.com/faker-js/faker/actions/runs/8602772045/job/23573087471?pr=2798#step:7:349 (Build & Unit Test: node-20, macos-latest)

   ❯ test/scripts/apidocs/verify-jsdoc-tags.spec.ts  (2410 tests | 2 failed) 17615ms
   ❯ test/scripts/apidocs/verify-jsdoc-tags.spec.ts > verify JSDoc tags > string > hexadecimal > 0 > verify @example tag
     → Test timed out in 5000ms.
   If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
   ❯ test/scripts/apidocs/verify-jsdoc-tags.spec.ts > verify JSDoc tags > string > hexadecimal > 0 > verify @deprecated tag
     → Test timed out in 5000ms.

• https://github.com/faker-js/faker/actions/runs/8605638978/job/23582324459?pr=2791#step:7:368

• https://github.com/faker-js/faker/actions/runs/8683199398/job/23808835187?pr=2811#step:7:461

Right now it is 100% inside test/scripts/apidocs/verify-jsdoc-tags.spec.ts 🤔

[Shinigami92]

• I would like to wait with this until fix: tree-shaking #2790 is merged and then see if it is already fixed

[Shinigami92]

It looks like the problem always and only appears in verify-jsdoc-tags.spec.ts > verify JSDoc tags
However it appears in verify @deprecated tag and verify @example tag

I suggest to investigate why it is failing there, are there specific load-times? Can we e.g. cache something for a run? Or specifically the other way: do we need to warmup something?

[ST-DDT]

I suggest to investigate why it is failing there, are there specific load-times? Can we e.g. cache something for a run? Or specifically the other way: do we need to warmup something?

The first dynamic import takes very long. For some strange reason it only happens with the module files, maybe because the are generated on the fly? So we have the option to either increase the hook timeout to 30s or the test timeout to 30s.
If we initialize it in the hook, then we get the console logs an additional time, so IMO it is best to just increase the test timeout.