Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: use normalize() for unicode paths #3007

Closed
wants to merge 0 commits into from

Conversation

silverwind
Copy link
Contributor

Went ahead and changed the failing test to use String.prototype.normalize so path strings are compared form-independently.

@srl295 The test will still fail on debug builds that don't include ICU. Is there a way I can feature-test if the current build includes ICU? Maybe we should add process.versions.icu?

Related: #2165
Docs Issue: nodejs/docs#42

@silverwind silverwind added test Issues and PRs related to the tests. macos Issues and PRs related to the macOS platform / OSX. i18n-api Issues and PRs related to the i18n implementation. labels Sep 22, 2015
@silverwind
Copy link
Contributor Author

CI: https://ci.nodejs.org/job/node-test-pull-request/362/

(Let's hope CI builds have ICU enabled)

@srl295
Copy link
Member

srl295 commented Sep 22, 2015

@silverwind they have small ICU enabled, because that's the default since v3.1.0.

See https://github.com/nodejs/node/blob/master/test/parallel/test-intl.js#L13 for an example of testing whether ICU was enabled.

@silverwind
Copy link
Contributor Author

@srl295 thanks. Can one assume that both the root and en locales will always be present when built with ICU? What's the root locale exactly, by the way?

@silverwind
Copy link
Contributor Author

Updated the test so it works on non-ICU builds too by using characters that can't be decomposed (emoji) when ICU is not present.

Another CI: https://ci.nodejs.org/job/node-test-pull-request/365/

@srl295
Copy link
Member

srl295 commented Sep 25, 2015

@silverwind no.. you can't really assume it. but see process.config.variables.icu_locales

 > process.config.variables.icu_locales.split(',')
 [ 'en', 'root' ]

Root is just the "base" locale ID. It should have been named und if iso-639-2 had been around way back when. It's not English, but its collation is (close to) UCA, and it contains items that are common to other locales.

@silverwind
Copy link
Contributor Author

I will rework this once #3089 is landed.

@silverwind
Copy link
Contributor Author

Using process.version.icu now. Any feedback?

@jbergstroem
Copy link
Member

LGTM

@silverwind
Copy link
Contributor Author

There's a slight issue here: the test change depends on process.versions.icu which will land in 4.2, while this test change could go in earlier in 4.1.x. Won't be an issue on master thought.

cc: @nodejs/release

@silverwind
Copy link
Contributor Author

It won't fail dramatically in 4.1.x thought, it just won't actually test .normalize, even if Intl is there, so might be a non-issue.

@rvagg
Copy link
Member

rvagg commented Oct 3, 2015

might leave it out of 4.1.2 eh? don't want the commit suggesting to people that it's actually doing something

@silverwind
Copy link
Contributor Author

Alright, will wait after 4.1.2 with landing this.

@Fishrock123
Copy link
Contributor

LGTM if CI is happy and others sign-off.

@silverwind
Copy link
Contributor Author

One more CI with the latest change: https://ci.nodejs.org/job/node-test-pull-request/418/

@thefourtheye
Copy link
Contributor

Change LGTM, hope it fixes the failure.

silverwind added a commit that referenced this pull request Oct 6, 2015
OS X 10.11 changed the unicode normalization form of certain code points
returned by system calls like getcwd() from NFC to NFD which made
results in this test failing.

The consensus of #2165 is to delegate
the task of unicode normalization to the user, and work will continue to
document how to handle unicode in a form-sensitive file system.

PR-URL: #3007
Fixes: #2165
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
@silverwind silverwind closed this Oct 6, 2015
@silverwind silverwind deleted the normalize-fix branch October 6, 2015 16:14
@silverwind
Copy link
Contributor Author

Landed in 81e98e9, thanks!

@Fishrock123
Copy link
Contributor

@silverwind this requires ICU right? Does this test still work properly without it?

Fwiw, the CI compiles without ICU for test runs.

@silverwind
Copy link
Contributor Author

@Fishrock123 it works in non-ICU builds too, where process.version.icu will be undefined, and in which case we write an emoji-filename which can't be decomposed 😉.

@silverwind
Copy link
Contributor Author

Also, .normalize() returns the string as-is in non-ICU builds, if you're wondering.

@jasnell jasnell mentioned this pull request Oct 8, 2015
29 tasks
silverwind added a commit that referenced this pull request Oct 8, 2015
OS X 10.11 changed the unicode normalization form of certain code points
returned by system calls like getcwd() from NFC to NFD which made
results in this test failing.

The consensus of #2165 is to delegate
the task of unicode normalization to the user, and work will continue to
document how to handle unicode in a form-sensitive file system.

PR-URL: #3007
Fixes: #2165
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
i18n-api Issues and PRs related to the i18n implementation. macos Issues and PRs related to the macOS platform / OSX. test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants