[wasm] Enable ICU sharding #51665

tqiu8 · 2021-04-22T03:04:24Z

Initial work to enable ICU sharding according to locale and features.

Added --enable-sharding flag in runtime-test.js to toggle feature on and off.
Test suites currently do not run with sharding enabled.
Added switches between setAppData and setCommonData, which allows shards like icudt_CJK.dat and icudt_EFIGS.dat to be loaded simultaneously (needs more investigating).

Current size differences:

Current Data Files	Size (Compressed)	Proposed Shard Files	Aggregate Sizes (Compressed)
icudt.dat	384	icudt_base.dat icudt_currency.dat icudt_normalization.dat icudt_coll.dat icudt_locales.dat	368
icudt_EFIGS.dat	192	icudt_base.dat icudt_efigs_locales.dat icudt_currency.dat icudt_normalization.dat icudt_efigs_coll.dat	172
icudt_CJK.dat	256	icudt_base.dat icudt_cjk_locales.dat icudt_currency.dat icudt_normalization.dat icudt_cjk_coll.dat	304
icudt_no_CJK.dat	256	icudt_base.dat icudt_no_cjk_locales.dat icudt_currency.dat icudt_normalization.dat icudt_no_cjk_coll.dat	252

We see size gains in EFIGS in no_CJK, but size increases in CJK due to increased size of collation data.

Currently references icu_dictionary.json which is a mapping between culture and relevant files (generated by in dotnet/icu#104).

Related to #49220 and #49221

Additional documentation can be found here.

ghost · 2021-04-22T03:04:33Z

Tagging subscribers to this area: @tarekgh, @safern
See info in area-owners.md if you want to be subscribed.

Issue Details

Initial work to enable ICU sharding according to locale and features.

Added --enable-sharding flag in runtime-test.js to toggle feature on and off.
Test suites currently do not run with sharding enabled.
Added switches between setAppData and setCommonData, which allows shards like icudt_CJK.dat and icudt_EFIGS.dat to be loaded simultaneously (needs more investigating).

Current size differences:

Current Data Files	Size (Compressed)	Proposed Shard Files	Aggregate Sizes (Compressed)
icudt.dat	384	icudt_base.dat icudt_currency.dat icudt_normalization.dat icudt_coll.dat icudt_locales.dat	368
icudt_EFIGS.dat	192	icudt_base.dat icudt_efigs_locales.dat icudt_currency.dat icudt_normalization.dat icudt_efigs_coll.dat	172
icudt_CJK.dat	256	icudt_base.dat icudt_cjk_locales.dat icudt_currency.dat icudt_normalization.dat icudt_cjk_coll.dat	304
icudt_no_CJK.dat	256	icudt_base.dat icudt_no_cjk_locales.dat icudt_currency.dat icudt_normalization.dat icudt_no_cjk_coll.dat	252

We see size gains in EFIGS in no_CJK, but size increases in CJK due to increased size of collation data.

Currently references icu_dictionary.json which is a mapping between culture and relevant files (generated by in dotnet/icu#).

Related to #49220 and #49221

Additional documentation can be found here.

Author:	tqiu8
Assignees:	-
Labels:	`area-System.Globalization`
Milestone:	-

kg · 2021-04-22T03:41:47Z

src/libraries/Native/Unix/System.Globalization.Native/pal_icushim_static.c

@@ -146,7 +154,7 @@ int32_t GlobalizationNative_LoadICUData(const char* path)

    fclose(fp);

-    if (load_icu_data(icu_data) == 0) {
+    if (load_icu_data(icu_data, strcasecmp("icudt.dat", path)) == 0) {


Will this always be named icudt.dat? I think in some cases the platform native .dat file might have a version number in its path.

should rename all the shards to something like .app.dat and assume anything that doesn't match that pattern should use setCommonData?

src/mono/wasm/runtime/library_mono.js

src/tasks/WasmAppBuilder/WasmAppBuilder.cs

src/mono/wasm/build/WasmApp.targets

src/tasks/WasmAppBuilder/WasmAppBuilder.cs

src/mono/wasm/build/WasmApp.targets

src/tasks/WasmAppBuilder/WasmAppBuilder.cs

marek-safar · 2021-04-22T10:34:12Z

Could you add test(s) that the feature works as expected?

CoffeeFlux · 2021-04-23T01:27:59Z

Do you know why the size of the collation data is increasing? Also, how hard would it be to split up locales further as a followup (and how much potential size would we save by doing so)?

eerhardt · 2021-04-23T15:48:13Z

src/libraries/Native/Unix/System.Globalization.Native/pal_icushim_static.c

 {

    UErrorCode status = 0;
-    udata_setCommonData(pData, &status);
+    if (type == 0) {


Can we give this enum some names? That way it reads better.

src/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props

tqiu8 · 2021-04-23T18:53:27Z

Do you know why the size of the collation data is increasing? Also, how hard would it be to split up locales further as a followup (and how much potential size would we save by doing so)?

Not completely sure....Splitting up the collation by locales on the EFIGS does not make a huge difference, but might make a difference in CJK.

src/tests/BuildWasmApps/Wasm.Build.Tests/ICUShardingTests.cs

lewing

Could we move the specifics of the file names and the shard resolution javascript to dotnet/icu (see my comment there) and then just import the props and .js here?

lewing

Could we move the specifics of the file names and the shard resolution javascript to dotnet/icu (see my comment there) and then just import the props and .js here?

lewing · 2021-05-05T22:42:17Z

src/libraries/Native/Unix/System.Globalization.Native/pal_icushim_static.c

@@ -146,7 +154,7 @@ int32_t GlobalizationNative_LoadICUData(const char* path)

    fclose(fp);

-    if (load_icu_data(icu_data) == 0) {
+    if (load_icu_data(icu_data, strcasecmp("icudt.dat", path)) == 0) {


should rename all the shards to something like .app.dat and assume anything that doesn't match that pattern should use setCommonData?

ghost · 2021-05-20T22:23:20Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Initial work to enable ICU sharding according to locale and features.

Added --enable-sharding flag in runtime-test.js to toggle feature on and off.
Test suites currently do not run with sharding enabled.
Added switches between setAppData and setCommonData, which allows shards like icudt_CJK.dat and icudt_EFIGS.dat to be loaded simultaneously (needs more investigating).

Current size differences:

Current Data Files	Size (Compressed)	Proposed Shard Files	Aggregate Sizes (Compressed)
icudt.dat	384	icudt_base.dat icudt_currency.dat icudt_normalization.dat icudt_coll.dat icudt_locales.dat	368
icudt_EFIGS.dat	192	icudt_base.dat icudt_efigs_locales.dat icudt_currency.dat icudt_normalization.dat icudt_efigs_coll.dat	172
icudt_CJK.dat	256	icudt_base.dat icudt_cjk_locales.dat icudt_currency.dat icudt_normalization.dat icudt_cjk_coll.dat	304
icudt_no_CJK.dat	256	icudt_base.dat icudt_no_cjk_locales.dat icudt_currency.dat icudt_normalization.dat icudt_no_cjk_coll.dat	252

We see size gains in EFIGS in no_CJK, but size increases in CJK due to increased size of collation data.

Currently references icu_dictionary.json which is a mapping between culture and relevant files (generated by in dotnet/icu#104).

Related to #49220 and #49221

Additional documentation can be found here.

Author:	tqiu8
Assignees:	-
Labels:	`arch-wasm`, `area-System.Globalization`
Milestone:	-

…et/icu, load ICU.DataFiles.props file generated by the icu

safern · 2021-06-17T22:16:28Z

src/mono/wasm/Makefile

@@ -178,7 +178,7 @@ run-tests-%:
 	EMSDK_PATH=$(EMSDK_PATH) PATH="$(JSVU):$(PATH)" $(DOTNET) build $(TOP)/src/libraries/$*/tests/ /t:Test $(_MSBUILD_WASM_BUILD_ARGS) $(MSBUILD_ARGS)

 run-build-tests:
-	PATH="$(JSVU):$(PATH)" $(DOTNET) build $(TOP)/src/tests/BuildWasmApps/Wasm.Build.Tests/ /t:Test $(_MSBUILD_WASM_BUILD_ARGS) $(MSBUILD_ARGS)
+	PATH="$(JSVU):$(PATH)" $(DOTNET) build $(TOP)/src/tests/BuildWasmApps/Wasm.Build.Tests/ /t:Test $(_MSBUILD_WASM_BUILD_ARGS) $(MSBUILD_ARGS) /p:XUnitMethodName=Wasm.Build.Tests.InvariantGlobalizationTests.Invariant_WithSharding


Is this an intentional change to check in?

safern · 2021-06-17T22:17:45Z

@tqiu8 What is needed to move this PR forward?

lewing · 2021-06-17T22:35:49Z

@safern it is blocked on me at the moment, @tqiu8 has moved on to greener pastures.

maryamariyan · 2021-07-08T22:10:27Z

@lewing the PR seems to have conflicts, I'll close the PR.

tqiu8 added 4 commits April 21, 2021 22:44

initial icu sharding

481fa72

integrate icu dictionary with icu data loading

64743ac

remove unnecessary files in platform manifest

bc78621

revert unnecessary changes

5b9cd96

tqiu8 requested review from radical, lewing, vargaz, kg, marek-safar, steveisok and CoffeeFlux April 22, 2021 03:04

dotnet-issue-labeler bot added the area-System.Globalization label Apr 22, 2021

tqiu8 mentioned this pull request Apr 22, 2021

[wasm] Enable Icu sharding dotnet/icu#104

Closed

5 tasks

kg reviewed Apr 22, 2021

View reviewed changes

src/mono/wasm/runtime/library_mono.js Outdated Show resolved Hide resolved

kg reviewed Apr 22, 2021

View reviewed changes

src/tasks/WasmAppBuilder/WasmAppBuilder.cs Outdated Show resolved Hide resolved

radical reviewed Apr 22, 2021

View reviewed changes

tarekgh requested a review from safern April 22, 2021 15:51

add explanations and add try catch in WasmAppBuilder

c8c4051

CoffeeFlux requested a review from eerhardt April 23, 2021 01:28

eerhardt reviewed Apr 23, 2021

View reviewed changes

src/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props Outdated Show resolved Hide resolved

add some tests for sharding, move sharding switch to WasmApp.targets

e4cdcce

radical reviewed Apr 27, 2021

View reviewed changes

Add test case for invariant globalization

8bc2730

lewing and others added 2 commits May 4, 2021 16:52

Merge branch 'main' into icu-sharding

63d39c7

Add checks for sharding property

ebc8e01

lewing reviewed May 5, 2021

View reviewed changes

tqiu8 added 2 commits May 19, 2021 17:59

wip

99597b6

Merge branch 'main' of github.com:dotnet/runtime into icu-sharding

9e11bf7

maryamariyan added the arch-wasm WebAssembly architecture label May 20, 2021

tqiu8 added 8 commits May 21, 2021 09:39

[wip] change data loading in library_mono.js to match changes in dotn…

22d1c32

…et/icu, load ICU.DataFiles.props file generated by the icu

Merge branch 'main' of github.com:dotnet/runtime into icu-sharding

c8b30d6

[wip] getting shard loading to work

ff4c37e

load ICU.DataFile.props from icu lib dir

35c1b56

move icu_dictionary.js declaration to ICU.DataFiles.props

4baa752

Merge branch 'main' of github.com:dotnet/runtime into icu-sharding

f2bbf96

move icu dictionary loading under conditional

4f22f23

more fixes for full icu scenario

331f452

safern reviewed Jun 17, 2021

View reviewed changes

lewing self-assigned this Jun 17, 2021

maryamariyan closed this Jul 8, 2021

karelz mentioned this pull request Jul 28, 2021

System.Net.Security test run killed on Mono Linux #56478

Closed

ghost locked as resolved and limited conversation to collaborators Aug 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wasm] Enable ICU sharding #51665

[wasm] Enable ICU sharding #51665

tqiu8 commented Apr 22, 2021 •

edited

Loading

ghost commented Apr 22, 2021

kg Apr 22, 2021

lewing May 5, 2021

marek-safar commented Apr 22, 2021

CoffeeFlux commented Apr 23, 2021

eerhardt Apr 23, 2021

tqiu8 commented Apr 23, 2021

lewing left a comment

lewing left a comment

lewing May 5, 2021

ghost commented May 20, 2021

safern Jun 17, 2021

safern commented Jun 17, 2021

lewing commented Jun 17, 2021

maryamariyan commented Jul 8, 2021

[wasm] Enable ICU sharding #51665

[wasm] Enable ICU sharding #51665

Conversation

tqiu8 commented Apr 22, 2021 • edited Loading

ghost commented Apr 22, 2021

kg Apr 22, 2021

Choose a reason for hiding this comment

lewing May 5, 2021

Choose a reason for hiding this comment

marek-safar commented Apr 22, 2021

CoffeeFlux commented Apr 23, 2021

eerhardt Apr 23, 2021

Choose a reason for hiding this comment

tqiu8 commented Apr 23, 2021

lewing left a comment

Choose a reason for hiding this comment

lewing left a comment

Choose a reason for hiding this comment

lewing May 5, 2021

Choose a reason for hiding this comment

ghost commented May 20, 2021

safern Jun 17, 2021

Choose a reason for hiding this comment

safern commented Jun 17, 2021

lewing commented Jun 17, 2021

maryamariyan commented Jul 8, 2021

tqiu8 commented Apr 22, 2021 •

edited

Loading