-
Notifications
You must be signed in to change notification settings - Fork 76
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adopt GB18030-2022 #336
Open
annevk
wants to merge
3
commits into
main
Choose a base branch
from
gb18030-2022-take-2
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Adopt GB18030-2022 #336
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This implements the Unicode Technical Committee recommendation around GB18030-2022 in a matter suitable for this standard, taking into account existing practice and the closeness between GBK and gb18030. In particular, using the text file attached to https://www.unicode.org/L2/L2023/23003r-gb18030-recommendations.pdf this does the following: 1. Merges the first set of 18 mappings, which are bidirectional, directly into index gb18030, replacing existing PUA entries. This ends up impacting GBK and gb18030. 2. The second set of 18 mappings (from PUA to bytes) are encoded as an encoder only table, for both GBK and gb18030. 3. The third set of 18 mappings (from bytes to code points) are ignored, as they are already covered by index gb18030 ranges. (Presumably they are included because the recommendation covers the transition from "Previous Mappings" to "Current Mappings" to "Recommended Mappings", whereas we are going directly from "Previous Mappings" to "Recommended Mappings".) The reason for changing GBK as well is because Chromium and WebKit have already code in the wild that impacts GBK to some degree (although the encoder only table is excluded for GBK only at the moment, including that would make the most sense compatibility-wise) and no fallout has been recorded. Additionally GBK is already positioned as a rough subset of gb18030 in this standard, with the decoder being shared completely. Tests: encoding/legacy-mb-schinese has some GB18030-2022 coverage already. The aim is to complete that with web-platform-tests/wpt#48239 and web-platform-tests/wpt#48240. This supersedes #335. This fixes #27 and fixes #312.
This was referenced Sep 18, 2024
Closed
annevk
added a commit
to annevk/WebKit
that referenced
this pull request
Sep 18, 2024
https://bugs.webkit.org/show_bug.cgi?id=279903 Reviewed by NOBODY (OOPS!). For GBK and gb18030 we have used the same backing table for quite a while now. This backing table was updated to account for GB18030-2022 at some point and this impacted GBK as well. However, the encoder side table was kept disabled for GBK, despite it actually allowing GBK to be more compatible with its former self. whatwg/encoding#336 now standardizes the behavior that GBK and gb18030 are to remain aligned in these matters and this change implements that. The corresponding tests are from this PR: web-platform-tests/wpt#48240 * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-decoder.any.js: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder-expected.txt: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder.html: * Source/WebCore/PAL/pal/text/TextCodecCJK.cpp: (PAL::gb18030AsymmetricEncode): (PAL::gbEncodeShared):
This was referenced Sep 18, 2024
hsivonen
reviewed
Sep 18, 2024
hsivonen
approved these changes
Sep 18, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Technically LGTM, but please 1) revise the informative description of index-gb18030-ranges.txt and 2) please re-run the visualization generator script.
Thanks.
xfq
added
the
i18n-tracker
Group bringing to attention of Internationalization, or tracked by i18n but not needing response.
label
Sep 19, 2024
hsivonen
approved these changes
Sep 19, 2024
webkit-commit-queue
pushed a commit
to annevk/WebKit
that referenced
this pull request
Sep 20, 2024
https://bugs.webkit.org/show_bug.cgi?id=279903 Reviewed by Alex Christensen. For GBK and gb18030 we have used the same backing table for quite a while now. This backing table was updated to account for GB18030-2022 at some point and this impacted GBK as well. However, the encoder side table was kept disabled for GBK, despite it actually allowing GBK to be more compatible with its former self. whatwg/encoding#336 now standardizes the behavior that GBK and gb18030 are to remain aligned in these matters and this change implements that. The corresponding tests are from this PR: web-platform-tests/wpt#48240 * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-decoder.any.js: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder-expected.txt: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder.html: * Source/WebCore/PAL/pal/text/TextCodecCJK.cpp: (PAL::gb18030AsymmetricEncode): (PAL::gbEncodeShared): Canonical link: https://commits.webkit.org/283987@main
annevk
added a commit
to web-platform-tests/wpt
that referenced
this pull request
Sep 21, 2024
See whatwg/encoding#336 for details.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
addition/proposal
New features or enhancements
i18n-clreq
Notifies Chinese script experts of relevant issues
i18n-tracker
Group bringing to attention of Internationalization, or tracked by i18n but not needing response.
normative
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements the Unicode Technical Committee recommendation around GB18030-2022 in a matter suitable for this standard, taking into account existing practice and the closeness between GBK and gb18030.
In particular, using the text file attached to https://www.unicode.org/L2/L2023/23003r-gb18030-recommendations.pdf this does the following:
The reason for changing GBK as well is because Chromium and WebKit have already code in the wild that impacts GBK to some degree (although the encoder only table is excluded for GBK only at the moment, including that would make the most sense compatibility-wise) and no fallout has been recorded. Additionally GBK is already positioned as a rough subset of gb18030 in this standard, with the decoder being shared completely.
Tests: encoding/legacy-mb-schinese has some GB18030-2022 coverage already. The aim is to complete that with web-platform-tests/wpt#48239 and web-platform-tests/wpt#48240.
This supersedes #335. This fixes #27 and fixes #312.
(See WHATWG Working Mode: Changes for more details.)
Preview | Diff