Upgraded rivo/uniseg to latest version, switched StringWidth/Truncate to speedier version #63

rivo · 2022-07-27T11:38:19Z

The rivo/uniseg package has received a major update which also includes methods for grapheme cluster parsing that are much faster than the previously used Graphemes class.

I've upgraded your package accordingly and updated the relevant code to use these faster methods. It would be great if you could merge these changes.

Thank you!

ps. I noticed that some automatic checks did not complete successfully because they are still running on Go 1.15. Would you like me to look into upgrading them to the current version (1.18)?

… to speedier version.

… in StringWidth. See mattn#59

codecov-commenter · 2022-09-11T11:06:37Z

Codecov Report

Merging #63 (0a21090) into master (f9d5553) will increase coverage by 0.25%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master      #63      +/-   ##
==========================================
+ Coverage   93.56%   93.82%   +0.25%     
==========================================
  Files           3        3              
  Lines         171      178       +7     
==========================================
+ Hits          160      167       +7     
  Misses          6        6              
  Partials        5        5

Impacted Files	Coverage Δ
runewidth.go	`95.10% <100.00%> (+0.25%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

rivo · 2022-09-11T11:20:23Z

I upgraded again to the latest uniseg version. Also implemented the "Quick Fix" solution outlined in #59 (comment).

Also updated CI to use current Go versions. Everything passes now.

mattn · 2022-09-11T16:54:34Z

script/generate.go

@@ -166,6 +167,16 @@ func emoji(out io.Writer, in io.Reader) error {
 		})
 	}

+	// We also want regional indicator symbols (flags) to be part of the Emoji
+	// table. They are U+1F1E6..U+1F1FF.


Can you add link to documentation of the specification?

http://www.unicode.org/reports/tr51/#def_emoji_flag_sequence documents how flags are constructed using Regional Indicator code points.

Regional Indicator code points are not classified as Extended_Pictographic so they don't show up in your emoji table. But for the sake of calculating the width, they behave the same as other emojis. So the simplest solution is to add them to your emoji table. Alternatively, you could add the detection of Regional Indicators to all other parts of your code. That would be overkill, in my opinion, but it's up to you.

In any case, you'll want StringLength("🇯🇵") == 2. That's what this is for.

You'll find them in the same file (look for "Regional Indicator"):

https://unicode.org/Public/13.0.0/ucd/emoji/emoji-data.txt

mattn · 2022-09-11T23:52:50Z

go.mod

@@ -1,5 +1,5 @@
 module github.com/mattn/go-runewidth

-go 1.9
+go 1.18


I hope to keep go1.16 but do you have something problem?

Unfortunately, rivo/uniseg uses generics and the new build tag syntax, both of which were introduced with Go 1.18.

I could probably downgrade it in go-runewidth to a previous version but that old version was much slower than the new version. If I do, Go 1.16 will work.

Let me know how you'd like to proceed.

What version of uniseg is it possible to build with go-runewidth with 1.16?

v0.3.4

But if you use v0.3.4, you also need to make adjustments to your code.

Here's my suggestion: I prepare a second PR with the same output as this one, but based on uniseg v.0.3.4 and the older Go version. We can leave this PR (#63) open and you can merge it once you're ready to switch to Go 1.18.

What do you think of this?

How about to separate files for runewidth_go118.go and runewidth_go117.go ?

I just submitted a commit that does this but it gives me merge conflicts. It looks like you made/accepted other changes in the meantime. I'm not able to resolve these conflicts, only you can.

I think I was able to resolve the merge conflict. Please have a look.

rivo · 2022-09-18T16:53:19Z

Do you still need information or any action from me here?

trim prefix

derekperkins · 2022-11-05T17:19:49Z

👍 to getting this merged

derekperkins · 2022-12-05T18:36:05Z

@mattn is there anything blocking this from merging and cutting a new release?

junegunn · 2024-01-04T04:10:15Z

@rivo Thanks for the patch. I can confirm that this fixes a display issue of my program.

junegunn · 2024-01-20T05:09:31Z

runewidth_go118.go

+		cl, s, _, state = uniseg.FirstGraphemeClusterInString(s, state)
+		for index, r := range cl {
+			if index == 0 && inTable(r, emoji) {
+				chWidth = 2 // Not the optimal solution but it will work in most cases.


I noticed that this incorrectly reports the width of some characters, such as ▶ (0x25b6), which is in both the emoji and ambiguous tables.

Related: junegunn/fzf#3588

Well, it turns out that emojis can have multiple representations. There is a default for each emoji and your example or, say, ☺, defaults to "text presentation". And there are code points that can be added to force a specific presentation. To make things worse, many systems don't respect the Unicode specification and then there is the question of whether fonts will support the different representations.

go-runewidth does not go into this detail. This is also why the comment here says "in most cases". It's a simple approximation. If you want this emoji presentation flag to be considered, you can also use uniseg directly which has had string width calculation for about two years now. uniseg.StringWidth("\u25b6") will report as "1".

From what I can see, uniseg will be much more accurate than go-runewidth. However, go-runewidth is mostly in line with most terminals which also tend to use the simple wcwidth functionality. Depending on your application, there may be some benefit to "making the same mistakes" as the environment it is run in. For example, iTerm2 on macOS which is a very popular terminal application, does not render the rainbow flag correctly:

This flag obviously has a width of 2 but iTerm2 assigns a width of 1. If you need your application to be in line with iTerm2, it may be better to make that same mistake. go-runewidth is probably your best bet in that case. (Although I don't know what it reports for this specific example, it might not have the same issue as iTerm2.) But if you need something that is accurate, no matter what, I would suggest using uniseg to calculate rune/string widths.

As a final note, VS Code renders the flag correctly:

There is no official spec for these widths. Therefore, everyone rolls their own implementation. It's a mess.

Thanks for the detailed comment. Really appreciated. I'll see if I can use uniseg.StringWidth instead.

jesseduffield · 2024-06-23T10:24:25Z

Hi @mattn just checking if there's anything specific blocking this merge?

@rivo Are you using this branch yourself in another project? Is your fork something I could safely use myself?

rivo · 2024-06-23T10:49:09Z

@jesseduffield I'm using this package but not this branch. After tons and tons of hours and effort getting into the details of Unicode and terminals, including writing my own version of a character width package (https://github.com/rivo/uniseg?tab=readme-ov-file#monospace-width), I realize that which package to use really depends on what you're trying to achieve. go-runewidth will be more or less compatible with most terminals out there. Most of them use the widespread wcwidth implementation which, unfortunately, is old and buggy. Many emojis don't render properly, spacing is wrong is lots of cases. And similar issues have been found here, in go-runewidth. But if your application runs in these terminals, you'll probably want to make the same "mistakes". It doesn't help if a character is really 2 spaces wide but the terminal will always advance by 1 space. So go-runewidth is your best bet in doing what the terminal is doing.

On the other hand, the uniseg package attempts to give you the "real" width of a character. If your application has control over where a character is placed, I would suggest using uniseg instead. For example, if you write your own text editor GUI (e.g. something like VSCode) where you don't rely on terminals, I would not suggest using go-runewidth but uniseg instead.

Regarding this branch, I don't know what the plans are. This whole thing is a complicated topic and I totally understand if @mattn doesn't have the time to get into the details. Looks like there's not much happening in this project anymore anyway.

Maybe I should add wcwidth compatibility to uniseg at some point. Then people could simply switch over.

mattn · 2024-06-23T12:04:43Z

I got some errors in this branch.

./runewidth_go118.go:188:28: undefined: uniseg.FirstGraphemeClusterInString
./runewidth_go118.go:214:33: undefined: uniseg.FirstGraphemeClusterInString
./runewidth_go118.go:244:33: undefined: uniseg.FirstGraphemeClusterInString

rivo · 2024-06-23T16:03:29Z

@mattn I haven't spent time on this PR anymore as it seemed like you didn't want to follow up on it / merge it. I can fix these issues but only if you indicate that you will merge it. Otherwise, it would be a wasted effort. Please let me know.

Also, your go.mod still lists Go 1.9 as a requirement. uniseg requires 1.18 due to the use of generics. Would you consider bumping your Go version to 1.18 as well? It would make things much easier.

jesseduffield · 2024-06-23T23:04:44Z

@rivo thanks for that detailed explanation, that's really helpful

rivo added 3 commits July 27, 2022 12:34

Upgraded rivo/uniseg to latest version, switched StringWidth/Truncate…

835809f

… to speedier version.

Adapted to short-notice change in rivo/uniseg.

a8413cf

Upgraded to latest rivo/uniseg. Also implemented basic emoji handling…

2e820a2

… in StringWidth. See mattn#59

Added a test for Truncate that includes emojis.

0a21090

This was referenced Sep 11, 2022

Width is 1 when it should be 2 #59

Open

Unwanted artifacts on screen when scrolling text with a lot of unicode characters rivo/tview#693

Closed

mattn reviewed Sep 11, 2022

View reviewed changes

tty2 added 7 commits September 15, 2022 18:08

trim prefix

f052c83

change tail to prefix

01f382a

change t.Run names

8f4e6bc

change tests to table-driven approach

669734a

if pos < w: add empty chars

b18c354

change function name to TruncateLeft

6dff02e

add spaces == width+chWidth-w

10a3ce9

rkfg mentioned this pull request Sep 17, 2022

Some node aliases are trimmed edouardparis/lntop#89

Closed

mattn and others added 4 commits September 20, 2022 21:35

Merge pull request mattn#65 from tty2/feature/trim-prefix

2c6a438

trim prefix

Split the code so it can be upgraded once we move to Go1.18+

73a4cd2

Resolved merge conflict.

5f0dad1

The wrong uniseg version was used. Fixed it.

bb452ae

junegunn reviewed Jan 20, 2024

View reviewed changes

jesseduffield mentioned this pull request Jun 23, 2024

Improve render performance jesseduffield/lazygit#3686

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgraded rivo/uniseg to latest version, switched StringWidth/Truncate to speedier version #63

Upgraded rivo/uniseg to latest version, switched StringWidth/Truncate to speedier version #63

rivo commented Jul 27, 2022 •

edited

Loading

codecov-commenter commented Sep 11, 2022 •

edited

Loading

rivo commented Sep 11, 2022

mattn Sep 11, 2022

rivo Sep 11, 2022 •

edited

Loading

mattn Sep 11, 2022

rivo Sep 12, 2022

mattn Sep 18, 2022

rivo Sep 18, 2022

mattn Sep 19, 2022

rivo Sep 20, 2022

rivo Sep 30, 2022

rivo commented Sep 18, 2022

derekperkins commented Nov 5, 2022

derekperkins commented Dec 5, 2022

junegunn commented Jan 4, 2024

junegunn Jan 20, 2024

rivo Jan 20, 2024

junegunn Jan 20, 2024

jesseduffield commented Jun 23, 2024

rivo commented Jun 23, 2024

mattn commented Jun 23, 2024

rivo commented Jun 23, 2024

jesseduffield commented Jun 23, 2024

Upgraded rivo/uniseg to latest version, switched StringWidth/Truncate to speedier version #63

Are you sure you want to change the base?

Upgraded rivo/uniseg to latest version, switched StringWidth/Truncate to speedier version #63

Conversation

rivo commented Jul 27, 2022 • edited Loading

codecov-commenter commented Sep 11, 2022 • edited Loading

Codecov Report

rivo commented Sep 11, 2022

Choose a reason for hiding this comment

rivo Sep 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rivo commented Sep 18, 2022

derekperkins commented Nov 5, 2022

derekperkins commented Dec 5, 2022

junegunn commented Jan 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jesseduffield commented Jun 23, 2024

rivo commented Jun 23, 2024

mattn commented Jun 23, 2024

rivo commented Jun 23, 2024

jesseduffield commented Jun 23, 2024

rivo commented Jul 27, 2022 •

edited

Loading

codecov-commenter commented Sep 11, 2022 •

edited

Loading

rivo Sep 11, 2022 •

edited

Loading