windows doesn't print emojis correctly

[nyctef]

Hi! This is my first time looking at Deno, and it seems to be very exciting :)

I was playing around with the examples from the tutorial and noticed this inconsistency: (using powershell and windows terminal):

> deno run https://deno.land/std/examples/welcome.ts
Welcome to Deno 🦕
> deno run --allow-net https://deno.land/std/examples/curl.ts https://deno.land/std/examples/welcome.ts
console.log("Welcome to Deno ­ƒªò");

I found that editing curl.ts to add the line

import { decode } from "https://deno.land/std/encoding/utf8.ts";

and replacing the Deno.stdout.write line with

console.log(decode(body));

seems to make the downloaded file print out correctly.

[r-tae]

Cannot reproduce on macOS 10.15.4:

~/Desktop ❯❯❯ deno run --allow-net https://deno.land/std/examples/curl.ts https://deno.land/std/examples/welcome.ts
console.log("Welcome to Deno 🦕");

but can reproduce on Windows 10 (in PowerShell):

> deno run --allow-net https://deno.land/std/examples/curl.ts https://deno.land/std/examples/welcome.ts
console.log("Welcome to Deno 🦕");

[MarkTiedemann]

Deno.stdout.write writes UTF-8 data but the console code page may be different.

If you run chcp 65001 (65001 = UTF-8) first, the output is as expected.

[nyctef]

If you run chcp 65001 (65001 = UTF-8) first, the output is as expected.

Yep, that works for me too 👍

Seems strange that console.log just works here though - is it doing something different that Deno.stdout.write could use? I tried browsing through the source to figure out how console.log was implemented, but got lost pretty quickly.

[MarkTiedemann]

Seems strange that console.log just works here though - is it doing something different that Deno.stdout.write could use?

I'd assume that it's calling SetConsoleOutputCP(CP_UTF8) before logging but I don't know the internals either.

[piscisaureus]

I think it uses WriteConsoleW.

Supposedly SetConsoleOutputCP(CP_UTF8) should work, but only on the most recent versions of windows.

[littletof]

I just run into the same issue trying to use characters like ∎ (U+220E) ▒ (U+2592) with stdout (win10 powershell).

If you run chcp 65001 (65001 = UTF-8) first, the output is as expected.

Worked for me too, as a temp solution.

[MarkTiedemann]

I think it uses WriteConsoleW.

I think WriteConsoleW doesn't set the console code page.

In fact, since other programs may run in the same console afterwards, I think it's best practice to reset the code page before exit:

UINT original_codepage;

void init() {
        original_codepage = GetConsoleOutputCP();
        SetConsoleOutputCP(CP_UTF8);
}

void cleanup() {
        SetConsoleOutputCP(original_codepage);
}

[SteelAlloy]

#6131
This problem is the same, I couldn't display characters with accents.

The details of the problem are explained, and how to solve it under Windows.

[piscisaureus]

I think WriteConsoleW doesn't set the console code page.

That's true, but here's the secret: WriteConsoleW doesn't depend on the code page, it always uses UTF16 (technically UCS2).

[MarkTiedemann]

WriteConsoleW doesn't depend on the code page, it always uses UTF16

Thanks for sharing this! That's good to know.

Well, back to the problem:

I'd assume that WriteConsoleW is used for console.log and something else is used for Deno.stdout.write. Otherwise the output would be the same, right? Perhaps instead of WriteConsoleW(hStdout, ...), it's WriteFile(hStdout, ...) for Deno.stdout.write? I don't think WriteFile handles unicode console output automatically like WriteConsoleW.

[MarkTiedemann]

@piscisaureus I just traced the two commands using API Monitor. I can confirm that console.log is using WriteConsoleW whereas Deno.stdout.write is using WriteFile.

Screenshots

> deno run https://deno.land/std/examples/welcome.ts

> deno run --allow-net https://deno.land/std/examples/curl.ts https://deno.land/std/examples/welcome.ts

[nitrospaz]

Hi I'm pretty new to all this but I thought this was interesting, see my attached screenshot. I found that the emojis did not display correctly in Windows Powershell (2), Powershell 7.1 (3) or the terminal in VS Code (4) but they do display correctly if you use Windows Powershell or Powershell 7.1 with the new Windows Terminal app (1). So maybe the Terminal is able to interpret the unicode properly where as the Powershell by itself isn't a full unicode native.

[pedropaulosuzuki]

#8199

[prcdpr]

[Console]::OutputEncoding = [Text.UTF8Encoding]::UTF8

did the trick for me while chcp 65001 didn't work (I was testing Windows Terminal using Powershell 7.1).

To keep this setting persistent, you can create a %userprofile%/Documents/PowerShell/profile.ps1 with the line above.

[LitoMore]

Same issue when using the request permissions API in Windows Terminal with WSL.

[midnightnoon]

Perhaps access to stdout could be standardized. Currently all the console.* methods use stdout().write_all() internally, but Deno.stdout.write uses Deno.File. Ideally a call to Deno.stdout.write would just proxy to stdout().write_all() which would have the benefit of automatically handling UTF8 support on Windows environments where switching the code page isn't an option due to legacy software crashing due to it being a "beta" feature from Microsoft.

deno/core/ops_builtin.rs

Line 76 in 4d6f412

pub fn op_print(

As a side note, it's becoming increasingly difficult to consider Deno as a serious platform for our cross-environment development since the core developers treat Windows issues as one of their lowest priorities. I'd gladly pick up some slack and dive into these issues to make Deno have parity on Windows, but I would need some incentive since it would involve taking time to learn Rust as well as time away from my other projects.

[bartlomieju]

@piscisaureus PTAL

[tch1121]

Invalid set code page