Bugs in handling non-ASCII characters in clipboard over restarts

[comodoro]

1. If I select "závorka", do "stoosh three", restart Dragon and do "spark three", I get zxe1vorka. It should be encoded as z\xe1vorka. Probably affects characters 128-255. Confirmed by @mrob95

2. Certain saved clipboard items give

<class 'toml.decoder.TomlDecodeError'>
Reserved escape sequence used (line 1 column 1 char 0)
<traceback object at 0x0EF9EB48>
  File "C:\workspaces\dns_userdir\caster\lib\utilities.py", line 85, in load_toml_file
    result = toml.loads(f.read())

  File "C:\Python27\lib\site-packages\toml\decoder.py", line 458, in loads
    raise TomlDecodeError(str(err), original, pos)

A clipboard file is pasted here: https://pastebin.com/jGeV5jvE

[Versatilus]

It looks like this is an issue upstream.
uiri/toml#201

If a fix isn't trivial to implement, it might be a good idea to switch this file and others which don't hold information usually edited by users to JSON while a fix is being worked on.

[comodoro]

Thanks, good find. I am not sure what would be more work.

Changing to unicode at https://github.com/synkarius/caster/blob/bc5b2b8bd4a2495c910a4beb4260d50847f922a9/caster/lib/navigation.py#L100 seems to eliminate hex literals, but I am not certain about the whole issue

[Versatilus]

I'll have to investigate further. I should've paid more attention to the clipboard changes in Dragonfly. It used to all be Unicode and just work.

[comodoro]

I don't know, I think with Json it worked. The upstream issue mention was spot on, because the string \x causes problems, being saved as \u00, exactly as described in the issue. Still I would maybe consider dirty temporary fix before temporarily moving back to Json

[LexiconCode]

@Versatilus @comodoro
It looks like we have two alternatives for Python implementations of toml.

• Python (alethiophile) - https://github.com/alethiophile/qtoml
• Python (sdispater) - https://github.com/sdispater/tomlkit

[comodoro]

@Versatilus @LexiconCode A little peek on them:

• qtoml only supports Python 3
• tomlkit seems to have problems with Unicode strings, but might be worth investigating if this is a trivial omission or not, as it has key sorting and there is Implement Sort keys when saving Toml files. #323, if I understand it correctly

[LexiconCode]

Thank you for taking a closer look. How about for now, we utilize tomlkit as it seems to be a more advanced implementation?

• generate comments as part of spec
• The ability to modify TOML documents.
  and so on.

Then for non-user facing files such as clipboard will use json until/if toml parsing libraries become a bit more mature.

[comodoro]

So it turns out I did not use Unicode literals, the tests in tomlkit are quite extensive and include Unicode without problems. So tomlkit seems to be OK everywhere.

[LexiconCode]

@comodoro
tomlkit is not going to be a straight port.
It Caster loads after deleting .toml files but on second load the following stack trace is produce. Which means there's an issue with how toml are read. Which means that our current implementation is going to need some tweaking.

[comodoro]

@LexiconCode This is really best solved upstream, I will post a patch there, in case of non-reaction there are options of Versatilus' going back to json, monkey patching or adapting tomlkit.

[LexiconCode]

Sounds like a plan.

[LexiconCode]

Let me know when you have a patch upstream and I'll comment supporting your implementation.

[comodoro]

OK so it was just a preliminary shot, however it fails one of the tests and is still without reaction: uiri/toml#244. I have noticed as well that it can encode strings into TOML that the decoder is unable to decode, do not know if the decoder would have to be rewritten as well. All in all I think it is not worth the trouble as JSON was perfectly fine.

[comodoro]

This was solved/cuircumvented by #582