CodeParser not opening source files with proper decoder

[nedbat]

Originally reported by Brett Cannon (Bitbucket: brettcannon, GitHub: brettcannon)

---

In CodeParser.init() you will notice that it is opening a source file and then reading it, relying on the default encoding for open(). This can trigger a UnicodeDecodeError if the source file specifies an explicit encoding other than Unicode (on Python 3).

For example, in Python's stdlib, Lib/sqlite3/test/dbapi.py has a specified encoding of ISO-8859-1. But because the CodeParser doesn't use something like tokenize.detect_encoding() (http://docs.python.org/py3k/library/tokenize.html#tokenize.detect_encoding) the read fails as there is some bytes in there not allowed under UTF-8 but are valid under ISO-8859-1.

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/107
• This issue had attachments: tokenize_open.diff. See the original issue for details.

[nedbat]

Original comment by Brett Cannon (Bitbucket: brettcannon, GitHub: brettcannon)

---

This also rears its head in CodeUnit.source_file().

[nedbat]

Original comment by Brett Cannon (Bitbucket: brettcannon, GitHub: brettcannon)

---

Attached is a patch that uses Python 3.2's tokenize.open() when available. A solution that works for Python 3.0 and 3.1 could be created by copying the implementation of tokenize.open(), but I went the easier route. =)

BTW, Ned, do you prefer patches or pull requests?

[nedbat]

Fixed in <<changeset bfb4640496bf (bb)>>. I made similar changes in a few more places that seemed like they would also need them.