Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

catch TokenError on parse #1788

Merged
merged 3 commits into from
May 20, 2024
Merged

catch TokenError on parse #1788

merged 3 commits into from
May 20, 2024

Conversation

tybug
Copy link
Contributor

@tybug tybug commented May 20, 2024 •
edited
Loading

closes #1787.

Reproduced by running atheris locally on the coverage.py oss-fuzz runner:

p = PythonParser(text="\r'\\\n'''")
p.parse_source()

This string passes ast.parse but fails tokenize.generate_tokens (on 3.10 but not 3.12):

s = "\r'\\\n'''"
ast.parse(s) # passes
generate_tokens(s) # errors on 3.10

so we can't rely on ast.parse to give us nice syntax/indentation errors.

nedbat
nedbat reviewed May 20, 2024
View reviewed changes
tests/test_parser.py Outdated Show resolved Hide resolved
nedbat
nedbat reviewed May 20, 2024
View reviewed changes
tests/test_parser.py Outdated Show resolved Hide resolved
@nedbat nedbat merged commit 364282e into nedbat:master May 20, 2024
35 checks passed
@tybug tybug deleted the parse-catch-tokenize branch May 20, 2024 16:07
nedbat added a commit that referenced this pull request May 20, 2024
@nedbat
docs: changelog for #1788 #1787. Thanks Liam DeVoe
0700018
@nedbat
Copy link
Owner

nedbat commented May 26, 2024

This is now released as part of coverage 7.5.2.

renovate bot added a commit to allenporter/flux-local that referenced this pull request May 26, 2024
@renovate
chore(deps): update dependency coverage to v7.5.2 (#692)
78d7624 
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [coverage](https://github.com/nedbat/coveragepy) | `==7.5.1` ->
`==7.5.2` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/coverage/7.5.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/coverage/7.5.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/coverage/7.5.1/7.5.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/coverage/7.5.1/7.5.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>nedbat/coveragepy (coverage)</summary>

###
[`v7.5.2`](https://github.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-752--2024-05-24)

[Compare
Source](https://github.com/nedbat/coveragepy/compare/7.5.1...7.5.2)

- Fix: nested matches of exclude patterns could exclude too much code,
as
    reported in `issue 1779`\_.  This is now fixed.

- Changed: previously, coverage.py would consider a module docstring to
be an
executable statement if it appeared after line 1 in the file, but not
executable if it was the first line. Now module docstrings are never
counted
as executable statements. This can change coverage.py's count of the
number
of statements in a file, which can slightly change the coverage
percentage
    reported.

- In the HTML report, the filter term and "hide covered" checkbox
settings are
remembered between viewings, thanks to `Daniel Diniz <pull 1776_>`\_.

-   Python 3.13.0b1 is supported.

- Fix: parsing error handling is improved to ensure bizarre source files
are
handled gracefully, and to unblock oss-fuzz fuzzing, thanks to `Liam
DeVoe <pull 1788_>`*. Closes `issue 1787`*.

.. \_pull
1776:[nedbat/coveragepy#1776
.. \_issue
1779[nedbat/coveragepy#1779
.. \_issue
178[nedbat/coveragepy#1787
.. \_pull
17[nedbat/coveragepy#1788

.. \_changes\_7-5-1:

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/allenporter/flux-local).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNjguMTAiLCJ1cGRhdGVkSW5WZXIiOiIzNy4zNjguMTAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbXX0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
renovate bot added a commit to allenporter/pyrainbird that referenced this pull request May 27, 2024
@renovate
Update dependency coverage to v7.5.2 (#447)
7a5834b 
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [coverage](https://github.com/nedbat/coveragepy) | `==7.5.1` ->
`==7.5.2` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/coverage/7.5.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/coverage/7.5.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/coverage/7.5.1/7.5.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/coverage/7.5.1/7.5.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>nedbat/coveragepy (coverage)</summary>

###
[`v7.5.2`](https://github.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-752--2024-05-24)

[Compare
Source](https://github.com/nedbat/coveragepy/compare/7.5.1...7.5.2)

- Fix: nested matches of exclude patterns could exclude too much code,
as
    reported in `issue 1779`\_.  This is now fixed.

- Changed: previously, coverage.py would consider a module docstring to
be an
executable statement if it appeared after line 1 in the file, but not
executable if it was the first line. Now module docstrings are never
counted
as executable statements. This can change coverage.py's count of the
number
of statements in a file, which can slightly change the coverage
percentage
    reported.

- In the HTML report, the filter term and "hide covered" checkbox
settings are
remembered between viewings, thanks to `Daniel Diniz <pull 1776_>`\_.

-   Python 3.13.0b1 is supported.

- Fix: parsing error handling is improved to ensure bizarre source files
are
handled gracefully, and to unblock oss-fuzz fuzzing, thanks to `Liam
DeVoe <pull 1788_>`*. Closes `issue 1787`*.

.. \_pull
1776:[nedbat/coveragepy#1776
.. \_issue
1779[nedbat/coveragepy#1779
.. \_issue
178[nedbat/coveragepy#1787
.. \_pull
17[nedbat/coveragepy#1788

.. \_changes\_7-5-1:

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/allenporter/pyrainbird).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNzcuOCIsInVwZGF0ZWRJblZlciI6IjM3LjM3Ny44IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
github-merge-queue bot pushed a commit to canonical/charmcraft that referenced this pull request Jun 10, 2024
@renovate
chore(deps): update bugfixes (#1694)
f46fb73 
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [coverage](https://github.com/nedbat/coveragepy) | `==7.5.1` ->
`==7.5.3` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/coverage/7.5.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/coverage/7.5.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/coverage/7.5.1/7.5.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/coverage/7.5.1/7.5.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
| [cryptography](https://github.com/pyca/cryptography)
([changelog](https://cryptography.io/en/latest/changelog/)) | `==42.0.7`
-> `==42.0.8` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/cryptography/42.0.8?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/cryptography/42.0.8?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/cryptography/42.0.7/42.0.8?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/cryptography/42.0.7/42.0.8?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
| [freezegun](https://github.com/spulec/freezegun)
([changelog](https://github.com/spulec/freezegun/blob/master/CHANGELOG))
| `==1.5.0` -> `==1.5.1` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/freezegun/1.5.1?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/freezegun/1.5.1?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/freezegun/1.5.0/1.5.1?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/freezegun/1.5.0/1.5.1?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
| [platformdirs](https://github.com/platformdirs/platformdirs) |
`==4.2.1` -> `==4.2.2` |
[![age](https://developer.mend.io/api/mc/badges/age/pypi/platformdirs/4.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/pypi/platformdirs/4.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/pypi/platformdirs/4.2.1/4.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/pypi/platformdirs/4.2.1/4.2.2?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>nedbat/coveragepy (coverage)</summary>

###
[`v7.5.3`](https://github.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-753--2024-05-28)

[Compare
Source](https://github.com/nedbat/coveragepy/compare/7.5.2...7.5.3)

- Performance improvements for combining data files, especially when
measuring
line coverage. A few different quadratic behaviors were eliminated. In
one
extreme case of combining 700+ data files, the time dropped from more
than
three hours to seven minutes. Thanks for Kraken Tech for funding the
fix.

- Performance improvements for generating HTML reports, with a side
benefit of
reducing memory use, closing `issue 1791`\_. Thanks to Daniel Diniz for
    helping to diagnose the problem.

.. \_issue
1791:[nedbat/coveragepy#1791

.. \_changes\_7-5-2:

###
[`v7.5.2`](https://github.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-752--2024-05-24)

[Compare
Source](https://github.com/nedbat/coveragepy/compare/7.5.1...7.5.2)

- Fix: nested matches of exclude patterns could exclude too much code,
as
    reported in `issue 1779`\_.  This is now fixed.

- Changed: previously, coverage.py would consider a module docstring to
be an
executable statement if it appeared after line 1 in the file, but not
executable if it was the first line. Now module docstrings are never
counted
as executable statements. This can change coverage.py's count of the
number
of statements in a file, which can slightly change the coverage
percentage
    reported.

- In the HTML report, the filter term and "hide covered" checkbox
settings are
remembered between viewings, thanks to `Daniel Diniz <pull 1776_>`\_.

-   Python 3.13.0b1 is supported.

- Fix: parsing error handling is improved to ensure bizarre source files
are
handled gracefully, and to unblock oss-fuzz fuzzing, thanks to `Liam
DeVoe <pull 1788_>`*. Closes `issue 1787`*.

.. \_pull
1776:[nedbat/coveragepy#1776
.. \_issue
1779[nedbat/coveragepy#1779
.. \_issue
178[nedbat/coveragepy#1787
.. \_pull
17[nedbat/coveragepy#1788

.. \_changes\_7-5-1:

</details>

<details>
<summary>pyca/cryptography (cryptography)</summary>

###
[`v42.0.8`](https://github.com/pyca/cryptography/compare/42.0.7...42.0.8)

[Compare
Source](https://github.com/pyca/cryptography/compare/42.0.7...42.0.8)

</details>

<details>
<summary>spulec/freezegun (freezegun)</summary>

###
[`v1.5.1`](https://github.com/spulec/freezegun/blob/HEAD/CHANGELOG#151)

[Compare
Source](https://github.com/spulec/freezegun/compare/1.5.0...1.5.1)

-   Fix the typing of the `tick()` method, and improve it's behaviour.

</details>

<details>
<summary>platformdirs/platformdirs (platformdirs)</summary>

###
[`v4.2.2`](https://github.com/platformdirs/platformdirs/releases/tag/4.2.2)

[Compare
Source](https://github.com/platformdirs/platformdirs/compare/4.2.1...4.2.2)

<!-- Release notes generated using configuration in .github/release.yml
at main -->

#### What's Changed

- Fix android detection when python4android is present by
[@&#8203;tmolitor-stud-tu](https://github.com/tmolitor-stud-tu) in
[tox-dev/platformdirs#277

#### New Contributors

- [@&#8203;tmolitor-stud-tu](https://github.com/tmolitor-stud-tu) made
their first contribution in
[tox-dev/platformdirs#277

**Full Changelog**:
tox-dev/platformdirs@4.2.1...4.2.2

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "every weekend" in timezone Etc/UTC,
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config help](https://github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/canonical/charmcraft).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zOTMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjM5My4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This was referenced Jun 20, 2024
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nedbat nedbat nedbat left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

oss-fuzz 69058: TokenError
2 participants
@tybug @nedbat