Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parsing markdown with mix of ordered and unordered list runs into RecursionError #356

Closed
ketanbhatt opened this issue Sep 21, 2023 · 1 comment
Closed

Parsing markdown with mix of ordered and unordered list runs into RecursionError #356

ketanbhatt opened this issue Sep 21, 2023 · 1 comment
Labels
bug

Comments

@ketanbhatt
Copy link

Hello 👋🏼

I am using mistune==3.0.1.

When I try to parse a markdown that is one big list of ordered and unordered list, mistune runs into Recursion Error. I suspect it might be taking it as one big nested list.

Here is a sample markdown:

- Fruits
1. Apple
2. Orange

- Vegetables
1. Eggplant
2. Onion

If this markdown continued like this to have more lines than the sys.getrecursionlimit(), parsing this markdown would result in a recursion error.

Here is some code that reproduces the error:

import mistune
import sys

list_to_repeat = """
- Fruits
1. Apple
2. Orange
"""

breaking_md = ""

# Get number of list items such that we hit the recursion limit
repeat_times = sys.getrecursionlimit() // len(list_to_repeat.splitlines()) + 10
print(f"Recursion Limit: {sys.getrecursionlimit()}")
print(f"Repeat times: {repeat_times}")

for idx in range(repeat_times):
    breaking_md += list_to_repeat

print("\nTesting breaking markdown...")
print(f"Number of lines in breaking markdown: {len(breaking_md.splitlines())}")

try:
    mistune.create_markdown(renderer="ast")(breaking_md)
except RecursionError as exc:
    print(f"RecursionError: {exc}")


non_breaking_md = ""

for idx in range(repeat_times):
    non_breaking_md += list_to_repeat
    non_breaking_md += "\n---\n"

print("\nTesting non-breaking markdown...")
print(f"Number of lines in non-breaking markdown: {len(non_breaking_md.splitlines())}")

mistune.create_markdown(renderer="ast")(non_breaking_md)
print("No error")

Output:

Recursion Limit: 1000
Repeat times: 260

Testing breaking markdown...
Number of lines in breaking markdown: 1040
RecursionError: maximum recursion depth exceeded while calling a Python object

Testing non-breaking markdown...
Number of lines in non-breaking markdown: 1560
No error
Here is the stacktrace when the recursion error is hit

I set the recursion limit to 100 so I can get a smaller full stacktrace. The stacktrace is similar with limit 1000.

Traceback (most recent call last):
  File "/Users/ketanbhatt/Library/Application Support/JetBrains/PyCharm2023.1/scratches/scratch_3.py", line 24, in <module>
    mistune.create_markdown(renderer="ast")(breaking_md)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/markdown.py", line 110, in __call__
    return self.parse(s)[0]
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/markdown.py", line 85, in parse
    self.block.parse(state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 446, in parse
    end_pos = self.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 91, in _parse_list_item
    text, continue_width = _compile_continue_width(text, leading_width)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 205, in _compile_continue_width
    text = expand_tab(text)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/util.py", line 18, in expand_tab
    return _expand_tab_re.sub(repl, text)
RecursionError: maximum recursion depth exceeded while calling a Python object

Thank you for your work : )
@lepture lepture added the bug label Sep 24, 2023
lepture added a commit that referenced this issue Sep 29, 2023
@lepture
fix: fix RecursionError for lists
44cc62b 
ref: #356
@lepture
Copy link
Owner

lepture commented Sep 29, 2023

Hi, thanks for your reporting. This issue has been resolved.

@lepture lepture closed this as completed Sep 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lepture @ketanbhatt