GHSA-wjmf-p669-5m5p
Protego has exponential backtracking ReDoS in robots.txt URL wildcard matching
Packages
Details
As written in the advisory (Markdown source, shown as plain text).
### Problem description
Protego constructs regular expressions to match URLs against `robots.txt` `Allow:` and `Disallow:` directives, see `protego._urlpattern._URLPattern._prepare_pattern_for_regex()`. Every `*` in the directive value is translated into a lazy `.*?` regex piece, thus a specially crafted directive value with many asterisks may produce a regex that freezes the parser due to exponential backtracking.
### Impact
Parsing a specially crafted `robots.txt` with `protego.Protego.parse()` and then trying to match an URL with `protego.Protego.can_fetch()` results in the latter call not returning for a period dependent on the length of the URL.
### Proof of concept
```python
from protego import Protego
robotstxt = f"""
User-agent: *
Disallow: /{"*1" * 12}*Z
"""
rp = Protego.parse(robotstxt)
url = "/" + "1" * 60
rp.can_fetch(url, "mybot") # freezes
```
References
External links from the advisory. They lead to third-party sites.