Contao crawler leaks auth credentials to external hosts

Summary

Contao's crawler tries to prevent confidential HTTP client options from being sent to external domains by creating a scoped client: full options for root page origins, cleaned options for everything else. The cleaner removes Cookie and Authorization headers, but it removes the non-Symfony option names basic_auth and bearer_auth instead of Symfony HttpClient's real auth_basic and auth_bearer options.

When contao.crawl.default_http_client_options contains Basic or Bearer authentication for a protected staging/production site, those credentials remain in the "clean" client used for external links or configured additional URIs. An attacker who can get an external URL crawled, for example through a link on a crawled page while the broken-link checker is enabled, can receive the crawler credentials.

Technical Detail

Root Cause

// core-bundle/src/Crawl/Escargot/Factory.php:175-209 @ e550b92a01ef625bd546e6c3956dd200af05ebf0
private function createHttpClient(array $options = []): HttpClientInterface
{
    $options = array_merge_recursive(
        [
            'headers' => [
                'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                'user-agent' => self::USER_AGENT,
            ],
            'max_duration' => 10,
        ],
        array_merge_recursive($this->getDefaultHttpClientOptions(), $options),
    );

    $cleanOptions = $this->cleanOptionsFromConfidentialData($options);

    if ($options === $cleanOptions) {
        return ($this->httpClientFactory)($options);
    }

    $scopedOptionsByRegex = [];

    foreach ($this->getRootPageUriCollection()->all() as $rootPageUri) {
        $scopedOptionsByRegex[preg_quote($this->getOriginFromUri($rootPageUri))] = $options;
    }

    return new ScopingHttpClient(($this->httpClientFactory)($cleanOptions), $scopedOptionsByRegex);
}

// core-bundle/src/Crawl/Escargot/Factory.php:226-247 @ e550b92a01ef625bd546e6c3956dd200af05ebf0
foreach ($options as $k => $v) {
    if ('headers' === $k) {
        foreach ($v as $header => $value) {
            if (\in_array(strtolower($header), ['authorization', 'cookie'], true)) {
                continue;
            }

            $cleanOptions['headers'][$header] = $value;
        }

        continue;
    }

    if ('basic_auth' === $k || 'bearer_auth' === $k) {
        continue;
    }

    $cleanOptions[$k] = $v;
}

Symfony HttpClient authentication options are auth_basic and auth_bearer; Contao's own manual documents auth_basic for crawler Basic Authentication. Because the cleaner only strips basic_auth and , the "clean" default client for non-root-page hosts still carries the real auth options. The existing factory test intends to assert that is not sent to , but its mock client factory ignores the argument, so it does not catch auth options that survive into .