FreeBSD VuXML: Documenting security issues in FreeBSD and the FreeBSD Ports Collection

py-Scrapy -- credentials leak vulnerability

Affected packages
py310-Scrapy < 1.8.3
2.0.0 <= py310-Scrapy < 2.6.2
py311-Scrapy < 1.8.3
2.0.0 <= py311-Scrapy < 2.6.2
py37-Scrapy < 1.8.3
2.0.0 <= py37-Scrapy < 2.6.2
py38-Scrapy < 1.8.3
2.0.0 <= py38-Scrapy < 2.6.2
py39-Scrapy < 1.8.3
2.0.0 <= py39-Scrapy < 2.6.2


VuXML ID 2ad25820-c71a-4e6c-bb99-770c66fe496d
Discovery 2022-07-29
Entry 2023-08-31

When the built-in HTTP proxy downloader middleware processes a request with `proxy` metadata, and that `proxy` metadata includes proxy credentials, the built-in HTTP proxy downloader middleware sets the `Proxy-Authentication` header, but only if that header is not already set.

There are third-party proxy-rotation downloader middlewares that set different `proxy` metadata every time they process a request.

Because of request retries and redirects, the same request can be processed by downloader middlewares more than once, including both the built-in HTTP proxy downloader middleware and any third-party proxy-rotation downloader middleware.

These third-party proxy-rotation downloader middlewares could change the `proxy` metadata of a request to a new value, but fail to remove the `Proxy-Authentication` header from the previous value of the `proxy` metadata, causing the credentials of one proxy to be leaked to a different proxy.

If you rotate proxies from different proxy providers, and any of those proxies requires credentials, you are affected, unless you are handling proxy rotation as described under **Workarounds** below.

If you use a third-party downloader middleware for proxy rotation, the same applies to that downloader middleware, and installing a patched version of Scrapy may not be enough;

patching that downloader middlware may be necessary as well.