Contents
Offload ZODB blob downloads to the front-end web server using the X-Sendfile / X-Accel-Redirect headers.
X-Sendfile (Apache, Lighttpd) and X-Accel-Redirect (nginx) let a backend application tell the front-end web server to send a file from disk, instead of sending the file content itself.
collective.xsendfile adds this to Plone:
- Plone still handles the request as normal: traversal, permission checks, response headers.
- Instead of streaming the file content through the proxy connection, Plone sends a response with a header holding the blob's file path. The front-end web server reads the file from disk and sends it to the user.
Note
Zope already streams blobs efficiently. This add-on goes further: the file data never passes through Zope or the proxy connection. The benefit depends on your use case, so you may want to run some benchmarks.
- The front-end web server must be able to read the ZODB blob files, so it needs access to the blob directory on the same filesystem (or a shared mount), with permissions that let its user read the files.
- Blobs must be committed. An image scale generated on the fly for the current request is streamed by Plone as usual; later requests for it are offloaded.
Version 2.x supports Plone 6.0, 6.1 and 6.2 on Python 3.10 or later.
For older Plone and Python versions, use version 1.4 (git tag 1.4) from GitHub.
- Apache, with mod_xsendfile (
X-Sendfileheader) - nginx (
X-Accel-Redirectheader) - Lighttpd (
X-Sendfileheader)
.../@@download/fieldname/filename.../@@display-file/fieldname/filename.../context/form/++widget++widgetname/@@download/filename.../@@images/fieldnameand image scales, e.g..../@@images/image/preview
Other urls will use the normal Zope download mechanism.
Install it with pip:
pip install collective.xsendfile
or, if you use buildout, include it in the buildout.cfg:
eggs =
collective.xsendfile
Its ZCML is loaded automatically through plone.autoinclude.
There are two ways to configure collective.xsendfile, either site by site, or globally per Zope instance.
- Install the add-on in your site(s) through the Plone add-on control panel
- Enable XSendFile on your front-end web server and virtual host configuration (see below)
- In the XSendFile Plone control panel, set the HTTP response header for your server and, if needed, the path rewriting
collective.xsendfile can also be configured globally for all Plone sites in a Zope instance using environment variables.
When XSENDFILE_RESPONSEHEADER is set, the per-site settings are ignored, and there is no need to install the add-on in your sites.
XSENDFILE_RESPONSEHEADER- Activates global configuration.
Either
X-Sendfile(Apache, Lighttpd) orX-Accel-Redirect(nginx). XSENDFILE_ENABLE_FALLBACKtrue(the default) oryesmeans the file is served by Zope as usual when the request has noX-Forwarded-Forheader, i.e. when it did not come through the front-end proxy. Any other value always uses the header.XSENDFILE_PATHREGEX_SEARCH- Regular expression applied to the blob's absolute file path.
Defaults to
(.*). XSENDFILE_PATHREGEX_SUBSTITUTE- Replacement for the matched path (Python
re.subsyntax). Defaults to\1. For nginx this has to produce a URL under aninternallocation, for example/xsendfile/\1.
The per-site control panel has the same four settings.
Code that needs Zope to serve a file itself can disable xsendfile for the current request:
from collective.xsendfile.utils import disable_xsendfile disable_xsendfile(request)
Install and enable the Apache module:
sudo apt-get install libapache2-mod-xsendfile sudo a2enmod xsendfile sudo systemctl reload apache2
Related virtual host configuration. Limit XSendFilePath to the blob directory, so the header cannot be used to send any other file:
<VirtualHost *:80>
ServerName example.com
XSendFile on
XSendFilePath /path/to/var/blobstorage
RewriteEngine On
RewriteRule (.*) http://127.0.0.1:8080/VirtualHostBase/http/example.com:80/VirtualHostRoot/$1 [L,P]
</VirtualHost>
Use the X-Sendfile header and leave the path settings at their defaults.
nginx needs an internal location that maps a URL prefix to the blob directory.
internal is essential: it stops browsers from requesting that location directly:
server {
listen 80;
server_name example.com;
location / {
proxy_pass http://127.0.0.1:8080/VirtualHostBase/http/$host:80/VirtualHostRoot/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /xsendfile/ {
internal;
alias /path/to/var/blobstorage/;
}
}
Then use the X-Accel-Redirect header and rewrite the blob path into that location:
- path regex search:
^/path/to/var/blobstorage/(.*) - path regex substitution:
/xsendfile/\1
If the site keeps its blobs in S3 with zodb-s3blobs, there is no blob file on disk for the web server to read. Instead, Plone hands nginx a short-lived presigned S3 URL, and nginx fetches the blob from S3 and sends it to the user. The blob is neither downloaded into Zope nor streamed through it.
Install the s3blobs extra, which requires zodb-s3blobs 1.2.0 or later (the first release that provides IMVCCStorage; with an older release the S3 path is never used):
pip install "collective.xsendfile[s3blobs]"
When the blob's storage is an S3BlobStorage, collective.xsendfile:
- presigns a
GETURL for the blob's committed revision, valid for 60 seconds, with the file's content type and filename signed into it (blobs are stored in S3 without metadata); - sends it in an
X-S3-Urlresponse header; - sets the configured response header to
/s3-internal/.
It does this without loading the blob, which on S3BlobStorage would download it from S3.
The path regex settings are not used for S3 blobs.
Plone streams the blob as usual when there is nothing it can presign: a blob added or changed in the current transaction (including an image scale generated for this request), a storage using SSE-C (the proxy can't send the customer key), or if signing fails.
Use the X-Accel-Redirect header: Apache and Lighttpd can only send files from disk.
Add an internal location that proxies to the URL in X-S3-Url.
Because proxy_pass uses a variable, nginx needs a resolver to look up the S3 host:
server {
...
resolver 127.0.0.53 valid=10s;
location /s3-internal/ {
internal;
# The presigned URL from Plone, used as it is: rewriting it
# would break the signature.
set $s3_url $upstream_http_x_s3_url;
proxy_pass $s3_url;
proxy_ssl_server_name on;
proxy_http_version 1.1;
proxy_set_header Host $proxy_host;
# Don't forward the user's credentials to S3.
proxy_set_header Authorization '';
proxy_set_header Cookie '';
proxy_buffering off;
proxy_pass_request_body off;
proxy_set_header Content-Length '';
}
}
Plone still checks permissions before handing over the URL, and the URL itself never reaches the browser.
The XSENDFILE_ENABLE_FALLBACK setting applies as for local blobs.
- https://github.com/collective/collective.xsendfile
- https://nginx.org/en/docs/http/ngx_http_core_module.html#internal
- https://kovyrin.net/2006/11/01/nginx-x-accel-redirect-php-rails/
- https://github.com/nmaier/mod_xsendfile
- https://blog.jazkarta.com/2010/09/21/handling-large-files-in-plone-with-ore-bigfile/
If you get an HTTP response like:
OK The requested URL /site-images/xxx/cairo.jpg was not found on this server.
it is probably a file permission issue: the web server user cannot read the blob file. Also check that the path in the response header, after rewriting, matches your web server configuration.
- Peter Holzer (@agitator)
- Georg Gogo. BERNHARD (@gogobd)
- Mikko Ohtamaa (@miohtama)
- Jens W. Klein (@jensens)
- Dylan Jay (@djay)
- Jon Pentland (@instification)
Special thanks to Kapil Thangavelu, we extensively borrowed from his code ;-)
GNU General Public License, version 2 or later (GPL-2.0-or-later). See LICENSE.