Web protection
Web protection is a set of checks es-python runs on every request your web application receives: who sent it, how often, whether it is framed in a way that can confuse a proxy, and what your application sent back. This page explains each check, how a client is identified, and how blocking works.
What web protection is
Web protection adds request-level checks to es-python, the kind a web application firewall makes, but from inside your application. Taint tracking follows attacker data to a dangerous operation; these checks look at the request and the response as a whole. Nothing is on until you switch it on, on the Filters page for your whole account or in a machine's Configure window for one machine.
A running program picks up a change within about 30 seconds. The one exception is a program that was started while every taint source and every web protection setting was off: it runs without the request checks, so after turning web protection on for the first time, restart it once.
The checks run in applications built on Django, Flask, FastAPI and other Starlette applications, and aiohttp. Each finding is reported as an event on the es-python list, with the source http, so rules can show, hide, drop or raise on it like any other event.
| Setting | What it does | On by default |
|---|---|---|
| Who sent the request | Picks the client address every other check uses | Header X-Forwarded-For |
| Rate limit | Refuses (or reports) a client sending too many requests | Off |
| Blocked clients | Refuses every request from an address on your list | Off |
| Checks | Reports ambiguous requests, reflected input, open redirects, forged host names, card numbers in responses and missing security headers | Off |
| Request on each event | Attaches the client, method and path to events | On |
| Routes | Lists the routes your application serves, beside the events on each | Off |
Who sent the request
Every check counts, blocks and reports a request under one client address. By default es-python reads it from the X-Forwarded-For header, which a load balancer or reverse proxy adds; you can name a different header, or use the address of the connection itself.
- A request header (the default). Each proxy appends the address it saw to the right of the header, so the right-most entries were written by your own proxies and the left-most one is whatever the client claimed. Set Proxies in front to the number of proxies between the internet and your application: with one load balancer, keep 1. If the header is missing, the connection address is used instead.
- Header name. Any header your proxy sets, for example
X-Real-IP,CF-Connecting-IPorTrue-Client-IP. A single-address header works the same way, andForwarded(RFC 7239) is read from itsfor=part. - The connection address. The address of the machine that opened the connection. Choose this when nothing sits in front of your application. Behind a proxy it would be the proxy's address for every visitor.
A header is only as trustworthy as the proxy that writes it. If your application can also be reached directly, a client can send any X-Forwarded-For it likes, so make sure traffic can only arrive through your proxy.
Rate limit
The rate limit refuses a client that sends more than a set number of requests within a set number of seconds, with 429 Too Many Requests and a Retry-After header. The default is 100 requests per 60 seconds. Set the action to Report only to record the event without refusing anything.
The count slides across time rather than resetting on a fixed clock, so a client cannot send twice the limit by timing requests around a boundary. Each client is reported once per window, as a rate limit event.
The count is kept in each application process. If your server runs four worker processes, a client can send up to four times the limit before every worker has refused it. Set the limit with that in mind, or keep a separate limit at your load balancer for exact totals.
Refusing requests is part of Raise. Without Raise on your plan, the rate limit reports only.
Blocked clients
The blocked clients list holds the addresses and ranges your application refuses, with 403 Forbidden, before your code runs. It is one list for your whole account, kept on the Filters page; switch Blocked clients to Refuse listed clients for the account or for one machine to enforce it.
Add an address such as 203.0.113.7 or a range such as 203.0.113.0/24, with an optional note and an expiry. A range wider than a /8 (IPv4) or a /32 (IPv6) is refused, so a typo cannot lock out everyone. You can also block a client straight from an event: open the event, and under Web request choose Block this client. A blocked client's first refusal in each minute is reported as a blocked client event.
A change reaches your machines within about 30 seconds. Enforcing the list is part of Raise; without it you can still build the list, and it applies once Raise is on your plan.
Auto-block
Auto-block adds a client to your blocked clients for a set number of hours after it does something only an attacker does. It is account-wide and off by default.
- Attack-shaped requests blocks a client whose request set off the reflected input, ambiguous request or forged host name check. Ordinary visitors do not send live markup, conflicting body lengths or someone else's host name.
- Also any high or critical event additionally blocks a client whose request led to a high or critical event. This catches more, but if ordinary traffic reaches a flaw in your application, ordinary visitors are blocked too. Use it once the events list is quiet.
Private, loopback and link-local addresses are never auto-blocked. When the client header is missing, es-python falls back to the connection address, which behind a proxy is the proxy itself, and blocking it would refuse everyone. An event that was made safe before it reached the risky call (suppressed) never triggers a block. Auto-blocked entries are marked auto in the list, expire on their own, and can be removed early.
Checks
Each check reports what it finds as an event and never refuses a request on its own. A Raise rule on an ambiguous request makes es-python refuse it with 400; the response checks run after the response has been sent, so a Raise rule on them is recorded but cannot take the response back.
| Check | Event | Reports |
|---|---|---|
| Ambiguous requests | request smuggling |
A request with two different Content-Length values, both Content-Length and Transfer-Encoding, or an unusual Transfer-Encoding. These are how request smuggling makes a proxy and your application disagree about where a request ends. |
| Reflected input | reflected xss |
A query parameter carrying HTML or script (a tag, a javascript: link, or a quote followed by an event handler) that appears unchanged in the HTML page your application returned. Escaped output does not match. |
| Open redirect | open redirect |
A redirect to another site whose address came from a query parameter. Redirects within your own host, and fixed redirects to a partner site, are not reported. |
| Forged host name | host header injection |
A request whose Host (or X-Forwarded-Host) is not one of your host names, when your application writes that name into a link or a redirect: how a password reset link gets pointed at an attacker's site. List your names in the field under the check, such as shop.example, *.shop.example; with no names the check cannot judge, so it stays off. |
| Card numbers in responses | card number in response |
A payment card number (it must pass the card checksum) in an HTML, text, JSON or XML response. A number the request itself sent is ignored, so a checkout page showing a visitor's own card back is not reported. The event shows the number masked to its first six and last four digits. Reported once an hour for each path. |
| Missing security headers | missing security header |
An HTML page without Content-Security-Policy, X-Frame-Options (or a CSP frame-ancestors) or X-Content-Type-Options, plus Strict-Transport-Security on HTTPS. Reported once an hour for each combination of missing headers. |
Request on each event
With Request on each event on, every event es-python reports while serving a web request carries that request: the client address, the address the connection came from when it differs, the method, the path with its query string, and the host. It is on by default, and only applies while some taint source is on, since without one there are no events to describe.
Query values whose names look like secrets (containing TOKEN, KEY, PASS, SECRET, AUTH, SESSION and similar) are replaced with [masked] on the machine and again when the event arrives, the same rule as the command line.
The event detail lists each request under Web request, with a Block this client button. One event keeps up to 500 distinct requests; after that it keeps counting the ones it has.
Routes
With Routes on, es-python keeps a list of the routes your application answered, how many requests each one had, and how many of its answers were errors, and sends it to EyalSec once a minute. The Web protection section of the Filters page lists them, with the number of events that happened on each.
The list shows what is reachable, which documentation does not always say: a route that answers traffic and has never been tested, or one that fails with 5xx far more than the rest. Variable parts of a path are folded, so /users/42 and /users/43 are one route, /users/{n}; FastAPI and Starlette routes use their own pattern, such as /items/{item_id}. Each application process keeps up to 500 routes and each machine up to 1,000.
Limits
Web protection works inside your application, which brings one strength and some limits. The strength is that it sees the request your code actually received, after your proxies have done their work. The limits:
- It protects only applications running under es-python on the frameworks above, and only traffic that reaches them. Floods that exhaust your network or your server before a request arrives need protection in front of the application.
- Rate limits count per process, as described above.
- The reflected input and open redirect checks read query parameters. A form body is not inspected.
- The response checks read the first 256 KB of each response.
- Django's test client calls Django's request handling directly, so these checks do not run under it.