Running your pytest suite under the EyalSec Python runtime: what it reports and how to read an event

To run a pytest suite under the EyalSec Python runtime you change one word: es-python -m pytest instead of python -m pytest. The tests run as before, and every time data from a source you switched on reaches a risky operation (a query, a command, a file path, a deserializer), the run reports an event with the value, where it entered and the call path. This tutorial walks through the commands, which sources make sense for a test suite, how to mark test input as untrusted, how to read each block of an event, and how to make CI act on the result.

Your test suite already describes how your application is meant to be used. Running it under es-python reuses that work: each test becomes a small dataflow experiment.

Before you start

You need a machine with es-python installed and reporting to your dashboard. The quick start covers the account, the machine and the one-line installer; Install es-python has the details. Confirm the install with the helper the installer adds:

eyalsec selftest
es-python -V
es-python --name

eyalsec selftest sends a harmless test event and prints a link to it. es-python --name prints the account and machine this runtime reports to (Running es-python).

es-python uses the packages installed for your regular Python of the same version, so install your project's dependencies and pytest the way you normally do. To keep things isolated, create the virtual environment with es-python itself:

es-python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt pytest
python -m pytest        # inside this venv, "python" is es-python

Run the suite

es-python -m pytest

Every pytest option works unchanged, because EyalSec options go before -m and everything after belongs to pytest (es-python command line):

es-python --report-code -m pytest tests/api -x -q
#         ^ EyalSec option  ^ pytest and its own arguments

Nothing is reported unless at least one source is on. A source is a place where untrusted data enters the program. A machine you add starts with socket (network input) and file (files read) on; the rest are off until you switch them on in the machine's Configure window, or for one run with a flag or an environment variable. A dashboard On or Off always wins over a flag; the flag decides only while the control is unset.

Choose sources that fit a test suite

Tests receive input differently from production, so the right sources differ too. The full list is on Taint sources; these are the ones that matter under pytest.

Environment variables are the convenient form under pytest, because child processes inherit them:

ES2_MAKE_FILE_VULN=1 ES2_MAKE_STDIN_VULN=1 es-python -m pytest

One point saves an afternoon of confusion. The socket source marks data returned by the socket receive calls. An in-process test client (Flask's app.test_client(), Django's Client, Starlette's TestClient) calls your application directly and reads no socket, so with only socket on, expect those tests to have nothing marked. If such a suite reports nothing, check this first. Two ways to give such a suite untrusted input follow.

Mark test input as untrusted with make_vuln

make_vuln() is a builtin that es-python adds: it marks the value you pass as untrusted and returns it. Set the make_vuln control to On for the machine; while it is unset or off, the call returns its argument unchanged. Under regular Python the name does not exist, so wrap it in a fixture that degrades to a no-op and the same tests run in both:

import builtins

import pytest


@pytest.fixture
def untrusted():
    """Return a function that marks a value as untrusted under es-python."""
    return getattr(builtins, "make_vuln", lambda value: value)

A test then says which of its inputs an attacker would control:

def test_search_handles_hostile_term(client, untrusted):
    term = untrusted("report' OR '1'='1")
    response = client.get("/search", query_string={"q": term})
    assert response.status_code == 200

The assertion is the one you already had. What changes is that the term is now followed: if any code path concatenates it into SQL text, opens it as a path or hands it to a shell, the run reports that flow, whether or not the test's own assertion noticed.

The second way is to test over a real connection: pytest-django's live_server fixture and similar helpers start a listening server, so the request bytes arrive through a socket and the socket source marks them.

What the run reports

In Report mode the program is never interrupted, and pytest prints its usual summary. The findings are on the dashboard's Events page. Each row is one distinct finding, not one occurrence: when ten tests reach the same flow, one row shows a count of ten (Events).

On es-python the row reads as a sentence, with the technical form underneath:

Data from a network connection reached a system command
socket → os system

The small line is the source (socket), an arrow, and the sink label (os system). Beside it are the impact tags, such as sqli or command injection, which name the class of vulnerability, and the severity, which EyalSec works out from how easily an attacker controls the source and how much damage the sink can do.

To see only what this test run found, click Runs in the toolbar and pick the run: every start of a program under es-python is one run with its own id, and a pytest worker pool belongs to the same run as the command that started it (Runs). While the list is narrowed to one run, each row's count is that run's own occurrences.

How to read an event

Click a row to open it. The blocks appear in this order on an es-python event, and a block is shown only when the event has something to put in it (Event detail).

A practical reading order for a test-suite event:

1. What happened names the call, for example os.system(). Search your code for it.
2. Call path shows which test reached it and through which functions. Under pytest the outer frames are pytest's own; the frames from your package are the ones to read.
3. Compare the two Value blocks. If the value at the sink is the test's input wrapped in SQL or shell syntax, the code built a statement from it. If the two are identical, the value was passed through untouched.
4. Was it made safe? tells you whether EyalSec could prove the value harmless at that operation. An event proven safe is suppressed: stored, and out of the default view.

The two code blocks need Source code switched on, with --report-code, ES2_REPORT_CODE=1 or the dashboard setting. It is off by default because it sends lines of your source; for a test suite on your own code it is usually worth turning on, since each event then shows the line that received the data and the line that used it, with five lines either side (Event context).

An event is a real flow that happened on that test's input. It is not, by itself, proof of an exploitable bug: a fixture file your own repository wrote is a trusted origin, and the event's severity and the Why this is risky sentence say so. Read the origin before you read the severity.

Decide what to do with each finding

Three outcomes cover almost every event from a test run:

• Fix the code. The What to do line names the safe API: a bound parameter, an argument list in place of a shell string, a resolved path checked against its base directory.
• Record that the flow is expected. A tool whose job is to run a command the operator typed will reach os system by design. Add a Hide rule for that sink on that machine, so the next run shows only what is new.
• Keep it as a regression signal. Leave it visible until it is fixed.

Gate a CI build on the run

For a pipeline, EyalSec provides a script that installs es-python on the runner with your API key, runs your command, waits for the events and exits non-zero when the run reported anything at or above a severity you choose. The GitHub Actions step, from Run es-python in CI:

- name: Tests under es-python
  env:
    EYALSEC_API_KEY: ${{ secrets.EYALSEC_API_KEY }}
  run: |
    curl -fsSL https://eyalsec.com/ci.sh -o es-python-ci.sh
    sh es-python-ci.sh run --machine ci-${{ github.event.repository.name }} \
      --distro ubuntu_24_04 --arch x86_64 \
      -- es-python -m pytest

The script switches on socket and file for the command it runs; --sources socket,file,env chooses others. --fail-on sets the threshold (high is the default), and --new-only reports only findings no earlier run had reached, which is the gentlest way to adopt it on a codebase with history. Exit code 0 means nothing at or above the threshold, 3 means findings, 4 means the script could not do its job, and any other code is your own command's, which always wins. The summary names each finding's sink label and event number and never prints the value, because CI logs are often public.

Give each pipeline its own machine name and run one job at a time per name (caveats). CI findings count toward your event quota; see pricing.

What to expect, honestly

• Coverage is your tests' coverage. A flow no test exercises produces no event. A run with no events means no watched data reached a risky operation on these inputs, nothing more.
• Sources cost time. Each source adds work, and file on a suite that reads many files adds the most. Measure your own suite before switching everything on.
• Dataflow only. A missing permission check or a missing CSRF token is not a flow of untrusted data into an operation, so it is outside what this reports.

If you would like to see your own suite's first run read through with you, you can book a live demo.