Your app security pipeline can pass every check and still leak a customer record through a test fixture, scanner request, or downloadable report. I would draw the data boundary before adding another scanner, because a green CI badge says nothing about who received the data used to produce it. That is an uncomfortable priority when you have been asked to ship tests quickly.
Your test fixture can become the first privacy breach
A staging database often looks safe because it is not production. That assumption fails when staging contains a restored production snapshot: a test that reads one account can copy its email address into a CI log, a screenshot, or a scanner finding. Before you run a security job, trace one fixture from its source through the runner, test output, uploaded artifacts, and anyone who can download them.
I would use the pipeline walkthrough, Build Your First App Security Test Pipeline From Scratch, to wire jobs after drawing that boundary. The order matters because a working upload step can preserve sensitive evidence long after the test itself has finished. Mark each input as synthetic, derived from real data, or unknown; treat “unknown” as real until someone can show how it was generated.
Synthetic does not mean typing a plausible person’s details into a JSON file. Reserve names and addresses deliberately: example.invalid is reserved for examples under RFC 2606, so it cannot accidentally deliver a test email to someone who owns a real domain. For authentication tests, issue credentials for disposable accounts through OAuth 2.0 or OpenID Connect rather than reusing a developer’s session cookie, because a copied personal cookie gives the runner that developer’s access.
This Python 3 check runs locally or in CI. It is a small guard for a deliberately synthetic fixture, not proof that every field in the repository is safe:
import json
from pathlib import Path
from tempfile import TemporaryDirectory
with TemporaryDirectory() as directory:
path = Path(directory) / "user.json"
path.write_text(json.dumps({
"email": "tester@example.invalid",
"token": "fixture-only-token"
}))
user = json.loads(path.read_text())
assert user["email"].endswith("@example.invalid")
assert user["token"] == "fixture-only-token"
For a team setting, start with 2 disposable test identities—one ordinary user and one administrator—and adjust that count when a permission boundary needs another role. Give those identities only synthetic records, because a low-privilege account can still expose real data through search results or error messages. Run Gitleaks on committed fixtures and Semgrep on code paths that log request bodies, but review their matches: neither tool can establish that an apparently harmless customer ID was copied from production.
I would not copy a production database into CI and promise to mask it later, because backups, job logs, and failed intermediate steps may retain the unmasked version. Generate the smallest records that exercise the behavior instead. OWASP ASVS 4.0.3 can help you name the authorization behavior you are testing; it does not require genuine customer data to test it.
A scanner’s network reach is a security decision
Dynamic testing sends requests, including malformed ones, to a running app. Against the wrong target, OWASP ZAP can trigger password-reset emails, change records, or put attack strings into audit logs; those effects follow from exercising real endpoints, not from the scanner’s label. Ask which hostname the runner can resolve, which credentials it receives, and whether its route to production is blocked outside the test script.
A hostname check inside a shell script is a weak last line of defense because a later edit can remove it. Prefer a separate nonproduction environment with network rules that deny production destinations, plus an allowlist in the scanner configuration. Set an initial request ceiling of 20 requests per minute at the test environment’s ingress and tune it after checking service logs, because a single scanner can compete with teammates using the same staging deployment. That limit is an operating choice, not a safe rate promised by ZAP.
Authentication makes the boundary harder. A scanner with an administrator token can reach more useful pages, but it can also perform more damaging writes if its route is wrong. Start an unauthenticated pass, then use a short-lived, scope-limited token for authenticated checks. With GitHub Actions, grant a job permissions: contents: read unless it specifically needs more; use id-token: write only for a job that obtains a short-lived credential through OIDC. Those settings reduce the privileges available if a dependency or test script is compromised.
Do not assume a “read-only” API test stays read-only because the HTTP method says GET. An endpoint may send mail, record a login, or increment a counter when fetched, so inspect its behavior before adding it to a crawl. Conversely, excluding every authenticated route makes the scan less useful because authorization failures often appear only after login. A practical compromise is to crawl a dedicated test tenant, exclude destructive paths explicitly, and review the resulting request list before expanding coverage.
Keep passive checks distinct from active attacks. ZAP Baseline is a reasonable first pass when you need a quick crawl without an active scan, but its requests can still cause side effects because crawling calls application endpoints. If the app includes a mobile client, OWASP MASVS v2 may guide client-side test cases, while the server still needs its own authorization checks; a mobile checklist cannot tell you whether the API accepted another user’s object ID.
Local and hosted scanners trade maintenance for disclosure
Consider ZAP Baseline on a self-managed runner against Burp Suite DAST as a hosted service. ZAP wins when policy forbids sending targets and findings to a third party, because you control the runner, traffic route, and output storage; it costs runner capacity, updates, and time spent validating noisy results. Burp Suite DAST wins when a team can approve that external processing and needs managed scan scheduling, because less scanner infrastructure must be maintained; it costs a paid subscription and a vendor review of what requests, credentials, and evidence leave your environment.
That is an explicit privacy comparison, not a claim that local is automatically safe. A self-managed runner with a public artifact bucket can disclose more than a properly configured hosted service, because storage access and retention matter as much as where the scan executes. I would put the buying guide, Cheap vs Proper App Security Testing What a CTO Should Buy or Build, beside a data-flow questionnaire rather than let scanner price decide.
Ask where the scanner runs, whether it retains captured responses, who can open its findings, and how deletion is verified. A finding may include a full URL, an authorization header, or part of a response body, because scanners need evidence to explain what triggered an alert. Before granting a hosted scanner access, create a test account with no production data, check its published retention terms, and run a controlled scan that plants a recognizable synthetic marker; search the downloaded report for that marker to see what evidence is copied.
The same review applies to dependency services. Trivy can scan dependencies in your own runner, while a SaaS dependency platform may upload manifests or project metadata; choose based on the data you are allowed to share, not a blanket belief that package names are harmless. A private dependency name can disclose an unreleased service or internal architecture because it describes what the team is building.
For a first trial, allow 1 nonproduction target rather than a wildcard domain, then expand the scope only after reviewing its captured requests. That number is a starting constraint to revise, not a coverage metric. Record the approved target and scanner identity in the repository so the next developer does not have to infer permission from an old CI job.
Findings need an expiry date and an owner
A scanner report is a new dataset with its own access problem. SARIF 2.1.0 makes findings portable between tools and code-hosting interfaces, but portability also makes it easy to copy sensitive file paths, URLs, and evidence into another system. Decide where a finding should live before enabling every export format.
GitHub documents 90 days as the default retention period for Actions artifacts and logs, although repository or organization settings can change it. That vendor-published default is a reason to inspect your settings, not a sensible automatic lifetime for reports containing tokens or response fragments. I would set an initial artifact retention of 7 days for a test pipeline and revisit it with whoever investigates findings, because evidence that disappears before review is useless while evidence kept indefinitely widens access.
Redaction needs testing at the point of capture. Masking a secret in one console line does not remove it from a ZAP HTML report, a SARIF attachment, or an OpenTelemetry trace emitted by the app. Place a unique synthetic token in a test request and search every output destination for its exact value; if you find it, fix the producer or remove that output rather than relying on a downstream regex that may miss another format.
Give reports a narrow audience and a named deletion path. For object storage, S3 server-side encryption with AWS KMS protects stored bytes under configured access controls, but it does not stop an authorized viewer from downloading a report; check bucket policy, IAM permissions, and access records such as AWS CloudTrail as well. Keep the raw scanner output available only to the people investigating it, and put a short, redacted finding in the developer ticket.
Do not delete evidence merely to make a dashboard look clean. Preserve enough detail to reproduce a real issue, because a finding without a request path, test identity, and expected result is difficult to fix. Preserve it in the smallest approved place for the shortest workable period, and make one person responsible for closing or renewing that exception.
Delete one artifact before adding another test
On your next CI run, open one downloadable security artifact and search it for an email address, token, and response body from a synthetic test account. If any appears, stop uploading that artifact until you can remove or restrict the evidence, because another scanner will multiply the copies. Then write down its owner and expiry setting before you add the next check.
