Practical guide · Harness Engineering
AI Agent Security Checklist: 10 Harness Engineering Checks Before Production
Before an AI coding agent receives shell, repository, cloud or deployment access, test the system around the model. A new empirical study of 3,171 public repositories found confirmed security defects in 16.0% of assembled agent setups. This checklist turns those measured failure classes into ten release gates.
Evidence classification: CONFIRMED: the study measured configuration artifacts, not runtime compromises. It confirmed file-level defects through independent re-derivation and review. NOT CONFIRMED: credential exfiltration or exploitation of every affected repository. ANALYSIS: the ten controls below extend the measured findings into a production threat model.
What the study measured
Across 2,660 assembled setups, 9.8% used an unpinned MCP package, 3.1% granted arbitrary execution behind scoped-looking syntax, and 3.8% installed a skill that pre-approved shell access. The union was 16.0%, not the sum, because repositories can contain more than one defect. The paper explicitly reports no confirmed credential-exfiltration path.
The 10 production checks
- 01
Pin every MCP dependency
Test: Reject npx, uvx and container declarations without an exact version, digest or lockfile-backed resolution.
Evidence: The study confirmed unpinned MCP packages in 9.8% of 2,660 assembled setups.
- 02
Expand permission patterns before approval
Test: Resolve what patterns such as Bash(python:*) can actually execute; do not trust a scoped-looking label.
Evidence: 3.1% of setups carried an arbitrary-execution grant disguised as a narrower permission.
- 03
Treat skills as executable dependencies
Test: Review SKILL.md instructions and bundled scripts before installation; record origin, commit and reviewer.
Evidence: 3.8% of setups and 3.7% of skill collections contained a skill that pre-approved shell access.
- 04
Separate model intent from tool authority
Test: Let the model propose; require deterministic policy to authorize filesystem, network, secrets and deployment actions.
Evidence: A prompt is advisory. The harness and tool layer decide what can actually execute.
- 05
Sandbox the blast radius
Test: Run agents with isolated workspaces, short-lived credentials, destination allowlists and resource limits.
Evidence: Installed harness components run with developer privileges unless the execution environment narrows them.
- 06
Bind approval to the exact side effect
Test: Show command, arguments, destination, files and expected mutation; invalidate approval when any field changes.
Evidence: Broad approvals convert a one-time human decision into reusable authority.
- 07
Isolate memory and secrets
Test: Prevent untrusted skills and subagents from inheriting credentials or unrelated project memory by default.
Evidence: Composition creates paths that no individual configuration file reveals.
- 08
Log proposal, decision and result
Test: Keep tamper-resistant records of the requested action, policy version, approval, tool response and resulting diff.
Evidence: Auditing configuration alone cannot prove what the runtime later executed.
- 09
Make retries idempotent and reversible
Test: Use idempotency keys, checkpoints and tested rollback for external writes, releases and infrastructure changes.
Evidence: Recovery can repeat an already completed side effect when state is ambiguous.
- 10
Gate the assembled harness in CI
Test: Scan context files, skills, hooks, MCP declarations and subagents together; fail builds on validated high-confidence defects.
Evidence: 16.0% of studied setups had at least one confirmed security defect; raw scanner output overestimated risk, so findings need validation.
Threat path: from installation to side effect
A secure harness places independent checks between every step. Reviewing the model alone cannot detect a package that changes later, a skill that carries shell permission, or an approval that authorizes more than the user saw.
Primary evidence and reproducibility
FAQ
What is an AI agent security checklist?
It is a set of verifiable release gates for the harness around an agent: dependencies, tools, permissions, sandbox, memory, approvals, logs and recovery.
Why is MCP version pinning important?
An unpinned launcher can fetch and execute a different package version on a later run. Exact versions, digests and lockfiles reduce that supply-chain drift.
Are AI agent skills just documentation?
No. Skills contain instructions and may bundle scripts or request tools. A privileged agent can turn those instructions into real execution.
Does passing this checklist prove an agent is safe?
No. It reduces known configuration risks. Runtime red-teaming, monitoring and incident response remain necessary.