Why public detail varies
Agent security vendors range from companies inside large security platforms to start-ups that left stealth this year. The younger ones often have a strong product and a thin website. On the pages we read, Bay's public site had two blog posts, no documentation portal and no published pricing, and it scores lowest of the nine on our public proof criterion. Bloom Security published limited company information and no pricing; its seed round was reported by trade press. None of the nine vendors publishes list prices on its own website; Harmonic Security publishes tier names, and Noma Security and Lasso Security list annual contracts on AWS Marketplace.
Our scores can only reflect what is public. Our limitations note says so: a capability a vendor has but does not describe publicly scores lower than it might. The evaluation is where you close that gap.
The request list
Send this list before the first technical session, in writing.
- Architecture. What runs on the device, for how long and with what privileges; what runs in the vendor's cloud; where decisions are made.
- Data handling. What leaves the device: prompts, file contents, command lines, metadata. Where it is stored, for how long, and in which region.
- Attestations. The current SOC 2 report and ISO 27001 certificate if the vendor lists them, with scope and dates. Noma Security and Onyx Security list attestations on their pages. Ask for the reports, not the logos.
- Supported agents. The current list of agents and versions, and for each one whether the product sees, governs or decides on its actions.
- Verdicts. Which verdicts are available on which agents, and what the user sees for each.
- Failure behaviour. What happens when the component is missing, crashes or cannot reach its policy: does the agent run uncontrolled, or stop?
- Vendor-stated figures. The method behind any figure on the website, such as deployment time, decision latency or false positive rate.
- Pricing. The unit (devices, users, agents), the minimum, and what is in each tier.
- References. Two customers of similar size who run the same agents you do.
How to read the answers
A vendor that answers the list quickly and specifically is usually ready for production use. Vague answers on failure behaviour or data handling are the ones to push on. Write the answers into your own comparison so you can hold every shortlisted vendor to the same list.
Vendor-stated figures
Our pages label vendor figures as vendor-stated, and so should your evaluation notes. A decision time or false positive rate measured on the vendor's test set may not hold on your agents and your policies. Measure the two or three that matter during the pilot.