The Missing Row: Why We Are Publishing Our AI Security Selection Criteria

It would be easy to claim this as a procurement failure. It was the opposite. The organization did what every capable IT buyer does.

Generative AI had already moved into production across the business and leadership wanted governance in place, but the window was short.

When the business adopts AI in weeks, security does not get quarters to catch up.

They built a requirements matrix, evaluated the field and selected a platform marketed as comprehensive AI security and governance. The evaluation was documented. The scoring was defensible. Every requirement on the matrix was met.

However, the failure sat underneath the matrix. The rows in it were assembled from vendor materials, datasheets, capability briefs, demonstration scripts and a vendor-licenced analyst report.

Vendors describe what they build. A capability the vendor lacks does not appear in the vendor's materials, which is why it never becomes a row.

Due to this, it is never scored against, making it invisible in the decision matrix. The vendor development pace is moving so fast that analyst reports can’t keep up with it. The missing control had no row to be missing from.

The bottom line is that buyers can't see what an AI security product doesn't do.

That's not a vendor trick. Every vendor naturally describes the market in terms of what its own product covers. The gaps only become visible after the purchase, when the money is spent and the architecture is locked. Published, neutral criteria move that solution discovery to the one point where it's cheap, but it mostly happens before the decision.

Ask a simple question of the last AI security evaluation your team ran. Where did the rows come from?

Let’s take a look at the sources that are used to evaluate AI security solutions.

If the answer is vendor documentation, your evaluation measured how well each vendor performs on the ground the vendors chose.

It measured nothing about the ground they avoided. This holds even when the process is rigorous and it holds hardest for the most diligent teams, because a thorough evaluation of an incomplete map produces the most confident wrong answer available.

Analyst research is a better answer and it is still not complete. The analysts see the same problem. Gartner named AI security platforms one of its top strategic technology trends for 2026 and observed that most vendors in this young market cover only part of the surface.

At the time of writing, there was no Magic Quadrant or Wave for AI security. Where AI security does appear in analyst scoring is as a criterion inside other categories: SASE, endpoint, XDR, identity, data security, etc.

Each report scores AI capability only within its own category boundary, so you can read all of them and still not lay the market on one page. And when the comparative reports arrive, they will rank vendors within a defined market.

Vendors below the inclusion bar will be absent, not scored at zero. The empty cells still won’t appear on the page.

The fix is a category set that exists independently of any vendor.

We built one for our own use: 12 capability domains for securing AI systems, validated against the published work of several industrial bodies.

This includes the Open Worldwide Application Security Project (OWASP), the National Institute of Standards and Technology (NIST) and Canadian regulatory instruments such as Quebec Law 25 and the federal privacy regime. The domains were derived from what the frameworks say can go wrong. Vendor coverage had no vote in what the categories are.

A fixed category set changes the mechanics of an evaluation. Every product gets a rating in every domain, including the domains where the honest rating is that nothing is there.

Therefore, all the empty cells appear on the page. In our own scoring across the market, no single vendor has earned a native rating in all twelve domains.

The strongest platforms in the field carry gaps and the gaps cluster in the domains vendors talk about least.

Neutral categories are necessary. They are not sufficient and we know because our own instrument failed on its first outing.

During an internal review, one of our security solutions architects challenged the score we had assigned to a major platform vendor. We had rated it at eight of 12 domains.

He pushed on specific cells. We went back to the primary documentation, product by product, release note by release note and re-scored the platform.

The vendor came out at 11 of 12. Five domain ratings changed, every one of them upward, and the original number had already been used in customer conversations before we caught it.

That correction taught us the operating cost of this work. General marketing research understates capability, acquisitions close monthly and integration lags the press release. A score is a photograph of a moving object. Our comparative ratings now expire every quarter and must be re-earned against fresh evidence, which means the assessment is never finished.

We accepted that cost deliberately, because the alternative is a confident document that ages in silence, and we had already seen what confident and wrong produces.

That is also why we stopped waiting on annual industry reports to anchor these decisions. An annual cadence was built for markets that move annually. This one does not.

The criteria, the method and an open floor, in sequence.

First, the criteria themselves: The 12 domains, the capability metric a product must clear in each and the evaluation questions to put to any vendor. These publish in full, in a whitepaper that documents the framework mappings behind every domain.

Second, the method: The scoring scale, the evidence standard and the refresh cadence, packaged so your team can run the assessment without us. The working files go to a public repository, free to download and use.

Third, an open floor: We are inviting vendors, including our closest partners, to respond in writing against the criteria, stating what they cover, what they partially cover and what they do not. We will publish those responses. A vendor willing to name its own empty cells is telling you something worth knowing and so is a vendor that declines.

First: We are practitioners before we are a reseller and practitioners know where good security comes from.

The best work in this field has never been secret sauce. It is OWASP, ATT&CK, the CVE program, open tooling maintained on nights and weekends by people who decided defenders should not start from zero. We built our careers on work other people published for free. This is a small payment on that debt.

We are not neutral about this market. We are neutral within it. CDW sells across most of the vendors these criteria score, so we have no incentive to bend the map toward any one of them. When a client lands on the right platform, we win either way. When they land on the wrong one, we inherit the escalation.

We are making the criteria available because the document is not the product. Our business is what happens after the scoring: deciding which gaps to accept, which to close, in what order and with what architecture. That is roadmap work. We would rather earn it by showing how we think than by claiming completeness in a datasheet.

Published criteria give vendors something concrete to respond to, buyers a shared language for gaps and the industry a public record of who covers what.

A public map on a quarterly refresh also means our ratings face the same scrutiny we are asking you to apply to vendors. We have corrected ourselves once in this document already. We expect to do it again.

That includes our competitors. If you compete with us and you can improve this map, reach out. We will take the correction and credit it. If you want to run these criteria with your own clients, run them. The point was never that the map is ours. The point is that buyers finally have one.

Take the criteria and run them against your incumbent platform and anything on your AI security roadmap. Score the domains, including the ones you don’t hear about.

Where you find an empty cell, decide on purpose whether to accept the gap, close it with another product or change course. Any of those three is a defencible position.

Over the next two years, Canadian organizations and their security teams will be judged on how deliberately they governed the AI they adopted and the judgment will come from regulators, insurers and boards at the same time. The organizations that hold up will be the ones that knew where their gaps were before someone else found them.

The criteria are published here and ready for you to use.