How to run a proof of concept for a crisis simulation or cyber range platform

6-1Why a proof of concept and not a demo?

A demo shows the product at its best, with a presenter who knows the script. A proof of concept shows how it behaves with your people, your scenario and your constraints. In this category that difference is large, because the thing you are buying is how the exercise reacts to real decisions and what evidence it leaves behind. Our buyer's checklist lists the questions to ask first; this post covers the test.

6-2Step 1: write the success criteria

Three criteria, agreed with the executive sponsor before any vendor is contacted. Tie each to the gap you are buying for. Examples:

• The exercise includes the incident commander, legal, communications and one executive in the same session.
• The scenario is built from our environment and takes no more than an agreed number of staff hours to prepare.
• The report grades our objectives and ties each gap to a plan section, an owner and a date.

For a range, swap in technical criteria: which of your tools the exercise runs on, and whether it measures detection and containment times.

6-3Step 2: pick one real scenario

Use a scenario your organization could plausibly face this year, and give every vendor the same one. Our 2026 scenarios article has five tied to current attack data. Write two or three objectives for it, as in the lesson on writing exercise objectives, so each vendor's exercise can be graded the same way.

6-4Step 3: use the same people

The same players, or the same roles, for every vendor. If one vendor's session has the general counsel and another's does not, you are comparing attendance, not platforms. Book the sessions within a few weeks of each other so the team's knowledge does not change much between them, and vary the order if you can so no vendor always goes first.

Before the sessions, ask each vendor what data the platform uses to build the scenario and where it is kept. Some build from public information about your organization, and some can add your telemetry; either way your security and privacy teams should approve it first.

6-5Step 4: measure the same things

• Preparation: staff hours from kickoff to a runnable exercise, including your people's time, not only the vendor's.
• Adaptation: at least once per session, have a player make a decision the scenario did not expect, and record what happens.
• Participation: who took part, and whether any absent role could be covered.
• Evidence: run the report through the five checks in how to read an after-action report.
• Player view: a short feedback form after each session. CISA's Tabletop Exercise Packages include one.

Hold a short hotwash after each session, as in the lesson on running a hotwash, and keep the notes with the vendor's report.

6-6Step 5: decide with price in hand

Most vendors in this category publish no price. Ask for a written quote before the proof of concept, scoped to the audiences and the number of exercises a year you expect, so the decision does not wait on a commercial process afterwards. Our pricing article lists what is published.

Then score each vendor against the three success criteria, with the measures from step 4 as evidence. If you want a second view, re-weight our criteria in the calculator to match your brief and see whether the shortlist order changes. Pick the vendor whose exercise produced the report you would be willing to show your board.

Sources

1. CISA Tabletop Exercise Packages: https://www.cisa.gov/resources-tools/services/cisa-tabletop-exercise-packages
2. NIST SP 800-84: https://csrc.nist.gov/pubs/sp/800/84/final