Vendors demo the path that works. These questions cover the paths that do not.
About failure
- What happens when your API times out mid process?
- Who gets alerted when something breaks, and what does the alert contain?
- What happens when the model returns a well formed answer that is wrong?
- Can I see the error rate for accounts like mine, not the aggregate?
- What is the longest outage a customer has had, and what did they do during it?
Error handling is treated as a first class concern by the platforms these products are built on, for a good reason: what has no error route fails in silence.
About your systems
- Exactly which fields do you write back to my CRM?
- Can a rep see what your product did without logging into your product?
- What happens to the data if we leave?
- Do you write outcomes, or only activity?
- How do you handle records with missing or bad data?
Question 7 matters more than it looks. If the outcome lives only inside the vendor's product, your team will work around it and you will pay for a system nobody uses.
About scale
- What changes between location one and location two? Configuration or rebuilding?
- Do you have a customer at our number of units, and can I speak to them?
- What percentage of your customers run this without customisation?
- How long from signature to first production result, for a customer like us?
- What blocks launches most often, in your experience?
For anything involving messaging in the United States, add: what is required for carrier registration, and who does it? Application to person messaging requires brand and campaign registration, and an unapproved registration blocks texting entirely.
About accountability
- Are you accountable for the software working, or for my outcome?
- Who owns this at my company when your implementation ends?
- What does your implementation cover, and where exactly does it stop?
- What do your customers usually have to build themselves?
- What would make you tell us we are not a good fit?
Question 20 is worth the whole call. Vendors who can answer it have seen failures and learned from them. Vendors who say everyone is a fit are describing their sales process, not their product.
How to read the answers
Specific beats confident. A vendor who says the second location takes about a day of configuration and here is the config file is telling you something checkable. A vendor who says it scales effortlessly is telling you nothing.
Watch for the boundary. Every product has one, and the work that blocks rollouts usually lives just outside it: your CRM, your routing, your data, your compliance. Knowing where their boundary sits tells you what you still need to own.
If the work outside the boundary is the problem
That is the job of a forward deployed engineer. MitHub publishes open profiles of people who have done exactly that, with the deployments written out.