The AI vendor checklist
Ten questions to ask any AI vendor before you sign. Each one has a good answer and a weak answer. Bring the list to every demo, including ours.
Every vendor now sells AI. The demos look alike. These ten questions separate a vendor that can prove an outcome from a vendor that can only show activity. Ask them in order. Write the answers down.
Evaluating the whole platform, not just the AI?
Institutional Success Platform Selection Checklist. Start with whether you are solving the right problem, then avoid five common buying traps, evaluate the platform, run a quick test on its AI, and evaluate the partner, not just the product. Read it on the page →
1. What is your model trained on: our students, or the sector?
Why it matters. A model built on national benchmarks or one vendor’s pooled customers knows what a typical student does. It does not know what your students do. Risk at an open-access commuter college looks nothing like risk at a residential university.
A good answer. “On your student information, learning management, and CRM history. We retrain it every term.”
A weak answer. “On millions of student journeys across our customer base.” That is a sector model with your logo on it.
2. How do you find a student at risk: rules, or a model?
Why it matters. A rule is a threshold somebody typed in: GPA under 2.0, a missed registration, an unopened email. A rule finds the students you already knew about. A model finds the student with a 3.2 GPA who dropped from fifteen credits to nine and stopped logging in to the LMS.
A good answer. “A model. Here is the list of signals it uses, and here is how the weights differ at your institution.”
A weak answer. “Our agent continuously scans records for risk signals.” Ask what the signals are. If the list is short and obvious, it is rules.
3. Show me a lift number against a comparison group.
Why it matters. A retention gain has to be measured against students who did not get the intervention but looked the same. Without a comparison group, a vendor takes credit for a good year.
A good answer. A persistence or completion number, the size of the comparison group, and a confidence interval. Something like “plus 3.1 points, n = 412, 95% interval 1.4 to 4.8”.
A weak answer. “Our partners report their agent exceeded expectations.” Or “months is the average time to see results.” Neither is an outcome.
4. Name an intervention of yours that did not work.
Why it matters. Some interventions have no effect. A vendor that measures honestly has found a few. A vendor that has never found one is not measuring.
A good answer. A named program, the number, and what the institution did next. Usually: moved the budget to something that did work.
A weak answer. Silence, or “every deployment shows value.”
5. What does one point of retention pay back at an institution our size?
Why it matters. The business case for AI is retention. If the vendor cannot turn a point into dollars, the vendor cannot tell you whether the contract paid for itself.
A good answer. Your headcount, your net price, one point, and a dollar figure you can check. Our calculator does this from public IPEDS data in ten seconds. Ask them to do the same on a whiteboard.
A weak answer. “Staff capacity, operational efficiency, and outcomes tied to your specific goals.”
6. Which of our systems do you replace, and which do you read from?
Why it matters. “Operating system”, “ontology”, and “unified data layer” are replacement language. A replacement is a two-year project before anyone sees a prediction.
A good answer. “None. We read from Banner, PeopleSoft, Workday, Slate, Canvas, Salesforce, and whatever else you run. Nothing gets replaced.”
A weak answer. “We unify your data into a single platform that every office builds on.”
7. How long until an advisor sees a prediction on a real student? Name an institution that hit that date.
Why it matters. “Weeks, not years” is easy to say. A named institution with a date is hard to fake.
A good answer. A number of weeks and a name. Lawrence Technological University was live in six weeks.
A weak answer. “It depends on your data readiness.”
8. How often is the model refreshed?
Why it matters. A model trained once at install decays. Your students change, your courses change, your aid policy changes. By year three the model describes an institution that no longer exists.
A good answer. “Every term, on your latest data, and here is how we report the change.”
A weak answer. “Our models are continuously improved across all customers.” That is the sector model again.
9. How many institutions run this, and how many published outcome numbers do you have? Can we call three?
Why it matters. “Hundreds of partners” and “millions of conversations” are volume. A published outcome with an institution’s name on it is proof. A reference call is proof you can check.
A good answer. A link and three names. We publish 31 measured outcomes, each with the number and how it was measured, and we will set up the calls.
A weak answer. A logo wall with no numbers under it.
10. What is your status under FERPA, and where is the data sharing agreement?
Why it matters. Your general counsel will ask. A vendor that has done this before has the answer and the document ready.
A good answer. “A school official you designate under FERPA. Here is the data sharing agreement, the platform specifications, and the standard data specifications.”
A weak answer. A trust center badge and a promise to follow up.
What a good vendor does with this list
Answers every question in the room. Shows the lift number with the comparison group on the screen. Names the intervention that did not work. Brings the FERPA paperwork to the first meeting.
Bring the list to our demo. Book one, and bring your questions.
Bring your questions. Leave with an estimate you can defend.
A 45-minute working session: we start from your public IPEDS figures, show what the platform found at institutions like yours, and size the outcome — before anyone touches your data.