Choosing a cognitive assessment
How to evaluate a cognitive assessment: four questions every vendor should answer
You're building a shortlist, and every vendor will claim to be validated. Here is a fair, defensible test to apply to all of them, including us.
The fair test
The four criteria for judging any cognitive assessment
Any vendor can copy the screens. None can copy the evidence beneath them, which is exactly why these four questions are the right test. Knowing how to evaluate a cognitive assessment comes down to asking them of every vendor on your shortlist, in order.
Is it standardised on a large sample that matches the people you'll use it on?
A cognitive assessment is only as sound as the population it was normed against. Ask for the size and make-up of the normative sample, and whether it is refreshed as the population changes.
A large, population-matched sample the vendor can state on the spot, refreshed as the population changes.Do the indexes that drive decisions measure consistently?
Reliability tells you whether a score would hold if the same learner sat the assessment again. Ask for the reliability coefficients on the indexes that actually drive decisions, not a blanket "it's validated".
Reliability coefficients stated for the decision-driving indexes, at a recognised high-stakes standard.Is there evidence it measures what it claims, honestly qualified?
Validity is where marketing and evidence part company. Ask for the theoretical model, whether analysis confirms it, and which strands of evidence are peer-reviewed and which are not.
A named model confirmed by analysis, plus clinical and predictive evidence at its true strength, including what is not peer-reviewed.Who, independent of the vendor, stands behind it?
Independent oversight is what separates a credible assessment from a self-graded one. Ask who developed it, which independent experts were involved, and what any professional-body review actually covers.
Named independent experts in its development and a recognised professional-body review, described for what it is, not overstated as peer review.
Take this to every vendor
The questions to ask any vendor
| The question to ask | What a strong answer looks like | How to check it yourself | |
|---|---|---|---|
| Normative sample | Is it standardised on a large sample that matches your population, and refreshed for drift? | A specific sample size and make-up, matched to your learners and recently refreshed. | Ask for the technical manual and the norm-sample size and composition. |
| Reliability | Do the indexes that drive decisions measure consistently? | Reliability coefficients stated for the decision-driving indexes, at a high-stakes standard. | Ask for the coefficients, not a vague "it's validated". |
| Validity | Is there evidence it measures and predicts what it claims? | A named model confirmed by analysis, plus clinical and predictive evidence at its true strength. | Ask which validity strands are peer-reviewed and which are not. |
| Independent oversight | Who, independent of the vendor, stands behind it? | Named independent experts in its development and a recognised professional-body review. | Ask who the experts are and what the review actually covers. |
| Administration conditions | Under what conditions was the evidence established, and does the assessment run under those conditions? | The conditions are stated, and the claims are scoped to them. Where more than one mode is offered, an equivalence study covers them. | Ask which devices and settings are supported, then ask for the published equivalence study for each. A standardised score only means something against the conditions it was normed under. |
| Whole-cohort assessment vs a screening or referral-only tool | Is every learner assessed, or only those who are referred or flagged? | A validated assessment that covers the whole cohort, not only a specialist referral or a raised flag. | Ask whether every learner is assessed, or only referrals. |
| End-to-end platform vs point solution | Does it cover assess, plan, do and review with one evidence chain, or just raise a flag? | One connected workflow from assessed need to evidenced impact, inside the tools you already run. | Ask whether it carries the whole graduated approach, or hands the workflow back to you. |
Category question
A screener, or a validated assessment?
A screen is not an assessment, and a flag is not a workflow. A screening-only or special educational needs and disabilities (SEND) tool serves the specialist, or raises a flag and stops. A validated whole-cohort assessment measures every learner and feeds the work that follows.
- A tool that only flags a referred few, by design, never sees learners who don't disclose or aren't referred.
- A whole-cohort assessment amplifies the specialist and covers everyone, rather than replacing them.
- So ask any vendor: is this a validated whole-cohort assessment, or a screener?
Category question
A point solution, or an end-to-end platform?
A validated assessment is only the start. The graduated approach, assess, plan, do and review (APDR), turns a flag into evidenced, deliverable support with one thread from assessed need to evidenced impact. A point solution that covers one stage hands the rest of the workflow, and the evidence chain, back to you.
- Ask any vendor: does it carry the whole graduated approach with one evidence chain, or just one stage?
- Support in a separate, parallel system fragments your data and rarely gets adopted.
- The real question is whether it works inside the tools you already run, or sits beside them.
Our own answers
Put these four questions to Cognassist
We built this test to be used on us too. In short: Cognassist is standardised on 26,301 UK apprentices (ages 14 to 77); the two primary decision indexes, Language & Numbers and Executive Function / Working Memory, meet the British Psychological Society (BPS) high-stakes reliability standard, with the coefficients set out in the technical manual; validity rests on three strands; and an independent Science Advisory Board (Professor John R. Crawford, Dr John Welch, Dr Clive Skilbeck and Professor Chris Petkov) developed it. It has been reviewed by PATOSS (the Professional Association of Teachers of Students with Specific Learning Difficulties) against exam-access requirements, and aligns to the standardised-score conventions accepted by most awarding organisations. Just as important is where Cognassist stops.
- Cognassist is not a diagnosis. It measures the cognitive differences associated with specific learning differences, as a signal for professional judgement, and does not by itself establish disability under the Equality Act 2010.
- The known-groups and criterion strands are doctoral research, independently supervised and not peer-reviewed. We don't claim they are.
- Formal exam access arrangements need a separate Level 7 specialist assessment. Cognassist is a recognised tool toward that, not the full route.
- Our evidence was established under a consistent set of administration conditions, and we scope our claims to those conditions rather than assuming they carry across every way an assessment might be taken. We would expect you to hold any vendor, including us, to that.
FAQ
The questions a sceptical buyer actually asks
How do I evaluate a cognitive assessment?
Apply four criteria to every vendor, in order: is it standardised on a large, population-matched sample; do the decision-driving indexes measure reliably; is there validity evidence, honestly qualified; and who, independent of the vendor, stands behind it? Then two category questions: is it a whole-cohort validated assessment or a screening tool, and does it cover the whole graduated approach or just one stage?What is the difference between a cognitive assessment and a screening tool?
A screen raises a flag and stops, or serves a specialist referral only. A validated whole-cohort assessment measures every learner and feeds the assess, plan, do and review workflow with one evidence chain. It is a difference of category, not degree: a screen is not an assessment, and a flag is not a workflow.What questions should I ask a cognitive-assessment vendor?
Ask for the technical manual and the normative sample size and composition. Ask for the reliability coefficients on the decision-driving indexes, not a vague "it's validated". Ask which validity strands are peer-reviewed and which are not. And ask who the independent experts are and what any professional-body review actually covers.Is Cognassist a diagnosis?
No. Cognassist measures the cognitive differences associated with specific learning differences, as a signal for a tutor's professional judgement. It is not a diagnosis and does not by itself establish disability under the Equality Act 2010. Formal exam access arrangements need a separate Level 7 assessment.Point solution or platform: does it matter?
Yes. A point solution that covers one stage leaves the rest of the workflow, and the evidence chain, to you. An end-to-end platform carries the whole graduated approach with one thread from assessed need to evidenced impact, and works inside the tools you already run rather than beside them.Does it matter which device a cognitive assessment is taken on?
It matters more than it looks. A standardised score is only interpretable against the conditions the assessment was normed under, so changing those conditions changes what you are comparing against. Screen size, whether input is touch or mouse and keyboard, and how interruptible the setting is are all real differences in how a timed task is performed. That does not make any one device wrong, but it does mean a vendor supporting several should be able to show a published equivalence study across them. Ask for it.Is Cognassist peer-reviewed?
No, and we don't claim it is. The reliability and validity evidence sits in our technical manual and validation white paper series, some of it doctoral research still in preparation, and the assessment has been reviewed by PATOSS against exam-access requirements. A professional-body review is not peer review, and we describe it as exactly that.
Now you have the test. Put it to Cognassist.
Book a demo and ask us the four questions yourself, or read how Cognassist answers all four in full.