The AI-readiness assessment in your class

Recommended for: The first weeks of a course in which students will work with generative AI. It fits any discipline: the assessment asks about use and judgement, not subject knowledge

Objective: The student can say where their own handling of generative AI is strong and where it is weak - prompting, checking output, privacy and academic integrity - and can name one habit they want to change in this course.

Time needed: 20 to 30 minutes for the assessment, 20 minutes for the debrief · Group size: from 8 students; debrief in groups of three or four

Background for the instructor

The AI-readiness assessment is not a questionnaire but a conversation. Fourteen questions, multiple choice and open-ended mixed together, which adapt to the answers: do well on a topic and the follow-up gets harder, get stuck and you get an explanation first. Along the way the student earns stars; at the end comes a report with strengths, areas to work on and next steps.

The questions cover four things that are rarely assessed together: knowing what a language model is and where it goes wrong, writing a usable prompt, checking an AI output before it goes into a piece of work, and the ethical side - which data you do not put in a chat window, and when you declare that you used AI. On top of that it asks about their own use: how often, what for, and with which rules in mind.

The teaching value is not in the score. Students who use AI daily almost always rate themselves highly, and then come unstuck on question 3 or 12, which are about checking what comes back. That gap between using and judging is what this exercise is about. The report is the teaching material, not the mark.

Two practical points. The fourteen questions are in English - it is a validated instrument, and a translation quietly turns a question into a different one - while the conversation, the feedback and the final report follow the student's language. Start in Dutch and the rest is in Dutch. And the assessment does not ask for a name or a student number; tell students they need not enter one.

For where the instrument comes from and what the Education Lab is testing with it, see the project page in the GenAI experimentation collection.

Procedure

Variant A: in class. This is the strongest form, because the debrief does the work.

1. Say at the start that nothing is handed in and nothing is marked. Without that sentence students answer what they think looks good, and then you measure nothing.
2. Laptops open, 25 minutes. The assessment is on Ask EduBase; students click the starter in the language they want.
3. Ask everyone to keep the final report - screenshot or copy and paste. The conversation is gone afterwards.
4. Groups of three or four, ten minutes: put the reports side by side. Do not compare stars, compare the Areas to work on. Which point comes back for several people? Where do you differ, and why?
5. Plenary, ten minutes: collect the recurring points on the board. Close with one agreement for this course - how AI use will be declared in assignments, say. Coming out of their own reports, it is not a rule imposed from above.

Variant B: formative, in their own time. Lighter, and usable when there is no class time.

1. Put the link in Canvas or in the course manual, with a deadline: before the week the first AI assignment starts.
2. Ask for one thing back, not the whole report: the line from Areas to work on that hit home, plus what the student is going to do about it. Three lines is enough.
3. Read those lines before you explain the AI assignment. You then know where the group stands and can aim your explanation at it.

Variant B is weaker, and it is worth knowing why: without a debrief the report stays a private piece of advice that most students read once. The conversation in the group is what turns a test into an exercise.

Discussion questions

• What did you think you were good at, and what did the report point at instead?
• At which question did you notice you were giving the answer you thought was "right" rather than what you actually do?
• Several questions ask about checking. How often do you really check an AI answer, and what tells you something is off?
• What have you put into a chat window that, thinking about it now, did not need to go there?

Assessment opportunity

Do not mark this. The assessment is self-reported: anyone who knows what the right answer looks like can score well without doing any of it, and the moment marks are attached you are only measuring what students think you want to hear.

What does work: have the student use the final report as the start of a learning agreement. Two points from Areas to work on, what they will do about them in this course, and half a page at the end on what came of it. That reflection can be marked and cannot be written by a model, because it refers to their own report and their own work.

Extension options

Have students take the assessment again at the end of the course and put the two reports side by side. A change in stars says little; a change in what they write themselves at the open questions says a great deal.

For a more critical variant: let a group pick one question from the assessment and argue why the "right" answer does not hold in their discipline. The instrument comes from another university and another field; that is open to challenge.

This exercise combines well with Five critical questions before using AI as a follow-up to the prompting and judgement questions, with Checking an AI answer with ACCUSE for the checking, and with Data privacy and AI infrastructure for the privacy questions. To start without a tool, run Ask students about their AI experiences first.

Reference

• Based on the AI-Readiness Agent of the University of Sydney. The fourteen questions and the design come from there; the final report points on to their own Canvas course.
• The VU version runs on the Education Lab's own environment, inside the EU. See the project page AI-readiness assessment for what is being tested with it.
• The activity above - the two variants, the questions and the learning agreement - is ours.