AI-Generated Summary Check

Recommended for: After students understand AI variability

Procedure

1. Assign students a peer-reviewed scientific paper relevant to course content
2. Require students to perform a close reading of the paper, taking detailed notes on key arguments, methodology, and conclusions
3. Instruct students to use an AI tool to generate a summary of the same paper
4. Have students systematically evaluate the AI summary for accuracy, completeness, and correctness by comparing it directly to the original text
5. Organize small groups (3-4 students) to discuss their findings, focusing specifically on what the AI missed, misrepresented, or oversimplified
6. Extension - Structured Prompt Variation: Provide students with a structured prompt template and repeat the exercise to examine how prompt engineering affects output quality:

Role

You are an expert in language and summarization of scientific articles

Task

You will make a summary of an article I will upload

Format: The summary has a maximum length of 500 words

Structure: Problem description, research design, results, conclusions, and implications

Style: Scientific writing style

Output: Format using Markdown

Discussion questions

• What critical nuances or methodological details did the AI fail to capture?
• How might relying solely on AI summaries impact students' comprehension and learning?
• How does prompt specificity and structure affect the quality and accuracy of AI outputs?

Assessment opportunity

Students produce a comparative analysis identifying at least five discrepancies between the AI summary and the original paper, with explanations of why these discrepancies matter.

Reference

• For detailed implementation examples, see Dr. Mohammad H. Rezazade Mehrizi's pedagogical approach: https://vu.nl/en/education/more-about/generative-learning-through-not-despite-generative-ai-a-real-life-experiment