Discipline-Specific Diagram Evaluation

Recommended for: After foundational disciplinary content has been established

Implementation Note: Select the discipline-appropriate version below or adapt the structure for your specific field.

Biomolecular Sciences Version

Procedure

1. Provide students with an AI-generated molecular pathway diagram or have them generate one using AI tools (e.g., a signal transduction pathway, metabolic pathway, or gene regulatory network)
2. Instruct students to systematically identify errors including:
   • Incorrect arrow directions indicating reversed causality or regulation
   • Missing regulatory molecules or cofactors
   • Mislocalized proteins (e.g., cytoplasmic proteins shown in nucleus)
   • Absent mechanistic details or intermediate steps
   • Incorrect molecular interactions or binding partners
3. Discuss why these specific errors occur and their implications for understanding biological mechanisms
4. Have students propose corrections based on authoritative sources

Discussion questions

• What biological knowledge was required to identify these errors?
• How might students who rely on AI-generated pathway diagrams develop fundamental misconceptions about cellular processes?
• What role should AI-generated diagrams play in learning molecular biology?

Computer Science/Software Engineering Version

Procedure

1. Provide an AI-generated UML class diagram, entity-relationship diagram, or system architecture diagram
2. Instruct students to systematically identify errors including:
   • Incorrect inheritance relationships or class hierarchies
   • Missing or inappropriate associations between classes
   • Violated design principles (e.g., circular dependencies, tight coupling, low cohesion)
   • Incorrect cardinality notations in relationships
   • Methods or attributes placed in inappropriate classes
   • Missing essential design patterns or architectural components
3. Discuss how these errors would manifest as actual problems in software development
4. Have students refactor the diagram to conform to sound design principles

Discussion questions

• What software engineering knowledge was necessary to identify these design flaws?
• How might flawed AI-generated diagrams lead to poorly architected software systems?
• Under what circumstances might AI-generated design diagrams be useful versus harmful?

Organic Chemistry Version

Procedure

1. Provide students with an AI-generated reaction mechanism diagram
2. Instruct students to systematically identify errors including:
   • Incorrect electron-pushing arrows (wrong direction, source, or destination)
   • Violations of arrow formalism conventions
   • Missing intermediate structures or transition states
   • Stereochemical errors or racemization where inappropriate
   • Incorrect formal charges on atoms or molecules
   • Thermodynamically implausible reaction steps
   • Violation of orbital symmetry or conservation principles
3. Discuss the fundamental importance of mechanistic accuracy in organic chemistry
4. Have students provide corrected mechanisms with justification from chemical principles

Discussion questions

• What understanding of organic chemistry was required to identify mechanistic errors?
• How might reliance on AI-generated mechanisms impede development of mechanistic reasoning skills?
• What verification methods should chemistry students employ when using AI-generated mechanisms?

Electrical Engineering Version

Procedure

1. Provide an AI-generated circuit diagram for analysis
2. Instruct students to systematically identify errors including:
   • Incorrect component symbols or non-standard notation
   • Improper polarity markings on capacitors, diodes, or voltage sources
   • Missing essential connections such as ground references
   • Components incorrectly placed in series when they should be parallel (or vice versa)
   • Voltage sources with conflicting polarities creating logical impossibilities
   • Violations of Kirchhoff's voltage or current laws
   • Missing current-limiting resistors or protection components
3. Discuss safety and functionality implications of these errors
4. Have students redesign the circuit correctly and verify using circuit analysis techniques

Discussion questions

• What electrical engineering knowledge was necessary to identify these circuit errors?
• What are the potential safety hazards of implementing AI-generated circuit designs?
• How should engineers verify AI-generated circuit designs before implementation?

Medicine/Clinical Practice Version

Procedure

1. Provide an AI-generated diagnostic algorithm or clinical decision flowchart (e.g., acute chest pain management, sepsis protocol, stroke assessment)
2. Instruct students to systematically identify errors including:
   • Incorrect decision thresholds or cut-off values
   • Missing critical assessment steps or diagnostic procedures
   • Inappropriate treatment sequences or medication orders
   • Contraindications not properly accounted for
   • Evidence-based clinical guidelines violated or ignored
   • Decision branches that could lead to patient harm or adverse outcomes
   • Missing considerations for special populations (pediatric, geriatric, pregnant)
3. Discuss patient safety implications and the critical importance of evidence-based practice
4. Have students revise the algorithm based on current clinical guidelines and evidence

Discussion questions

• What clinical knowledge and experience was required to identify these potentially dangerous errors?
• What are the patient safety implications of clinicians relying on unverified AI-generated protocols?
• How should the medical profession approach AI-generated clinical decision support tools?
• Social Sciences (Sociology/Political Science) Version:

Procedure

1. Provide an AI-generated causal diagram or theoretical framework model (e.g., factors influencing voter behavior, determinants of social mobility, causes of political polarization)
2. Instruct students to systematically identify errors including:
   • Reversed causal arrows indicating incorrect directionality
   • Missing confounding variables that affect multiple factors
   • Oversimplified relationships that ignore important contextual factors
   • Inappropriate assumptions of direct causation without mediating mechanisms
   • Excluded moderating factors that alter relationship strength
   • Misrepresentation of well-established theoretical relationships
   • Absence of feedback loops or reciprocal causation
3. Discuss how these errors might lead to flawed policy recommendations or invalid research conclusions
4. Have students reconstruct the model based on established social science theory and empirical evidence

Discussion questions

• What theoretical and methodological knowledge was required to identify these conceptual errors?
• How might oversimplified AI-generated models lead to ineffective or harmful social policies?
• What role should AI play in developing theoretical frameworks in the social sciences?
• Cross-Disciplinary Discussion Questions:
• What depth of disciplinary knowledge is required to effectively identify errors in AI-generated representations?
• How might students who uncritically accept AI-generated diagrams develop fundamental misconceptions?
• What verification and validation processes should researchers employ when using AI-generated disciplinary content?
• Under what circumstances might AI-generated representations be pedagogically useful versus harmful?

Assessment opportunity

Students produce corrected versions of AI-generated diagrams with detailed annotations explaining each error and its correction, supported by authoritative disciplinary sources.