Critical Evaluation of AI Experimental Predictions
Develop students' ability to assess the scientific reasoning, mechanistic logic, and plausibility underlying experimental predictions.
Recommended for: Upper-level research-focused courses
Procedure
- Students design a specific experimental scenario or receive one from the instructor (e.g., "Predict the phenotypic outcome of treating cultured cells with rapamycin while simultaneously inhibiting mitophagy" or "Predict the behavior of a pendulum system with non-linear damping")
- Students request that an AI system predict the experimental outcome and provide a detailed explanation of the underlying mechanisms
- Students systematically evaluate the AI's prediction and reasoning:
- Plausibility: Is the predicted outcome consistent with established principles, laws, or empirical patterns in the field?
- Mechanistic logic: Is the causal explanation mechanistically sound and free from logical inconsistencies?
- Completeness: Are all relevant mechanisms, pathways, or factors considered?
- Overlooked factors: What has the AI missed, oversimplified, or failed to integrate?
- Literature support: Can the prediction be verified or refuted using published empirical research?
- Quantitative reasoning: Where applicable, are any quantitative predictions mathematically sound?
- Students propose alternative experimental outcomes with mechanistic justifications
- Students identify what additional information, controls, or measurements would be needed for a reliable prediction
Extension options
- Students conduct systematic literature searches to empirically verify or refute AI predictions
- Compare predictions for identical scenarios across different AI systems
- When feasible, compare predictions to actual experimental results from published literature or laboratory work
Discussion questions
- What factors made certain predictions more or less scientifically plausible?
- How did the AI handle experimental complexity, uncertainty, and the integration of multiple mechanisms?
- What depth of domain expertise was required to adequately evaluate the predictions and reasoning?
- How should researchers appropriately integrate AI predictions into experimental planning and hypothesis development?
Assessment opportunity
Students produce a scientific evaluation of the AI prediction, including mechanistic critique, alternative hypotheses, and a proposed experimental approach to test competing predictions.