Many courses now incorporate online methods for assessing student learning. Some instructors assign questions prior to lecture (e.g., pre-lecture quizzes) to obtain information that helps them tailor their lessons to the current state of student understanding. To assess curriculum efficacy, physics education researchers often use ‘pretests’, inventories, etc. that provide baseline data that can be compared to results from post-tests. These assessments were initially given on paper but moved online in the early 2000s. At the University of Washington, we found that the impact of this change was minimal; the results of our pretests stayed essentially the same when given on paper or online and were consistent from quarter to quarter and instructor to instructor. However, we have begun to observe systematic changes in pretest performance that are consistent with students using AI tools to answer pretests, even though they are scored only on effort, not correctness. Using two-proportion z-tests, mixed-effects logistic regression, and Welch’s t-tests to analyze response length of explanations, this paper documents the nature and extent of these changes across 39 questions in two introductory algebra-based physics courses. This paper presents results that suggest caution to those using online quizzes to probe student understanding - both before and/or after instruction. We have seen increases in correct responses on pretests as great as 36%, but we have also seen significant decreases on questions that AI tools have tended to answer incorrectly. These findings raise important questions about the validity of online assessments and are intended to raise awareness for anyone using online tests to assess student learning. It remains an open question as to whether, and how, AI can be leveraged to improve instruction.