Assessing astronomy students ’ views about the nature of scientific inquiry

[This paper is part of the Focused Collection on Astronomy Education Research.] Students taking a second astronomy course for nonscientists were asked to reflect on the nature of scientific inquiry three times during the first half of the semester. First, they were assigned a short paper in which they were asked to argue for or against the thesis that observers of the night sky in ancient civilizations were scientists, and nearly all of the students argued that yes, they were. Second, they were asked to write a second paper in which they argued the opposite of their first thesis. Lastly, they were asked, on a midterm exam, to write an essay about whether the Ancient Greek astronomers were good scientists. In writing these papers and essays, many students included definitions of science or scientists, and these definitions progressed from paper to paper. In addition, the evidence students used to argue that ancient observers were scientists changed, with students becoming more expertlike.


I. INTRODUCTION
As early as the nineteenth century, educational leaders were arguing for the inclusion of science in the general education curriculum as a method for students to learn reasoning skills, such as drawing conclusions from observations.This was contrasted with appeals to authority as a way to learn, which (it was argued) was how the thencurrent curriculum of humanities and classical languages were taught [1].Over the centuries since, three dominant approaches to science teaching have emerged: science is taught as a body of knowledge, as a set of processes, or as a human activity.All three of these are useful for those who are to become scientists themselves.However, many of those who teach students who are not STEM majors think it is more important to teach about the processes of science and how science is a human endeavor [1].They want to teach the nature of science.
It is often difficult to define the nature of science.Many refer to Lederman, who spoke of the "values and assumptions inherent to the development of scientific knowledge [2], (p.331)."Even expert ideas about the nature of science are context dependent [3].Given those caveats, there is enough of an agreement about the nature of science to include it in popular K-12 curricula [4,5].Science is based on observations, as observations improve facts can change.Science is grounded in theories, and "theory" means something much stronger in science than it does when it is used nonscientifically.Lastly, it is people who do science, so it is both grounded in cultures and driven by human imagination [6].
When nature of science is taught explicitly at the university level, it is often done in astronomy classes [7].Many of the students who are required to take a science class as a distribution requirement at their college or university take astronomy; therefore, introductory astronomy is one of the most popular science courses taken at the university level.The numbers have been reported as approximately 250 000 students per year and as about 10% of all college students [8].In sharp contrast, there are fewer than 200 students graduating with a major in astronomy each year [9].Not only are most of the students in introductory astronomy classes not majoring in astronomy, typically only about 15% of them are majoring in STEM fields [10].
The population of these courses guides the faculty who design them.Many of us have learned to start course design with goals and student learning outcomes.Student learning outcomes are things that the students should be able to accomplish, and the instructor should be able to assess by the end of the course.Some outcomes, like "the student should be able to evaluate the evidence for a hypothesis," might be more important than "the student should be able to explain the phases of the Moon" [11].Many professors explicitly teach the methods, process, and nature of science in their introductory astronomy course [12].They teach about making observations and drawing inferences from them and about how technology changes the kinds of observations that can be made and, therefore, facts can change.Sometimes they have time to talk about the people who came up with the models and theories we work with today, the people who came up with models and theories that have since been discredited, and the cultures in which they lived.Workshops of department chairs held in 2001 resulted in a set of agreed-upon goals for the introductory astronomy course, and fewer than half of them can be seen as fact based, like "familiarity with the night sky and how its appearance changes with time and position on Earth," with the majority about the nature of science, like "the notion that the world is knowable, and that we are coming to know it through observations, experiments and theory" [9].
Most of the research on astronomy education has been on student understanding of topics in astronomy [13].While most of the research on the views that students and teachers have about the nature of science and the nature of scientific inquiry has mostly been done at the K-12 level [14] some has been done at the university level.There are instruments that attempt to objectively assess student views of the nature of science, though many of those have been evaluated and found not to be valid [15].In fact, many argue that standardized questionnaires are not the best way to explore student understanding in this area [16].
In one study, Abd-El-Khalick and Lederman gave an open-ended survey, with questions like "What, in your view, is science?What makes science different from other disciplines of inquiry?" to students taking history of science courses at a university [6].They wanted to know what views about the nature of science students entered their courses with and whether taking a history of science course would change those views.They found that, contrary what was predicted by others [17], simply taking history of science courses did not significantly change students' naïve views of the nature of science; that only happened when the nature of science was explicitly addressed in those courses [6].
The current study also concerns university student views of the nature of scientific inquiry.Students were taking a course on the history of astronomy, during which they did explicitly discuss the nature of scientific inquiry.Student views of the nature of science, scientists, and scientific inquiry were gleaned not from standardized questionnaires or from surveys but from real course assignments.During the first half of the course, students wrote two short papers and an essay on a midterm exam answering the question of whether people in a given ancient civilization were scientists.This gave students a chance to explicitly state their definition of science, think about what people did, and did not do, as they did their science, and discuss how the culture influenced the practice of science.Student definitions of science, and their evidence of what makes people scientists, were extracted from their papers for analysis.

II. METHODS
Miami University is a public university in southwestern Ohio.In the fall of 2016, it enrolled nearly 17 000 undergraduates and more than 2000 graduate students [18].Admission is competitive, so the student body was successful in high school.The class of 2020 is more than 10% international, though more than 40% of the students still come from Ohio [19].
The general education and liberal education requirements are called the Global Miami Plan.In addition to other distribution requirements, each student, no matter their major, must take at least one course in biological science and at least one in physical science (and one of those must have a lab component).Additionally, each student without a second major or minor must complete a Thematic Sequence.This is a set of related courses outside the student's home department [20].One of the Thematic Sequences offered in the Department of Physics is about astronomy.It is called Our Place in the Universe, and consists of these three courses: Introduction to Astronomy and Space Science, Observational Foundations of Astronomy, and Contemporary Astronomy.The first, introductory, course is taught in a large auditorium, but the second and third are capped at 24 and mostly taken by students in the thematic sequence.Students get no credit for completing a thematic sequence in their own major; therefore, almost no students in these courses are physics majors.
This study was done over two years of the second class in the sequence, Observational Foundations of Astronomy.The first course in the sequence, Introduction to Astronomy and Space Science, is a prerequisite, so students are assumed to have some background knowledge.Regardless, the course starts with a short review, concentrating on what patterns in the sky can be seen with the unaided eye.Students also start reading, including a chapter on ancient observers of the sky [21] and Neil deGrasse Tyson's article on stick-in-the-mud astronomy [22].During the second week of the class, students do a mini research project, in which the students research ancient civilizations (excluding Ancient Greece), noting what they observed and recorded.This is a very short project; in the course of one 80-minute class period, the student groups choose a civilization, research it online, and present their findings to the rest of the class.I end with a brief discussion of what the ancient civilizations have done, which leads into a discussion of what scientists do.Some things students have come up with are gather information, evaluate information, use logic and reasoning, explain how the natural world works, ask questions, find answers, find patterns, try to become more efficient and find better ways to do things, make predictions, experiment, and avoid superstitions and spiritual explanations.That leads into conversations about the relationship between religion (and astrology) and science, given that many civilizations have not always separated those.I am no longer surprised when students do not come out firmly in favor of separating science from astrology and religion; see below for more on this.The class usually agrees that astrologers and priests certainly contributed to early astronomy with their record keeping, whether or not they were scientists themselves.After that, students are assigned their first short (2-3 page) paper.Here is the assignment: Choose a civilization other than the one on which you gave your report.We have briefly discussed Mesopotamian/Babylonian, Chinese, Roman, Egyptian, and Aztec civilizations.If you really want to, you can choose another, but not Greece-that's coming later.Use this thesis: Observers of the night sky in ancient civilizations were scientists.Present an argument that supports, rejects, or modifies the given thesis, and support your response with factual evidence.
They turn that paper in during the third week of the semester.During the two semesters over which this study was conducted, Fall 2015 and Fall 2016, there was only one student who made the initial argument that observers in ancient civilizations were not scientists.As soon as the first paper is collected, the second short paper is assigned.The assignment is as follows: For this paper, take the argument you used in the first paper and flip it around: If you argued that the Ancient Egyptians were scientists in the first paper, then in the second paper your thesis will be that the Ancient Egyptians were NOT scientists.
The goal of this assignment is for students to become proficient at making arguments using evidence about science.Writing this second paper so soon after the first one will also give the students a chance to fully inhabit a point of view that was not initially their own; if they start out generous in their definition of scientist, they will need to also construct an argument using a stricter standard.When the first paper is returned in the fourth week, students spend class time in small groups discussing their strategies for flipping their arguments.They talk about how they have defined science, what evidence they used in their first paper and other things they have learned about the civilizations they wrote about, and what they can use to change their theses.
While students are writing the second papers, I start a two-week-long discussion of the Ancient Greek astronomers, introducing the concept of model building and the use of mathematics, and ending with the Ptolemaic model of the solar system.At the end of the sixth week there is a midterm exam, and one of the questions on the exam was Were the Ancient Greek astronomers good scientists?As you did in your papers, be sure to include your definition of science and back your thesis up with examples.Please do use complete sentences for this question!
The majority of the students, 33 (89%), argued that the Ancient Greek astronomers were good scientists.
With permission from the Institutional Review Board (IRB), students from the Fall 2015 and Fall 2016 classes of the Observational Foundations of Astronomy course consented to have their papers and midterm essay used for research.There were 37 students who turned in both papers, a midterm exam, and a consent form, 15 (41%) from the 2015 class and 22 (59%) from the 2016 class.Of these students 6 (16%) were women and 31 (84%) were men; there was one first-year student (3%), eleven sophomores (30%), eight juniors (21%), and seventeen seniors (46%).The majority of the students, 22 (59%), were from the business school, with twelve (32%) from arts and sciences and one each from the schools of engineering, creative arts, and education.There was one physics major, and a total of four (11%) of the students from STEM majors.This compares with the national trends for student taking introductory astronomy; nationally 15% of student in introductory astronomy classes are majoring in STEM fields, while about 20% of them major in business [10].

A. Definitions of science
There was only one student in the two years of the course who said that the observers of the sky they wrote about in their first paper were not scientists.Many of the students included definitions of science or scientist(s) in these papers, some using definitions from online dictionaries and some writing their own.To analyze the definitions, words were counted.First the articles, pronouns, linking words, and prepositions were removed.Then the words were put in alphabetical order, to organize them.Separate lists were made for the first paper, the second paper, and the midterm essay.Words with the same stems, like "experiment" and "experimentation" or "nature" and "natural", were treated as the same word.The word "science" and its variants, including "scientific" and "scientist", was the most common word in the definitions of science and scientist used in the short papers (though not in the midterm essays).It was excluded from analysis.The remaining words that appeared at least three times on the lists were put into Wordle (Wordle.net), a graphic organizer that shows the most-used word as the largest word in the figure, while words used less often are proportionally smaller.
Sixteen of the students (43%) included a definition of science and/or scientists in their paper; six wrote their own definition and ten pulled definitions from online dictionaries.All of the definitions from this paper are included in Appendix A, and the Wordle created from the commonly used words is shown in Fig. 1.At the time of the first paper, FIG. 1. Wordle created from student definitions of science and scientists, from their first short paper.students were using definitions like "person with knowledge of the natural and physical world," with some emphasis on research, studying, problems, and training.
For the second paper, there were also sixteen students who included definitions of science and/or scientists; five wrote their own definitions and eleven used definitions from online dictionaries.All of the definitions from this paper are included in Appendix B, and the Wordle created from the commonly used words is shown in Fig. 2.There was a large increase in the number of words used at least three times by the second paper; the words in the first paper are still common, but now students are talking more about research, problem solving, activities, systematic work, methods, and structure.This makes sense, given the conversations the students had with one another and as a class between the papers.They spontaneously discussed the scientific method and remembered the steps they had been taught as children.Further, as the students had been forced to reverse their theses for their second paper, since most of them choose a generous definition of science or scientist in their first paper they had to get stricter here.
On the midterm, twenty-six students (70%) included a definition of science; all wrote their own, as the students had no access to the Internet during their exam.All of the definitions from the midterm essay are included in Appendix C, and the Wordle created from the commonly used words is shown in Fig. 3.The number of frequently used words went down again after the second paper, perhaps reflecting in part the fact that they were all writing their own definitions, not resorting to quotes from online dictionaries.It is still easy to write definitions like "someone with knowledge of the world" from these commonly used words, but now scientists are predicting, hypothesizing, reasoning, testing, and studying, and there is lots of emphasis on experimentation and data.

B. Evidence of scientific behavior
In addition to defining science or scientists, which not all of the students did, students constructed arguments in their short papers and their midterm essay about whether ancient observers of the sky were scientists.In the 111 papers (37 students writing three each), there were 178 pieces of evidence given to support a thesis that ancient observers were scientists, and only 91 pieces of evidence supporting a thesis that they were not.Recall that, although between the first and second papers there would be equal numbers of pro and con arguments, only four students argued that the Ancient Greeks were not scientists when writing their midterm.Categories of evidence of scientific behavior, or the lack thereof, were generated inductively.The papers were read several times, and notes taken in a spreadsheet on the evidence students used.Coding was used to simplify the data, bringing out general themes [23].The evidence was then read several times, and commonly used words like "observations" or "astrology" were noted.Eventually, student phrases were replaced with these commonly used words.Then judgment was used, along with reference to the original texts, to group some words together into categories.For example, students used the words "observations" (or "observe"), "measurement" (or "measure"), "data", and "recording" (or "records") in nearly identical contexts, so one category became "observations, measurement, data, recording" (see Table I).
All of the evidence students used, over the two papers and the midterm, is summarized in Table I.The majority of the evidence students used to argue that observers of a given civilization were scientists, 60%, was tied to the processes of scientific inquiry: questions, predictions, hypotheses, observations, measurements, data, recording, experiments, testing, analysis, reasoning, math, models, explanations, and theories.A further 34% of the evidence was in the form of lists or descriptions of what the people of those civilizations had accomplished, such as creating a calendar using their observations of the lunar month and the solar year.
When students argued that observers of the sky from ancient civilizations were not scientists, nearly half of them (48%) said something about religion, astrology, priests, or gods.Much of the astronomy the class had discussed by the second paper was motivated by questions about what the gods wanted people to do and was hard to distinguish from astrology, so that makes sense.A further 37% of students argued that people were not scientists because of a lack of predictions, hypotheses, experiments, testing, models, explanations, and theories.
Most of the students used both the first short paper and the midterm to argue that the ancient observers of the night sky were scientists.Most of the evidence supporting an argument that anyone was not a scientist was in the second paper, so it was not possible to trace the student arguments there.However, it is possible to compare the evidence used by the majority of students on the first short paper, where 36 students used 94 pieces of evidence that ancient observers were scientists, to the evidence on the midterm exam, where 33 students used 81 pieces of evidence to argue the same thesis.The categories of evidence, with percentages and frequencies for each, are shown in Table II.
The evidence students used to support the arguments about scientific activity changed between the first paper and the midterm.No one mentioned experiments or testing on their first paper, while a few people did on the midterm.In addition, mention of analysis, reasoning, and math increased, as did mention of models, explanations, and theories.Most of this increase was compensated for by a decrease in mentions of the discoveries, invention, and knowledge of the ancient people.This makes sense, as the question on the midterm exam was about the Ancient Greek astronomers, who were among the first to use math and models to analyze and explain the world around them.
Interestingly, only six definitions of science, four on the second paper and two on the midterm, used the phrase "scientific method."These were all definitions the students had written on their own.Of those six essays, each done by a different student, only one used aspects of the scientific method in their evidence.She said that the civilization she was discussing was not scientific since they did not make predictions or hypothesize.

IV. CONCLUSIONS
This study concerned students in a history of science course that explicitly addressed the nature of science in class discussion and assignments.Student views of the nature of on scientific inquiry were gleaned from their course assignments, rather than from surveys or questionnaires.They were asked to reflect on the nature of scientific inquiry three times during the first half of the semester, with these three prompts: (1) Use this thesis: Observers of the night sky in ancient civilizations were scientists.Present an argument that supports, rejects, or modifies the given thesis, and support your response with factual evidence.(2) For this paper, take the argument you used in the first paper and flip it around: If you argued that the Ancient Egyptians were scientists in the first paper, then in the second paper your thesis will be that the Ancient Egyptians were NOT scientists.(3) Were the Ancient Greek astronomers good scientists?As you did in your papers, be sure to include your definition of science and back your thesis up with examples.One of the interesting outcomes from these papers was the most prominent reason students used to say that a civilization did not have good science: references to religion, astrology, priests, or gods (see Table I).For many of the civilizations discussed in class, there was not much separation between science and religion; the astrologers, priests, philosophers, and scientists were all the same people.This was not a problem for the students as they wrote their first paper, as nearly all of the students said that these civilizations did good science.It was only when they were pressed by the second assignment to flip their argument that half of the students used the conflation of science and religion to say that people were not good scientists.This is consistent with the work that Brickhouse and Shipman have done.In a university astronomy class, they gave students assignments in which they explicitly wrote about the relationship between science and religion, in a present-day context.They found that about half of their students did not see science and religion as incompatible; these students thought they could use both science and religion to explain the Universe around them.The other half of the students thought that science and religion were distinct; they argued that the questions answered by science and by religion were in separate domains [24,25].This is also the distinction made in some introductory science textbooks [26].Interestingly, none of the students in their study were in the third category they anticipated, thinking that only science, and not religion, was useful in making sense of the Universe [24].This finding was similar to that found by Hansson and Redfors, who surveyed students and found that more than half of them thought that science and religion were compatible; the students found it possible to accept the methods and results of science while also holding religious beliefs.Interestingly, most of the students predicted that physicists would find science and religion to be incompatible [27].
In general, as the semester progressed to the halfway point, the words students used in their definitions of science and scientist represented higher levels of scientific activity.They moved from definitions like "a scientist is a person with knowledge" to definitions that included more about the process of science, including hypothesis, experiment, study, and data.The evidence of scientific activity that students provided also changed between their second paper and their midterm.Of course, that is, in part, because of what was discussed in class about the Greeks, the subject of the midterm essay.The readings and discussions about Greek astronomy were largely about the use of mathematics and the introduction of models, both spoken of in positive terms as improvements that built on Babylonian astronomy.Students may well have thought, especially on a midterm exam, that the right answer was that mathematics and modeling made good scientists, and that is what most students said.
In fact, that brings in an important caveat for this whole paper.All three essays, the two short papers and the essay on the midterm exam, were done as graded work.The students may well have been "playing school," especially on the second paper and the midterm, rather than saying what their true feelings were.This is, of course, a problem with the study of any graded work.It would be interesting to repeat these assignments with a new group of students, outside of a graded course setting, to see if their responses were substantially different.
In general, though, this study reinforces the ideas of Abd-El-Khalick and Lederman [6], who saw students' naïve views of the nature of science changing only when the nature of science was explicitly discussed during their course.This research can be a valuable lens into the thoughts of nonscientists.It could have implications for how general education science courses are taught across the curriculum, as a common student learning objective in these courses is to be able to demonstrate an understanding of what science is and/or what scientists do.
• "Someone who asks the right questions" (BrainyQuote.com-attributedto Levi-Strauss) • Someone who has knowledge or studies a particular field of nature

FIG. 2 .
FIG.2.Wordle created from student definitions of science and scientists, from their second short paper.

TABLE I .
Evidence students used to argue that ancient people were or were not scientists-overall.