Browsing by Author "Willoughby, Shannon D."

Now showing 1 - 17 of 17

Applicability of the Newtonian Gravity Concept Inventory to Introductory College Physics Classes
(2016-06) Williamson, Kathryn E.; Prather, Edward E.; Willoughby, Shannon D.
The study described here extends the applicability of the Newtonian Gravity Concept Inventory (NGCI) to college algebra-based physics classes, beyond the general education astronomy courses for which it was originally developed. The four conceptual domains probed by the NGCI (Directionality, Force Law, Independence of Other Forces, and Threshold) are well suited for investigating students' reasoning about gravity in both populations, making the NGCI a highly versatile instrument. Classical test theory statistical analysis with physics student responses pre-instruction (N = 1,392) and post-instruction (N = 929) from eight colleges and universities across the United States indicate that the NGCI is composed of items with appropriate difficulty and discrimination and is reliable for this population. Also, expert review and student interviews support the NGCI's validity for the physics population. Emergent similarities and differences in how physics students reason about gravity compared to astronomy students are discussed, as well as future directions for analyzing the instrument's item parameters across both populations.
Changing Epistemological Beliefs with Nature of Science Implementations
(2018-02) Johnson, Keith; Willoughby, Shannon D.
This article discusses our investigation regarding nature of science (NOS) implementations and epistemological beliefs within an undergraduate introductory astronomy course. The five year study consists of two years of baseline data in which no explicit use of NOS material was implemented, then three years of subsequent data in which specific NOS material was integrated into the classroom. Our original study covered two years of baseline data and one year of treatment data. Two additional years of treatment course data have revealed intriguing new insights into our students’ epistemic belief structure. To monitor the evolution of belief structures across each semester we used student pre-post data on the Epistemological Beliefs About the Physical Sciences (EBAPS) assessment. The collected data were also partitioned and analyzed according to the following variables: college (Letters of Science, Business, Education, etc.), degree (BA or BS), status (freshman, sophomore, etc.), and gender (male or female). We find that the treatment course no longer undergoes significant overall epistemic deterioration after a semester of instruction. We also acquire a more detailed analysis of these findings utilizing the aforementioned variables. Most notably, we see that this intervention had a pronounced positive impact on males and on students within the college of Education, Arts & Architecture, and those with no concentration. Lastly, whether or not students believe their ability to learn science is innate or malleable did not seem to change, remaining a rigid construct with student epistemologies.
A classical test theory and item response theory comparison of the brief electricity and magnetism assessment and the conceptual survey of electricity and magnetism
(2019-01) Eaton, Philip; Johnson, Keith; Frank, Barrett; Willoughby, Shannon D.
For proper assessment selection understanding the statistical similarities amongst assessments that measure the same, or very similar, topics is imperative. This study seeks to extend the comparative analysis between the brief electricity and magnetism assessment (BEMA) and the conceptual survey of electricity and magnetism (CSEM) presented by Pollock. This is accomplished by using large samples (NBEMA ¼ 5368 and NCSEM ¼ 9905) within classical test theory (CTT) and item response theory (IRT) frameworks. For the IRT comparison, after consideration of the conceptual content addressed in each assessment, it was assumed that each of these assessments are measuring the same student latent ability (θ), specifically a student’s ability to do introductory electricity and magnetism. Via a CTT and IRT analysis it was found that both assessments are essentially equal in overall difficulty. Classical item analysis applied to 7 questions used by both assessments revealed that each assessment functions slightly differently internally. The test information curves found from IRT show that the CSEM has superior information compared to the BEMA in estimating student latent abilities for the entire range of typical latent abilities achieved by students on each assessment, θ ≈ −2 to θ ≈ 3. Information in this case is interpreted as how well a student’s latent ability was estimated by an assessment as a function of latent ability. When the circuits questions are removed from the BEMA the majority of the information is lost in the θ ≈ 0 to θ ≈ 2 range. This means the circuits questions on the BEMA are information heavy for higher ability scores. So, special considerations should be made as to which assessment a study uses depending on the specific questions a researcher is attempting to answer.
Comparing exploratory factor models of the Brief Electricity and Magnetism Assessment and the Conceptual Survey of Electricity and Magnetism
(2019-10) Eaton, Philip; Frank, Barrett; Johnson, Keith; Willoughby, Shannon D.
While numerous studies have analyzed the conceptions probed by the Force Concept Inventory (FCI), assessments dedicated to electricity and magnetism lack similar analyses. This paper investigated the conceptions explored by the Brief Electricity and Magnetism Assessment (BEMA) and the Conceptual Survey of Electricity and Magnetism (CSEM) using exploratory factor analysis techniques. Exploratory factor analysis (EFA) was performed on both assessments using 5368 and 4941 postinstruction student responses for the BEMA and CSEM, respectively. A 6-factor EFA generated model was found for the CSEM, and was fit against another sample of 4964 student responses using confirmatory factor analysis to supply evidence for the possible generalizability of the model. The 5-factor EFA generated model for the BEMA could not be fit against another sample when trying to check for generalizability. The EFA generated factor models for the BEMA and CSEM were then compared and found to be similar in conceptual content, with the exception of one or two factors. Thus, from a factor analysis perspective, the BEMA and CSEM were found to be similar in conceptual content as revealed by student responses. With a better understanding of these electricity and magnetism assessments, future research into this domain of physics will then be able to make stronger conclusions based on students’ results within these assessments.
Confirmatory factor analysis applied to the force concept inventory
(2018-04) Willoughby, Shannon D.; Eaton, Philip
In 1995, Huffman and Heller used exploratory factor analysis to draw into question the factors of the Force Concept Inventory (FCI). Since then several papers have been published examining the factors of the FCI on larger sets of student responses and understandable factors were extracted as a result. However, none of these proposed factor models have been verified to not be unique to their original sample through the use of independent sets of data. This paper seeks to confirm the factor models proposed by Scott et al. in 2012, and Hestenes et al. in 1992, as well as another expert model proposed within this study through the use of confirmatory factor analysis (CFA) and a sample of 20 822 postinstruction student responses to the FCI. Upon application of CFA using the full sample, all three models were found to fit the data with acceptable global fit statistics. However, when CFA was performed using these models on smaller sample sizes the models proposed by Scott et al. and Eaton and Willoughby were found to be far more stable than the model proposed by Hestenes et al. The goodness of fit of these models to the data suggests that the FCI can be scored on factors that are not unique to a single class. These scores could then be used to comment on how instruction methods effect the performance of students along a single factor and more in-depth analyses of curriculum changes may be possible as a result.
Development of the Newtonian Gravity Concept Inventory
(2013-12) Williamson, Kathryn E.; Willoughby, Shannon D.; Prather, Edward E.
We introduce the Newtonian Gravity Concept Inventory (NGCI), a 26-item multiple-choice instrument to assess introductory general education college astronomy (“Astro 101”) student understanding of Newtonian gravity. This paper describes the development of the NGCI through four phases: Planning, Construction, Quantitative Analysis, and Validation. We discuss the evolution of the instrument through three versions, including the refinement of a set of four concept domains and nine examples of items to illustrate how expert review, student interviews, and Classical Test Theory statistics informed our approach. We conclude that the NGCI is a reliable and valid instrument.
Encouraging student participation in large Astronomy courses
(2012-03) Willoughby, Shannon D.
Introductory astronomy is one of the most widely taught classes in the country1 and the majority of the students who take these classes are non-science majors. Because this demographic of students makes up the majority of astronomy enrollments, it is especially important as instructors that we do our best to make sure these students don't finish this course thinking that they can't do science, that only experts can understand scientific concepts, or that science is just an elaborate belief system. Ideally, at least some classroom time would be spent having the students use skills they already possess to explore concepts about astronomy. Realistically, we understand the time constraints and pressures felt by the average college instructor and the incredible amount of time and energy that can be required to rework a course.
Epistemic belief structures within introductory astronomy
(2018-06) Johnson, Keith; Willoughby, Shannon D.
The reliability and validity of inventories should be verified in multiple ways. Although the epistemological beliefs about the physical science survey (EBAPS) has been deemed to be reliable and valid by the authors, the axes or factor structure proposed by the authors has not been independently checked. Using data from a study sample we discussed in previous publications, we performed exploratory factor analysis on 1,258 post-test EBAPS surveys. The students in the sample were from an introductory Astronomy course at a mid-sized western university. Inspection suggested the use of either a three-factor model or a five-factor model. Each of the factors is interpreted and discussed, and the factors arc compared to the axes proposed by the authors of the EBAPS. We find that the five-factor model extrapolated from our data partially overlaps with the model put forth by the authors of the EBAPS, and that many of the questions did not load onto any factors.
Epistemic beliefs of non-STEM majors regarding the nature of science: Where they are and what we can do
(2017-06) Willoughby, Shannon D.; Johnson, Keith
Students enrolled in introductory astronomy at a mid-sized land grant institution were surveyed at the start and end of each semester for three years regarding their epistemic beliefs about the physical sciences. After collecting data for two years in the control (baseline) course, the course was revised to include regular discussions of the nature of science, practice identifying pseudoscientific claims, and practice with metacognition. Survey data on epistemic beliefs were collected for one more academic year after the course was revised. This work details how the course was structured during the baseline and revised portions of the study. Data from both portions of the study are analyzed as a whole as well as broken down by gender. Negative trends seen in the baseline data are mostly eradicated in the revised portion of the course. The data analysis motivates a discussion of why including content regarding the nature of science should be integrated in classes geared toward non-science majors.
Exploring gender differences with different gain calculations in astronomy and biology
(2009-07) Willoughby, Shannon D.; Metz, Anneke
To investigate differences in learning gains by gender, we collected data in large introductory astronomy and biology courses. Male astronomy students had significantly higher pre- and post-test scores than female students on the astronomy diagnostic test. Male students also had significantly higher pretest and somewhat higher post-test scores than female students on a survey instrument designed for an introductory biology course. For both courses, males had higher learning gains than female students only when the normalized gain measure was utilized. No differences were found with any other measures, including other gain calculations, overall course grades, or individual exams. Implications for using different learning gain measures in science classrooms, as well as for research on learning differences by gender are discussed.
How should we train and assess our STEM graduate students in oral communication?
(2020-01) Willoughby, Shannon D.; Davis, Kent; Green, Jennifer; Hughes, Bryce; LaMeres, Brock; Sterman, Leila B.
A poster presented at a IGE annual PI's reporting meeting in January 2020. Today’s STEM graduate students need to be able to effectively communicate their research with the public. How do we develop and assess a curriculum that fosters these skills in tomorrow’s science professionals?
Identifying the Stereotypical Who, What, and Why of Physics and Biology
(2018-12) Bruun, Megan; Willoughby, Shannon D.; Smith, Jessi L.
Supporting efforts to grow the scientific workforce means articulating and comparing the content of science field stereotypes. To do this, data were collected from the general public [undergraduates (n=121) and Amazon Mechanical Turk workers (n=223)] as well as from people within science [attendees of an undergraduate conference for women in physics (n=34)]. Participants were randomly assigned to consider either biologists or physicists and then produce both spontaneous judgments and rate various person traits (e.g., ratings related to looks and personality and hobbies) and field characteristics (e.g., ratings related to the working conditions, norms, and expectations for the field). Analyses show stereotypes of the scientist and the science field were statistically significantly negative overall, with stereotypes about physicists and the field of physics more negative than biology. Compared to biologists, physicists were perceived as statistically significantly more competent, but statistically significantly more unattractive, tech oriented, awkward, and loners. Furthermore, compared to biology, a job in physics was viewed as having fewer opportunities for working with and helping others, but more opportunities for agency, a greater requirement for innate brilliance and effort to succeed, and as more difficult. That said, physicists were more envied than biologists. Data were triangulated with open-ended responses illustrating that across samples, people are more likely to reproduce science stereotypes for physicists. Implications for stereotype research and broadening participation of the science workforce are discussed, with a focus on the utility of role models and classroom interventions that negate stereotypes such as writing activities and encouraging students to approach physics with a growth mindset. Instructors are encouraged to consider what stereotypes students have about the field of physics and physicists. At the department level, instructors are encouraged to consider hosting a Conference for Undergraduate Women in Physics sponsored in part by the American Physical Society.
Listening to student conversations during clicker questions: what you don’t hear may surprise you!
(2011-01) James, Mark C.; Willoughby, Shannon D.
When instructors provide time for students to discuss their ideas in Peer Instruction, instructors minimally expect that the conversation partners will discuss their opinions relating to the physical attributes posed in a question and submit clicker responses that coincide with individual opinions. We defined conversations that met these two criteria as “standard conversations.” In our study of 361 recorded Peer Instruction conversations from large introductory astronomy classrooms taught by experienced instructors, we found that 38% of student conversations were standard conversations. Of the remaining 62%, we identified three broad categories consisting of ten types of “nonstandard” conversations. The first category of conversations describes student ideas that were not reflected in any of the given multiple choice answers. The second category includes issues related to the interpretation of the statistical feedback provided by electronic classroom response systems. The third category describes common pitfalls experienced by students during conversations that led to unproductive interactions. Our analysis of nonstandard Peer Instruction conversations will be useful to practitioners and researchers seeking to improve the implementation of Peer Instruction.
Quantifying Scientific Jargon
(SAGE Publications, 2020-07) Willoughby, Shannon D.; Johnson, Keith; Sterman, Leila B.
When scientists disseminate their work to the general public, excessive use of jargon should be avoided because if too much technical language is used, the message is not effectively conveyed. However, determining which words are jargon and how much jargon is too much is a difficult task, partly because it can be challenging to know which terms the general public knows, and partly that it can be challenging to ensure scientific accuracy while avoiding esoteric terminology. To help address this issue, we have written an R script that an author can use to quantify the amount of scientific jargon in any written piece and make appropriate edits based on the target audience.
Student Understanding of Gravity in Introductory Astronomy
(2012-12) Williamson, Kathryn E.; Willoughby, Shannon D.
Twenty-four free-response questions were developed to explore introductory college astronomy students’ understanding of gravity in a variety of contexts, including in and around Earth, throughout the solar system, and in hypothetical situations. Questions were separated into three questionnaires, each of which was given to a section of introductory college astronomy with 143, 137, and 32 respondents, respectively. Combined with 15 interviews, the exploratory, open-response format allowed themes to emerge naturally, with both documented and undocumented misconceptions observed. The breadth of questions allowed for descriptions of possible student mental frameworks, including alternative models and misapplication of the scientific model
Technology talks: clickers and grading incentive in the large lecture hall
(2009-02) Willoughby, Shannon D.; Gustafson, Eric
Two sections of an introductory astronomy class were given different grading incentives for clicker participation for two consecutive semesters. In the high stakes classroom points were awarded only for correct answers, in contrast to the low stakes classroom in which points were awarded simply for participating. Self-formed groups of four students each were recorded in both sections several times during the spring 2007 semester and their conversations were transcribed and categorized into nine topics to analyze the variations between the sections. Performance on clicker questions and tendency to block vote were correlated with class grades and gains for the pre- and post-test scores on the Astronomy Diagnostic Test.
Workbook for ASTR 110IN Introduction to Astronomy: Mysteries of the Sky
(Montana State University, 2020) Willoughby, Shannon D.; Johnson, Keith
An introduction to contemporary astronomy that explores the nature, methods, and limitations of scientific inquiry within the context of our struggle to understand the structure and evolution of the Universe. Topics include the history of astronomy, motions of the night sky, the solar system, stellar evolution, galaxies, and cosmology.