Browsing by Author "Johnson, Keith"

Now showing 1 - 7 of 7

Changing Epistemological Beliefs with Nature of Science Implementations
(2018-02) Johnson, Keith; Willoughby, Shannon D.
This article discusses our investigation regarding nature of science (NOS) implementations and epistemological beliefs within an undergraduate introductory astronomy course. The five year study consists of two years of baseline data in which no explicit use of NOS material was implemented, then three years of subsequent data in which specific NOS material was integrated into the classroom. Our original study covered two years of baseline data and one year of treatment data. Two additional years of treatment course data have revealed intriguing new insights into our students’ epistemic belief structure. To monitor the evolution of belief structures across each semester we used student pre-post data on the Epistemological Beliefs About the Physical Sciences (EBAPS) assessment. The collected data were also partitioned and analyzed according to the following variables: college (Letters of Science, Business, Education, etc.), degree (BA or BS), status (freshman, sophomore, etc.), and gender (male or female). We find that the treatment course no longer undergoes significant overall epistemic deterioration after a semester of instruction. We also acquire a more detailed analysis of these findings utilizing the aforementioned variables. Most notably, we see that this intervention had a pronounced positive impact on males and on students within the college of Education, Arts & Architecture, and those with no concentration. Lastly, whether or not students believe their ability to learn science is innate or malleable did not seem to change, remaining a rigid construct with student epistemologies.
A classical test theory and item response theory comparison of the brief electricity and magnetism assessment and the conceptual survey of electricity and magnetism
(2019-01) Eaton, Philip; Johnson, Keith; Frank, Barrett; Willoughby, Shannon D.
For proper assessment selection understanding the statistical similarities amongst assessments that measure the same, or very similar, topics is imperative. This study seeks to extend the comparative analysis between the brief electricity and magnetism assessment (BEMA) and the conceptual survey of electricity and magnetism (CSEM) presented by Pollock. This is accomplished by using large samples (NBEMA ¼ 5368 and NCSEM ¼ 9905) within classical test theory (CTT) and item response theory (IRT) frameworks. For the IRT comparison, after consideration of the conceptual content addressed in each assessment, it was assumed that each of these assessments are measuring the same student latent ability (θ), specifically a student’s ability to do introductory electricity and magnetism. Via a CTT and IRT analysis it was found that both assessments are essentially equal in overall difficulty. Classical item analysis applied to 7 questions used by both assessments revealed that each assessment functions slightly differently internally. The test information curves found from IRT show that the CSEM has superior information compared to the BEMA in estimating student latent abilities for the entire range of typical latent abilities achieved by students on each assessment, θ ≈ −2 to θ ≈ 3. Information in this case is interpreted as how well a student’s latent ability was estimated by an assessment as a function of latent ability. When the circuits questions are removed from the BEMA the majority of the information is lost in the θ ≈ 0 to θ ≈ 2 range. This means the circuits questions on the BEMA are information heavy for higher ability scores. So, special considerations should be made as to which assessment a study uses depending on the specific questions a researcher is attempting to answer.
Comparing exploratory factor models of the Brief Electricity and Magnetism Assessment and the Conceptual Survey of Electricity and Magnetism
(2019-10) Eaton, Philip; Frank, Barrett; Johnson, Keith; Willoughby, Shannon D.
While numerous studies have analyzed the conceptions probed by the Force Concept Inventory (FCI), assessments dedicated to electricity and magnetism lack similar analyses. This paper investigated the conceptions explored by the Brief Electricity and Magnetism Assessment (BEMA) and the Conceptual Survey of Electricity and Magnetism (CSEM) using exploratory factor analysis techniques. Exploratory factor analysis (EFA) was performed on both assessments using 5368 and 4941 postinstruction student responses for the BEMA and CSEM, respectively. A 6-factor EFA generated model was found for the CSEM, and was fit against another sample of 4964 student responses using confirmatory factor analysis to supply evidence for the possible generalizability of the model. The 5-factor EFA generated model for the BEMA could not be fit against another sample when trying to check for generalizability. The EFA generated factor models for the BEMA and CSEM were then compared and found to be similar in conceptual content, with the exception of one or two factors. Thus, from a factor analysis perspective, the BEMA and CSEM were found to be similar in conceptual content as revealed by student responses. With a better understanding of these electricity and magnetism assessments, future research into this domain of physics will then be able to make stronger conclusions based on students’ results within these assessments.
Epistemic belief structures within introductory astronomy
(2018-06) Johnson, Keith; Willoughby, Shannon D.
The reliability and validity of inventories should be verified in multiple ways. Although the epistemological beliefs about the physical science survey (EBAPS) has been deemed to be reliable and valid by the authors, the axes or factor structure proposed by the authors has not been independently checked. Using data from a study sample we discussed in previous publications, we performed exploratory factor analysis on 1,258 post-test EBAPS surveys. The students in the sample were from an introductory Astronomy course at a mid-sized western university. Inspection suggested the use of either a three-factor model or a five-factor model. Each of the factors is interpreted and discussed, and the factors arc compared to the axes proposed by the authors of the EBAPS. We find that the five-factor model extrapolated from our data partially overlaps with the model put forth by the authors of the EBAPS, and that many of the questions did not load onto any factors.
Epistemic beliefs of non-STEM majors regarding the nature of science: Where they are and what we can do
(2017-06) Willoughby, Shannon D.; Johnson, Keith
Students enrolled in introductory astronomy at a mid-sized land grant institution were surveyed at the start and end of each semester for three years regarding their epistemic beliefs about the physical sciences. After collecting data for two years in the control (baseline) course, the course was revised to include regular discussions of the nature of science, practice identifying pseudoscientific claims, and practice with metacognition. Survey data on epistemic beliefs were collected for one more academic year after the course was revised. This work details how the course was structured during the baseline and revised portions of the study. Data from both portions of the study are analyzed as a whole as well as broken down by gender. Negative trends seen in the baseline data are mostly eradicated in the revised portion of the course. The data analysis motivates a discussion of why including content regarding the nature of science should be integrated in classes geared toward non-science majors.
Quantifying Scientific Jargon
(SAGE Publications, 2020-07) Willoughby, Shannon D.; Johnson, Keith; Sterman, Leila B.
When scientists disseminate their work to the general public, excessive use of jargon should be avoided because if too much technical language is used, the message is not effectively conveyed. However, determining which words are jargon and how much jargon is too much is a difficult task, partly because it can be challenging to know which terms the general public knows, and partly that it can be challenging to ensure scientific accuracy while avoiding esoteric terminology. To help address this issue, we have written an R script that an author can use to quantify the amount of scientific jargon in any written piece and make appropriate edits based on the target audience.
Workbook for ASTR 110IN Introduction to Astronomy: Mysteries of the Sky
(Montana State University, 2020) Willoughby, Shannon D.; Johnson, Keith
An introduction to contemporary astronomy that explores the nature, methods, and limitations of scientific inquiry within the context of our struggle to understand the structure and evolution of the Universe. Topics include the history of astronomy, motions of the night sky, the solar system, stellar evolution, galaxies, and cosmology.