Journal of Agricultural Education, 65(1), 320-336 https://doi.org/10.5032/jae.v65i1.138 Journal of Agricultural Education 320 Volume 65, Issue 1, 2024 Impact of Professional Development of School-Based Agricultural Education Teachers: In-service Implications for Safety Training Rebecca G. Lawver 1 Scott W. Smalley 2 Dustin K. Perry 3 Michael L. Pate 4 Abstract Vocational and technical education programs continue to play a pivotal role in developing workers’ occupational safety and health skills in all industries. The Agricultural Safety Education Initiative was first conducted in the summer of 2017 as a multi-year “Train the Trainer” program to improve teachers’ tractor and machinery knowledge. The National Safe Tractor and Machinery Operations Program (NSTMOP) Curriculum was used to develop and organize the training program. The purpose of this study was to determine the efficacy of a multi-year agricultural safety education professional development model. A total of 85 teachers participated in the third year of the training program. Teachers’ average NSTMOP test score was 41.9 out of 50 (SD = 3.62). Teachers attending the training for the first time in 2019 scored lower (40.8, SD = 4.41) than teachers who had attended the training during all three offerings (43.2, SD = 3.00), though the difference was not statistically significant (Kruskal-Wallis H was 5.91 (2) p = .052). Post- experience qualitative data findings showed many teachers reported curriculum obtainment as a motivating factor for continued attendance. A benefit expressed by teachers participating in the professional development focused on higher-order instructional and alternative assessment methods for tractor and machinery safety. For these SBAE participants, the involvement in a multi-year approach to professional development is influenced when curriculum is offered. Introduction/Background Unlike other industries, agriculture continues to employ youth less than 16 years of age, who may have underdeveloped safety knowledge and technical skills, as a significant part of the labor force, whether paid or unpaid, in a variety of tasks deemed hazardous by the Department of Labor (DOL) (National Institute for Occupational Safety and Health [NIOSH], 2014; U.S. Department of Labor, n.d.). An example of a hazardous task commonly performed by youth is the operation of tractors and machinery. However, vocational and technical education programs have historically played a pivotal role in developing workers’ occupational safety and health skills in all industries (Schulte et al., 2005). Within the context of school- 1 Rebecca G. Lawver is a Professor of Agricultural Education and Department Head in the Department of Applied Sciences, Technology, and Education at Utah State University, 2300 Old Main, Logan, UT, 84322, rebecca.lawver@usu.edu, https://orcid.org/0000-0001-8226-2810 2 Scott W. Smalley is an Associate Professor of Agricultural Education in the Department of Agricultural Education, and Studies at Iowa State University, 217 Curtiss Hall, Ames, IA 50011, smalle16@iastate.edu, https://orcid.org/0000-0001-8386-4266 3 Dustin K. Perry is an Associate Professor of Agricultural Education and Department Head in the Department of Agricultural Education and Technology Education at Montana State University, Linfield Hall 230E, Bozeman, MT, 59717, dustin.perry@montana.edu, https://orcid.org/0009-0008-4398-5571 4 Michael L. Pate is a Professor of Agricultural Systems Technology in the Department of Applied Sciences, Technology, and Education at Utah State University, 2300 Old Main, Logan, UT, 84322, michael.pate@usu.edu, https://orcid.org/0000-0002-2904-5347 Lawver et al., Impact of Professional Development … Journal of Agricultural Education 321 Volume 65, Issue 1, 2024 based agricultural education, Supervised Agricultural Experiences (SAEs) facilitate youth’s occupational skill development, such as safe tractor and machinery operation. Related to hazardous work associated with students’ SAEs, the Federal Fair Labor Standards Act provides two applicable exemptions. The first exemption allows 14 and 15-year-old youth who have completed a tractor and machinery certification (TMC) program to work for a farm employer who is not a parent. The second allows a student-learner exemption such as a Supervised Agricultural Experience project (Wage and Hour Division, 2016). These exemptions specify that safety training can only be validated and provided under the supervision of the Cooperative Extension Service or “vocational agriculture teachers” as part of a “bona fide” secondary school program (Wage and Hour Division, 2016, p. 5). NIOSH (2002) and Miller (2012) challenged the adequacy of these educational exemptions and proposed revisions (Wage and Hour Division, 2011). The major changes proposed on September 2, 2011 by the Wage and Hour Division of the Department of Labor included a notice of proposed rulemaking to update the agricultural child labor regulations. The major revisions proposed included: 1) requiring that all tractors operated be equipped with proper rollover protection structures and seatbelts, 2) removing two youth training certifications, 3) revising the type of equipment they may operate, 4) revising and expanding prohibitions against working with animals, 5) bringing parity between agricultural and nonagricultural child labor rules, 6) prohibiting all tasks which fall with the job of pesticide handler, 7) prohibiting the use of most electronic devices while operating power-driven machinery, and 8) prohibiting all work in tobacco production and curing. Research documenting the impact of tractor and machinery youth certification programs was limited (76 Fed. Reg. 171, 2011; NIOSH, 2002). Public comments on the proposed changes to revise the student learner exemption and remove the TMC training exemptions documented an adverse reaction from organizations such as the National Association for Agricultural Educators (Agriculture Coalition, 2011). These comments noted that the proposed changes would inhibit teachers’ ability to train students for careers in agriculture without focused efforts on SAE safety. Ultimately, the DOL child labor revisions were withdrawn (Wage and Hour Division, 2012). Following the withdrawal of the child labor revisions, a refocused collaborative effort on agricultural safety curriculum for youth was led by the USDA NIFA’s Youth Farm Safety Education and Certification Program (YFSEC). In 2013, USDA NIFA awarded funds supporting the Safety in Agriculture for Youth (SAY) project, led by Pennsylvania State University. The outcomes of this funded project were the Safety in Agriculture for Youth Curriculum Clearinghouse and updated safety training curriculum, including the Supervised Agricultural Experience Risk Assessment Protocol and the online CareerSafe® 10-hour Occupational Safety and Health Administration General Industry Agriculture Credential. However, there remains a need to examine the impact of professional development on agricultural education teachers’ knowledge of the safety content provided in the curriculums. Bush and Andrews (2013) noted the immense variation between states’ career and technical education program structures, resulting in integration discrepancies among occupational safety and health curricula. Jepsen (2012) reported the impact of the TMC program is linked to implementation efforts of community-based instructors and emphasized that program evaluation efforts are daunting due to a variety of community-based implementation factors. This has been documented for secondary agricultural education programs as researchers have acknowledged the variation in safety programming and student training (Lawver et al., 2016; Mann & Jepsen, 2019; Pate et al., 2016). Murphy (1992) noted that since the inception of the TMC training, the DOL has provided limited surveillance to ensure program fidelity. The focus on youth farm safety through the USDA NIFA-funded SAY project has generated a greater repository of multiple safety curricula with increased visibility, access, and curriculum standard alignment. Yet, research efforts are still needed to determine how to best prepare and develop TMC Lawver et al., Impact of Professional Development … Journal of Agricultural Education 322 Volume 65, Issue 1, 2024 instructors for delivering safety curriculum that meets the DOL standards. Unlike the OSHA Trainer certification requirements (Occupational Safety and Health Administration [OSHA], n.d.), the TMC training program requires no experience or professional development requirements for instructors other than the requirement of being employed as an extension agent or an agriculture instructor. Yet, instructor preparation can vary widely (Rasty et al., 2017) and many teacher preparation programs have reduced capacity to provide teachers with agricultural mechanics training, such as safe tractor and machinery operations. This project aligns with the 2016-2020 American Association for Agricultural Education National Research Agenda under research priority 3.2 (Stripling & Ricketts, 2016). Specifically, this research seeks to address the issue of agricultural teacher safety training and support their success in enhancing youth farm safety at all career stages. The Agricultural Safety Education Initiative Project was funded as part of the High Plains Intermountain Center for Agricultural Health and Safety (HICAHS) to develop an innovative educational program to be utilized by local agricultural teachers to improve the safety practices and work environments of businesses and companies that employ young agricultural workers between the ages of 14- 18. This fits within the priority of workforce preparedness given the requirements established by the hazardous occupations in agriculture order issued by the Department of Labor. Intending to gain insight into a professional development model for agricultural safety education, a specific aim of this project was to investigate the impact of a multi-year “Train the Trainer” program on teachers’ tractor and machinery knowledge. The Agricultural Safety Education Initiative was first conducted in the summer of 2017 and organized around the National Safe Tractor and Machinery Operations Program (NSTMOP) Curriculum (Pate et al., 2019). In 2017, the training focused on the safety theme of tractor stability and roll-over protection. Pate et al. (2019) initially found teachers’ (N = 116) average NSTMOP pre-test score was 35.2 out of 48 (SD = 3.3) before completing the training, and an average NSTMOP post-test score of 40.3 out of 48 (SD = 4.1) upon completing the training. The resulting increase in test scores was significant (t(109) = 11.9, p < 0.001). Training was offered again during the summers of 2018 and 2019. Each seminar focused training activities on a specific safety theme. The safety theme for 2018 was All-terrain Vehicle Stability and Operation for Agricultural Tasks. The safety theme of the 2019 training was hitching/backing tractors and agricultural implement connections. It has been documented that increased student achievement is directly related to the length and type of professional development that teachers experience (Loucks-Horsley, et al. 2003). Supovitz and Turner (2000) identified a statically significant relationship can be identified based on professional development experiences and changes in teaching practices. When a sustained professional development opportunity is available teachers will improve their instructional practices (Johnson, 2006) compared to one-and-done professional development opportunities. The change in instructional practice once a teacher participates in a multi-year professional development requires time for the information to be effectively transferred into practice (Johnson, 2006). The life cycle of the career teacher model (Steffy & Wolfe, 2001) is focused on the premise that teachers will continue to grow and develop throughout their teaching careers. The model is broken down into six phases of development including novice, apprentice, professional, expert distinguished, and emeritus. Teachers who progress throughout their professional lifetime achieve the goals of the National Commission on Teaching and America’s Future. It is undocumented how a multi-year teacher-based teacher training program may influence instructor development related to tractor and machinery safety knowledge. It is important to determine the efficacy of having teachers attend an agricultural safety education professional development over multiple years. Will attending multiple professional development seminars increase TMC instructors’ capacity to serve youth in developing safety competencies related to tractors and machinery operations? Lawver et al., Impact of Professional Development … Journal of Agricultural Education 323 Volume 65, Issue 1, 2024 Conceptual Framework A large body of literature has emerged related to in-service teacher professional development, teacher learning, and teacher change (Desimone et al., 2002; Desimone & Stuckey, 2014; Richardson, 2001; Richardson & Placier, 2001). While describing teacher professional development can be complex (Desimone et al., 2002) teacher professional development intends to enhance professional knowledge, skills, and attitudes of teachers so that they might, in turn, impact student learning (Guskey, 2002). Delivering effective teacher professional development is one way to increase educator skills (Hammond, et al., 2017). The researchers outline elements of effective professional development after reviewing 35 rigorous studies that demonstrated a positive link between teacher professional development and student outcomes. The areas outlined in professional development are content-focused, incorporate active learning, support collaboration, use models and modeling, provide coaching, offer opportunities for feedback and reflection, and sustain duration. Professional development planners must conduct evaluations of teacher outreach programs to assess the value of continuing professional development or to determine the effect of teacher preparation efforts (Myers & Roberts, 2004). The effectiveness of teacher professional development programs is critical for program improvement and should be guided by a program theory of action. Evaluators have used program theory and program logic models to study the underlying assumptions about how programs are expected to work (Rogers et al., 2000). Weiss (1998) defined program theory as the mechanisms that mediate between a program’s delivery (and receipt) and the emergence of the outcomes of interest. Ideally, program theory guides an evaluation by identifying key program elements and articulating how these elements are expected to relate to each other (Cooksy et al., 2001). Program evaluation assesses whether a program is designed in a way that it can achieve its intended outcomes (Brousselle & Champagne, 2011). The program theory of action is often visualized in the form of a program logic model providing an integrative framework for evaluation (Cooksy et al., 2001). Weiss (1972) suggested using path diagrams to model the sequence of steps between program intervention and the intended outcomes. A logic model depicts assumptions about the resources needed to support program activities and produce outputs and the activities and outputs to realize the intended outcomes of a program (Wholey, 1995). Within the logic model, data is collected by different methods or from different sources on the same program to determine if a pattern of relationships exists (Denzin, 1970; Trochim, 1985). We used a logic model design to guide the program, which is seen in Figure 1. The logic model links theories and assumptions with the inputs, activities, outputs, and outcomes of a project (W.K. Kellogg Foundation, 2004). This systematic evaluation of the agricultural safety professional development program aims to ascertain outcomes and justify resources associated with the program to key stakeholders. Within local communities, school-based agriculture teachers interact with students often and may serve a vital role in protecting student safety by providing safety training and helping their students develop logical thinking and sound decision-making (Schwebel & Pickett, 2012). Students learn safe behaviors by modeling their teachers’ behaviors, particularly during supervised work experiences. Supervised agricultural experiences facilitate experiential learning to develop skills and abilities leading to an agricultural career (Barrick et al., 1992; Burke et al., 2006). Sanderson et al. (2010) concluded that as children become young adults they learn to farm safely through observation and modeling of mentors. Lawver et al., Impact of Professional Development … Journal of Agricultural Education 324 Volume 65, Issue 1, 2024 Figure 1 Logic Model for Agricultural Safety Education Initiative Tractor and Machinery Safety Professional Development Model Under Kennedy’s (2016a) framework, teacher professional development should be guided by a theory of action comprised of a central problem of practice and pedagogy to help teachers enact new ideas into the context of their practice. For this multi-year professional development program, the theory of action guiding the program was informed by a central problem of practice concerning supervised agricultural experience safety using a hands-on pedagogy to facilitate contextualization of safety instruction for students. Our workshop model followed Desimone’s (2009) core features of professional development by addressing the teaching problem (coherence) of how to improve student comprehension of tractor and machinery safety (content) using hands-on table-top demonstrations (active learning), tractor operations walk-through examples (collective participation) of student activities during a 10-hour session (duration). The enactment component of our program theory of action was guided by a prescriptive approach for integrating agricultural safety curriculum within school-based supervised agricultural experiences (Kennedy, 2016b) using the National Safety Tractor Machinery Operations Program and the Supervised Agricultural Experience Risk Assessment Protocol. A prescriptive approach provides the teacher with pre- established lesson plans that utilize closed-ended prompts for teachers and expected responses by students. Perspective curriculum approaches include core units and lessons ready for teachers’ immediate implementation without modification for their respective students. Due to hazards associated with operating machinery, the prescriptive approach reduces the amount of judgment needed by teachers on the implementation of the teaching strategy and focuses on program fidelity as needed to meet DOL exemption Lawver et al., Impact of Professional Development … Journal of Agricultural Education 325 Volume 65, Issue 1, 2024 requirements (Kennedy, 2016b). Kennedy noted that this approach could backfire if professional development is limited in addressing teachers’ challenges. Kennedy (2016b) also warned that from a teacher’s perspective, the educational system creates a variety of educational requirements and establishes conflicting curriculum priorities on them. Sustaining the implementation effort by teachers could prove daunting as teachers maintain district and state requirements for their students. Purpose The purpose of this study was to determine the effect of implementing a professional development program over multiple years on sustaining teachers’ knowledge of safe tractor and machinery operations and the motivations for attending. The research objectives guiding the study were: 1. Describe selected demographic characteristics of school-based agricultural education teachers who participated in a multi-year agricultural safety professional development. 2. Determine if differences in participants’ average age and teacher experience occur when attending a multi-year professional development. 3. Determine the effect of sustained teacher participation in professional development on teachers’ knowledge of safe tractor and machinery operation. 4. Describe motivational factors for school-based agriculture teachers to continue participation in a multi-year professional development focused on tractor and machinery safe operation. Methodology Participants During the summer of 2019, a convenience sample of secondary agricultural educators, who had participated in a hands-on agricultural safety training experience from Montana, South Dakota, and Utah was recruited for participation in this study. Maximum enrollment was set at 50 participants per state. Due to the convenience sampling which violates the assumptions of parametric statistical tests, non-parametric tests were used to check for statistical significance. Methods Each state’s training seminar occurred separately; however, the content and delivery were uniform following learning activities organized so that teachers participated together through a series of hands-on exercises (Pate et al., 2019). Teacher educators from each state represented in the study served as trainers. Trainers met face-to-face with the lead researcher to establish the lesson plan and format of the workshop and were all trained. Each workshop was provided to the teachers in June after the school year. Incentives primarily comprised of safety materials and supplies, as well as flash drives loaded with workshop curriculum. Additional incentives included professional development credit towards licensure and safety educational resources for students. The human subject research protocol was reviewed and approved under Utah State University’s Institutional Review Board protocol 10514. An Institutional Review Board reliance agreement was established and approved between Montana State University and Iowa State University with Utah State University as the institution of record. Informed consent forms were provided to teachers. The curriculum used during the 2019 training focused on the safe operation of equipment and tractors. The training seminar format followed the 2017 seminar procedures described by Pate et al. (2019). A lesson plan was developed that included large group activities and a rotation between small group hands- on station modules. Teachers performed tabletop exercises focused on the backing of tractors and the hitching of equipment. Tabletop exercises are a common instructional strategy to provide a scaled proxy for emergency preparedness and public health events (Savoia et al., 2009). This strategy affords teachers a highly effective strategy to address student training limitations such as engaging many participants (>30) Lawver et al., Impact of Professional Development … Journal of Agricultural Education 326 Volume 65, Issue 1, 2024 with limited availability of agricultural tractors and machinery. These tabletop exercises incorporated microsimulations of tractor rollovers, obstacle course setup, and hitching to provide educators with practical hands-on experiences to model with their students (Wojtowicz & Wallace, 2010). Teachers practiced using and recognizing the American Society for Agricultural and Biological Engineers standardized hand signals to communicate with operators. Teachers were given demonstrations on backing equipment and connecting agricultural implements, followed by hands-on practice using the equipment. Teachers were provided instructions on how to lay out a driving course for tractor safety and then practiced implementing the strategy. Teachers were required to incorporate NSTMOP tractor operations course layout elements of navigating around three obstacles using a serpentine maneuver and backing a trailer into a designated area without striking barriers. Course layout was contextualized around the equipment dimensions to estimate the distance needed between obstacles and course barriers to safely complete the activity at their schools. Teachers completed the seminar by driving a utility tractor through the obstacle course, which was created as part of the training. Each teacher had the chance to complete the obstacle course and provide one another with hand signals. Instrument After the training, participants completed a paper-based exam constructed of NSTMOP knowledge items, which had reliability established previously with youth. This exam also collected demographic questions and questions on motivation for returning to the training seminar. Knowledge questions were focused on safe tractor operation, machinery safety, and general health and safety. The KR-20 alpha was used to measure internal consistency reliability for measurers with dichotomous choices. KR-20 alpha was used due to the dichotomous answer scoring (0 = incorrect, 1 = correct). The exam items were tested with 1,400 youth ages 14-18 years old, the KR-20 alpha for the instrument was .89 (Smalley et al., 2022). For post-hoc reliability with the current sample of teachers KR-20 alpha was .54. For teacher-made tests, acceptable score reliability averages a KR-20 value of .50 (Frisbie, 1988). Factors that have been noted to affect reliability estimates include test content, group heterogeneity, item difficulty, and item discrimination (Frisbie, 1988). A factor we noted that may have contributed to the low reliability score was the multidimensional content assessed by the items which were pulled from each of the seminar’s safety lessons (Tractor Stability, ATV/UTV, and Safe Operation of Equipment). Tests that measure multiple content areas tend to yield lower reliability scores (Frisbie, 1988). Additionally, this group of teachers were similar in expertise resulting in a rather homogenous group. Frisbie (1988) noted when groups are very homogeneous it is more difficult to achieve a spread of scores making detecting interindividual differences difficult. Item analysis was conducted to identify specific exam items of concern with regard to item discrimination using point-biserial correlations and the item difficulty index, which are provided in Table 1. For items with non-significant correlations below .20, an average of 10 individuals answered incorrectly. One individual did not answer seven items which were scored as incorrect. The average item difficulty index was .84 which met credibility for items used in a certification exam (Crocker & Algina, 1986). Twenty-one items had significant correlations at or above .20, which indicates good discrimination between the upper and lower thirds of the participants (McGahee & Ball, 2009). Based on the literature and prior reliability scores, we concluded that the instrument was reliable. Lawver et al., Impact of Professional Development … Journal of Agricultural Education 327 Volume 65, Issue 1, 2024 Table 1 Safety Exam Item Discrimination and Difficulty Index Item Point Biserial Correlation Item Difficulty Index In order to prevent falls when mounting the tractor, you should have at least ____ body part(s) in contact with the tractor at one time. .177 .88 What is the purpose of personal protective equipment? .120 .99 The “point of no return” for a rear tractor overturn is reached in how many seconds? .122 .76 If a ditch is 6 feet deep, how far away should you keep the tractor from the embankment? .052 .87 Heavy draft loads (i.e. tillage equipment) should be attached which of the following. .453** .62 When working on a skid steer loader with the bucket in the raised position, the following safety practice is expected of all workers: .329** .85 According to the North American Guidelines for Children’s Agricultural Tasks (NAGCAT), what is the recommended minimum age for operating a PTO-powered implement? .369** .48 How would you describe work on the farm? .476** .74 What percent of tractor-related fatalities are a result from tractor overturns? .104 .82 Start a tractor engine with the: .200 .93 When releasing a two-pedal direction and speed control, what position should you return it? .173 .82 Before starting your tractor you should: .210 .96 To stop a diesel engine: .302** .75 Rear tires of older tractors may contain a commonly used corrosive liquid in the inner tube to add weight to the tractor to improve its traction. What is the liquid? .207 .88 If a mechanical push-pull fuel switch is used, where should this switch be located? .233* .82 To prevent runaways when parked with heavy tractor loads, you should: .207 .88 What may happen if you crank the tractor’s starter motor too long? .019 .84 To prevent heat-related illness, you should: .275* .95 What information about your tractor engine is shown in this picture? (RPM) .237* .87 Throttle controls next to the tractor seat increase engine speed when moved: .324** .75 The letters “ROPS” stand for: .309** .88 Which of the following increases the chance of a run-over? .228* .87 Which of the following scenarios is NOT a designed use of a farm tractor? .120 .87 When using wheel-type tractors on silage surfaces, do NOT use with slopes greater than: .348** .85 When operating a high-lift bucket, where should you keep the bucket while the tractor is in motion? .110 .93 To prevent untrained operators, children, and visitors from accidentally starting the tractor you should: .186 .89 Lawver et al., Impact of Professional Development … Journal of Agricultural Education 328 Volume 65, Issue 1, 2024 Table 1 Safety Exam Item Discrimination and Difficulty Index Item Point Biserial Correlation Item Difficulty Index Working as a non-family member farm employee, youth who are younger than 16 or older can fell trees with a butt diameter up to: .319** .84 Nationally, what fraction of all farm work fatalities are tractor- related? .104 .56 According to the North American Guidelines for Children’s Agricultural Tasks (NAGCAT), which age group should not operate a medium/large tractor (more than 70hp) .370** .71 The ROPS on a tractor: .305** .92 A safe work site should include which of the following: .182 .95 Rather than an occupation, farming is commonly viewed as: .227* .78 A factor(s) that affect(s) your reaction time is/are: -.154 .98 What has to exist in order for OSHA to apply to a business or operation? .353** .78 You should avoid driving an ATV on: .352** .80 This pictorial warns you about which of the following potential hazards: (run over) .059 .93 If you raise your arm vertically overhead (palm to the front) and rotate it in large horizontal circles, what hand signal are you using? (come to me) .261* .60 What personal protective equipment is recommended for ATV driving? -.003 .96 Identify this public road hand signal. (stop) .123 .93 Farm shops need adequate: -.004 .94 What unit is used to measure sound? .012 .95 Identify this public road hand signal. (left turn) .015 .96 If someone draws his/her right-hand palm down across his/her neck in a throat-cutting motion from left to right, what should you do? (stop engine) .079 .94 What does an arm extended horizontally sideward with palm down, waving a downward signal? (slow down) .010 .94 Identify this public road hand signal. (left turn) .038 .94 What does this symbol indicate? (alert) .042 .87 Which of the following are ground-motion controls and should be orange color-coded? .140 .78 Lifting heavy loads with the skid steer bucket can result in the center of gravity: .347** .75 PTO controls are designed to move rearward or downward to: .341** .60 Engine speed controls are operated with which of the following? .074 .75 * p < . 05; ** p < .01 Data Analysis Descriptive statistics including frequencies, percentages, means, and standard deviations were used to report teacher performance, demographic data, and participation status. To accomplish research objective two, a Kruskal-Wallis test was used to determine if there was a significant difference in knowledge of safe tractor and machinery operations between teachers participating in multiple years of the professional Lawver et al., Impact of Professional Development … Journal of Agricultural Education 329 Volume 65, Issue 1, 2024 development and teachers who participated less frequently. To address research objective three, researchers used an open-ended item to assess participants’ motivation for returning to the training seminar. Respondents were asked to indicate what would keep them returning to the training. Open responses were coded based on priori established themes developed by the researchers using the conceptual framework established regarding teacher professional development (L. M. Desimone, 2009; Kennedy, 2016b). Researchers coded responses as 1 = knowledge and 2 = curriculum. Data was initially compiled in Microsoft Excel and then analyzed in SPSS version 21. Findings/Results A total of 85 teachers participated in year three of the training program. Table 2 provides the distribution of teachers from each state. Over half (57.6%, f = 49) of the participants identified as “female.” Only two individuals failed the exam. These individuals were self-identified as female. The chi-square test of association was used to determine if there was a significant association between first-year attendees and multi-year attendees. There was no significant association between years of attendance and gender (𝜒𝜒2 (2) = 2.98, p = .084). The average age of participants was 35.0 years (SD = 12.39). Participants were asked how many times, including the current year participation, had they participated in the training program. Most participants (f = 31, 36.5%) had participated in the training at least twice. There were no teachers from [State] that had attended the training for all three offerings. Table 3 provides the distribution of teachers’ participation experience in the training program. Table 2 Demographics of Professional Development Participants f % Distribution of teachers by State Montana 32 37.6 South Dakota 33 38.8 Utah 20 23.5 Gender Female 49 57.6 Male 36 42.4 Years of Experience Beginning (1-5) 33 38.8 Mid Career (6-15) 33 38.8 Veteran (16 plus) 19 22.4 Age Young Adult (21-29) 38 46.9 Middle Aged Adult 30-39) 20 24.7 Older Adult 23 28.4 Lawver et al., Impact of Professional Development … Journal of Agricultural Education 330 Volume 65, Issue 1, 2024 For years of experience teaching, five participants had 30 or more years of experience teaching. There were 17 (20.0%) participants with less than or equal to one year of experience teaching. For all participants, the median teaching experience in years was 7 (IQR = 13). First-year attendees’ average age was 33.1 years (SD = 2.66) with an average of 8.17 years of teaching experience (SD = 2.23). Second-year attendees’ average age was 32.9 years (SD = 1.84) with an average of 7.7 years of teaching experience (SD = 1.30). Third-year attendees’ average age was 39.1 (SD = 2.60) with an average of 15.4 years of teaching experience (SD = 2.26). Kruskal-Wallis test was used to check for significant differences between first, second-, and third-year attendees on age (𝜒𝜒2 (2) = 6.38, p = .041) and years of teaching experience (𝜒𝜒2 (2) = 10.15, p = .006). Teaching experience was collapsed as an ordinal variable and renamed “Teacher Life Cycle Stage” with 1-5 years of teaching experience classified as a beginning teacher, 6-15 years as a mid-career, and 16 or more years as a veteran. Most teachers participating were classified as either beginning (f = 33, 38.8%) or mid-career (f = 33, 38.8%). There were 19 participants (22.4%) who were classified as veteran teachers. Age was recoded into an ordinal variable and renamed “age category” with 21-29 as “young adult”, 30-39 as “middle-aged adult”, and ≥ 40 as “older adult.” The average test score was 41.9 (SD = 3.62) out of 50. Table 4 provides mean scores by classification of attendee. Third-year attendees scored an average of 43.2 (SD = 3.00). Participants passed by correctly answering greater than 70% of the questions. Only two individuals (8.0%) failed the exam. To determine the effect of sustained teacher participation on teachers’ knowledge of safe tractor and machinery operation, a Kruskal-Wallis H test was used because the Levene’s Test for equal variance was significant (F(2,82) = 4.0, p = .02) indicating a violation of the ANOVA assumption of homogeneity of variance. The result of the Kruskal-Wallis H(3) = 5.91, p = .052. The difference was statistically non-significant. Table 4 Test Score Averages by Teacher Attendance Category Attendance Category Test Score M SD First Time Attendee 40.8 4.41 Second Year Attendee 41.7 3.16 Third Year Attendee 43.2 3.00 A Kruskal-Wallis test was used to determine if there were significant differences in safety knowledge between teacher life cycle stages (𝜒𝜒2 (2) = 2.110, p = .348). Another Kruskal-Wallis test was used to determine if there were significant differences in safety knowledge between teacher age categories Table 3 Distribution of Teacher Attendee Category by State Montana South Dakota Utah Attendance f % f % f % First time attending 8 25.0 8 24.2 9 45.0 Second time attending 12 37.5 8 24.2 11 55.0 Third time attending 12 37.5 17 51.5 - - Lawver et al., Impact of Professional Development … Journal of Agricultural Education 331 Volume 65, Issue 1, 2024 (𝜒𝜒2 (2) = 2.189, p = .335). The difference was non-significant. A Fisher’s exact test was used to determine if teacher gender was significantly associated with a passing score. Fisher’s exact test results revealed no significant association between gender and passing score (p = .506). There were no significant differences in safety knowledge between teacher age categories or teacher life cycle stages. When participants were asked on the post-experience NSTMOP exam what attracted them to attend the training experience, 31.6% (f = 12) reported knowledge acquisition while 68.4% (f = 26) reported curriculum obtainment. Other responses included incentives such as PD credit, gift cards, or food (f = 9, 10.6%) and fun (f = 3, 3.5%). There were 13 participants (15.9%) who responded with an affirmative “yes” but did not indicate factors that would bring them back to the training. Three participants noted their return was dependent on the scheduling of other professional development. Table 5 provides frequencies and percentages of attendees’ justification for returning to the training. Conclusions/Recommendations/Implications The purpose of this study was to determine the effect of implementing a professional development program over multiple years on sustaining teachers’ knowledge of safe tractor and machinery operations and the motivations for attending. Limitations should be noted which prevents the generalization of these results beyond the participants of this study. There was no pre-test to determine any preexisting differences in knowledge between first-year attendees and those who had attended previous training. Future research should include a pre/posttest design to reduce the influence of extraneous variables and isolate the impact of the training program. However, it was assumed that multiple-year attendees would have higher knowledge due to their experience with the program. Based on Darling-Hammond et al. (2017) the duration of the professional development was noted as an element of effective professional development. The element of duration focuses on professional development delivered over weeks, months, and years versus one-off workshops. We examined differences in age and teaching experience between first-year and multi- year attendees. This was examined to see if age/teacher experiences could cause for additional investigation to see if educators could have been exposed to different experiences, professional development, or curriculum during their preparation. There was no significant association between gender and frequency of training attendance. Only two individuals failed the exam. These individuals were self-identified as female. We conclude that age and teaching experience did not influence safety knowledge gains seen in the post- test results. Based on the findings, all participants viewed the training favorably. Third-year attendees were more likely to be older and have more teaching experience. While third-year attendees averaged a higher score, the result was not statistically significant. Based on Steffy and Wolfe (2001) a teacher will continue to grow and develop throughout their professional life. Due to the convenience sampling which violates the assumptions of parametric statistical tests, non-parametric tests were used to check for statistical significance. The Kruskal-Wallis test is noted to have low power which requires a larger sample size to detect a statistically significance difference in test scores. We note the p-value approached the .05 alpha level but conclude that a larger sample size is needed to determine if this effect is sustained. Based on these Table 5 Frequencies and Percentages of Attendees’ Justification for Returning to the Training Knowledge Acquisition Curriculum Obtainment Attendance f % f % First time 1 16.7 5 83.3 Second time 8 42.1 11 57.9 Third time 3 23.1 10 76.9 Lawver et al., Impact of Professional Development … Journal of Agricultural Education 332 Volume 65, Issue 1, 2024 results, we recommend continued and sustained professional development for these participating teachers to facilitate gains in safety knowledge and curriculum updates. This is in line with Stripling and Rickets’s (2016) suggestions for linking existing knowledge to new concepts through the use of multiple and varied representations of concepts and tasks as well as providing engagement through challenging tasks under supportive guidance. For first-year participants, a “one and done” approach to professional development targeting teachers’ safety training is not encouraged if the goal is to facilitate gains in safety knowledge and curriculum updates. Continued research is needed to determine effective methods for addressing safety knowledge over a teacher’s life cycle. This will be critical to addressing stakeholder concerns related to the efficacy of teacher safety preparation to ensure the safety of students engaged in school-based agricultural education (NIOSH, 2002). This research suggests that instructors are more likely to attend sustained professional development activities over their teaching career if there are incentives. It should be noted that 68.4% of attendees acknowledged curriculum obtainment as a motivational factor for attending the workshop in the future. A qualitative approach to understanding teachers’ motivation and interest in safety training professional development is recommended to describe the documented variation in safety programming and student training among school-based agriculture education teachers (Lawver et al., 2016; Mann & Jepsen, 2019; Pate et al., 2016). Teacher educators should note from this research that active engagement is a desired feature of professional development among these participating teachers. A benefit for participating teachers in this professional development was focused on higher-order instructional and alternative assessment methods for tractor and machinery safety. This is consistent with Desimone et al. (2002) findings that professional development characterized by “active learning,” where teachers are not passive “recipients” of information increases the impact of the professional development activities. Reconnecting to our conceptual framework, the results of this study are similar to Desimone et al. (2002) finding that teachers benefit from participation in professional development which focuses on higher-order instructional or alternative assessment methods to impact student learning. Imel (2000) suggested: Adult educators frequently act as change agents, although they may not be conscious that they are playing this role. Like learning, change is a complex process, and understanding the relationship between learning and the change process can help adult educators be more purposeful in assisting with change. (p. 5) Regarding Program Evaluation Theory, the teachers’ training intends for the creation and implementation of a safety curriculum with their students. Labor exemptions provided to school-based agricultural education programs require further validation through additional data collection to determine how participating teachers implement the tractor and machinery curriculum acquired from participating in the workshop. This must be assessed by examining how SBAE students’ safety behaviors change and the reduction of injuries. This will address a concern that Kennedy (2016b) identified regarding implementation barriers such as conflicting curriculum priorities that would limit teachers’ use of the safety curriculum. Teachers must maintain district and state requirements for their students which could place tractor and machinery safety at a lower priority. Future research efforts among professional development specialists should investigate teacher demands and potential areas to facilitate the integration of student safety training within existing curriculum structures used by teachers. References Agriculture Coalition. (2011). Public Submission Wage and Hour Division. Department of Labor. https://www.regulations.gov/comment/WHD-2011-0001-9175 Barrick, R., Arrington, L., Heffernan, T., Hughes, M., Moody, L., Ogline, P., & Whaley, D. (1992). Experiencing agriculture: A handbook on supervised agricultural experience. Alexandria, VA: Lawver et al., Impact of Professional Development … Journal of Agricultural Education 333 Volume 65, Issue 1, 2024 National Council for Agricultural Education. Brousselle, A., & Champagne, F. (2011). Program theory evaluation: Logic analysis. Evaluation and Program Planning, 34(1), 69–78. https://doi.org/10.1016/j.evalprogplan.2010.04.001 Burke, M. J., Sarpy, S. A., Smith-Crowe, K., Chan-Serafin, S., Salvador, R. O., & Islam, G. (2006). Relative effectiveness of worker safety and health training methods. American Journal of Public Health, 96(2), 315–324. https://doi.org/10.2105/AJPH.2004.059840 Bush, D., & Andrews, K. (2013). Integrating occupational safety and health training into career technical education in construction. University of California Berkeley Labor Occupational Health Program and CPWR‐The Center for Construction Research and Training. https://www.cpwr.com/wp- content/uploads/publications/Integrating_SandH_in_CTE_Training_Bush_2013.pdf Cooksy, L. J., Gill, P., & Kelly, P. A. (2001). The program logic model as an integrative framework for a multimethod evaluation. Evaluation and Program Planning, 24(2), 119–128. https://doi.org/10.1016/S0149-7189(01)00003-9 Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. ERIC. Darling-Hammond, L., Hyler, M.E., & Gardner, M. (2017). Effective Teacher Professional Development. Learning Policy Institute Research Brief. https://learningpolicyinsitutte.org/sites/default/files/product- files/Effective_Teacher_Professional_Development_BRIEF.pdf Desimone, L. M. (2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181–199. Desimone, L., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of professional development on teachers’ instruction: Results from a three-year longitudinal study. Educational Evaluation and Policy Analysis, 24(2), 81–112. https://doi.org/10.3102/01623737024002081 Desimone, L., & Stuckey, D. (2014). Sustaining teacher professional development. In L. E. Martin, S. Kragler, D. J. Quatroche, & K. L. Bauserman (Eds.), Handbook of Professional Development in Education: Successful Models and Practices, PreK-12 (pp. 467–482). Guilford Publications. Frisbie, D. A. (1988). Reliability of scores from teacher‐made tests. Educational Measurement: Issues and Practice, 7(1), 25–35. Guskey, T. R. (2002). Professional development and teacher change. Teachers and Teaching, 8(3), 381– 391. https://doi.org/10.1080/135406002100000512 Imel, S. (2000). Change: Connections to adult learning and education. ERIC Digest No. 221. For full text: http://www. https://eric.ed.gov/?id=ED446252 Jepsen, S. D. (2012). The U.S. department of labor’s tractor and machinery certification program: Management styles and perceptions held by community stakeholders and instructors. Journal of Agricultural Safety and Health, 18(3), 217–232. https://doi.org/10.13031/2013.41958 Johnson, C.C. (2006). Effective professional development and change in practice: Barriers teacher Lawver et al., Impact of Professional Development … Journal of Agricultural Education 334 Volume 65, Issue 1, 2024 encounter and implications for reform. School Science and Mathematics, 106, 1-12. Kennedy, M. (2016a). Parsing the practice of teaching. Journal of Teacher Education, 67(1), 6–17. https://doi.org/10.1177/0022487115614617 Kennedy, M. (2016b). How does professional development improve teaching? Review of Educational Research, 86(4), 945–980. https://doi.org/10.3102/0034654315626800 L. Sanderson, L., R. Dukeshire, S., Rangel, C., & Garbes, R. (2010). The farm apprentice: agricultural college students recollections of learning to farm “safely.” Journal of Agricultural Safety and Health, 16(4), 229–247. https://doi.org/10.13031/2013.34835 Lawver, R. G., Pate, M. L., & Sorensen, T. J. (2016). Supervisor involvement and professional development needs associated with SAE programming and safety. Journal of Agricultural Education, 57(1), 150–166. https://doi.org/10.5032/jae.2016.01150 Loucks-Horsley, S., Love, N., Stiles, K.E., Mundry, S., & Hewson, P.W. (2003). Designing professional development for teachers of mathematics and science. Thousand Oaks, CA: Corwin Press. Mann, A. J., & Jepsen, S. D. (2019). Hazardous agricultural tasks completed by youth as part of their supervised agricultural experience (SAE): A descriptive study. Journal of Agricultural Safety and Health, 25(3), 107–116. https://doi.org/10.13031/jash.12998 McGahee, T. W., & Ball, J. (2009). How to read and really use an item analysis. Nurse Educator, 34(4). https://journals.lww.com/nurseeducatoronline/Fulltext/2009/07000/How_to_Read_and_Really_U se_an_Item_Analysis.12.aspx Murphy, D. J. (1992). Safety and health for production agriculture. ASAE. Myers, B. E., & Roberts, T. G. (2004). Conducting and evaluating professional development workshops using experiential learning. NACTA Journal, 48(2), 27–32. JSTOR. National Institute for Occupational Safety and Health [NIOSH]. (2014). Childhood agricultural injury survey (CAIS) results. National Institute for Occupational Safety and Health. https://www.cdc.gov/niosh/topics/childag/cais/default.html NIOSH. (2002). National institute for occupational safety and health (NIOSH) recommendations to the U.S. department of labor for changes to hazardous orders (pp. 1–176). National Institute for Occupational Safety and Health. https://www.cdc.gov/niosh/docs/nioshrecsdolhaz/default.html Occupational Safety and Health Administration [OSHA]. (n.d.). OSHA Outreach Training Program— Authorized Trainer | Occupational Safety and Health Administration [Government]. How to become an authorized trainer. Retrieved December 15, 2020, from https://www.osha.gov/dte/outreach/authorized.html Pate, M. L., Lawver, R. G., Smalley, S. W., Perry, D. K., Stallones, L., & Shultz, A. (2019). Agricultural safety education: Formative assessment of a curriculum integration strategy. Journal of Agricultural Safety and Health, 25(2), 63–76. https://doi.org/10.13031/jash.13113 Pate, M. L., Lawver, R. G., & Sorensen, T. J. (2016). Safety in high school supervised agricultural experiences: Teachers’ training and students’ injury awareness. Journal of Agricultural Safety and Lawver et al., Impact of Professional Development … Journal of Agricultural Education 335 Volume 65, Issue 1, 2024 Health, 22(1), 61–74. https://doi.org/10.13031/jash.22.11142 Rasty, J., Anderson, R. G., & Paulsen, T. H. (2017). How the quantity of agricultural mechanics training received at the secondary level impact teacher perceived importance of agricultural mechanics skills. Journal of Agricultural Education, 58(1), 36–53. https://doi.org/10.5032/jae.2017.01036 Richardson, V. (2001). Handbook of Research on Teaching (4th ed.). American Educational Research Association. Richardson, V., & Placier, P. (2001). Teacher change. In Handbook of Research on Teaching (4th ed., p. 1278). American Educational Research Association. Rogers, P. J., Petrosino, A., Huebner, T. A., & Hacsi, T. A. (2000). Program theory evaluation: Practice, promise, and problems. New Directions for Evaluation, 2000(87), 5–13. https://doi.org/10.1002/ev.1177 Savoia, E., Testa, M. A., Biddinger, P. D., Cadigan, R. O., Koh, H., Campbell, P., & Stoto, M. A. (2009). Assessing public health capabilities during emergency preparedness tabletop exercises: Reliability and validity of a measurement tool. Public Health Reports, 124(1), 138-148. https://doi.org/10.1177/00333549091240011 Schulte, P. A., Stephenson, C. M., Okun, A. H., Palassis, J., & Biddle, E. (2005). Integrating occupational safety and health information into vocational and technical education and other workforce preparation programs. American Journal of Public Health, 95(3), 404–411. https://doi.org/10.2105/AJPH.2004.047241 Schwebel, D. C., & Pickett, W. (2012). The role of child and adolescent development in the occurrence of agricultural injuries: An illustration using tractor-related injuries. Journal of Agromedicine, 17(2), 214–224. https://doi.org/10.1080/1059924X.2012.655120 Smalley, S. W., Perry, D. K., Shultz, A., Lawver, R. G., Pate, M. L., Hanagriff, R., & Ewell, C. (2022). Assessing youth safety knowledge with the agricultural experience tracker (AET). Journal of Agricultural Safety and Health, 28(2), 87–98. https://doi.org/10.13031/jash.14801 Steffy, B. E.& Wolfe, M.P. (2001) A Life-Cycle Model for Career Teachers, Kappa Delta Pi Record, 38:1, 16-19, doi:10.1080/00228958.2001.10518508 Stripling, C., & Ricketts, J. (2016). Sufficient scientific and professional workforce that addresses the challenges of the 21st century. In T. G. Roberts, A. Harder, & M. T. Brashears (Eds.), American Association for Agricultural Education National Research Agenda: 2016-2020 (pp. 30–33). Department of Agricultural Education and Communication. http://aaaeonline.org/resources/Documents/AAAE_National_Research_Agenda_2016-2020.pdf Supovitz, J.A., & Turner, H.M. (2000). The effects of professional development on science teaching practices and classroom culture. Journal of Research in Science Teaching, 37, 963-980. U.S. Department of Labor. (n.d.). Youth in Agriculture—agriculture eTool. Retrieved December 9, 2020, from https://www.osha.gov/SLTC/youth/agriculture/index.html Wage and Hour Division. (2012, April 26). Labor Department statement on withdrawal of proposed rule dealing with children who work in agricultural vocations | U.S. Department of Labor. Lawver et al., Impact of Professional Development … Journal of Agricultural Education 336 Volume 65, Issue 1, 2024 https://www.dol.gov/newsroom/releases/whd/whd20120426-0 Wage and Hour Division. (2016). Child labor bulletin 102: Child labor requirements in agricultural occupations under the fair labor standards act. Department of Labor. https://www.dol.gov/agencies/whd/child-labor/agriculture Wage and Hour Division. (2011). 76 Fed. Reg. 171. Child Labor Regulations, Orders and Statements of Interpretation; Child Labor Violations—Civil Money Penalties (Proposed Rules RIN 123-AA06; pp. 1–211). https://www.govinfo.gov/content/pkg/FR-2011-09-02/pdf/2011-21924.pdf#page=2 Weiss, C. (1998). Evaluation: Methods for studying programs and policies (2nd ed.). Prentice-Hall. Weiss, C. H. (1972). Evaluation research: Methods for assessing program effectiveness. Prentice-Hall. Wholey, J. S. (1995). Assessing the feasibility and likely usefulness of evaluation. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation. Jossey-Bass. W.K. Kellogg Foundation. (2004). Logic model development guide. W.K. Kellogg Foundation. https://www.wkkf.org:443/resource-directory/resources/2004/01/logic-model-development-guide Wojtowicz, J., & Wallace, W. A. (2010). Traffic Management for Planned Special Events Using Traffic Microsimulation Modeling and Tabletop Exercises, Journal of Transportation Safety & Security, 2(2), 102-121, doi: 10.1080/19439962.2010.487635 Conceptual Framework Tractor and Machinery Safety Professional Development Model Purpose Methodology Participants Methods Instrument Data Analysis Findings/Results Conclusions/Recommendations/Implications