FLATHEAD WATERSHED CURRICULUM DEVELOPMENT: KNOWLEDGE, SKILLS, AND DISPOSITION RESULTS FROM THE FLATHEAD WATERSHED DELPHI SURVEY by Rosanna Rohrs Vallor A thesis presented in partial fulfillment of the requirements for the degree of Master of Education in Education MONTANA STATE UNIVERSITY Bozeman, Montana April 2016 ©COPYRIGHT By Rosanna Rohrs Vallor 2016 All Rights Reserved ii ACKNOWLEDGEMENTS I would like to acknowledge and thank my advisor, Dr. Michael Brody, for all his thoughts, guidance and insights that helped bring this work to completion. I would like to thank my committee members, Dr. Lynn Kelting-Gibson and Dr. Sarah Schmitt Wilson, for their positive attitudes, suggestions and support through this process. I would like to thank Dr. Kimberly Yates for her work on the Delphi survey and Amanda Obery for her thoughts about the Flathead Watershed curriculum; I appreciated the opportunity to work with two great collaborators. Finally, I would like to deeply thank and acknowledge the unquestioning support from my family, Greg and Fiona, who dealt with this work with grace in every way. iii TABLE OF CONTENTS 1: INTRODUCTION .......................................................................................................... 1 The Flathead Watershed Delphi Survey ......................................................................... 1 The Flathead Watershed ................................................................................................. 1 People in the Flathead Watershed ................................................................................... 4 The Flathead Watershed Sourcebook ............................................................................. 8 The Flathead Watershed Delphi Survey ......................................................................... 9 Place-based Education in the Flathead Watershed ....................................................... 10 Flathead Watershed Curriculum Development ............................................................. 11 Research Questions ....................................................................................................... 11 Purpose of the Study ..................................................................................................... 12 Definition of Terms....................................................................................................... 12 Assumptions .................................................................................................................. 14 Limitations .................................................................................................................... 15 Significance of the Study .............................................................................................. 16 Expected Outcomes ...................................................................................................... 16 2: LITERATURE REVIEW ............................................................................................. 17 The Watershed: Services and Management .................................................................. 17 Watershed Curricula, An Overview .............................................................................. 20 Place-Based Education and Community in Curriculum Development......................... 22 The Use of Delphi Surveys to Develop and Inform Curriculum .................................. 25 History and Nature of the Delphi Survey ..................................................................... 27 The Three Domains of Education ................................................................................. 31 3: METHODOLOGY ....................................................................................................... 34 Research Method and Design Appropriateness ............................................................ 34 Validity and Reliability ................................................................................................. 34 Participant Selection ..................................................................................................... 37 Round One. ..................................................................................................................... 41 Round Two ..................................................................................................................... 43 Round Three ................................................................................................................... 45 4: RESULTS ..................................................................................................................... 46 Flathead Delphi Statement Content Analysis ............................................................... 46 Research Question One ................................................................................................... 46 Research Question Two .................................................................................................. 51 Research Question Three ................................................................................................ 54 iv TABLE OF CONTENTS - CONTINUED Research Question Four .................................................................................................. 58 Research Question Five................................................................................................... 60 5: DISCUSSION ............................................................................................................... 62 Results Discussion ........................................................................................................ 62 Methodology Discussion .............................................................................................. 67 Conclusion .................................................................................................................... 70 REFERENCES ................................................................................................................. 72 APPENDICES .................................................................................................................. 78 APPENDIX A: Delphi Survey Participants .................................................................. 79 APPENDIX B: Round One Sample Responses ............................................................ 82 APPENDIX C: Round Two Ranked Statements .......................................................... 85 APPENDIX D: Round Three Knowledge Statements ................................................ 101 APPENDIX E: Round Three Skill Statements ........................................................... 106 APPENDIX F: Round Three Disposition Statements ................................................ 109 APPENDIX G: Round Three Teaching Statements ................................................... 113 v LIST OF TABLES Table Page 1. Comparison of Number of Statements from Round 2 to Round 3 ................... 37 2. Number and Category of Participants in the three Delphi Rounds ................... 40 3. Example of Participant Responses for Round One, Disposition Prompt .......... 42 4. Example of a Round Two Statement and Comments ....................................... 44 5. Example of the Sourcebook Delphi Statement Matrix ..................................... 61 vi LIST OF FIGURES Figure Page 1. Outline and Regional Location of the Flathead Watershed .............................. 2 2. Land Ownership Patterns in the Flathead Watershed ....................................... 6 3. Literature Review Concept Map .................................................................... 18 4. Knowledge Statement Means with +1 and -1 Standard Deviation................. 47 5. The Three Themes in the Knowledge Statements .......................................... 49 6. Skill Statements Means with +1 and -1 Standard Deviation. ......................... 51 7. The Four Themes in the Skill Statements ....................................................... 52 8. Disposition Statement Means with +1 and -1 Standard Deviation................. 55 9. The Two Themes Within the Disposition Statements .................................... 56 10. Teaching Statement Means with +1 and -1 Standard Deviation .................. 58 vii ABSTRACT This research focused on the establishment of foundational knowledge, skills and dispositions for the Flathead Watershed Educators Guide, a place-based watershed curriculum for middle school grades based on the Flathead Watershed Sourcebook. The methodology of this research was the consensus building process known as the Delphi survey. Survey participants (n=33) were chosen based on their expertise as educators, resource managers and scientists living and practicing in the Flathead Watershed in northwestern Montana, USA. Participants’ responses were gathered through the three- round survey by the Montana State University (MSU) research team using MSU’s online software program Desire 2 Learn (D2L), an anonymous, asynchronous platform with distance accessibility. Round One responses were gathered through the D2L discussion function so that participants could read each other’s responses and reply if desired, allowing an exchange and development of ideas. Round One discussion responses were formatted into statements, which were then made available to the participants to rate through two successive rounds using 1-5 Likert scales. Of the initial 142 statements, 91 statements were retained in the final round. The 91 final statements were matched to Flathead Watershed Sourcebook contents to identify the learning objectives for the Flathead Watershed Educators Guide. The final statements listed the knowledge, skills, and dispositions survey participants felt were most important for students in the Flathead Watershed to learn. Statements showed concern for conservation and protection of the natural environment through place-based watershed education. 1 CHAPTER 1: INTRODUCTION The Flathead Watershed Delphi Survey This thesis examines the Flathead Watershed Delphi survey responses that form the foundation for development of the place-based watershed curriculum called the Flathead Watershed Educators Guide, based on the Flathead Sourcebook. This research study sought to establish through consensus the knowledge, skills and dispositions necessary for a conservation-based curriculum. Conservation and education-minded community members living in the Flathead Watershed of northwestern Montana, USA, worked with education professionals in the Department of Education at Montana State University who designed and instigated the Flathead Delphi survey. Thesis results show survey responses in the foundational curriculum areas indicated strong consensus around place-based educational principles valuing community awareness and service, ecosystem conservation and environmental stewardship. The Flathead Watershed The Flathead Watershed has its headwaters in the pristine regions of Glacier National Park, the Great Bear Wilderness Area and the Bob Marshall Wilderness Area. The watershed of the Flathead River extends throughout a significant portion of northwestern Montana, with one of its tributaries, the North Fork of the Flathead River, originating in the Canadian province of British Columbia. The North, Middle and South Forks of the Flathead River join with the Stillwater, Swan and Whitefish Rivers to feed 2 Flathead Lake, at 191.5 sq. miles one of the largest freshwater lakes in the Western United States. Flowing out of Flathead Lake, the lower Flathead River joins the Clark Fork River, after which its waters flow west into the Columbia River, forming the boundary between the states of Washington and Oregon, and eventually into the Pacific Ocean (Curtis, 2010). See Figure 1 for an illustration of the extent and location of the Flathead Watershed in the northwestern United States. Figure 1. Outline and Regional Location of the Flathead Watershed 3 The climate of the Flathead Watershed is a major influence determining ecosystem composition. The Flathead Watershed is located just west of the Continental Divide. Pacific maritime air masses moving east often meet up with southerly trending arctic continental conditions, creating the opportunity for wide weather fluctuations. The northern latitude of the watershed assures that much of the year precipitation in the higher elevations will fall as snow. Flathead Lake has a strong moderating effect on local area weather so that areas near the water, such as the town of Polson, MT, remain warmer throughout the winter and cooler in the summer (http://flatheadlake.us/). Drier landscapes exist in the southern areas of the watershed around the town of Camas Prairie, MT, due to rain shadows created through the blocking of moist airflows by the high Cabinet Mountains to the west. The varied conditions created by the wide range of climatic influences lead to highly diverse ecosystems within the watershed. (http://www.wrcc.dri.edu/narratives/MONTANA.htm). The Flathead Watershed is one of the largest, most biologically intact ecosystems in North America with over 400 terrestrial wildlife species, including 11 amphibians, 11 reptiles, 319 birds, and 71 mammals (Curtis, 2010). The Montana portion of the Flathead Watershed is part of the Northern Rocky Mountain Forest-Steppe-Coniferous Forest- Alpine Meadow Province and can be found on the Ecoregions of Montana map. Habitats in the watershed range from grasslands through a variety of forest ecotypes to alpine tundra. Highly diverse wetlands play a significant role in the Flathead Watershed, enhancing the region’s biodiversity and the watershed’s functionality. The Flathead Watershed is a major stopping point on migratory bird routes. Threatened, endangered 4 and at risk species are present in the watershed, as are a large number of Species of Concern. People in the Flathead Watershed Population studies in the Flathead Watershed show that the land and the people living on it are inextricably tied together. Rich natural resources located in the dramatic natural beauty of the Flathead Watershed create strong draws for people to move to and stay in the Flathead area. People choose to live in the Flathead Watershed for what the land offers and they remain in the Flathead Watershed for cultural traditions of family and tribe. The quality of the natural environment is highly valued throughout the watershed and is considered a chief asset (Curtis, 2010). The highest concentrations of people in the watershed live in Flathead County, including the northern end of Flathead Lake up to the Canadian border, and Lake County, at the southern end of Flathead Lake. Flathead Lake is located roughly in the center of the watershed. The reservation of the Confederated Salish and Kootenai Tribes lies in the southern half of the watershed. The Flathead Watershed fully encompasses Flathead and Lake Counties, with much less acreage in Sanders, Missoula and Powell Counties. Population in the Flathead Watershed is approximately 125,000 people. Flathead and Lake Counties populations are 94,924 and 29,099 respectively, with a small population in Sanders County (2014 estimate) (http://factfinder.census.gov/). The largest cities in the Flathead Watershed are Kalispell, pop. 21,640, (http://flathead.mt.gov/) at the northern end of Flathead Lake, with 93,068 people in the greater Kalispell area 5 (Kalispell, Montana, 2014) and Polson, pop. 4585, (http://www.city-data.com/) at the southern end of Flathead Lake. There are 47 public schools in Flathead County, serving 13,752 students, and 13 private schools serving 981 students. Lake County has 21 public schools serving 4,439 students, and 6 private schools serving 281 students. Minority enrollment in the public school system is 53%, with the majority Native American (http://www.publicschoolreview.com 2016). Flathead Community College, with 2,539 students (https://www.cappex.com/2016) is located in Kalispell. Kalispell is the location of the Kalispell Regional Medical Center. The Flathead Lake Biological Station (FLBS) on the eastern shore of Flathead Lake is an academic department and center of University of Montana. It was founded in 1908 and is one of the oldest active biological field research stations in the United States. It is a world-renown research and education facility in freshwater ecology and environmental issues relating to human impact on freshwater systems and lakes. Over 40 researchers and scientists are situated at or affiliated with the FLBS. The Flathead Reservation of the Confederated Salish and Kootenai Tribes (CSKT) consists of over 1.2 million acres and comprises the southwestern quarter of the watershed. There are approximately 7,753 enrolled tribal members, with about 5000 members living on or near the reservation (http://tribalnations.mt.gov/cskt). The Salish Kootenai College, with 951 students (http://www.skc.edu/ 2016) is located in the town of Pablo on the reservation. The Flathead Reservation is the site of the National Bison Range/Pablo National Wildlife Refuge, instrumental in bison survival through the past century. The Ninepipe Wildlife Refuge is located on the Flathead Reservation; a wetlands 6 complex important for waterfowl populations. Ninepipe National Wildlife Refuge and the National Bison Range are two of the four refuges that make up the National Bison Range Complex. The Mission Mountain Wilderness, in the Flathead Reservation, was established in 1979 and is comprised of 73,877 acres. It is the only tribal wilderness in the United States, (http://www.visitmt.com/2015). Figure 2 shows the range and locations of major landownerships in the Flathead Watershed. Figure 2. Land Ownership Patterns in the Flathead Watershed 7 Economic endeavors in the Flathead Watershed include forestry, farming, fishing, orchards, and manufacturing. The service industry, construction and medical resources are important in the Flathead region. Recreation plays a significant role in the economy. Sport fishing and water sports on Flathead Lake and several other lakes and their tributaries, and flyfishing in the streams in the watershed draw numbers of visitors to the watershed. Hiking in Glacier Park and nearby wilderness areas draw people to the area to spend tourist dollars. Snow sports at Big Mountain and Blacktail Mountain Ski Areas, and Izaak Walton Inn in Glacier Park and on other Flathead National Forest lands bring visitors during the winter season. Livelihoods in the Flathead Watershed have traditionally been based on land resources such as timber and agriculture. The area is remains home to a largely rural population, and a shift has occurred in employment to the service industry, medical trades, and government employment, particularly in the northern part of the watershed (Curtis, 2010). Trends in the past half of the 1900’s show a decline in agriculture revenues with increasing amounts of agriculture land converted to development in the form of 40 acre and smaller subdivisions, (Curtis, 2010). This change in land use has been accompanied by significant population growth, particularly in the north and northwest portions of the watershed around Kalispell. Due to the changing use of land in the watershed, and the very high value both long-time residents and new arrivals place on the natural characteristics of the environment, there is a strong concern in the watershed communities that growth is balanced with environmental awareness and resource conservation. Citizen groups such 8 as the Community of Resource Educators (CORE), the Flathead Lakers and the Crown of the Continent, are active in the watershed working in education and outreach, helping to preserve and restore environmental quality and informing citizens about policy decision- making. The Flathead Watershed Sourcebook Developing awareness of the Flathead Watershed’s unique natural qualities through education was the impetus for the creation of the Flathead Watershed Sourcebook (http://www.flatheadwatershed.org/) (Curtis, 2010) (hereafter known as the Sourcebook) and the proposed environmental education curriculum called the Flathead Watershed Educators Guide (Educators Guide) based on the Sourcebook’s contents. The Sourcebook is a “rich, watershed-focused textbook that intertwines complex social, historical, economic, and ecological aspects of the Flathead Watershed with over 50 personal vignettes representing a variety of perspectives” (http://swcdmi.org/2016). The Sourcebook contains information about Flathead Watershed ecology, natural history, cultural history, communities, recreation opportunities, water resources and usage, current land use practices and economics. The Perspectives of the Sourcebook are 1-2- page length informational pieces focusing on individual residents, specific attributes and conservation accomplishments of the Flathead Watershed. The perspectives add dimensions such as personal histories of life in the watershed, expressions of individual philosophies of stewardship, in-depth looks at partnerships in conservation, agencies’ 9 explanations of regional stances on conservation action, and local industry concerns and actions for the environment. The Sourcebook was researched and written by Lori S. Curtis, resident of the town of Whitefish in the Upper Flathead Valley, as part of the requirements for her Masters of Science degree for Green Mountain College in Vermont. Curtis was guided and advised by Lex Blood, Professor of Geology at Flathead College and founder of Crown of the Continent Ecosystem Education Consortium, as well as members of the organization known as the Flathead Community of Resource Educators (CORE). The CORE members who worked with Curtis were or are active in or employed locally with agencies and organizations such as the Forest Service (Teresa Wenum), the Flathead Conservation District (Patti Mason), the Flathead Lakers (Robin Steinkraus), the Whitefish Institute (Mike Koopal), Flathead Audubon Society (Nancy Zapatocki), and the Flathead Basin Commission (Caryn Miske). Together with Curtis, the advisory group is known as the Steering Committee. The Flathead Watershed Delphi Survey With the completion of the Sourcebook, the Steering Committee began planning the development of the Flathead Watershed Educators Guide, a set of curriculum material aimed at middle school grades based on the information contained in the Sourcebook. The committee contacted Dr. Michael Brody, Montana State University professor in science and environmental education, to consult on development of the Educators Guide. Dr. Brody had worked on previous curriculum projects based on Delphi surveys, notably 10 Project WET (Water Education for Teachers) and Project Archaeology. Dr. Brody recommended the use of a Delphi survey to develop a conceptual basis for the Educators Guide. Steering Committee members chose Dr. Brody, along with Dr. Kimberly Yates and graduate student Rose Vallor, to implement and manage the Delphi survey procedure. Dr. Brody suggested that a Delphi survey ask selected Flathead Watershed community members for their opinions about the important knowledge, skills and dispositions that middle-school students should know as citizens living in the Flathead Watershed. The Steering Committee agreed to identify and contact people who would be appropriate Delphi survey participants. Delphi survey participant selection would be based on community members’ expertise in areas pertaining to education and to the Flathead Watershed, such as local middle school teachers, natural resource scientists and managers and historians. Dr. Brody, Dr. Yates and Rose Vallor became the MSU research team for the Delphi survey and creation of the Educators Guide. The results of the Delphi survey that informs the development of the Educators Guide are the subject of this research. Place-based Education in the Flathead Watershed Place-based education taught through the lens of a watershed recognizes that students are part of and can learn about their place within the natural and cultural boundaries of a local watershed. In a defining quote about the meaning and value of ‘place’ from the book: Places, Linking Nature, Culture and Planning, Nelson and Lawrence (2009) write: The way people understand and value places strongly influences decisions they make about their conservation and development. Places are more than mere locations. They consist of a broad set of 11 natural and human processes and features, whose interactions and characteristics change through time. Bedrock, weather and climate, plants, animals, soils, human land uses, technology, institutions and social learning, or culture, vary among and define places as we know them. (pg. 1) The defining attributes of the Flathead Watershed draw people who choose to live, work and recreate in and near its native ecosystems. Flathead Watershed residents active in organizations dedicated to conservation and education in the Flathead Watershed work to protect the relatively pristine quality of their surroundings. Place- based curriculum about the Flathead Watershed constructed with input from watershed community members creates a circle of connection to the watershed and to the students living there. Flathead Watershed Curriculum Development The research questions for this study developed either from the initial prompts used in Round One of the Flathead Watershed Delphi survey (questions 1-3), were emergent from the analysis of the participant responses (question 4), or reflect the purpose of the Flathead Watershed Delphi survey to determine a foundation for a curriculum utilizing the Flathead Watershed Sourcebook (question 5). Research Questions 1. What is the essential knowledge, related to the Flathead Watershed, that students should learn? 12 2. What are the essential skills, related to the Flathead Watershed, that students should be able to do? 3. What are the essential dispositions related to the Flathead Watershed that students should have? 4. What strategies could be used to guide Flathead Watershed curriculum? 5. How do the Delphi survey results relate to the Flathead Watershed Sourcebook? Purpose of the Study The purpose of this study is to determine the knowledge, skills and dispositions considered most important to community members who are considered watershed experts to inform the development of the Flathead Watershed Educators Guide based on the Flathead Watershed Sourcebook. These would be considered the essential knowledge, skills and dispositions that can and should be taught and learned by students in the watershed. The Flathead Watershed Educators Guide intends to be a place-based education curriculum grounded in the environment and the communities of the watershed; by being locally focused and locally informed the curriculum is highly relevant for watershed communities. Definition of Terms 1. Watershed: the entire geographical area drained by a river and its tributaries; an area characterized by all runoff being conveyed to the same outlet. (http://www.vocabulary.com/dictionary/watershed). In the terms of this study, the 13 watershed referred to is the Flathead Watershed, the geographic region defined by the divides from which the water in all tributaries flows into the common waterway of the Flathead River and then exits the watershed where the Flathead River meets the Clark Fork River in northwestern Montana. The term watershed in this study is used to convey not only the geological, biological and ecological definition but also the inclusive communities of people living in a connected geographic area. 2. Curriculum: Curriculum refers to the means and materials with which teachers and students will interact for the purpose of achieving identified educational outcomes. 3. Values: Important and lasting beliefs or ideals shared by the members of a culture about what is good or bad and desirable or undesirable. Values have major influence on a person's behavior and attitude and serve as broad guidelines in all situations. 4. Knowledge: knowledge refers to the body of information that teachers teach and that students are expected to learn in a given subject or content area, such as English language arts, mathematics, science, or social studies. Content knowledge generally refers to the facts, concepts, theories, and principles that are taught and learned in academic courses. 5. Skills: From a 21st Century position, skills students should be capable of include complex thinking, learning, and communication. These abilities are also commonly referred to as higher-order thinking skills, deeper learning outcomes, and complex thinking and communication skills. They consist of critical thinking and problem solving, collaboration and leadership, agility and adaptability, initiative and entrepreneurialism, 14 effective oral and written communication, accessing and analyzing information and curiosity and imagination. (Saavedra and Opfer, 2012) 6. Dispositions: According to the Thorndike-Barnhart Dictionary dispositions are defined as qualities that characterize a person as an individual; the controlling perceptual (mental, emotional, spiritual) qualities that determine the person’s natural or usual ways of thinking and acting. Also defined as: a tendency to exhibit frequently, consciously, and voluntarily a pattern of behavior that is directed to a broad goal. (Katz, 1993) 7. Place-based Education: place-based education is learning that is focused on the local community of a student. It provides learners with a path for becoming active citizens and stewards of the environment and the place where they live (Sobel, 2004). 8. Indigenous Perspectives: Indigenous perspectives in this research refer to the Native American tribes who live in the Flathead Watershed, called the Confederated Salish and Kootenai Tribes (CSKT). They are the Bitterroot Salish, the Pend d’Oreille, and the Kootenai tribes. Assumptions An assumption made in this study is that a Delphi survey is an appropriate instrument to determine curriculum based on the watershed concept. It is also assumed that participants would develop consensus opinions about essential knowledge, skills and dispositions for a watershed-based curriculum. 15 A further assumption is that Delphi survey participants included sufficient representation from a diversity of experts in the Flathead Watershed community in the areas of watershed science, education, cultural and natural history and resource management. It is assumed that participants responded honestly and thoroughly to the initial survey prompts and that they were diligent in reading the responses of other participants and contributing to discussions. It was assumed that survey participants read and ranked all subsequent statements in each iterative round. Limitations Limitations in the Flathead Watershed Delphi survey include the response behavior of survey participants. The selection process for survey participants was determined by members of the Curriculum Steering Committee based on their knowledge of Flathead Watershed community members, but whether the selected community representatives chose to participate was outside the control of the Steering Committee; a fact that may have limited the range and representation of community viewpoints. An example of this is that several members of the Confederated Salish Kootenai Tribes (CSKT) were invited to participate; two began the survey but did not complete all three rounds. The behavior of survey participants to respond honestly and to complete all the rounds of the survey was also outside of the control of the Steering Committee and research team. A further limitation to Delphi responses may have been frustration with the online delivery system of the survey, which was a factor in the ability of participants to fully 16 access and answer survey rounds. Restrictions to the length of time allotted for participants to respond may have limited participants’ ability to engage with the survey and to complete all rounds of the survey. Significance of the Study The significance of the Flathead Watershed Delphi survey for place-based watershed curriculum development is that community members could provide their opinions based on their expertise and community knowledge, share their thoughts with other community members, and as a group reach a consensus about the knowledge, skills and dispositions that students as citizens within a watershed should be taught. Expected Outcomes The expected outcome of this study is that the Delphi survey will produce a set of statements reached by consensus of qualified community members that will serve as a foundation for the development of the curriculum of the Flathead Watershed Educators Guide. 17 CHAPTER 2: LITERATURE REVIEW The literature supporting this study explores six areas that relate to this research. Those areas are:1) the watershed: services and management 2) watershed curriculums, an overview, 3) environmental place-based education in curriculum development, 4) the use of Delphi surveys to develop and inform curriculum development, 5) the history and nature of Delphi surveys, and 6) the three domains of education. See Figure 3 for the concept map of the literature review. The Watershed: Services and Management By definition, a watershed is an area of land where all the water that flows across and through the land exits the common area at a single point. It can be thought of as an area that "catches" water and routes it to a common basin, channel, or network of channels (Haury, 2000). Latitude and longitude directly influence the prevailing climate of a watershed, accounting for the form and amount of precipitation and the length and temperature of the seasons. The physical characteristics of a watershed, soil, geology, vegetation and slope, control the quantity and quality of water that flows from it (Vandas, 1997). These factors determine the types of ecosystems of a watershed, the plants and animals that live interrelated lives within a watershed. The flowing water and its associated environments are a mosaic of habitats in which matter and energy are transferred through the system (Brody, 1997). 18 Figure 3. Literature Review Concept Map Ecosystems History and Nature of the Delphi Survey Dispositions The Use of Delphi Surveys to Develop and Inform Curriculum Development Agriculture Knowledge Skills Natural Resources Use Culture and Community The Watershed: Services and Management Watershed Curricula: An Overview The Three Domains of Education Continuum of human presence and use in the watershed Community in education Watershed Education Community in curriculum Local watershed experts Statement categories Curriculum development Place-Based Education and Community in Curriculum Development 19 A watershed is a complex system in which water withdrawals, land use, developments, and industries cumulatively affect all the people, animals, and plants that rely on the river and watershed system. These vital but conflicting ecosystem services result in tradeoffs between water quality, water quantity, ecosystem conservation and human benefits (Brandes, Ferguson, M'Gonigle, & Sandborn, 2005). Efforts to quantify tradeoffs are exemplified in a model developed by Bai, Zheng, Ouyang, Zhuang and Jiang (2012) that shows the effects of increases in forestry, urban expansion, agriculture and hydropower compared to riparian restoration and native vegetation rehabilitation for maintaining ecosystem benefits and functions. Research into watersheds primarily utilized by agriculture recognizes the significant impact of agriculture on ‘downstream users’ (Schulte, Donahey, Gran, Isenhart and Tyndall, 2010). Schulte, et al (2010)’s models attempt to balance the multiple functions of the watershed to enhance commodity productivity, produce a wider array of ecosystem goods and services for a broad suite of societal stakeholders, while minimizing (or eliminating) trade-offs. Watershed management is a recognized area in the field of resource management. One of many tools available in watershed management is integrated watershed management (IWM) (Gregersen, Ffolliott, Brooks, 2007) a management approach that advocates for a balanced, sustainable use of watershed services. Community values surrounding a watershed range from monetary benefits to cultural services. Land use decisions based on monetary return and resource utilization can be at odds to watershed services that are highly valued by community members. Research by Zagarola, Anderson and Veteto (2014) into social values of a local 20 watershed in Patagonia revealed higher values for esthetics, food and fiber from indigenous plants for community members while scientists and resource managers valued watershed services such as hydrologic functions that did not take in the full range of values held by the community members. Both groups showed high values for watershed conservation and were similarly concerned about industrialization and loss of habitat. An example from the U.S. Midwest is the Raccoon River, a watershed that provides economic benefits for private farm owners and operators, but also provides drinking water and recreational opportunities for more than 400,000 people in the Des Moines area (Schulte, et al, 2010). In the summary of the City of Calgary’s municipal water supply study, Brandes, et al (2005) concluded that sustainable water management required managers to “think like a watershed”, to consider the complex interaction of human activities and natural processes, and to manage human behavior rather than manipulate the watershed to achieve sustainable use and maintain ecological functionality. Watershed Curricula, An Overview Watersheds are an ideal concept to use to educate about the local landscape and community values. As early as 1974, Gail wrote: A Curriculum Activities Guide to Watershed Investigations and Environmental Studies, an extensive guide of watershed science, investigations and land use planning tools for middle and high school students. In 1981, leading environmental educator William Stapp wrote the first edition of a widely used watershed curriculum, Field Manual for Water Quality Monitoring, which includes water-monitoring protocols that is currently in its 13th edition. Stapp also created the non- 21 profit organization Global Rivers Environmental Education Network (GREEN) that has supported watershed health through citizen science since 1984 (Stapp, 2000). GREEN is now an arm of the non-profit organization Earth Force. The Project WET (Water Education for Teachers) program was founded in 1984 by the North Dakota State Water Commission in the agency’s planning division to educate the public about water resources and its management (http://www.projectwet.org), by offering an integrated water science and cultural studies curriculum guide. Project WET continues today as a non-profit organization offering curriculum guides based on watersheds in general: Discover a Watershed: Watershed Manager, and specific watersheds in the United States: Discover a Watershed: The Colorado Educators Guide, and Discover a Watershed: The Missouri Educators Guide, along with the Project WET Curriculum and Activity Guide 2.0 and various water conservation curriculum guides. Growing concern for watershed degradation issues prompted various city, state and federal governmental organizations such as the Department of Natural Resource Conservation (DNRC), United States Geologic Service (USGS) and the Environmental Protection Agency (EPA) to develop curricula based on watersheds. An example of one state’s response to the need for locally relevant curriculum, Pennsylvania Department of the Conservation of Natural Resources (PA DCNR) and the Pennsylvania Bureau of State Parks (PA BSP) developed a state-specific curriculum in 1998 entitled Watershed Education (WE). (Gruver, 2008). An example on the federal government level, the Watershed Academy, is offered through the EPA website, (https://www.epa.gov/watershedacademy) as an extensive curriculum of training modules 22 about the science and management of watersheds geared toward resource managers and other professionals and others. Watersheds are a successful organizing principle for curriculum. Everyone on earth lives within a watershed; the quality of life is greatly affected by the condition of the local watershed; and watersheds can serve as an instructional focus for active learning in science, mathematics, social studies, environmental education, and other subject areas (Haury, 2000). Place-Based Education and Community in Curriculum Development The theoretical framework of place-based education is exemplified in community-generated curriculum serving local students. The conceptual and epistemological framework that informs this research is the idea that place-based education develops closer regard and appreciation for the local environment and community. Placed-based education is the process of using the local community and environment as a starting point to teach concepts in language arts, mathematics, social studies, science and other subjects across the curriculum…. This approach to education increases academic achievement, helps students develop stronger ties to their community, enhances students’ appreciation for the natural world, and creates a heightened commitment to serving as active, contributing citizens. (Sobel, 2004, pg. 11). One of the basic premises of place-based education is that education should prepare people to live and work to sustain the cultural and ecological integrity of the places they inhabit (Sobel, 2004). In order to do that, Orr (1994) proposes that place- based education ensures that people become aware of ecological patterns, systems of 23 causation, and the long-term effects of human actions on those patterns and systems. Proponents of place-based education often envision a role for it in achieving local ecological and cultural sustainability (Woodhouse and Knapp, 2000) The assumptions of place-based education stem from constructivist theory, the building of knowledge on known concepts and information. John Dewey, in his ‘Pedagogic Creed’ stated in that “all school life should grow out of home life… giving a background of past experience to the new ideas given in school.” (Dewey, 1926). Constructivist educators contend that the background knowledge, previous experiences and fundamental world view of students profoundly affect their interpretation of subject matter (Theobald and Curtiss, 2000). In support of that view, Gruenewald (2003) states: A multidisciplinary analysis of place reveals the many ways that places are profoundly pedagogical. That is, as centers of experience, places teach us about how the world works and how our lives fit into the spaces we occupy. Further, places make us: as occupants of particular places with particular attributes, our identity and our possibilities are shaped. (p. 631) Gruenewald (2003) goes on to say that place-based education “aims to enlist teachers and students in the firsthand experience of local life and in the political process of understanding and shaping what happens there” (p. 620). In an illustration of Gruenewald’s understanding of the central role of place, Sobel (2005 p.25-26) writes of the Lubuc High School in Lubuc, Maine, where students in the school aquaculture study facility research the issues surrounding the historical fishing industry, actions that have led to revitalization of the threatened economic base of their town. Smith (2002) discusses five thematic patterns in placed-based learning efforts: cultural studies, nature studies, real-world problem solving, internships and 24 entrepreneurial opportunities and induction into community process. In his opinion, induction into the community process brings place-based education full circle in creating citizens who are educated within their community, by community members, and who then become acting community members themselves. Smith goes on to state that “The primary value of place-based education lies in the way that it serves to strengthen children's connections to others and to the regions in which they live.” (p. 594). Sobel (2005) concurs in his statement about community in his definition of place-based education: “Community vitality and environmental quality are improved through the active engagement of local citizens, community organizations, and environmental resources in the life of the school.” (p. 11). Place-based education has advantages over standardized curriculum through increased student engagement leading to greater student achievement. A significant barrier to place-based education stems from standardized curriculum in school districts, measured through standardized testing imposed through federal law by the No Child Left Behind (NCLB) Act of 2001 (McInerney, Smyth and Down, 2011). Generic curricular models are inappropriate in a place-based educational model (Smith, 2002); an insight that points to the appropriateness of the local watershed concept to develop a place-based curriculum. Place-based education is by its nature specific to particular locales. It reinforces the validity of asking knowledgeable community members about what values they feel are most important on which to build local watershed curriculum. 25 The Use of Delphi Surveys to Develop and Inform Curriculum Curriculum development results based on a Delphi survey depend on the participants chosen to respond as well as the initial prompts that begin the discussion. Based on the goals of the survey, participants may be stakeholders in the survey outcomes. In curriculum development, stakeholders participating in a Delphi survey may be people invested in the community or profession who will be sharing responsibilities with future colleagues or community members trained in the proposed curriculum (Bachmann, et al. 2012). Delphi surveys have been found useful in curriculum development in business, nursing, medicine, agriculture, and technology along with environmental education. As early as 1978, Reeves and Jauch, at Southern Illinois University, report using a Delphi method in curriculum development in business education. The authors cite using the Delphi as a means to achieve consensus on curriculum in a business school by asking the opinions of members of the business school advisory council, using the rationale that community members who work with business school graduates have opinions as to important areas of study within the curriculum. The Delphi results indicated a movement toward consensus among survey participants though there was a low participation rate for the final round (Reeves and Jauch, 1978). Miller, Repinski, Hayes, Bliss and Trexler (2011) discuss a Delphi survey to design curriculum in a plant-breeding program in which private sector stakeholders were asked to make recommendations about knowledge, experiences, skills and specialties they deemed most important for graduates entering the profession. Forty-six stakeholders from 31 countries participated in the survey, with 59% completing all three rounds. The 26 Delphi survey was successful at deriving a solid basis of curriculum content considered important to employers worldwide. The Delphi survey has been used in curriculum development in several areas in environmental education. Project WET (Water Education for Teachers) is an example of curriculum development based on the skills, knowledge and dispositions related to water and water resources education considered important by survey participants. The study sample of 268 included a range of people in natural resource management, management education, elementary, secondary and nonformal education, university researchers and natural resource agency managers representing every state in the United States (Brody, 1995). Of the 80 criteria identified out of the 5,446 initial responses, participants had a high level of agreement for each criterion (Brody, 1995); the result being the 80 criteria became the framework for the Project WET curriculum. Project WET is a widely respected curriculum used by school and natural resource educators in the United States and in many other countries (http://www.projectwet.org/). Archaeological Science for All (ASFA) developed a Delphi survey to establish a national framework that delineated the essential science concepts, skills, and dispositions that can be addressed in informal archaeology education activities (Brody, Fisher, Moe, and Keremedjiev, 2010). The ASFA project consulted experts in the fields of informal science, archaeological science and archaeological education, choosing participants who were geographically situated across the United States. The ASFA Delphi was successful in reaching a consensus regarding archaeological content, skills and dispositions related to teaching science inquiry to under-represented populations. Results of the ASFA survey 27 inform the development of a national plan to assist informal science educators (ISE) professionals throughout the nation (Brody, et. al., 2010). History and Nature of the Delphi Survey The Delphi survey is named for the Delphic Oracle who was active in the 5th century B.C. in the Greek city of Delphi. In Greece at that time, the Delphic Oracle served as an institutional process to help make policy decisions (Marchais-Roubelat and Roubelat, 2011). The oracle gathered information from a number of informants to deliver a truthful statement, compiled as a result of data from many sources (Kennedy, 2004). Within the past century, the Rand Corporation first used the Delphi survey in the 1950s as a prediction method for the implications of the use of warfare technology on armament targets. Experts on the topic of nuclear weapons were consulted and their opinions pooled to formulate a prediction or consensus opinion (Dalkey and Helmer, 1963). Delphi surveys as census-building instruments can be characterized as “a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem.” (Linstone and Turoff, 1975, pg. 3). Since Linstone and Turoff’s study (1975) of the Delphi survey process, Delphi surveys have become widely used in more academic domains and for more purposes (Rowe and Wright, 2011). Delphi surveys, in seeking to uncover a reliable consensus, solicit the opinions of acknowledged experts in the field (Andranovich, 1995). Knowledgeable participants can be experts in the field of study as well as being 28 community members with personal stakes in the outcomes of the application of the survey results (Baker, Lovell and Harris, 2006). A Delphi survey is an anonymous, iterative process, where initial responses are discussed and then ranked by individuals in the group as to their importance. This ranking process was initially described in Delbecq, Van de Ven and Gustafson (1975). Rated responses are winnowed down by group consensus determined by ranked scores to a core set of statements that participants feel are most valuable (Linstone and Turoff, 1975). A Delphi survey is comprised of a minimum of two separate groups of individuals, the survey manager or managing team and the survey participants. A third interested party could include the entity for whom the data is being gathered (Delbecq et al., 1975). The participants in the Delphi survey are chosen for their expertise in the area being studied or the complex problem being analyzed. Baker et al. (2006) in researching Delphi participants in nursing studies discusses the definition of expert in terms of recognized knowledge, experience, and/or the ability to influence policy in the field. An expert has been defined in various, related ways, such as a group of informed individuals, specialists in their field, or someone who has knowledge about a specific subject (Keeney, Hasson and McKenna, 2001). In the matter of participants who are both willing to engage in the process of a Delphi survey and also considered experts in the subject area, Keeney et al. go on to say that: Simply because individuals have knowledge of a particular topic does not necessarily mean that they are experts. In fact, those who are willing to engage in discussion are more likely to be affected directly by the outcome of the process and are also more likely to 29 become and stay involved in the Delphi. Hence, the commitment of participants is related to their interest and involvement with the question or issue being addressed. (p. 196). The same consideration is expressed by Moore (1987)"A nuclear physicist is an appropriate expert if the Delphi concerns atomic energy and a resident of a neighborhood is an expert on what should be a community's goals" (p. 51). Four key features define a survey as a ‘Delphi’. These are anonymity, iteration, controlled feedback, and the statistical aggregation of group response, (Rowe and Wright, 1999). These features can offer specific advantages in a group decision-making process. Anonymity prevents domination by individuals, is an advantage when participants may be hostile to each other, and it alleviates situations where personality styles would be distracting in a face-to-face setting (Delbecq, et. al., 1975) Anonymity also contributes to the democratic nature of a Delphi (Day and Bobeva, 2005). Since a Delphi survey can be conducted online or at a distance, it is appropriate to use when personal contact is not possible due to distance, time and cost constraints. Online surveys can offer the advantage of asynchronous participation. The Delphi survey structure of iterative feedback, the repeated exposure of participants to statements generated by the group, develops insights that in the end are more than the sum of the parts (Day and Bobeva, 2005). Theoretically, the Delphi research process can be continually iterated until consensus is achieved. However, typically three iterations are adequate to collect sufficient data and reach consensus (Custer, Scarcella and Stewart 1999). In a Delphi survey, feedback is controlled through the survey structure put in place by the survey manager. Clayton (1997) states that “The 30 Delphi process aims to arrive at a level of consensus among the panel members. Consensus is assisted by the researcher providing feedback to each panel member of their previous rating together with a group measure of central tendency.” (p. 382). Statistical aggregation of responses occurs through the analysis of ratings, often determined with Likert scales, to determine mean scores for items in the survey. Rigor as determined by measures of reliability, validity, and trustworthiness of Delphi surveys was researched by Hasson and Keeney (2011) in a study of nursing research Delphi surveys. According to Hasson and Keeney, two characteristics of the Delphi survey create issues with establishing rigor. The first characteristic that challenges measures of rigor is the dual nature of classical Delphi surveys, depending on whether the survey is used for prediction or to explore and understand situations. Hasson and Keeney express the duality in philosophy as “juxtaposition of a classical Delphi technique between positivist and naturalistic paradigms that raises a problem about which standards to adopt” (p.1696). The second characteristic of Delphi surveys leading to dilemmas around rigorous testing is the continual modifications of the method, with at least ten different recognized variants and applications of Delphi designs. While methods to establish rigor through quantitative means remains problematic (Hasson and Keeney, 2011), the qualitative measures of trustworthiness, obtained through the characteristics of the classical Delphi survey such as iterative rounds, the range and representativeness of participants, and detailed description of the collection and analysis process, are considered a more credible means of establishing rigor. 31 The Three Domains of Education The three prompts of the Flathead Delphi survey asked participants to relate their thoughts about the important knowledge, skills and dispositions students should learn about the Flathead Watershed. The three prompts have their basis in the domains recognized by Bloom in his work with others in developing what is now called Bloom’s Taxonomy. The three domains are: the cognition domain, consisting of knowledge and skills, the affective domain, also called dispositions, and the psychomotor domain. The domains were determined by Bloom and other educators when they came together in the 1950s to begin to determine a standardized basis for assessment for comprehensive exams (Bloom, 1956). Bloom and others, notably Krathwohl, combined efforts to create the Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook 1: Cognitive Domain (Bloom & Krathwohl, 1956), an attempt to delineate and organize the educational objectives of the cognitive domain (Owens, 2016). Bloom believed the Taxonomy could also serve as a common language about learning goals, a basis for curriculum, a means to align course objectives, and a broad view of educational possibilities (Krathwohl, 2001, pg. 212). The original Taxonomy was solely concerned with the cognitive domain. The two other domains of learning, Affective and Psychomotor, were identified by the same group of researchers during their years of meetings under Bloom’s direction but not developed at that time (Bloom, 1956). Bloom’s Taxonomy is a hierarchal listing of 6 cognitive processes beginning with simpler cognition leading to more complex processes. The six processes in their original order are: knowledge, comprehension, application, analysis, synthesis and evaluation 32 (Bloom, 1956). The original Taxonomy of Cognitive Learning was revised by Anderson and Krathwohl (2002) to incorporate later distinctions in cognitive psychology. Where the original Taxonomy combined both content knowledge and skills into one statement of cognition at the knowledge level, the revised Taxonomy separated the knowledge level into four areas: factual knowledge, conceptual knowledge, procedural knowledge and metacognitive knowledge. The revised Taxonomy partially restructured and renamed the original six cognitive processes into a new hierarchy consisting of: remember, understand, apply, analyze, evaluate and create (Anderson and Krathwohl, 2001, pg. 215). A Taxonomy of the Affective Domain was developed and published by Krathwohl, Bloom and Masia in 1973. In this work they identified the five hierarchal divisions of the affective domain as: receiving, responding, valuing, organization and characterization (Krathwohl, Bloom and Masia, 1973). Harrow developed a Taxonomy of the Psychomotor Domain that also has a hierarchal organization. The six divisions of the psychomotor domain are: reflex movements, fundamental movements, perceptual abilities, physical abilities, skilled movements, and nondiscursive communication (Harrow, 1972). While Bloom, Krathwohl and others recognized and organized the objectives of learning, Novak and Gowin (1984), in Learning How to Learn, determined four commonplaces of learning that described an educational experience. The four commonplaces are teaching, learning, curriculum and the milieu, also called governance. In their view, each commonplace is an integral element in the process of education. 33 Teaching is an obligation of the teacher, learning is an act done by the learner, the milieu, or governance, is the setting of the educational process, and “curriculum is comprised of the knowledge, skills and values of the educative processes that meet a standard of excellence that makes them worthy of study” (Novak and Gowin, 1984, pg. 6). Gowin (1981) defines the fundamental basis of the educational process as “the deliberate connection-making between thinking, and feeling which leads to intentional human acting.” (pg. 49). These foundational education processes inform the development of curriculum. The knowledge, skills and dispositions considered important by Flathead Delphi survey participants, matched with the Sourcebook, become the basis for the Educators Guide. 34 CHAPTER 3: METHODOLOGY The research method for this study is the Delphi survey methodology. The Delphi survey is a descriptive research methodology using a mixed data acquisition approach. This chapter explains the research design, the selection process of the Delphi survey participants, the Delphi survey process and the correlation process of Delphi statements to the Sourcebook. Research Method and Design Appropriateness A Delphi survey was the method employed to solicit opinions about the essential foundational knowledge, skills and dispositions for the Flathead Watershed Educators Guide. A Delphi survey method was chosen because of its capability of building a consensus opinion about the topic under consideration. The Delphi survey research method was appropriate for this use because it created a circumstance in which knowledgeable stakeholders within the Flathead Watershed were able to provide input in an endeavor that may impact the education of their children about the watershed they live in and care about. As an anonymous and iterative consensus building tool, participants were able to learn about the views of other participants and respond with their own thoughts without pressure to conform. Validity and Reliability Descriptive validity is achieved through maintaining factual accuracy of the collected data (Gay, Mills and Airasian, 2012). Descriptive validity was ensured when 35 researchers kept the original wording and intent of participants’ responses to the initial Round One prompts in creating statements for Round Two of the survey. An example from the knowledge section of the survey shows participants’ responses made into statements while keeping the intended concepts (participants’ actual responses are italicized): “I believe in the four C's- clear, connected, complex and cold. Those four things are essential for the bull trout, an essential animal of the Flathead Watershed.”, written by FP participant 003, was combined with “Students seem to grasp the 4 C's quickly and are able to relate the concepts to the health of Big Creek. Students come back to the class room and create a presentation slide show illustrating the 4 C's and why they are essential components for the native bull trout.”, from FP Participant 024. The two comments were transformed into the positive declarative statement for Round Two: The four C's- clear, connected, complex and cold, are an essential concept for understanding of the Flathead watershed, particularly for animals such as the bull trout, an essential animal of the Flathead Watershed. Evaluative validity is concerned with “whether the researcher was objective enough to report the data in an unbiased way, or without making judgments and evaluation of the data” (Gay, et.al. 2012, pg. 392). The research team created evaluative validity when each member of the team agreed on the accuracy of the statements in capturing the intent of the Delphi participants when transformed into Round Two Delphi statements. An additional measure of evaluative validity was maintained because the research team did not rate statements in either round two or round three. 36 Construct validity was established through internal measures, where it became evident that similar or identical themes were repeated throughout participants’ comments. Participants then rated those same themes highly when evaluating them on Likert scales, resulting in the common themes remaining in the final data collected in the Delphi survey. As a further measure of validity, MSU researchers kept an audit trail of the research, starting with selection of participants (Wenum email, 2015), through initial prompts and responses and ending with the final rating and ranking of the Delphi survey statements. Information was captured electronically through the online software program D2L and research team meeting notes were recorded and kept in hard drive storage. Conference call notes and email transcripts with Steering Committee meetings were also collected during the Delphi survey process. Reliability in this study is defined as the degree in which the research instrument consistently measures what it is designed to measure (Gay, et.al. 2012). In a Delphi survey, reliability can be measured through the repeated ranking of statements by Likert scale in the rounds of the survey. In Round Two of the Flathead Delphi survey, reliability was shown by the stability of scores achieved through asking participants to rerate the high ranking statements from Round One for Round Two of the survey, creating a test- retest scenario for the high ranking statements in the survey. Table 1 shows the percentage of statements that received a repeated measure of rating of important or very important (at or near 4.0) on the 1-5 Likert scale, with 5 being considered very important for students to learn. Table 1 also shows the inclusion of a fourth category of statements 37 related to teaching that emerged from the analysis of participant responses from Round One. See Table 1 for a comparison of the number of statements in each category. Table 1. Comparison of Number of Statements from Round 2 to Round 3 Category of Response Number of Round 2 Statements after initial comments Number of Statements remaining after final ranking Cut-off point in Likert Scale Ranking (1-5 scale) % of statements with ‘Important’ or above rating Knowledge 57 36 4.06 65% Skills 40 19 4.06 48% Dispositions 34 27 3.94 79% Teaching 12 12 3.94 100% Participant Selection Enactment of a Delphi survey for the purposes of curriculum development required the participation of individuals with specialized knowledge and skills in areas for which information and opinions were collected. In the Flathead Watershed, the individuals identified were people who were knowledgeable about the Flathead Watershed in a wide range of fields such as natural science, economics, culture and history. Local educators would be important contributors to the process. The Sourcebook Steering Committee and the CORE group that became the Curriculum Steering Committee (Steering Committee) worked together to identify the individuals they felt would bring the needed experience to the survey. The Steering Committee decided on three categories of participants: educators, resource managers, and scientists. Members of 38 the Steering Committee each contributed names of individuals that they knew from personal or professional contact within the watershed. All of us on the committee submitted suggestions, within the groups we identified…teachers, mgrs. and scientists…using our own knowledge of the community and asking people that participated in the original Sourcebook development….and wanted to get representation across the watershed. (Wenum, email correspondence, Oct. 2014). Decisions of who to invite to participate in the Delphi survey were made through consensus of the opinions of the Steering Committee. The proposed participants were asked to join the Delphi survey through invitations by the CORE group. Several members of CORE also participated in the survey. The initial number of respondents in the Delphi survey was 29 community members and 4 MSU researchers. See Appendix A for the list of Delphi survey participants. Participants in the survey came from a wide variety of backgrounds, but had common elements within their experiences. Those elements were engagement with the natural world, teaching experiences, typically in the sciences, a history of community service, and/or employment in natural resources infrastructures such as dams or fish hatcheries. Initially, twelve educators, ten resource managers and seven scientists were asked to participate. Final numbers of respondents in the first round in each category were nine educators, eight resource managers, and four scientists. Of the nine educators who responded in the first round, five taught middle school science, one taught special education in middle school, and one was a retired administrator in public schools throughout the Flathead Watershed. Two informal or non – traditional educators participated in the first round of the survey, one of whom was 39 Native American and taught at the college level on a reservation and one a natural resource informal science educator. The resource managers involved in the first round of the survey included people who held managerial roles in resource infrastructure. One person each worked in fish hatcheries, wastewater treatment plants and dams. Two people worked in the private enterprises of lumber production and ranching. Three participants worked in the Forest Service, Tribal Lands and for the Flathead Lake Biological Station; two were employed in the nonprofit conservation field. The smallest representation of participants was in the scientist category. These people came from the widest range of backgrounds, including a city engineer, a local historian, a research scientist/ program coordinator working at the university level in environmental and fisheries biology, and an environmental scientist- educator working in the nonprofit resource conservation realm. Those who were invited to participate but were not able to contribute to the survey were in the same categories as those people who did participate: middle school science teachers, resource managers of the types listed above, and scientists/educators in nonprofit conservation organizations, colleges and tribal positions. Final participation in round one consisted of 25 people, down from 33 initially. Eight of the original participants either dropped out (2) or attempted to log on to the server carrying the survey and were not able to (6). Three participants did not complete either of the next two rounds after the first round. Sixteen participants completed the third and last round. See Table 2 for data about numbers of participants in each round of the Delphi Survey. 40 Table 2. Number and Category of Participants in the three Delphi Rounds Professional Association Total Number Invited Initially Participants in Round 1 Participants In Round 2 Participants in Round 3 Educators 12 9 (75%) 8 (66%) 4 (33%) Resource Managers 10 8 (80%) 9 (90%) 7 (70%) Scientists 7 4 (57%) 4 (57%) 4 (57%) Total Number of FH Participants 29 21 (72%) 21 (72%) 16 (55%) Research Team 4 4 (100%) 0 0 See Appendix A for Delphi Survey Participants. Flathead Delphi Survey Process The Flathead Watershed Delphi survey was set up as a three round survey. The online educational management system employed by Montana State University, Desire 2 Learn (D2L), was chosen as the survey implementation tool since it could be used at a distance, asynchronistically, with the option of enabling participants to engage in online dialogues anonymously. Participants were randomly assigned numbers by the MSU management team to log on to D2L and for purposes of anonymity and confidentiality in the Delphi survey. CORE group members and Steering Committee members who selected Delphi participants, while being aware of who the participants in the survey were initially, did not know which responses were posted by which participants and did not know which participants completed all the rounds of the survey. In the three rounds of the survey, the first round consisted of responses written by participants to the survey prompts and comments made to other participants’ responses. 41 In the following two rounds participants rated statements created from Round One responses on a 1 to 5 scale based on their opinions of the importance of the statement. In the second round participants were able to comment about individual statements, but only the research team was able to view the comments. In the third round participants had only the option of rating the statements retained from the second round. Round One The first round asked that survey participants respond to three prompts, one each in the areas of knowledge, skills and dispositions. The prompts were:  What are the essential concepts/issues that students should learn related to the Flathead Watershed? (for example; water quality or fish species);  What skills should students develop that would help them learn about and live in the Flathead Watershed? (for example; reading maps or writing a letter to city council);  What attitudes and values should students develop related to the Flathead Watershed? (for example; appreciating indigenous perspectives or valuing diverse opinions about resource management). During Round One participants were able to read and respond to each other’s comments so that ideas could be expressed and developed through dialogue and discussion. Responses from participants were not restricted by any means. Participants were able to write at length about their ideas pertaining to the prompts and to respond to other posts uploaded by other participant. There were no restrictions to the number of 42 times or the length of replies of responses to others or their own postings. See Table 3 for an example of the comments and responses. Table 3. Example of Participant Responses for Round One, Disposition Prompt Appreciating Where They Live Created by FP Participant 021 on Feb 28, 2014 12:42 PM Students should first be aware of the Watershed and the incredible gift they have been given in living within it. Many don’t recognize that where they live is one of the most incredible places in the world. People migrate back to the Flathead more often than not after experiencing differences in other parts of the country or they experience a sense of awe when flying back into the valley. Getting them to appreciate where they live is paramount in order for them to buy into taking care of their environment. A big part of appreciating the Flathead Watershed will come through honing their observation skills. Response by FP Participant 001 Mar 2 at 5:42 AM I like the idea of explicitly creating learning opportunities that develop real awareness of this remarkable place. Sometimes when you see it and benefit from it every day, you take it for granted. Good point made. Response by FP Participant 025 Mar 3 at 9:26 AM I concur as well. We have an amazing place here. The watershed is as good as it gets ecologically in the Lower 48. So I always stress that first and foremost, before I get into the threats and concerns that we have about our aquatic ecosystem. Response by FP Participant 018 Mar 4 at 6:29 AM I agree. As a person who grew up here I can vouch for the fact that it's easy to not understand at a young age how fortunate we are to live in this place. When this is all you've ever known, you don't really have good perspective. Response by FP Participant 004 Mar 6 at 6:56 PM Appreciation is a hard thing to develop in a young person but it is worth trying. 43 During this round the research team was actively but anonymously participating to help encourage discussions. Round One was open for participation for 14 days, from February 17 to March 2, 2014. After closing Round One, the participants’ comments in the three categories were transformed into positive, declarative statements by the research team. Statements were constructed to preserve the original intent and wording whenever possible. Efforts were also made to eliminate redundancy. A total of 143 statements were formed from participant responses. During Round One comments analysis, the research team recognized the need to create a fourth category due to the number of statements related to teaching methods. Teaching was added as a fourth category for Rounds Two and Three. See Appendix Three for Round One sample responses. Round Two For Round Two, the Round One statements in the four categories were uploaded to D2L in a quiz format with a 1-5 Likert scale option. Survey participants were invited to respond to each statement by rating it on the 5-point scale, with 5- very important, 4- important, 3- neutral, 2- unimportant, and 1- very unimportant. Participants were also encouraged to comment on and explain their answers and respond to other comments during this round. The research team did not participate in rounds two and three. Round Two was open for participation for 14 days, from April 7 through April 20, 2014. The values from the Likert scale ratings were not visible to other participants. At the close of Round Two mean scores from the Likert scale ratings were calculated and the statements were ranked by those scores. Rankings were examined, and it was 44 determined by the research team that natural breaks in the mean score rankings would be chosen as cut-off points. Each category revealed natural breaks at different numerical values, but all those values were at or above the 3.94 range, indicating statements were considered important or very important by participants. Participant comments were taken into consideration to determine if any of the lower ranking statements had strong appeal for participants. There were no comments that suggested reasons to keep lower ranking statements. See Table 4 for an example of comments received during the Round Two survey. Table 4. Example of a Round Two Statement and Comments. After eliminating the lowest ranking statements, the remaining statements became the final Round Three materials. See Appendix C for Round Two scores. 19. Indian Education for All is knowledge that students should have. Comments # 19: Native American values on water and its importance to life is powerful. Important to demonstrate how curriculum would meet Indian Education for All standards; There should be more resources/training available for teachers. I believe that some teachers are reluctant to teach Indian Ed. because they don't feel comfortable with the amount of knowledge they have on the subject. Students will connect better with the landscape if they understand that it has been important to people for thousands of years. The more they are taught that they are a part of the landscape just as Native Americans are & were, the more they (hopefully) will realize that this isn't a place for them to take for granted. Students need to understand that it is the SAME watershed. I think that somehow (perhaps because of elapsed time when Native Americans were the only ones here), students get the impression that Native Americans were part of a different world. I prefer having the Indian Ed for All mixed in with the rest, not taught separately. 45 Round Three In Round Three participants responded to the remaining statements in the four categories by rating the statements on a 1-5 Likert scale, as in Round Two. Participants were given 5 days, from May 19 to May 23, 2014, to participate in the third round. At the closing of Round Three, mean scores were calculated from the Likert scale ratings and statements were placed in rank order. The results were examined for natural breaks. Breaks appeared at around the 4.0 rating, corresponding to important on the Likert scale. The research team consulted with the Curriculum Steering Committee by teleconference to discuss the results, and it was decided by consensus to include a few selected statements from just below the cut-off point of 4.0, since those statements contained concepts that were considered important to include in the Educators Guide and were not represented in other statements. A total of 95 statements were retained after eliminations in Round Three. See Appendix D for Round Three Knowledge Statements. See Appendix E for Round Three Skill Statements See Appendix F for Round Three Dispositions Statements See Appendix G for Round Three Teaching Statements 46 CHAPTER 4: RESULTS Flathead Delphi Statement Content Analysis The research questions this study sought to answer were: 1. What is the essential knowledge, related to the Flathead Watershed, that students should learn? 2. What are the essential skills, related to the Flathead Watershed, that students should be able to do? 3. What are the essential dispositions related to the Flathead Watershed that students should have? 4. What strategies could be used to guide Flathead Watershed curriculum? 5. How do the Delphi survey results relate to the Flathead Watershed Sourcebook? Statements taken directly from the Delphi survey are shown in italics. The statement-identifying letter and number are indicated in parentheses following the italicized statement. Research Question One 1. What is the essential knowledge, related to the Flathead Watershed, that students should learn? The Delphi survey knowledge prompt was: What are the essential concepts/issues that students should learn related to the Flathead Watershed? (for example; water quality 47 or fish species). Figure 4 shows the knowledge statements in rank order with standard deviation (SD) bars indicating plus or minus one SD from the mean score. Figure 4. Knowledge Statement Means with +1 and -1 Standard Deviation The shorter lengths of the bars on the highest-ranking statements indicate less SD from the mean and a higher degree of agreement of the importance of that statement. The increasing lengths of the bars indicate a wider range of opinion about the importance of that statement. In general, standard deviations are within one score range, indicating that participants were in fairly close agreement about the relative importance of each statement. Several knowledge statements stand out as having larger SDs than typical for their rank. The first statement is K11, Students need to know the basic premise of a watershed, that it is where they hunt and fish and recreate, and work, and live, and eat and grow food and consume food, and that as a result; they are also influencing the watershed with all their everyday actions, M=4.50. This statement received two scores of 3, neutral on the Likert scale, 4 scores of 4, important on the scale, and 10 scores of 5, very important 2.00 3.00 4.00 5.00 6.00 K 2 K 1 K 3 0 K 3 2 K 1 1 K 1 5 K 2 7 K 3 3 K 3 9 K 4 K 1 4 K 2 0 K 4 3 K 8 K 2 3 K 2 4 K 2 6 K 2 8 K 3 5 K 3 7 K 4 2 K 4 4 K 1 0 K 4 7 K 1 7 K 3 1 K 5 1 K 3 K 9 K 2 1 K 3 4 K 4 8 K 1 6 K 2 9 K 4 0 K 4 9 L ik er t S ca le Knowledge Statement Means with +/-1 SD bars 48 on the scale. The next statement with a wide SD is K4, Students should be aware that a major threat to the Flathead watershed is water quality degradation, predominantly nutrient and sediment additions caused by human activities. The scores for this statement are 1- unimportant, 1- neutral, 4- important and 10 very important. The third statement with a wide range of SD is K21, Students should be aware of the stormwater conveyance system and that storm drains flow directly into our water bodies without treatment, M=4.19. Scores for this statement are 1- very unimportant, 1- neutral, 7 – important and 7- very important. One participant considered this statement very unimportant. K36 is a repeat of the statement K21. It received scores of 1- very unimportant, 1-unimportant, 0- neutral, 9- important and 5- very important for its second appearance on the list, and a mean score of 4.00. Having similar scores each time this statement was rated indicates consistency in participants’ decisions about the statements. In the knowledge category the top ranking statement, with a mean score of 4.69, was: Students should be knowledgeable about the plants and animals of the watershed, including invasive species and their impact on the watershed. (K2). The next three statements have mean scores of 4.63. Those three statements are: 1) Students should be aware why the watershed is important to the species of fish and wildlife that utilize the watershed. (K1); 2) It is important for students to know how humans impact the watershed. (K30); 3) Students should understand the characteristics of a healthy watershed. (K32). An analysis of the final knowledge statements, those receiving a mean score of 4.00 and above, showed three main themes emerging from the compiled knowledge 49 statements. The first theme evident in the knowledge statements was that students should learn about the various species of plants and animals that live in the Flathead Watershed. The second theme was that students should understand human impacts on watersheds. The third theme was that students should know characteristics about watersheds. Figure 5 shows a graphical representation of the themes in the knowledge statements. Figure 5. The Three Themes in the Knowledge Statements The first theme, that students should know the plants and animals of the watershed, while being the top-ranked theme, was not the most frequently mentioned. This theme was enlarged upon in several subsequent statements to address invasive species and indicator species who are also endangered, specifically the bull trout, an iconic species in the Flathead Watershed. Biodiversity was specifically mentioned as an important topic. The statement: A key concept for students would be an understanding of Knowledge Statements Theme 1: Students should learn about the various species of plants and animals that live in the Flathead Watershed. Theme 2: Students should understand human impacts on watersheds. Theme 3: Students should know characteristics about watersheds. 50 the nature and need for biodiversity (K10), is an example of statements that expressed the idea of students learning about the living components of the watershed. Statement K10 received a mean score of 4.31. The second theme expressed in the knowledge statements was concerned with students learning about how humans impact the watershed. This theme was expressed in a number of ways in many statements. Participants felt it was important that students know that ‘humans influence the watershed with their actions such as hunting, fishing, working and living in the watershed’ (K11). It was also stated that students should know that ‘humans are not separate from their watershed’ (K33), that they all ‘live in a watershed and their actions affect downstream users, water quality and aquatic ecosystems’ (K33), that: Students need to have an understanding of the human impact, both positive and negative on the watershed. K39), and that humans manage water resources in a watershed. Knowledge of ‘stormwater conveyance’ was mentioned specifically (K36). One other statement in this theme was that Students should be aware that there is a finite supply of clean water. (K37). The third theme was the most pervasive theme in the knowledge statements. It is that students should know factual information, or ‘the characteristics of watersheds’. Characteristics of watersheds mentioned included basic premises of watersheds, geography, water cycles, bodies of water, underground aspects, land and land masses, aspects of water such as wetlands, water quality and water resources (K27, K14, K24, K26, K44). 51 Within the three themes were statements that indicated that students should learn about the Flathead Watershed culture and communities. Those included statements about natural and cultural history (K8) (K35) (K34) (K38), local geography including international aspects (K40), effects of climate change and competing water users. Climate change and its effects on the needs of competing water users. (K17), received a wide range of scores compared to the majority of statements, as did international aspects of the Flathead Watershed. Research Question Two 2. What are the essential skills, related to the Flathead Watershed, that students should be able to do? The Delphi survey prompt to solicit ideas about important skills was: What skills should students develop that would help them learn about and live in the Flathead Watershed? (For example; reading maps or writing a letter to city council). Figure 6 shows a graphical representation of the ranking order for skill statements for Round 3 of the Delphi survey with error bars showing +1/-1 standard deviation. Figure 6. Skill Statements Means with +1 and -1 Standard Deviation 2.00 3.00 4.00 5.00 6.00 S20 S26 S16 S1 S21 S10 S22 S25 S28 S7 S8 S27 S4 S2 S12 S11 S23 S13 S5 L ik er t S ca le Skill Statement Means with +/- SD bars 52 As noted with the knowledge statements, statements with the highest Likert scale ranking also show the smallest standard deviations, meaning survey participants were in closest agreement on the importance of those skills for inclusion in the Flathead Watershed Educators Guide curriculum. All the statements in the skills category received a mean score rating of 3.94 or above with the exception of S5 "Nature journaling" or "field journaling" gives young students the tools they need to quietly observe what is going on around them. With a mean score of 3.69, field journaling ranked below 3.94, but in consultation with the Steering Committee, it was considered an important skill to keep and incorporate into the Educators Guide curriculum. Delphi participants made suggestions in four broad skill areas they felt students in the Flathead Watershed should learn. Figure 7 shows the themes in the skill statements. Figure 7. The Four Themes in the Skill Statements Skill Statements Theme 1: Students should learn to do scientific inquiry Theme 2: Students should be able to think critically Theme 3: Students should be able to employ process skills and science practices Theme 3: Students should be able to read maps, graphs and charts 53 The first and most agreed on of those areas was that students should learn to do scientific inquiry. Five out of the top 17 statements addressed learning how to do scientific inquiry. The second broad area mentioned was critical thinking. The top ranking skill statement included both those ideas: Basic scientific inquiry and critical thinking are important skills for students. (S20), and had a mean score of 4.88. Critical thinking was mentioned three times in the top ten statements. The third ranked statement was: Critical thinking is important. Students should be taught how to ask the tough questions and then learn how to look for answers through research. (S16). This statement had a mean score of 4.69. Survey participants ranked eighth, with a mean score of 4.38, the statement: It is important to teach students to look beyond the rhetoric/popular opinion and ask their own questions, as well as how to search for their own answers or solutions provides a life skill. (S25). The third area most often mentioned in skill statements was the ability to read, interpret and create maps, graphs and charts. Six out of the 26 skill statements specified these as skills students should have. Survey participants ranked the statement A student should be able to interpret data and graphs. (S10), sixth in rank order and The ability to create tables, picture graphs, bar graphs and picture maps to record and organize information and to be able to read, interpret, and use tables and graphs to identify patterns and trends, draw conclusions, and make predictions are important skills for students to have. (S22), seventh in rank order. These statements had mean scores of 4.50 and 4.44, respectively. 54 The fourth area mentioned in the skills statements was that students should be able to employ process skills and science practices. Some of the skills specifically mentioned were ‘listening, observing’ (S28), and ‘describing and explaining’ (S23). Skills mentioned in the fourth ranked statement, with a mean score of 4.63, were: Scientific investigations involving developing questions, collecting data, analyzing data, and comparing data are skills students should have. (S1). There were also several specific skills mentioned in this category. One specific skill was: Students, when looking to the Internet for information, need to always evaluate the source of information. (S26). This was the second highest ranked statement, with a mean score of 4.75, indicating participants felt it was either an important or very important skill for students to learn. Other specific skills and practices mentioned were: Field studies are great. (S7), M=4.25, Students should begin to conduct "research". (S2), M= 4.13, and Observation skills are essential so that students can sit quietly and observe the world around them. (S11), M=4.06. Research Question Three 3. What are the essential dispositions related to the Flathead Watershed that students should have? The Delphi survey prompt for the disposition statements was: What attitudes and values should students develop related to the Flathead Watershed? (for example; appreciating indigenous perspectives or valuing diverse opinions about resource management). Mean scores vary from M=4.56 to M=3.81, less than a full point difference from the highest ranked score to the lowest ranked statement. The relatively 55 flat mean score line and short range of distribution of SD bars in Figure 5 show consistent ratings for disposition statements in Round 3. SD bars become longer for most of the statements as the rank order of the statement becomes lower. In Figure 8 the disposition statements are shown in rank order by mean score, with +1/-1 SD bars. Figure 8. Disposition Statement Means with +1 and -1 Standard Deviation An exception to that trend is Statement D3: It is important that students engage in life styles (choices and decisions) that protect and sustain a healthy Flathead Watershed. M=4.06, that had the highest standard deviation, SD =1.12. This statement was considered very unimportant by one Delphi survey participant; the complete range of scores are: 1 – very unimportant, 3 – neutral, 5 – important, and 7 – very important. The last three statements in the dispositions ranking had mean scores of 3.88, 3.88 and 3.81, respectively. These statements were included in the final statements in consultation with the Steering Committee because it was felt the ideas expressed in them were important relative to the Flathead Watershed. Those ideas dealt with indigenous perspectives, such as the statement: Salish, Kootenai and Pend d’Oreille history and cultural resources, such as films, books, have a tribal perspective and are important sources of information on science, history and contemporary 2.00 3.00 4.00 5.00 6.00 D 2 3 D 2 8 D 2 D 4 D 6 D 1 3 D 1 5 D 8 D 1 1 D 1 2 D 1 7 D 2 7 D 5 D 9 D 1 0 D 1 9 D 1 D 1 6 D 2 0 D 2 4 D 3 D 2 6 D 2 5 D 2 9 D 1 4 D 2 1 D 7 D 2 2 D 1 8 L ik er t S ca le 1 -5 Disposition Statement Means with +/- 1 SD bars 56 information, including topics such as Indian Water Rights and native place names, which should be used in classrooms in the Flathead watershed. (D7), M =3.88, and major pollution issues in the watershed, with the statement: The major issues affecting watersheds today, such as water rights, toxic spills from rail or truck, toxic cleanup, industrial pollution, are important issues for students to learn about. (D22), M = 3.88. The mean scores of the last three statement fall below the lower end of the standard deviation error bar of the first statement, indicating those statements have a lower rating of importance. Two themes emerged in the analysis of the disposition statements. The two areas were: developing stewardship ideals and behavior about the watershed, and understanding and appreciating multiple perspectives, particularly indigenous perspectives. Figure 9 shows the two themes evident in the dispositions statements. Figure 9. The Two Themes Within the Disposition Statements The top ranked disposition statement captures the idea of stewardship: It is important for students to know what they can do to help protect our local watersheds. Dispositions Statements Theme 1: Students should develop stewardship ideals and behavior about the watershed Theme 2: Students should understand and appreciate multiple perspectives, particularly indigenous perspectives 57 (D23), with a mean score of 4.56. Four of the top eight statements express environmental stewardship (D23, D2, D6, and D8). Statements that contained stewardship ideas and behaviors spoke of ‘developing ‘a sense of place’ in students through understanding the unique social and environmental history of the Flathead Valley’ (D5) and that: It is important to create a value for stewardship to leave a lasting legacy and a history of protective use, but not abuse. (D17). Participants felt that: Students should see and feel the need to protect and sustain a healthy Flathead Watershed. (D6), particularly recognizing the importance of the ‘’intact’ nature of the Flathead Watershed’ (D27). Statements included this area also mentioned ‘appreciation for the Flathead Watershed’ (D8) and that students should ‘develop a proper attitude of caring’ (D2). Getting students outside was suggested as a vehicle for developing stewardship values (D13, D10). The second area that disposition statements addressed was ensuring students were aware of and understood the multiple perspectives in the Flathead Watershed. The second top ranking statement, with a mean score 4.38, was: Students should realize that all points of view should be treated with equal respect and value, using careful listening skills and asking questions. Indigenous perspectives were mentioned in four of the seven statements regarding multiple perspectives. Participants stated: It is valuable to know the Native American perspectives and to understand how the ecosystem has changed; such knowing could lead to a greater appreciation for the Flathead Watershed. M=3.81 (D18). In the statement: Students need to learn to recognize and appreciate their own and other people's values about the same things, like the Flathead Watershed; for instance, the 58 values of Native/Indigenous people who have lived here for 12,000 years, families who have lived here for generations, and people who have moved here last week (D29). M=4.00, participants stated that the perspectives of a wide range of community members were worthy of recognition. Understanding multiple perspectives was considered important in land management issues: Teaching students about multiple perspectives is necessary; there are issues such as management of lake trout in Flathead Lake in which not all the stakeholders agree with the tribal point of view. (D19), M=4.19. Research Question Four 4. What strategies could be used to guide Flathead Watershed curriculum? The teaching statement category was an emergent category created after Round One of the Delphi survey when it was determined that a number of comments contained specific suggestions on ways to teach knowledge, skills or dispositions in a watershed curriculum. Figure 10 shows the ranked teaching statements. Figure 10. Teaching Statement Means with +1 and -1 Standard Deviation 2.00 3.00 4.00 5.00 6.00 T10 T5 T3 T7 T2 T1 T4 T11 T8 T9 T6 L id er t S ca le Teaching Statement Means with +/- 1 SD bars 59 The line for the ranked mean scores for the teaching statements trends slightly downward, while the SD bars do not show a strong trend to lengthening or shortening with descending rank. Several statements, such as T1 and T4, have slightly longer SD bars. T1 statement is: A great Socratic Seminar topic for students could be the statement: "Whatever happens to the water happens to the people.", M= 3.94. This statement received one score of 1 – very unimportant, 5 – neutral, 3 – important, and 7 - very important. The T4 statement is: Teaching about diversity can be accomplished through expression in subject matter teaching. An example is the multiple ways water quality can be introduced: explaining what scientists look for in their work, explaining the cultural importance of native fish and their waterways to indigenous people, or discussing various approaches to resource management of the same water body and why each manager might choose this approach. M=3.94. Analysis of the teaching statements reveals no specific themes or areas into which these statements can be classified. The teaching statements ranged from specific suggestions such as the second ranked statement: It would be a science learning experience as well as fun for students to tour the players in our watershed, such as the Forest Service, wastewater treatment plants, storm water programs (the City of Kalispell has such a program), the County Landfill, etc.(T5), M=4.38, to the general approaches expressed in the top and third ranked statements: Bringing resource people in as guest speakers is a way to get students interested and engaged. M=4.44, (T10), and A project based approach would be a way to get students involved in learning the connections between the past and the present. (T3), M=4.31. 60 The teaching statements showed a range of topics in science, social studies, history, including indigenous perspectives: ‘integration of some of the traditional ecological knowledge’ (T2), M=4.00, and ‘teaching about diversity through explaining what scientists look for in their work, explaining the cultural importance of native fish and their waterways to indigenous people, or discussing various approaches to resource management’ (T4), M=3.94. Research Question Five 5. How do the Delphi survey results relate to the Flathead Watershed Sourcebook? The creation of the Educators Guide, a curriculum based on information compiled in the Flathead Watershed Sourcebook, was the goal of the Steering Committee and the motivation for conducting the Flathead Delphi survey. To determine how the Delphi survey results related to the Sourcebook, the Sourcebook contents were analyzed by major content areas. An outline of the Sourcebook was developed with the Sourcebook contents delineated into chapters, sub-chapters, sections and topics in each sub-section. The final Delphi statements were then matched to the Sourcebook topics in each sub- section in a matrix format. See Table 5 for an example of the Delphi statements matched to the Sourcebook for Chapter 1 of the Sourcebook. The Sourcebook contents were continually checked throughout the process for accuracy in capturing Sourcebook topic content and to ensure that Delphi statement interpretations were suitably matched to relevant Sourcebook content. 61 Table 5. Example of the Sourcebook Delphi Statement Matrix Chapter 1 Sections Topics Delphi Statement codes Learning Objectives What is a Watershed? We All Live Downstream Watershed Description pg. 2 K32, k20, k44, s4, d5, d26, t8, t5, Learning Objective 2: Learning About the Flathead Watershed Students will be able to define and describe what a watershed is, be able to identify the Flathead watershed, and learn about the characteristics of the ecosystem of the Flathead Watershed. The Watershed Community Living Eco-Web of the Watershed pg. 2 k2, k1, K32, K11, K15, k8, k24, k28, k44, k10, k9, s4, s23, d2, d6, d27, d5, d26, t1, t8, t9, t5, The Flathead Watershed Geography of the Flathead Watershed pgs. 3-6 K27, k20, k8, k26, k37, k44, k31, k16, k40, s8, s4, s13, d2, d27, d26, t5, Mapping the Flathead - GIS maps (perspective) pg. 3 k20, k16, s10, s8, s4, s2, s13, Life along a River - riparian zones pg. 7 k32, k14, k2, k24, k26, k10, k31, k16, t10, Twenty-three learning objectives were developed for the areas of greatest match between Delphi statements and Sourcebook content. The learning objectives sought to be inclusive of the maximum amount of both Sourcebook contents and Delphi statements. The Sourcebook Delphi Statement matrix was reviewed by the Steering Committee, who gave their approval of the scope and content of the learning objectives. The twenty-three learning objectives became the basis of the Flathead Watershed Educators Guide. 62 CHAPTER 5: DISCUSSION The purpose of this study was the analysis of the Flathead Watershed Delphi survey statements in the areas of knowledge, skills and dispositions on which to base the place-based Flathead Watershed Educators Guide curriculum. This discussion of the Flathead Delphi study will be divided into examining the value of the survey results and discussing some values of the methodology of the survey. Results Discussion There are five research questions for this study. The first question is: What is the essential knowledge, related to the Flathead Watershed, that students should learn? Participants in the survey expressed a wide range of topics that they felt students should learn about the watershed. This category contained the most statements of all the categories (n=36) and participants rated the statements highly. Participants wrote about key concepts of the watershed such as characteristics of the watershed, biodiversity, ecological cycles within the watershed, geography and geology of the watershed, fire impacts and water quality and quantity; values of the watershed that reflect the unique and pristine environmental conditions of the Flathead Watershed as written about by Curtis (2010) in the Flathead Watershed Sourcebook. The remarkable environmental aspects of the Flathead Watershed provide a highly relevant immersive classroom to teach watershed conservation values to local students. This agrees with the determination by Haury (2000) that the watershed is an appropriate organizing principle for curriculum that instills local ecological and cultural knowledge. Learning about the attributes of the 63 Flathead Watershed was highly valued by Delphi survey participants, reflecting the community values surrounding watersheds expressed by Brandes, et al. (2005). Delphi participant responses reflected the idea behind the statement by Brandes, et al. that it is important to ‘think like a watershed’ in considering the complex interactions of watershed systems, including human roles. It is widely evident in survey responses that participants cared deeply about students knowing about, valuing, and protecting the largely intact ecosystems of the Flathead Watershed, which is an important value in place-based education (Sobel, 2004). Survey participants showed that they recognized and felt the importance of students knowing about their ‘place’, (Gruenewald, 2003), and that the watershed concept worked as a viable theoretical and physical construct to understand and value ‘place’. Human impacts in the watershed were considered an important and relevant topic for students to learn about, both what those impacts are and what actions can be taken to minimize them. The Delphi survey results clearly advocate that students become aware of human impacts and engage in action to conserve and protect their local environment, which is an important tenet of place-based education (Gruenewald, 2003; Orr, 1994; Sobel, 2002). The results of the knowledge category form a solid basis of factual information to include in the Educators Guide about the Flathead Watershed. The second research question of this study is: What are the essential skills, related to the Flathead Watershed, that students should be able to do? This category received considerably fewer statements (n = 19) than the knowledge category, but participants valued the key skills of critical thinking and inquiry, skills that are considered higher 64 order skills in the Revision of Bloom’s Taxonomy, (Krathwohl, 2002). Skills mentioned in the statements centered around learning through scientific inquiry, practicing science process skills, and learning mapping skills, and chart and graph reading skills; areas that are highly important for conducting research, interpreting scientific information and understanding the spatial scope of the Flathead Watershed. These skills are basic to developing an inquiring, scientifically oriented mindset for students. Howley, et al. (2011), describes these same skills as inherent in place-based and environmental education, their importance being that students can think critically, conduct scientific research and work through problem-based projects about issues they face in their local environments. Teachers utilizing place-based education in Howley, et al. mentioned the importance of cultivating students’ ability to think and develop a sense of responsibility. Several other skills were also mentioned in the Delphi statements. Those were evaluating internet sources, using imagination for problem solving, observation skills, and field journaling. The skills listed by Delphi participants represent a small but important spectrum of skills Delphi participants felt students in the Flathead Watershed needed to master to understand issues related to maintaining and enhancing Flathead Watershed attributes. The third research question in this study is: What are the essential dispositions related to the Flathead Watershed that students should have? Delphi participants expressed a tremendous concern for the Flathead Watershed and the environment. They wrote strongly about students valuing and sustaining community in the watershed. This mirrors the benefits attributed to locally based education focused on community process 65 in dealing with issues. Writing about examples of the same philosophy, Smith (2002) speaks of studies researching the State of Maine’s deep commitment through many programs to preparing young people to assume active roles in community processes (pg. 591). Smith goes on to point out that the involvement of students in community process highlights the value and role of those students in the community. Appreciation for the environment, attitudes of caring, feeling the need to protect and sustain a healthy Flathead Watershed, and creating a value of stewardship, are examples of what participants wrote about of which Flathead Watershed students should be aware. One Delphi survey statement in particular sums up the prevailing attitudes and thoughts of the participants: Creating the next generation of "stewards" is a priceless goal that can be reached through establishing a relationship between the student and the outdoors when students are on the land and by the water (D10). It is this idea that is at the heart of place- based education about the local environment. It is clearly a value that Delphi participants seek to instill in their local watershed community students. As Brody (1997) explains it: ‘Educating people for the future in the watershed means helping to empower them to make positive decisions and promote equality and justice for living things. The facts of our lives are embodies in the land, in our “place,”’ (pg. 128-129). Through the connection with the land and people right outside their own doors, place-based education about the local environment seeks to instill the values of conservation and stewardship in the young people living in local, watershed communities. The fourth research question of this study is: What strategies could be used to guide Flathead Watershed curriculum? 66 This question emerged from the statements of survey participants as the research team analyzed participants’ responses in Round One in the initial three categories. Delphi participants are professionals in a range of fields; scientists working with the land and resources available on the land, cultural historians and community resource managers, and educators both in the classroom and in the outdoors. Their knowledge of their own fields and their communities was evident in the ideas they presented about teaching strategies for important concepts and in visits to facilities and natural sites for learning opportunities. This element in the Delphi responses expresses general and Flathead Watershed-specific teaching strategies that can be used in constructing curriculum. Their suggestions provide a valuable source of activities and resources to include in the Educators Guide. The fifth and last research question asks: How do the Delphi survey results relate to the Flathead Watershed Sourcebook? The Sourcebook is a truly rich compilation of information about the Flathead Watershed. It is this abundance of knowledge that inspired the instigation of the Flathead Delphi survey as a means to determine what would be important in the Sourcebook to form a basis for the Educators Guide. After matching the final Delphi statements to the Sourcebook contents, it was evident what areas of information in the Sourcebook would serve to provide strong background and reference material for the Educators Guide. The 23 knowledge, skill, disposition and teaching objectives formulated by the MSU research team from the match of the Delphi statements to the Sourcebook capture the intent of the Delphi statements while 67 incorporating the content of the Sourcebook to create a comprehensive and relevant outline on which to build the Educators Guide. Methodology Discussion The success of the Flathead Watershed Delphi survey in compiling a basis for the Educators Guide is shown by the richness and depth of the statements that constitute the compiled final round results. The initial responses to the Round One prompts were thoughtful, heartfelt opinions that indicated the value participants placed on stewardship of the environment and community in the Flathead Watershed. The Delphi survey methodology created the opportunity for community members to express their own concern for and commitment to the physical and cultural Flathead Watershed attributes discussed in Chapter 1 of this thesis. The effectiveness of the Delphi research methodology in surveying experts to determine a foundation for curriculum is corroborated by evidence from several studies on curriculum development from Delphi consensus statements (Brody, 1995, Brody, et al. 2010, Miller, et al. 2001, Reeves and Jauch, 1978). Examples of Delphi studies come from Brody (1995), who discusses developing a water curriculum based on the expert opinion of educators and scientists from across the United States, and Miller, et al. (2001) whose study determined a training curriculum within a specified field of enterprise, based on the consensus of experts in the field. Common features of the Delphi studies that contributed to their success were the identification of experts in the field, the advantages 68 in conducting the survey online, and the consensus building aspects created from the initial open discussion and multiple ratings of survey statements. A unique and innovative element in the Flathead Watershed Delphi survey is the contributions of local community experts informing the Educator Guide curriculum for the essential knowledge, skills and dispositions that their own community youth should acquire. Building a census of local experts’ opinions about what they consider important for their own community youth to know about their communities and environment is an unexplored area in curriculum development. An advantage of organizing a Delphi survey among people with common concerns and interests is the creation of a social construct that may not have existed prior to the survey. In the case of the Flathead Watershed Delphi survey, participants became members of a local network sharing the same concerns for the environment of their watershed and the education of the youth in the communities of the watershed. A further unique and innovative aspect of the Flathead Watershed Delphi is using the concept of a local watershed as an organizing principle for the Delphi survey. Curricula focused on watersheds is available in a number of forms, such as USGS, Project WET, DNRC, and from non-profit organizations concerned with conservation and resource management. There are examples of curricula for specific watersheds in the United States, such as the Colorado and Missouri watershed curricula from Project WET. In the case of the Flathead Watershed Delphi survey, Flathead Watershed experts who are watershed community members played a key role in the quality of the statements generated from their responses to the survey prompts and to each other. 69 Local experts know their watershed and their communities and care about the young people growing up in the watershed and the communities. Local experts who are community members themselves have a stake in the stewardship of their environment and the successful continuation of their communities. In a sense, asking local experts’ thoughts and opinions for curriculum for their youth is giving them the opportunity of passing on their knowledge and wisdom to educate and nurture the young people who will succeed them in management and conservation of the watershed. One aspect of expert solicitation and participation that would have helped make the Delphi statements more inclusive would have been the successful recruitment and sustained participation of more members of the Confederated Salish and Kootenai Tribes (CSKT). Several members joined the survey initially but did not stay through all the rounds. The contributions of more elementary and middle schoolteachers would have been welcomed as well. Representation was limited to 5 by Round 3 of the survey. There are several areas for further research related to the Flathead Watershed Delphi survey. One area to explore is further analysis of the data collected in the Flathead Watershed Delphi survey. Potential further research would be comparing the responses and rating data for each sub-group of participants, educators, resource professionals and scientists, to each other. There may be trends in the responses for each group that would point to consistent behavior and choices for each group. Another area of potential future analysis of Flathead Watershed data would be to look at statements receiving higher rating in comparison to statements rated with lower scores. Patterns in comparisons may point to methodology issues or trends in participant choices. 70 Other research areas could be assessing of the effectiveness of place-based watershed curriculum, a possibility with the piloting of the Flathead Watershed Educators Guide; and repeating a Delphi survey in another designated watershed as a means to establish a community-based Delphi as a viable option to determine curriculum foundations for watershed place-based curriculums. Conclusion The Flathead Delphi survey to determine a basis for the Flathead Watershed Educators Guide was created to develop a consensus of opinions about the knowledge, skills and dispositions about the Flathead Watershed considered important by conservation- minded and environmentally-minded local resident watershed experts. The emergence of similar or identical concepts and suggestions from a number of participants, combined with participants rating those themes highly, validates the hypothesis that there is foundational knowledge, skills, and dispositions that can be identified to form the basis for the Educators Guide. This study shows that surveying community members who are experts in aspects of their watershed results in a substantial pool of knowledge, skill and disposition statements, agreed upon by consensus, to form a basis for a place-based watershed curriculum. In essence, the Flathead Delphi survey is a research tool to learn about and examine what motivates resident stakeholders to conserve and protect environmental attributes and causes them to expend time and energy on education through curriculum that enhances appreciation and protection of local watershed attributes. 71 A local watershed-based curriculum embodies the concept of place-based education. A place-based curriculum based on the watershed concept, looked at in both biological and cultural terms, recognizes the interdependence of the people and the ecosystems within the physical parameters defined by the flow of water through the landscape. The goal of a place-based watershed curriculum is education to create awareness that leads to stewardship. The use of a Delphi survey for curriculum content development based on the attributes of a watershed is a new concept in education. Determining the values community members hold in relationship to the watershed they live in is an unexplored area of study in environmental education. Further research into the implications of the place-based education and community watershed concept is needed to understand and validate the approach. 72 REFERENCES 73 Anderson, L. W. and Krathwohl, D. R., et al (Eds.) (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Boston, MA: Allyn & Bacon. (Pearson Education Group) Andranovich, G. (1995). Developing Community Participation and Consensus: The Delphi Technique, retrieved from: http://research.wsulibs.wsu.edu/xmlui/handle/2376/4399 Azano, A. (2011). The possibility of place: One teacher’s use of place-based instruction for English students in a rural high school. Journal of Research in Rural Education, 26(10). Retrieved from http://jrre.psu.edu/articles/26-10.pdf. Bachmann, C., Abramovitch, H., Barbu, C. G., Cavaco, A. M., Elorza, R. D., Haak, R., Loureiro, E. Ratajska, A. Silverman, J. Winterburn, S., & Rosenbaum, M. (2013). A European consensus on learning objectives for a core communication curriculum in health care professions. Patient education and counseling, 93(1), 18-26. Bai, Y., Zheng, H., Ouyang, Z., Zhuang, C., & Jiang, B. (2012). Modeling hydrological ecosystem services and tradeoffs: a case study in Baiyangdian watershed, China. Environmental Earth Science, 70, 709–718. doi 10.1007/s12665-012-2154-5 Baker, J., Lovell, K., & Harris, N. (2006). How expert are the experts? An exploration of the concept of ‘expert’ within Delphi panel techniques. Nurse Researcher, 14(1), 59-70. Bloom, B.S. & Krathwohl, D. R. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals, by a committee of college and university examiners. Handbook I: Cognitive Domain. NY, NY: Longmans, Green Brandes, M. O., Ferguson, K., M’Gonigle, M., & Sandborn, C. (2005). At a Watershed: Ecological Governance and Sustainable Water Management in Canada–POLIS Project on Ecological Governance. Urban Water Demand Management. University of Victoria, Victoria, BC, Canada. Brody, M. (1995). Development of a curriculum framework for water education for educators, scientists, and resource managers. The Journal of Environmental Education, 26(4), 18-29. Brody, M. (1997). Descending the watershed. Canadian Journal of Environmental Education, 2, 114-131. Brody, M., Fisher Jr., J., Moe, J., & Keremedjiev, H. (2010). Archaeological Science for All Phase I: National Delphi Survey Project Report NSF Award# 0840065 74 https://www.academia.edu/826611/ArchaeologicalScienceforAllPhaseINationalD elphiSurveyProjectReport123110NSFAward0840065 Clayton, Mark J. (1997). Delphi: a technique to harness expert opinion for critical decision‐making tasks in education, Educational Psychology, 17(4), 373-386. doi:10.1080/0144341970170401 Cotton, D.R.E. (2006). Implementing curriculum guidance on environmental education: the importance of teachers' beliefs, Journal of Curriculum Studies, 38(1), 67-83. doi: 10.1080/00220270500038644 Curtis, Lori S. (2010). Flathead Watershed Sourcebook, Retrieved from: http://www.flatheadwatershed.org/ Custer, R. L., Scarcella, J. A., & Stewart, B. R. (1999). The modified Delphi technique: A rotational modification. Journal of Vocational and Technical Education, 15(2), 1-10. Dalkey, N., & Helmer, O. (1963). An Experimental Application of the Delphi Method to the Use of Experts, Management Science, 9(3) 458-467. Day, J. & Bobeva, M. (2005). A Generic Toolkit for the Successful Management of Delphi Studies. The Electronic Journal of Business Research Methodology 3(2), 103-116. Retrieved from http://www.ejbrm.com/main.html Delbecq, A., Van de Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program planning: A guide to nominal group and Delphi processes (Management applications series). Glenview, Ill.: Scott, Foresman. Dewey, J. (1926). My Pedagogic Creed, The Journal of Education 104(21) 542. Ernst, J. (2007). Factors Associated with K-12 Teachers Use of Environment-based Education, The Journal of Environmental Education, 38(3), 15-32. doi:10.3200/JOEE.38.3.15-32 Flanders, F. (1988). Determining curriculum content for nursery/landscape course work in vocational agriculture for the 21st century: a futures study utilizing the Delphi technique. (Doctoral dissertation). Retrieved from http://search.proquest.com/docview/303556966/ Gail, P. A. (1974). A Curriculum Activities Guide to Watershed Investigations and Environmental Studies, Volume 6. Revised. (ERIC Number: ED378054) Gay, L. R., Mills, G. E., & Airasian, P. W. (2011). Educational research: Competencies for analysis and applications. Pearson Higher Ed. 75 Gregersen, H. M., Ffolliott, P. F., & Brooks, K. N. (2007). Integrated watershed management: Connecting people to their land and water. CABI. Gowin, D. B. (1981). Educating. Ithaca, New York: Cornell University Press. Gruenewald, D. A. (2003). Foundations of place: A multidisciplinary framework for place-conscious education. American Educational Research Journal, 40, 619 – 654. Gruver, J. & Luloff, A. E (2008). Engaging Pennsylvania Teachers in Watershed Education. The Journal of Environmental Education, 40(1), 43-54. doi: 10.3200/ JOEE.40.1.43-54 Haney, J., Czerniak, C., & Lumpe, A. (1996). Teacher beliefs and intentions regarding the implementation of science education reform strands. Journal of Research in Science Teaching, 33, 971–993. Harrow, A. (1972). A Taxonomy of Psychomotor Domain: A Guide for Developing Behavioral Objectives. New York: David McKay. Hasson, F., & Keeney, S. (2011). Enhancing rigor in the Delphi technique research. Technological Forecasting and Social Change, 78(9), 1695-1704. Haury, D. L. (2000). Studying Watersheds: A Confluence of Important Ideas. (ERIC Digest No. 12). Retrieved from ERIC database. (ED463950) Howley, A., Howley, M., Camper, C., & Perko, H. (2011). Place-Based Education at Island Community School. The Journal of Environmental Education, 42(4), 216- 236. Katz, L. (1993). Dispositions: Definitions and Implications for Early Childhood Practices Retrieved from: http://files.eric.ed.gov/fulltext/ED363454.pdf Keeney, S., Hasson, F., & McKenna, H.P. (2000). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies, 38, 195-200. Kennedy, H. (2004). Enhancing Delphi research: methods and results. Journal of Advanced Nursing, 45(5), 504-511. Krathwohl, D. R. (2002). A Revision of Bloom's Taxonomy: An Overview. Theory into Practice, 41(4), 212 -218. Linstone, H., & Turoff, M. (1975). The Delphi method: Techniques and applications. Reading, Mass.: Addison-Wesley Pub., Advanced Book Program. 76 Lucero, M., Valcke, M., & Schellens, T. (2013). Teachers' Beliefs and Self-Reported Use of Inquiry in Science Education in Public Primary Schools. International Journal of Science Education, 35(8), 1407-1423, doi: 10.1080/09500693.2012.704430 Marchais-Roubelat, A., & Roubelat, F., (2011). The Delphi method as a ritual: Inquiring the Delphic Oracle, Technological Forecasting & Social Change, 78, 1491–1499 McInerney, P., Smyth, J., & Down, B. (2011). ‘Coming to a place near you?’ The politics and possibilities of a critical pedagogy of place-based education. Asia-Pacific Journal of Teacher Education, 39(1), 3-16. doi:10.1080/1359866X.2010.540894 Miller, J. K., Repinski, S. L., Hayes, K. N., Bliss, F. A., & Trexler, C. J. (2011). Designing Graduate-Level Plant Breeding Curriculum: A Delphi Study of Private Sector Stakeholder Opinions. Journal of Natural Resources and Life Science Education, 40, 82–90. doi:10.4195/jnrlse.2010.0031g Moore, C. M. (1987). Delphi technique and the mail questionnaire. Group Techniques for Idea Building: Applied Social Research Methods, 50-77. National Research Council. (2009). Learning science in informal environments: People, places, and pursuits. Washington, DC: The National Academies Press. Nelson, J. G., & Lawrence, P. L. (2009). Places: Linking nature, culture and planning. Calgary, Alta., University of Calgary Press. Novak, J. D. & Gown, D. B. (1984). Learning How to Learn. New York, NY: Cambridge University Press. Orr, D. W. (1994). Earth in mind: On education, environment, and the human prospect. Washington, DC: Island Press. Reeves, G., & Jauch, L.R. (1978). Curriculum Development Through Delphi, Research in Higher Education, 8, 157-168. Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: issues and analysis. International Journal of Forecasting, 15, 353 –375. Rowe, G., & Wright, G. (2011). The Delphi technique: Past, present, and future prospects - Introduction to the special issue. Technological Forecasting & Social Change, 78, 1487–1490. Saavedra, A. R. & Opfer, V.D (2012). Learning 21st-century skills requires 21st-century teaching. Phi Delta Kappan, 94(2) doi: 10.1177/003172171209400203 77 Schulte, L., Donahey, J., Gran, L., Isenhart, T., & Tyndall, J. (2010). People in Ecosystems/Watershed Integration: A dynamic watershed tool for linking agroecosystem outputs to land use and land cover. Journal of Soil and Water Conservation, 65(2), 33A-36A. Smith, G. (2002). Place-based education: Learning to be where we are. Phi Delta Kappan, 83, 584-594. Semken, S. and Freeman, C. B. (2008). Sense of place in the practice and assessment of place-based science teaching. Science Education, 92, 1042–1057. doi: 10.1002/sce.20279 Sobel, D. (2004). Place-based education: Connecting classrooms and communities. Nature Literacy Series Vol. 4. New Patriotism Series 4. Great Barrington, MA: The Orion Society. Stapp, W. B., & Polunin, N. (1991). Global environmental education: Towards a way of thinking and acting. Environmental Conservation, 18(01), 13-18. Theobald, P., & Curtiss, J. (2000). Communities as curricula. Forum for Applied Research and Public Policy, 15(1), 106-111. Wilson, L. O. (2016). The second principle, Retrieved from http://thesecondprinciple.com/instructional-design/threedomainsoflearning/ Wither, S. B. (2001, April). Local curriculum development: A case study. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA. (ERIC Document Reproduction Service No. ED 456 022). Vandas, S. (1997). Where we live. Science and Children, 34(7), 28 Woodhouse, J. L. & Knapp, C. E. (2000). Place-based curriculum and instruction: outdoor and environmental education approaches. ERIC Digest. ED448012 Zagarola, J. P., Anderson, D. B., & Veteto, J.R. (2014). Perceiving Patagonia: an assessment of social values and perspectives regarding watershed ecosystem services and management in southern South America. Environmental Management, 53(4), 769-782. 78 APPENDICES 79 APPENDIX A DELPHI SURVEY PARTICIPANTS 80 Delphi Participants Last Name First Name Position Affiliation County Cajune Julie Educator Confederated Salish & Kootenai Tribes (CSKT) Lake Fassnacht Deb Educator Salish Kootenai College Lake Bahr Melissa Educator Polson Middle School Lake Hill Kathy Educator Olney-Bissell School Flathead Jakes Randy Educator Helena Flats School Flathead Blank Dani Educator Somers Middle School Flathead Garlie Gina Educator Kalispell Middle School Flathead McGunagle Tim Educator Whitefish Middle School Flathead Dover Mary Caye Educator West Valley School Flathead Wenum Teresa Educator Flathead National Forest Flathead Sedivy Heidi Educator Flathead Lakers Flathead McCormick Sue Educator Retired Teacher Flathead Muhlfeld Clint Resource Manager/Educator US Geological Survey/ Flathead Lake Biological Station Flathead Becker Dale Resource Manager CSKT Wildlife Manager Lake Emrick Joni Resource Manager Kalispell Wastewater Flathead Brenneman Joe Resource Manager Rancher Flathead 81 Last Name First Name Position Affiliation County Maskill Mark Resource Manager Creston National Fish Hatchery Flathead Kendall Craig Resource Manager US Forest Service Flathead Mackenzie Paul Resource Manager F H Stolze Land & Lumber Flathead Philmon Dennis Resource Manager Hungry Horse Dam Flathead Curtis Lori Resource Manager Whitefish Lake Institute Flathead Mason Patti Resource Manager Flathead Conservation District Flathead Gubits Josh Technical Person/Educator Whitefish Lake Institute Flathead Wargo Rebekah Technical Person City of Kalispell Flathead Hoffman Tris Technical Person Flathead National Forest Flathead Wondrow Carolyn Technical Person Museum at Central School Flathead Bansak Tom Technical Person Flathead Lake Biological Station Lake Burch Susie Technical Person Flathead Valley Community College Flathead White Germaine Technical Person CSKT Lake Brody Michael Investigator Montana State University Gallatin Yates Kim Investigator Montana State University Gallatin Vallor Rose Investigator Montana State University Gallatin Obery Amanda Investigator Montana State University Gallatin 82 APPENDIX B ROUND ONE SAMPLE RESPONSES 83 Round I: Knowledge Discussion Thread Participant No. Response What are the essential concepts/issues that students should learn related to the Flathead Watershed? (For example; water quality or fish species) Participant 003 I believe in the four C's clear, connected, complex and cold. Those four things are essential for the bull trout, an essential animal of the Flathead Watershed. The program I taught focused on water quality. We tested turbidity, temperature and dissolved oxygen at each site we visited. We discussed each body of water's quality. We also focused on the plants and animals of the watershed. We discussed invasive species and their impact on the watershed. Geography is also another important concept. Landforms, water connections streams to rivers to lakes are essential for students to connect what they learning about. Fire can be an essential topic too for our area. Participant 029 I love this idea. What grade-level do think this type of investigations appropriate for? Do you think classroom teachers would be able to conduct something similar to this on their own or would it require some training? Participant 026 I agree that geography is another key concept...so students learn and understand the water cycle and the 4C's and are able to put it together for their 'place', the waters in the Flathead....like the connection to Big Creek that was mentioned. Having students know the location of key rivers, lakes, naming them, knowing where the headwaters are, the unique features, know there is a shallow aquifer and its location...making a connection to these places where they may fish, swim or boat, or drink the water, then how to take care of these places... 84 Round I: Skills Discussion Thread Participant No. Response What skills should students develop that would help them learn about and live in the Flathead Watershed? (For example; reading maps or writing a letter to city council) Participant 027 A student should be able to: - create, read and interpret a map, from the schoolyard map to the watershed - read literature related to the landscape/ecology and write about where they live - interpret data and graphs, see page 7 in the Sourcebook… Participant 020 I agree, this study should be done across the curriculum with the coordination of activities planned with teachers from all areas of study, not just science, but math, history and language arts. Round I: Dispositions Discussion Thread Participant No. Response What attitudes and values should students develop related to the Flathead Watershed? (For example; appreciating Indigenous perspectives or valuing diverse opinions about resource management) Participant 006 Students need to develop an appreciation for the importance of the Flathead Watershed ecosystem and a desire to maintain it in good condition. Knowing the Native American perspectives and understanding how the ecosystem has changed over the years would be valuable as well and could lead to a greater appreciation for the Flathead Watershed. Participant 010 All points of view should be treated with equal respect and value, using careful listening skills and asking questions. 85 APPENDIX C ROUND TWO RANKED STATEMENTS 86 Round II: Knowledge Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 K33 Students need to know the basic premise of a watershed, that it is where they hunt and fish and recreate, and work, and live, and eat and grow food and consume food, and that as a result; they are also influencing the watershed with all their everyday actions. 0 0 1 4 17 4.52 K1 Students should be aware why the watershed is important to the species of fish and wildlife that utilize the watershed. 1 0 0 8 14 4.48 K43 Students should know the effects on the watershed of our actions; examples are: when we flush, put things down the drain, fertilize our lawns, have oil leaks. 0 0 1 5 16 4.48 K2 Students should be knowledgeable about the plants and animals of the watershed, including invasive species and their impact on the watershed. 0 1 1 8 13 4.43 K39 Students need to have an understanding of the human impact, both positive and negative on the watershed. 0 0 0 8 14 4.43 K47 It is important for students to engage in fieldwork because it is more effective in building enthusiasm, helping with future career choices and fostering stewardship. 0 0 0 8 14 4.43 K48 How the watershed impacts their lives and why they should care about it is essential information for students. 0 0 0 8 14 4.43 K54 It is important for students to understand that we all live in a watershed and cannot separate ourselves from it, and that our actions can have effects on downstream users, water quality, and aquatic ecosystems. 0 0 0 8 14 4.43 K4 Students should be aware that a major threat to the Flathead watershed is water quality degradation, predominantly nutrient and sediment additions caused by human activities. 0 0 0 9 13 4.39 K55 Students should understand the characteristics of a healthy watershed. 0 0 0 9 13 4.39 K52 It is important for students to know how humans impact the watershed. 0 0 0 10 12 4.35 87 Round II: Knowledge Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 K14 Students should learn that the underground aspects of the watershed are important and that the underground reaches of the Flathead River can be more than a mile from its banks with water flowing in the underground soils and gravels, therefore pollutants entering the watershed in the upper reaches will potentially make their way into the underground river as well. 0 0 1 9 12 4.30 K56 Understanding what a watershed is and what bodies of water contribute to our watershed is essential knowledge for students. 0 0 1 10 11 4.26 K44 Students should be aware of the aspects of water Q water cycle, bioregions, watersheds, water bodies (headwaters, streams, rivers, tributaries, marshes, sloughs, lakes, oceans), alluvium, surface water, ground water, downstream, upstream, water pollution (point source, nonpoint source) nutrient load. 0 0 3 8 11 4.17 K15 Students should be knowledgeable about the cycles in a watershed. 0 1 2 8 11 4.13 K23 It is important that students study the relationship of water quality and economic vitality in the Flathead. 0 0 3 9 10 4.13 K42 Where the water comes from when the faucet is turned on and how it is treated to be drinkable is important information for students. 0 1 2 8 11 4.13 K21 Students should be aware of the stormwater conveyance system and that storm drains flow directly into our water bodies without treatment. 1 1 0 9 11 4.09 K24 Students should be exposed to the biology and chemistry that relate to the Flathead Watershed, specifically water quality, organisms, ecosystems and diversity within the watershed. 0 0 3 10 9 4.09 K13 Students should learn about "indicator" species such as bull trout and about its health and continued persistence as indicators of the health of the watershed. 0 0 3 11 8 4.04 K45 It is important for students to recognize the difference between "belief systems" and "facts". 0 1 3 8 10 4.04 88 Round II: Knowledge Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 K40 Students should be aware of the international aspects of the Flathead Watershed and its reach into Canada, including human impacts, since a watershed does not recognize international borders. 0 0 4 10 8 4.00 K49 Knowledge gained through what is referred to as "STEM" (science, technology, engineering and math) courses is important to being able to make informed decisions about our watershed ecology and water quality. 0 0 3 12 7 4.00 K51 The history of the Flathead Watershed is important information for students. 0 0 2 14 6 4.00 K18 Fire is an essential topic for students. 0 0 6 7 9 3.96 K34 Knowledge of resources, natural, historical, cultural, land and water use (agriculture, mining, and roads, residential, commercial), recreation, conservation, dams, and irrigation is essential information for students. 0 0 2 10 9 3.96 K38 Learning about the influences of man through building communities, agriculture, recreation, and industry is relevant knowledge. 0 0 3 13 6 3.96 K3 Nonnative species (mostly introduced intentionally by managers) are major threats that are important for students to learn about in the Flathead Watershed. 2 0 0 12 8 3.91 K9 It is important to teach about natural environmental change in general. 0 1 3 11 7 3.91 K17 Understanding the needs of competing users of water in the Flathead Watershed is relevant information, especially with the challenges of climate change in the future. 0 1 5 7 9 3.91 K16 It is important that students learn that a river system affects the whole watershed not only by flowing partially below the surface, but also by how wildlife transfers nutrients from the stream to the upland. 0 0 6 9 7 3.87 89 Round II: Knowledge Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 K35 Management of the watershed for ecological, social, and economic values is important information for students to be aware of. 0 2 2 11 7 3.87 K6 Mitigation and restoration methods are important information for students to learn and know. 0 2 1 14 5 3.83 K10 A key concept for students would be an understanding of the nature and need for biodiversity. 1 0 3 7 10 3.83 K12 The four C's (clear, connected, complex and cold) are an essential concept for understanding of the Flathead watershed, particularly for animals such as the bull trout, an essential animal of the Flathead Watershed. 1 0 2 14 5 3.83 K29 Learning, how has this watershed changed over time, geologic, historic, present, is important information for students to be informed about. 0 0 3 11 7 3.83 K31 Students should understand that the stream system is the lifeblood of the watershed; it feeds it and is fed by it. 0 0 5 12 5 3.83 K36 Students should be aware of the storm water conveyance system and that storm drains flow directly into our water bodies without treatment. 1 1 1 8 10 3.83 K37 Students should be aware that there is a finite supply of clean water. 2 0 4 6 10 3.83 K46 Students should understand how systems function, including biotic and abiotic subsystems and how they are inextricably linked. 0 0 5 12 5 3.83 K53 The concept of international cooperation in protecting a watershed is an important piece for students to learn. 0 1 4 11 6 3.83 K57 Students should know the unique natural and cultural features of the Flathead Watershed. 0 0 3 11 7 3.83 K20 Learning to test water quality by testing turbidity, temperature and dissolved oxygen at sites along water bodies throughout the watershed is relevant knowledge for students. 0 0 6 11 5 3.78 90 Round II: Knowledge Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 K28 The concept that there is one environment, not a "human environment" and a "natural environment". 0 2 5 7 8 3.78 K50 It is important that students know about watershed communities. 0 0 7 9 6 3.78 K7 An essential concept for students to know is how water was used to establish the Flathead basin. 1 0 3 14 4 3.74 K19 Indian Education for All is knowledge that students should have. 1 1 6 5 9 3.74 K25 Understanding the history of how water has been managed is important. 0 2 3 12 5 3.74 K41 Have students know the names and location of key rivers and lakes, where the headwaters are, the location of the shallow aquifer, and other unique features within the watershed. 0 1 5 11 5 3.74 K32 Students should know the species composition of the watershed. 0 1 3 12 5 3.65 K5 Knowledge of the agencies and groups that are there to help and nurture the watershed is important information for students to know. 0 1 9 8 4 3.52 K22 Water is a natural connector, both physically and conceptually. 0 4 7 4 6 3.26 K30 Students should understand that essential concepts and issues in the Flathead watershed would start with the essential nature of water. 0 1 11 5 4 3.26 K8 Politics is not important to include in the education guide. 3 3 7 4 4 2.87 K11 It is important to teach about water quality, organisms, ecosystems and diversity within the watershed without discussing the issue of climate change. 5 1 6 4 5 2.87 91 Round II: Skills Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 S20 Basic scientific inquiry and critical thinking are important skills for students. 0 0 0 5 15 4.75 S28 Listening and observing is an important skill. 0 0 0 6 14 4.7 S16 Critical thinking is important. Students should be taught how to ask the tough questions and then learn how to look for answers through research. 0 0 0 7 13 4.65 S26 Students, when looking to the Internet for information, need to always evaluate the source of information. 0 0 1 5 14 4.65 S21 Students should be able to make observations, identify problems, formulate relevant questions, use appropriate tools and technology to collect and analyze data, make inferences, draw conclusions, communicate and defend findings. 0 0 0 8 12 4.6 S10 A student should be able to interpret data and graphs. 0 0 1 7 12 4.55 S25 It is important to teach students to look beyond the rhetoric/popular opinion and ask their own questions, as well as how to search for their own answers or solutions provides a life skill. 0 0 1 8 11 4.5 S27 Imagination is important to encourage as students think of possible solutions. 0 0 1 10 9 4.4 S22 The ability to create tables, picture graphs, bar graphs and picture maps to record and organize information and to be able to read, interpret, and use tables and graphs to identify patterns and trends, draw conclusions, and make predictions are important skills for students to have. 0 1 0 10 9 4.35 S8 A student should be able to create, read, and interpret a map, from the schoolyard map to the watershed. 0 0 3 10 7 4.2 S7 Field studies are great. 0 0 2 13 5 4.15 S23 Describing and explaining how biodiversity in animal and plant life affects a watershed. 0 0 3 11 6 4.15 92 Round II: Skills Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 S24 Recognize and propose explanations for patterns of change in a watershed over time, including but not limited to changes in climate, animal and plant life, human activities and interactions. 0 2 3 5 10 4.15 S29 Learning skills such as how to fish, hunt, hike, raft, etc. takes you to places that gives you a real appreciation for the area, and helps people to come to their own conclusions about the importance of conservation of our resources. 0 1 3 9 7 4.1 S35 Observation skills are essential so that students can sit quietly and observe the world around them. 0 0 4 10 6 4.1 S33 Students should be able to read maps and landmarks and know the reservation watersheds. 0 0 6 7 7 4.05 S37 Practicing observation skills through participation in a variety of observation exercises using a variety of senses can start in the classroom, then move to the school yard, and finally students can implement their newly developed skills out in the field in the in the Flathead Watershed. Flathead Watershed. 0 0 4 11 5 4.05 S1 Scientific investigations involving developing questions, collecting data, analyzing data, and comparing data are skills students should have. have. 3 1 0 5 11 4 S19 Teach students skills to help them explore new places, like how to use a GPS. 0 0 6 8 6 4 S40 Students should begin to conduct "research". 0 1 4 9 6 4 S17 Learning how to use a compass is an important and engaging outdoor skill. 0 1 3 12 4 3.95 S18 Students can learn about and appreciate the Flathead Watershed through skills such as canoeing, rafting, kayaking, swimming, fishing, bird watching, boating etc. 0 1 4 10 5 3.95 93 Round II: Skills Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 S39 Students should be able to know how to delineate a watershed; this means being able to read a map and decide where the ridges are and where the valleys are, and where a droplet of water would flow were it to drop at any point on that map. 0 1 5 8 6 3.95 S9 A student should be able to read literature related to the landscape/ecology and write about where they live. 0 2 3 10 5 3.9 S3 Conducting stream surveys, including water quality tests, stream flow, water depth, and sampling for macro invertebrates, is an essential skill. 0 0 7 9 4 3.85 S30 Format the curriculum to make information available, but do not build perceptions. 0 0 8 7 5 3.85 S34 Create homework assignments that require students to investigate their environment. 0 0 9 6 5 3.8 S38 Nature journaling" or "field journaling" gives young students the tools they need to quietly observe what is going on around them. 0 1 7 9 3 3.7 S6 Nature journaling, poetry, art, and similar activities help with connection to caring for the watersheds we live in. 1 0 6 11 2 3.65 S21 Identification skills using keys are important. 1 0 8 8 3 3.6 S31 Students should learn technical skills; analyzing aerial imagery, utilizing information resources such as USGS flow data websites and getting in the field to take measurements at sites such as a gaging station and a snowtel site, so the data is understood. 0 2 6 10 2 3.6 S4 Activities that are important in watershed education are planting trees, digging up invasive weeds, soil sampling, native and non-native id, restoration projects, bird watching, and tracking. 1 1 4 14 0 3.55 S15 Take a picture of nature and then describe it to friends using words. 0 3 8 7 2 3.4 S36 Skills should include mapping an area, knowing the lentic and lotic habitats in the region and collecting data about species diversity. 0 3 9 5 3 3.4 94 Round II: Skills Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 S14 Diverse learners can be engaged through learning visually and then communicating what they have learned in more than one format, such as through art and photography. 0 5 6 6 3 3.35 S12 A student should be able to take a picture of nature and share it with friends. 1 3 9 3 4 3.3 S11 Students should be able to understand and apply mathematical algorithms to tasks such as calculating sediment load. 0 7 4 6 3 3.25 S32 Having students go through the permitting process at least in a conceptual manner. 1 5 8 5 1 3 S13 A student should be able to express their relationship with the watershed through art. 2 3 9 6 0 2.95 S5 Designing filters for cleaning water is an important skill. 1 6 11 1 1 2.75 95 Round II: Dispositions Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 D23 It is important for students to know what they can do to help protect our local watersheds. 0 0 0 6 13 4.68 D25 Raising student awareness of watershed issues will hopefully influence their future actions in addressing the issues as adults. 0 0 1 8 10 4.47 D27 It is important that students in the Flathead Watershed understand that this ecosystem is one of the most intact, with exceptional water quality, and that rivers, floodplains, and aquatic habitats are well connected and functional; and that many similar ecosystems in the lower 48 have been degraded. 0 0 2 8 9 4.37 D15 It is important to understand the balance between using the land and abusing the land. 0 0 1 11 7 4.32 D26 Learning about the local ecosystem will create a sense of stewardship. 0 0 1 11 7 4.32 D32 It is important that teachers be committed to getting kids outside, for several reasons; first, the lesson takes on life. Second, being at that place establishes a relationship between the student and that site. 0 0 2 9 8 4.32 D16 Nurturing a balance of use and protection within the watershed, of providing for our needs while caring for the watershed's health, is valuable to instill and develop in students. 0 1 2 7 9 4.26 D30 Teaching students about multiple perspectives is necessary; there are issues such as management of lake trout in Flathead Lake in which not all the stakeholders agree with the tribal point of view. which not all the stakeholders agree with the tribal point of view. 0 0 3 8 8 4.26 D33 Creating the next generation of "stewards" is a priceless goal that can be reached through establishing a relationship between the student and the outdoors when students are on the land and by the water. 0 0 4 6 9 4.26 D4 An appreciation for the natural and cultural history of the Flathead Watershed is an essential attribution for students. 0 0 3 9 7 4.21 96 Round II: Dispositions Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 D9 Explicitly create learning opportunities that develop real awareness of this remarkable place so that the Flathead environment is not taken for granted. 1 0 1 9 8 4.21 D11 Students should understand indigenous perspectives, but also understand more recent ties to the land, including early homesteaders, farmers, loggers, and industrial. 0 0 5 5 9 4.21 D22 The major issues affecting watersheds today, such as water rights, toxic spills from rail or truck, toxic cleanup, industrial pollution, are important issues for students to learn about. 0 1 3 6 9 4.21 D3 It is important that students engage in life styles (choices and decisions) that protect and sustain a healthy Flathead Watershed. 0 0 4 8 7 4.16 D5 For students to develop a strong 'sense of place', they should understand the unique social and environmental history of the Flathead Valley. 0 0 4 8 7 4.16 D20 History is important to learn about because it shows what human influence has done to the watershed and how it left it today. 0 0 5 6 8 4.16 D24 Learning about amounts and types of water consumption is relevant for students. 0 0 3 10 6 4.16 D28 Students should realize that all points of view should be treated with equal respect and value, using careful listening skills and asking questions. 0 1 4 5 9 4.16 D2 Show students how the watershed impacts their lives and how they impact the watershed to create and share a proper attitude of caring. 1 0 1 11 6 4.11 D6 Students should see and feel the need to protect and sustain a healthy Flathead Watershed. 0 1 3 8 7 4.11 D8 For students, developing a sense of appreciation for the Flathead Watershed is paramount to instilling a desire to take care of their environment environment. 0 1 3 8 7 4.11 97 Round II: Dispositions Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 D1 It is important that students understand the profound wisdom and far reaching implications of the statement: "Whatever happens to the water happens to the people." 0 1 4 7 7 4.05 D12 It is important that students develop the concept of conservation and understand the difference between conservation and preservation. 0 1 4 8 6 4.00 D17 It is important to create a value for stewardship to leave a lasting legacy and a history of protective use, but not abuse. 0 0 3 8 7 4.00 D18 It is valuable to know the Native American perspectives and to understand how the ecosystem has changed; such knowing could lead to a greater appreciation for the Flathead Watershed. 0 0 5 9 5 4.00 D21 Legacy is a great concept to teach and discuss; let students define what they want their legacy of stewardship to be. 0 1 5 6 7 4.00 D14 Learning from the pioneers who are still living in the Flathead Valley is important, as they are able to provide historical perspective. 0 0 6 8 5 3.95 D29 Students need to learn to recognize and appreciate their own and other people's values about the same things, like the Flathead watershed; for instance, the values of Native/Indigenous people who have lived here for 12,000 years, families who have live here for generations, and people who have moved here last week. 0 2 5 4 8 3.95 D34 Salish, Kootenai and Pend d'Oreille history and cultural resources, such as films and books, have a tribal perspective and are important sources of information on science, history and contemporary information, including topics such as Indian Water Rights and native place names, which should be used in classrooms in the Flathead Watershed. 1 1 5 6 6 3.79 98 Round II: Dispositions Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 D31 Students should recognize their own and all people's relationship to place and should be helped to share the feelings, attitudes, values and beliefs that have developed because of their relationships to their environment. 0 1 6 9 3 3.74 D19 Flathead Watershed students have a significant opportunity to learn about Native opinions and actions, historical and current day, because native peoples have lived there continuously. 0 1 5 8 4 3.63 D13 Understanding the history of wise use in the Flathead Valley, especially in the past 120 years, is an essential understanding for students. 1 0 7 10 1 3.53 D10 Appreciation for the environment is a hard thing to develop in a young person when the land they have grown up in is all they have ever known, but it is worth trying. 2 0 7 4 5 3.37 D7 Students should advocate for and support organizations, programs and initiatives that protect and sustain a healthy Flathead Watershed. 3 0 7 6 3 3.32 99 Round II: Teaching Methods Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 T11 Bringing resource people in as guest speakers is a way to get students interested and engaged. 0 0 1 11 8 4.35 T6 It would be a science learning experience as well as fun for students to tour the players in our watershed, such as the Forest Service, wastewater treatment plants, storm water programs (the City of Kalispell has such a program), the County Landfill, etc. 0 1 1 9 9 4.3 T3 A project based approach would be a way to get students involved in learning the connections between the past and the present. 0 0 2 13 5 4.15 T1 A great Socratic Seminar topic for students could be the statement: “Whatever happens to the water happens to the people." 0 1 4 7 8 4.1 T8 Appropriate guides should be used for students to learn about the plants and animals in the Flathead watershed, instead of memorizing. 0 1 3 10 6 4.05 T9 Reading about nonfiction topics is a good way to learn about animals and the watershed and satisfy common core standards requirements for young students. 0 1 5 8 6 3.95 T10 It is possible to address climate change in a neutral way through water and the interconnected nature of surface and groundwater. Precipitation decreases can affect surface flows and groundwater recharge/discharge. If temperature changes the storage of water through decreased snowpack, that to can affect groundwater availability as well as surface water flows. 0 1 5 10 4 3.85 T4 Teaching about diversity can be accomplished through expression in subject matter teaching. An example is the multiple ways water quality can be introduced: explaining what scientists look for in their work, explaining the cultural importance of native fish and their waterways to indigenous people, or discussing various approaches to resource management of the same water body and why each manager might choose this approach. 0 1 4 13 2 3.8 100 Round II: Teaching Methods Statement Number Statement Likert Scale Response Mean Score 1 2 3 4 5 T12 Resource education events can and should be conducted outside on the lawns of schools or in neighborhood parks that may be in walking distance. 0 0 7 11 2 3.75 T2 Integration of some of the traditional ecological knowledge that has been passed down through generations of native people might best be incorporated into any learning process by directly engaging the native people in the development of educational and informational programs. 0 1 7 9 3 3.7 T7 Students can discover 'snow towers' and the progression of snow melt through field trips and via visual presentations. 0 1 6 11 2 3.7 T5 Teaching the 'crumple a watershed' activity is a good way to communicate the watershed concept and develop vocabulary. 0 0 15 3 2 3.35 101 APPENDIX D ROUND THREE KNOWLEDGE STATEMENTS 102 Round III: Knowledge Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 K2 Students should be knowledgeable about the plants and animals of the watershed, including invasive species and their impact on the watershed. 0 0 0 5 11 4.69 0.48 K1 Students should be aware why the watershed is important to the species of fish and wildlife that utilize the watershed. 0 0 0 6 10 4.63 0.50 K30 It is important for students to know how humans impact the watershed. 0 0 0 6 10 4.63 0.50 K32 Students should understand the characteristics of a healthy watershed. 0 0 0 6 10 4.63 0.50 K11 Students need to know the basic premise of a watershed, that it is where they hunt and fish and recreate, and work, and live, and eat and grow food and consume food, and that as a result; they are also influencing the watershed with all their everyday actions. 0 0 2 4 10 4.50 0.73 K15 Students should be knowledgeable about the cycles in a watershed. 0 0 0 8 8 4.50 0.52 K27 Geography, landforms, water connections to streams to rivers to lakes are essential concepts for students. 0 0 0 8 8 4.50 0.52 K33 It is important for students to understand that we all live in a watershed and cannot separate ourselves from it, and that our actions can have effects on downstream users, water quality, and aquatic ecosystems. 0 0 1 6 9 4.50 0.63 K39 Students need to have an understanding of the human impact, both positive and negative on the watershed. 0 0 0 8 8 4.50 0.52 K4 Students should be aware that a major threat to the Flathead Watershed is water quality degradation, predominantly nutrient and sediment additions caused by human activities. 0 1 1 4 10 4.44 0.89 103 Round III: Knowledge Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 K14 Students should learn that the underground aspects of the watershed are important and that the underground reaches of the Flathead River can be more than a mile from its banks with water flowing in the underground soils and gravels, therefore pollutants entering the watershed in the upper reaches will potentially make their way into the underground river as well. 0 0 1 7 8 4.44 0.63 K20 Understanding what a watershed is and what bodies of water contribute to our watershed is essential knowledge for students. 0 0 0 9 7 4.44 0.51 K43 Students should know the effects on the watershed of our actions; examples are: when we flush, put things down the drain, fertilize our lawns, have oil leaks. 0 0 1 7 8 4.44 0.63 K8 Students should know the unique natural and cultural features of the Flathead Watershed. 0 0 1 8 7 4.38 0.62 K23 It is important that students study the relationship of water quality and economic vitality in the Flathead. 0 0 0 10 6 4.38 0.50 K24 Students should be exposed to the biology and chemistry that relate to the Flathead Watershed, specifically water quality, organisms, ecosystems and diversity within the watershed. 0 0 0 10 6 4.38 0.50 K26 Knowing about the land and landmasses, soil, rock, sediment, mountains, valleys, plains, flood plains, and wetlands is important for students. 0 0 1 8 7 4.38 0.62 K28 The concept that there is one environment, not a "human environment" and a "natural environment". 0 0 2 6 8 4.38 0.72 K35 Management of the watershed for ecological, social, and economic values is important information for students to be aware of. 0 0 0 10 6 4.38 0.50 K37 Students should be aware that there is a finite supply of clean water. 0 1 0 7 8 4.38 0.81 104 Round III: Knowledge Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 K42 Where the water comes from when the faucet is turned on and how it is treated to be drinkable is important information for students. 0 0 0 10 6 4.38 0.50 K44 Students should be aware of the aspects of water F water cycle, bioregions, watersheds, water bodies (headwaters, streams, rivers, tributaries, marshes, sloughs, lakes, oceans), alluvium, surface water, ground water, downstream, upstream, water pollution (point source, nonpoint source) nutrient load. 0 0 2 6 8 4.38 0.72 K10 A key concept for students would be an understanding of the nature and need for biodiversity. 0 0 2 7 7 4.31 0.70 K47 It is important for students to engage in fieldwork because it is more effective in building enthusiasm, helping with future career choices and fostering stewardship. 0 0 1 9 6 4.31 0.60 K17 Understanding the needs of competing users of water in the Flathead watershed is relevant information, especially with the challenges of climate change in the future. 0 1 1 7 7 4.25 0.86 K31 Students should understand that the stream system is the lifeblood of the watershed; it feeds it and is fed by it. 0 0 2 8 6 4.25 0.68 K51 The history of the Flathead Watershed is important information for students. 0 0 2 8 6 4.25 0.68 K3 Nonnative species (mostly introduced intentionally by managers) are major threats that are important for students to learn about in the Flathead watershed. 0 0 3 7 6 4.19 0.75 K9 It is important to teach about natural environmental change in general. 0 0 3 7 6 4.19 0.75 K21 Students should be aware of the stormwater conveyance system and that storm drains flow directly into our water bodies without treatment. 1 0 1 7 7 4.19 1.05 105 Round III: Knowledge Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 K34 Knowledge of resources, natural, historical, cultural, land and water use (agriculture, mining, roads, residential, commercial), recreation, conservation, dams, and irrigation is essential information for students. 0 0 2 9 5 4.19 0.66 K48 How the watershed impacts their lives and why they should care about it is essential information for students. 0 0 2 9 5 4.19 0.66 K16 It is important that students learn that a river system affects the whole watershed not only by flowing partially below the surface, but also by how wildlife transfers nutrients from the stream to the upland. 0 0 3 9 4 4.06 0.68 K29 Learning, how has this watershed changed over time, geologic, historic, present, is important information for students to be informed about. 0 0 3 9 4 4.06 0.68 K40 Students should be aware of the international aspects of the Flathead watershed and its reach into Canada, including human impacts, since a watershed does not recognize international borders. 0 1 2 8 5 4.06 0.85 K49 Knowledge gained through what is referred to as "STEM" (science, technology, engineering and math) courses is important to being able to make informed decisions about our watershed ecology and water quality. 0 1 3 6 6 4.06 0.93 106 APPENDIX E ROUND THREE SKILL STATEMENTS 107 Round III: Skills Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 S20 Basic scientific inquiry and critical thinking are important skills for students. 0 0 0 5 11 4.88 0.34 S26 Students, when looking to the Internet for information, need to always evaluate the source of information. 0 0 0 6 10 4.75 0.45 S16 Critical thinking is important. Students should be taught how to ask the tough questions and then learn how to look for answers through research. 0 0 0 6 10 4.69 0.48 S1 Scientific investigations involving developing questions, collecting data, analyzing data, and comparing data are skills students should have. 0 0 0 6 10 4.63 0.50 S21 Students should be able to make observations, identify problems, formulate relevant questions, use appropriate tools and technology to collect and analyze data, make inferences, draw conclusions, communicate and defend findings. 0 0 2 4 10 4.63 0.62 S10 A student should be able to interpret data and graphs. 0 0 0 8 8 4.50 0.63 S22 The ability to create tables, picture graphs, bar graphs and picture maps to record and organize information and to be able to read, interpret, and use tables and graphs to identify patterns and trends, draw conclusions, and make predictions are important skills for students to have. 0 0 0 8 8 4.44 0.73 S25 It is important to teach students to look beyond the rhetoric/popular opinion and ask their own questions, as well as how to search for their own answers or solutions provides a life skill. 0 0 1 6 9 4.38 0.81 S28 Listening and observing is an important skill. 0 0 0 8 8 4.38 0.62 S7 Field studies are great. 0 1 1 4 10 4.25 0.58 S8 A student should be able to create, read, and interpret a map, from the schoolyard map to the watershed. 0 0 1 7 8 4.25 0.68 108 Round III: Skills Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 S27 Imagination is important to encourage as students think of possible solutions. 0 0 0 9 7 4.25 0.58 S4 Students should be able to know how to delineate a watershed; this means being able to read a map and decide where the ridges are and where the valleys are, and where a droplet of water would flow were it to drop at any point on that map. 0 0 1 7 8 4.19 0.83 S2 Students should begin to conduct "research". 0 0 1 8 7 4.13 0.81 S12 Create homework assignments that require students to investigate their environment. 0 0 0 10 6 4.13 0.62 S11 Observation skills are essential so that students can sit quietly and observe the world around them. 0 0 0 10 6 4.06 0.93 S23 Describing and explaining how biodiversity in animal and plant life affects a watershed. 0 0 1 8 7 4.06 0.77 S13 Students should be able to read maps and landmarks and know the reservation watersheds. 0 0 2 6 8 3.94 0.77 S5 "Nature journaling" or "field journaling" gives young students the tools they need to quietly observe what is going on around them. 0 0 2 8 6 3.69 1.01 109 APPENDIX F ROUND THREE DISPOSITION STATEMENTS 110 Round III: Dispositions Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 D23 It is important for students to know what they can do to help protect our local watersheds. 0 0 0 7 9 4.56 0.51 D28 Students should realize that all points of view should be treated with equal respect and value, using careful listening skills and asking questions. 0 0 1 8 7 4.38 0.62 D2 Show students how the watershed impacts their lives and how they impact the watershed to create and share a proper attitude of caring. 0 0 1 9 6 4.31 0.60 D4 An appreciation for the natural and cultural history of the Flathead Watershed is an essential attribution for students. 0 0 2 7 7 4.31 0.70 D6 Students should see and feel the need to protect and sustain a healthy Flathead watershed. 0 0 2 7 7 4.31 0.70 D13 It is important that teachers be committed to getting kids outside, for several reasons; first, the lesson takes on life. Second, being at that place establishes a relationship between the student and that site. 0 0 2 7 7 4.31 0.70 D15 It is important to understand the balance between using the land and abusing the land. 0 0 1 9 6 4.31 0.60 D8 For students, developing a sense of appreciation for the Flathead watershed is paramount to instilling a desire to take care of their environment. 0 0 3 6 7 4.25 0.77 D11 Students should understand indigenous perspectives, but also understand more recent ties to the land, including early homesteaders, farmers, loggers, and industrial. 0 0 1 10 5 4.25 0.58 D12 It is important that students develop the concept of conservation and understand the difference between conservation and preservation. 0 0 2 8 6 4.25 0.68 D17 It is important to create a value for stewardship to leave a lasting legacy and a history of protective use, but not abuse. 0 0 2 8 6 4.25 0.68 111 Round III: Dispositions Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 D27 It is important that students in the Flathead watershed understand that this ecosystem is one of the most intact, with exceptional water quality, and that rivers, floodplains, and aquatic habitats are well connected and functional; and that many similar ecosystems in the lower 48 have been degraded. 0 0 3 6 7 4.25 0.77 D5 For students to develop a strong 'sense of place', they should understand the unique social and environmental history of the Flathead Valley. 0 0 3 7 6 4.19 0.75 D9 Explicitly create learning opportunities that develop real awareness of this remarkable place so that the Flathead environment is not taken for granted. 0 0 3 7 6 4.19 0.75 D10 Creating the next generation of "stewards" is a priceless goal that can be reached through establishing a relationship between the student and the outdoors when students are on the land and by the water. 0 0 3 7 6 4.19 0.75 D19 Teaching students about multiple perspectives is necessary; there are issues such as management of lake trout in Flathead Lake in which not all the stakeholders agree with the tribal point of view. 0 1 0 10 5 4.19 0.75 D1 It is important that students understand the profound wisdom and far reaching implications of the statement: "Whatever happens to the water happens to the people." 0 0 4 6 6 4.13 0.81 D16 Nurturing a balance of use and protection within the watershed, of providing for our needs while caring for the watershed's health, is valuable to instill and develop in students. 0 0 2 10 4 4.13 0.62 D20 History is important to learn about because it shows what human influence has done to the watershed and how it left it today. 0 1 1 9 5 4.13 0.81 D24 Learning about amounts and types of water consumption is relevant for students. 0 0 2 10 4 4.13 0.62 112 Round III: Dispositions Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 D3 It is important that students engage in life styles (choices and decisions) that protect and sustain a healthy Flathead watershed. 1 0 3 5 7 4.06 1.12 D26 Learning about the local ecosystem will create a sense of stewardship. 0 0 3 9 4 4.06 0.68 D25 Raising student awareness of watershed issues will hopefully influence their future actions in addressing the issues as adults. 0 0 4 8 4 4.00 0.73 D29 Students need to learn to recognize and appreciate their own and other people's values about the same things, like the Flathead watershed; for instance the values of Native/Indigenous people who have lived here for 12,000 years, families who have lived here for generations, and people who have moved here last week. 0 0 3 10 3 4.00 0.63 D14 Learning from the pioneers who are still living in the Flathead Valley is important, as they are able to provide historical perspective. 0 0 4 9 3 3.94 0.68 D21 Legacy is a great concept to teach and discuss; let students define what they want their legacy of stewardship to be. 0 0 4 9 3 3.94 0.68 D7 Salish, Kootenai and Pend d'Oreille history and cultural resources, such as films and books, have a tribal perspective and are important sources of information on science, history and contemporary information, including topics such as Indian Water Rights and native place names, which should be used in classrooms in the Flathead watershed. 0 1 4 7 4 3.88 0.89 D22 The major issues affecting watersheds today, such as water rights, toxic spills from rail or truck, toxic cleanup, industrial pollution, are important issues for students to learn about. 1 0 3 8 4 3.88 1.02 D18 It is valuable to know the Native American perspectives and to understand how the ecosystem has changed; such knowing could lead to a greater appreciation for the Flathead watershed. 0 2 3 7 4 3.81 0.98 113 APPENDIX G ROUND THREE TEACHING STATEMENTS 114 Round III: Teaching Methods Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 T10 Bringing resource people in as guest speakers is a way to get students interested and engaged. 0 0 0 9 7 4.44 0.51 T5 It would be a science learning experience as well as fun for students to tour the players in our watershed, such as the Forest Service, wastewater treatment plants, storm water programs (the City of Kalispell has such a program), the County Landfill, etc. 0 0 2 6 8 4.38 0.72 T3 A project-based approach would be a way to get students involved in learning the connections between the past and the present. 0 0 1 9 6 4.31 0.60 T7 Appropriate guides should be used for students to learn about the plants and animals in the Flathead watershed, instead of memorizing. 0 0 3 7 6 4.19 0.75 T2 Integration of some of the traditional ecological knowledge that has been passed down through generations of native people might best be incorporated into any learning process by directly engaging the native people in the development of educational and informational programs. 0 1 4 5 6 4.00 0.97 T1 A great Socratic Seminar topic for students could be the statement: "Whatever happens to the water happens to the people." 1 0 5 3 7 3.94 1.18 T4 Teaching about diversity can be accomplished through expression in subject matter teaching. An example is the multiple ways water quality can be introduced: explaining what scientists look for in their work, explaining the cultural importance of native fish and their waterways to indigenous people, or discussing various approaches to resource management of the same water body and why each manager might choose this approach. 0 2 3 5 6 3.94 1.0 115 Round III: Teaching Statement Number Statement Likert Scale Response M SD 1 2 3 4 5 T11 Resource education events can and should be conducted outside on the lawns of schools or in neighborhood parks that may be in walking distance. 0 0 7 6 3 3.94 .77 T8 Reading about nonfiction topics is a good way to learn about animals and the watershed and satisfy common core standards requirements for young students. 0 1 7 4 4 3.69 .95 T9 It is possible to address climate change in a neutral way through water and the interconnected nature of surface and groundwater. Precipitation decreases can affect surface flows and groundwater recharge/discharge. If temperature changes the storage of water through decreased snowpack that too can affect groundwater availability as well as surface water flows. 1 1 3 8 3 3.69 1.08 T6 Students can discover 'snow towers' and the progression of snow melt through field trips and via visual presentations. 0 0 8 7 1 3.56 .63