Running Head: LUNAR EXPLORATION PROGRAM FOR LIBRARIANS

 

 

 

 

 

 

 

 

 

 

 

Program Evaluation

 

NASA Lunar Exploration Librarian Training

 

 

Abstract

            The lunar exploration program for public librarians is a two-day seminar focusing on increasing librarian knowledge of STEM content through NASA materials. The seminar taught 70 librarians from three states, Delaware, Maryland, and Pennsylvania, STEM subject matter through the context of exploring the Moon’s surface features, the processes that caused the features, the possible sites for future human habitation on the moon and the distribution of resources that may aid human habitation. This evaluation surveyed the librarians and focused on three main questions:

  • Did the training help the participating librarians learn more about NASA lunar exploration content?
  • Are the NASA materials being used as intended? and
  • What audiences are the librarians reaching with NASA content?

            The librarians felt less than comfortable with STEM content before the seminar, but were very comfortable with STEM content after learning the material given in the seminar. Although the librarians had difficulty with the process of borrowing lunar samples, overall they expressed a great deal of enthusiasm for lunar exploration content because of the content they learned in the seminar. The librarians use the materials as NASA intended in addition to expanding the materials for younger age groups. The audiences the librarians reach with lunar exploration material are usually 6 to 12 year old children. The librarians mainly reach children with NASA materials during Saturday and after school programs. It is recommended that the program continue running as it has in the past with the exception of improving communication channels involved in borrowing lunar samples. This program reaches a need that public librarians have for learning and being excited about STEM content and the program expresses a strong message that exploration is needed and the moon is the most logical place to begin.

Introduction

            The National Aeronautics and Space Administration (NASA) Exploration Systems Mission Directorate (ESMD) awards funding for a series of 2-day seminars designed to teach public librarians content and process knowledge about lunar exploration. NASA invests in such informal types of education programs in order to get younger audiences excited about their current mission, robotic and human presence on the moon and later on Mars. This program is funded specifically by Exploration Systems Mission Directorate (ESMD), an engineering organization, because of the promotion of the engineering of robotics content. The Lunar Exploration Librarian Training program supports the ESMD education department’s mission of encouraging engineering literacy.

Purpose of the Evaluation

            The Lunar Exploration Librarian Training program has been running for several years and has a successful reputation at NASA. The purpose of this evaluation is to gather information so that the program may be further fine-tuned so that it can be transferred to other informal education communities. The ESMD education team has interest in the evaluation to inform budget decisions for future fiscal years. The model selected for this evaluation is a product evaluation that is management-based designed to facilitate management’s decisions about resource allocation in the future. The reasons for this choice take into consideration that the stakeholders as well as the ESMD education team are in a management position and are concerned with management issues.

The Lunar Exploration Librarian Training Program

            The Lunar Exploration Librarian Training Program initially began with an effort through NASA Explorer Institute which is designed to engage library youth program providers in NASA content. The module that was developed was designed to reach children ages 8 to 13 through public library programs. Last year, education specialists at Goddard Space Flight Center planned and held three 2-day seminars for librarians in Maryland, Delaware and Pennsylvania as well as developing a supporting website. The three seminars taught 70 librarians science, technology, engineering and mathematics curriculum through the context of exploring the Moon’s surface features, the processes that caused the features, the possible sites for future human habitation on the moon and the distribution of resources that may aid human habitation. The module was leveraged through an existing community of librarians and after school providers and long-term support of the librarians who attended the workshop began through an online network and newsletter. Librarians have been trained to use the materials in their programs, and to train other librarians to use the materials. The training involves a 2-day workshop at Goddard Space Flight Center, which allows participants to interact with NASA scientists and education specialists.

Limitations of the Study

            The response rate for the survey was lower than anticipated. The first call for completion of the survey yielded only fourteen of 67 possible responses. The second call for completion of the survey produced eleven more responses, resulting in a 37% response rate. The recommendations made in this report reflect only a small portion of the participants in the librarian training, and therefore need to be treated conservatively. Since less than half of the librarians responded, these data only represent feedback from a few of the participants. The information gathered does not represent the majority of the librarians.

Goals of the Program

            The Librarian training through the NASA Explorer Institute is designed to engage library youth program providers in, and excite them about, space science and NASA’s return to the Moon. In turn, these librarians will be prepared by the program to engage and excite children and their families involved in their youth and family programs, as well as to train other librarians and after-school program providers.

Stakeholders

            The Lunar Exploration Librarian Training has several groups of stakeholders. There are two groups of stakeholders at Goddard Space Flight Center, the program manager and the program trainers. Member checks of the evaluation plan were conducted by these groups of stakeholders. In addition, the NASA ESMD education team is a client because the education team is the primary funding source of the program. The students who participate in the program are the audience; however they will not be included in this plan because of Human Review Board guidelines.

Information Needed by the Stakeholders

            After the first member check, questions reported dealt with clarity of the content from the perspective of the librarians, fidelity of the content when the program is implemented, the range of audiences the librarians’ reach, frequency of times the program is offered, and the ease of communication with NASA after the workshop. After sorting through the questions to find convergence, a product evaluation was selected as the focus of this evaluation. A survey for the participating librarians (Appendix 1) was developed using the converging questions and sent to the stakeholders for another member check. Stakeholders wanted to know if the librarians were using the materials as intended.  

Evaluation Questions

            Evaluation questions will be addressed by electronically sending a survey to the librarians who took the training. After two weeks, the participants who have not responded will receive an electronic and a hard copy of a reminder. The questions focus on the comfort level of librarians regarding science, technology, engineering and mathematics (STEM) both before and after the training, the difficulty of the content presented in the training, the manner in which the librarians use the materials in their own communities, and the awareness of the librarians regarding NASA messages after the training. The questions have been generated by the evaluation plan author and the management team at Goddard Space Flight Center through three face-to-face meetings and two member checks. Three overall questions guided this program evaluation:

1)      Did the training help the participating librarians learn more about NASA lunar exploration content?

2)      Are the NASA materials being used as intended?

3)      What audiences are the librarians reaching with NASA content?

Evaluation Plan and Procedures

            Initial questions for the evaluation plan were drafted by the evaluator and sent by email to the program team at Goddard Space Flight Center and ESMD education at NASA Headquarters. After electronic comments were incorporated into the draft questions, they were separated into context, input, process and product categories so that the evaluation could be focused. Since the program had already been implemented, the questions that were most valuable occurred in the product category. The evaluator met with the program management team at Goddard Space Flight Center so that the questions the team had about the librarian training were included in the study. The questions were placed in a worksheet that indicated the questions, information sources, as well as data collection technique and were sent back to the team for a member check. After the worksheet input was gathered by the program management team, a survey was drafted. The survey questions were sent to the program management team and the ESMD education department for final approval. Feedback was incorporated and the survey was converted into an electronic version that was distributed on http://www.surveymonkey.com software. Librarians were sent the survey through a group email in order to maintain confidentiality and were asked to complete the ten minute survey within ten days of the initial mailing. After ten days, only fourteen of the 67 librarians responded. A thank you email was sent that also served as a reminder to the librarians who did not complete the survey. Three days after the reminder email, eleven more librarians responded resulting in a total of 25 responses. Results were disaggregated into the three guiding questions of the evaluation plan.

Benchmarking

            Two other studies have been located in the literature that matches well with the NASA Lunar Exploration Librarian Training program. The first describes the training element of a project funded by the W. K. Kellogg Foundation to improve the delivery of career services to adult learners in libraries (Sampson, 1991). The “train-the-trainer” model for a workshop is described in this study and it matches well with the format of training for the librarians in this program. The other study was conducted in Sweden and informs the role of children’s libraries in the social training of children and youth (Klasson, 1969). Since the NASA Lunar Exploration Librarian Training is concerned with getting students excited about future missions to the moon, this study is a comparable match. A large part of the study addresses the needs of librarians in training programs.

 

 

 

 

 

 

 

 

Results

            Seminars were given in three states, Delaware, Maryland and Pennsylvania. The majority of respondents were librarians from Pennsylvania, followed by Maryland.

Demographics of Librarians

 

Question #1: Did the training help the participating librarians learn more about NASA lunar exploration content?

            The majority of librarians reported that they were uncomfortable (2.7 out of a 5 point scale, 5 being the highest comfort) with STEM content before the training began. After the training, librarians reported that they were more comfortable with STEM content (3.7 out of 5). A large majority of the librarians (74%) reported that they had a large increase in interest in the lunar content presented at the workshop and that the two-day workshop was sufficient to meet their needs in terms of content. However, librarians feel that they have only some of the resources needed to continue offering lunar exploration content at their facility and that more resources would be needed to provide a sustainable program. All of the librarians reported that the material presented at the training was at their ability level. The librarians found the STEM content presented to be understandable and it considerably piqued their interest in lunar exploration.

Comfort Level with STEM Content

 

 

 

 Increase in Interest Level in Lunar Exploration

 

 

Difficulty of the Content Presented at the Seminar

 

            Librarians signed up for the training because they felt it would increase their ability to offer a variety of programs at their library. Several of them reported a personal interest in space exploration, but most of the participants felt a need to offer more science related programs at their facility as a motivation for participating. Communications about the training came from a state or division level and several of the librarians were “volunteered” for the training by their division director. Seven of the twenty five librarians chose to answer this question on the survey and all of the respondents stated that the state or division director signed them up for the training. Although the librarians felt that the training would help them expand their repertoire, they were placed in the program by a supervisor.

Question #2: Are the NASA materials being used as intended?

The NASA materials given to the librarians at the training are being used as intended with some minor adaptations. The projected audience age range is 8 to 13, which is a very wide range. Several of the librarians felt the need to adapt the materials to fit their audience better. Most of the librarians use the materials as given, and several have adapted the materials to fit a younger audience.

            Communication between NASA personnel and librarians after the librarians returned to their facility was extremely positive. Questions about content were answered promptly and resolved problems effectively. One problem that was reported consistently in the surveys was the delivery of the lunar samples. Three librarians reported either no response to requests for the lunar samples or a late delivery of the samples. Although there is very positive communication between librarians and the NASA personnel at Goddard Space Flight Center, there seems to be miscommunication in the requisition of the lunar samples.

Librarians overwhelmingly expressed a uniform message as the most important content provided in the training. They reported the message that the exploration of space is important and to begin this research more information is needed about the moon. One librarian reported, “. . . further exploration of our solar system starts with finding resources for a space station on the moon.” Another reports, “There are very good reasons to continue space exploration, beginning with the moon.” A very strong message about space exploration is being expressed by the trainers and is utilized by the librarians in their programs. These data show that the intended message of the importance of exploration to future development in aerospace engineering was clearly communicated to the librarians and the librarians have fidelity with the message.

Question #3: What audiences are the librarians reaching with NASA content?

            Most of the librarians reach audiences aged 6 to 12 and their families. In addition, a few librarians reported having programs for Girl Scout groups and home school communities. All librarians reported the highest rating for student enthusiasm about lunar exploration. Approximately half of the respondents reported holding programs for groups over 100, the largest group being 200 children. The other respondents reported holding programs for small groups under 30. All of the librarians reported that they held three or four programs at their home libraries, and some of the librarians expressed interest in helping other librarians in their area to hold similar programs.

Perceived Excitement Level of Children at Librarian’s Program

 

            All of the librarians are planning on offering a lunar exploration program between two and four times next year. Some are expanding their offerings to teachers in their school district. For example, one librarian is offering her program to all of the second grade teachers in her area because the standards for space content are written for second grade. Another public librarian is preparing to perform a puppet show to three area schools at the 1st and 2nd grade levels. The trained librarians continue to express enthusiasm with their lunar exploration programs. Librarians who have attended the seminar continue to offer lunar exploration programs at their facilities several times a year and some are expanding programs in order to reach out to more audiences.

A huge benefit of the program is that librarians increase their knowledge of STEM content and that a strong message is being conveyed about the need for exploration. The librarians have much more confidence in presenting STEM programs and they express excitement for space exploration to young people. The programs are being offered by the librarians at their facilities between two and four times a year and are reaching hundreds of students. The lunar exploration program for librarians is successfully reaching all of the proposed goals.

Conclusions and Recommendations for Action

Strengths.  Judging from the comfort level of the librarians regarding STEM content before and after the seminar, it is clear from these data that the content at the workshop is presented at an appropriate level for public librarians and helps to increase both knowledge of and interest in lunar exploration content. Librarians expressed commitment to future offerings and have shown in various ways how they are spreading their enthusiasm for lunar exploration to the communities they serve. This program fills a need expressed by the librarians for more STEM content at the public library level. Many of the librarians wrote that they felt a need to offer more STEM content and the lunar exploration “story” gave them something exciting to present to children.

Weaknesses.  One suggestion for expansion of lunar exploration content within public libraries is to leverage this community of enthusiastic, knowledgeable librarians to speak with other public librarians. Developing a community of learners from this pool of librarians could help more librarians get excited about STEM content and provide more offerings at more remote facilities. Many of the librarians expressed that they were already trying to encourage others to engage in STEM education.

One of the only disappointments expressed by librarians was the lack of availability of the lunar samples. Several of the librarians completed the necessary steps to borrow the lunar samples, but the samples did not arrive on time or did not arrive at all. It is recommended that program management take steps to facilitate the acquisition of the lunar samples for the librarians who attend the training..

Another more extensive survey is recommended to further evaluate the lunar exploration librarian program due to the low response rate for this evaluation. Sixty three percent of the librarians did not respond resulting in a lack of information from the majority of those attending the seminar. An incentive should be given to the librarians who complete the more extensive survey in order to increase the amount of information used to make decisions on the program. An additional survey would have the benefit of gathering more responses from the group of librarians as well as refining the questions of the original survey.

 

References

 

Klasson, M. (1969). The role of children’s libraries in the social training of children and youth. Pedagogisk-Psykologiska Problem, (87), pp. 35 – 55.

 

Sampson, J. P. (1991). Training librarians to deliver career services. Journal of Career Development 18 (1), pp. 19-30.

 

Appendix 1

To the Moon and Beyond: Engaging Children in Current NASA Lunar Exploration in the Library Environment

Librarian Survey

 

In which state are you a librarian?

 

_____ Delaware

 

_____ Maryland

 

_____ Pennsylvania

 

 

Likert-Scale response. Please rate your experiences with the Lunar Exploration in the Library Environment program by indicating the number that corresponds with your answer. One (1) is the lowest rating and five (5) is the highest rating.

 

  1. Before the training began, how comfortable did you feel running programs with science, technology, engineering and mathematics (STEM) content?

Not comfortable

 

Comfortable

 

Very comfortable

1

2

3

4

5

 

 

. . .

  1. Now that you have experienced the training How comfortable do you feel running programs with science, technology, engineering and mathematics (STEM) content after the training?

Not comfortable

 

Somewhat comfortable

 

Very comfortable

1

2

3

4

5

 

 

  1. How has your interest level in lunar exploration increased since the training?

Interest level has stayed the same

 

Interest level has increase slightly

 

Interest level has increase a great deal

1

2

3

4

5

 

 

  1. Please rate the degree to which you feel that you have the resources to continue to offer lunar exploration education at your facilities?

Not enough resources

 

 Enough resources

 

Plenty of resources

1

2

3

4

5

 

 

  1. How sufficient was the two-day training for you to become skillful with the lunar exploration content?

Not sufficient

 

Sufficient

 

Very Sufficient

1

2

3

4

5

 

 

  1. How difficult do you find the content in the lunar exploration (training?)materials?

Very difficult to understand

 

Somewhat difficult to understand

 

Easily understood

1

2

3

4

5

 

 

  1. How often do you use the website that accompanies the lunar exploration materials?

Do not use the website

 

Use the website regularly

 

Use the website often

1

2

3

4

5

 

 

  1. How excited do you feel the students become about the content when they attend your lunar exploration programs?

Not  excited

 

Excited

 

Very excited

1

2

3

4

5

 

 

  1. How often do you find you need to communicate with NASA personnel after the training in order to clarify understandings?

Do not need to communicate

 

Periodically need to communicate

 

Need to communicate often

1

2

3

4

5

 

 

  1. If you try to communicate with NASA personnel after the training, are you able to communicate with them to meet your needs?

Not helpful

 

Helpful

 

Extremely helpful

1

2

3

4

5

 

Please explain your response: 

 

 

 

  1. How many times have you offered lunar exploration programs at your facility?

 

            _____ times

 

  1. Approximately how many children have you reached with your lunar exploration programs?

            _____ children

 

 

  1. Do you have plans to offer the lunar exploration program in the future?

      _____ yes

      _____ no

 

If so, how frequently would you offer the program?

 

            ____ times

 

 

Open ended response. Please write in your thoughts to the following questions regarding the Lunar Exploration in the Library Environment program.

 

Before the training began . . .

  1. Why did you decided to attend the training?

 

 

 

  1. What did you expect to get out of the training?

 

 

 

  1. How did you hear about the training?

 

 

 

 

Now that you have experienced the training . . .

Offering the training

 

  1. In what venues have you offered the lunar exploration program?

 

 

  1. Who typically attends your lunar exploration programs?

 

 

 

NASA materials

  1. Do you use the NASA materials given to you at the training, or do you adapt them?  Yes/No

 

 

 

 

  1. If you adapt the NASA materials, briefly explain how you do this.

 

 

 

 

  1. What do you consider the most important message of the lunar exploration program?

 

 

 

 

Future suggestions

  1. Do you have any suggestions to improve the lunar exploration program?