Science Teacher Experiences in Developing Stem Literacy Assessment

Abstract

STEM education gives practice experiences for students. However, the assessment in identifying its impacts has not been developed in Indonesia widely. Teachers are responsible to assess students’ achievements as an impact of learning activities. They said that they have difficulties in developing the assessment instruments. Therefore, workshop activity was designed to minimize and identified the difficulties. This study aims to evaluate teachers’ experiences in developing assessments in STEM learning. Workshop activities were conducted over three days. It covered three main topics: the best practice of STEM education from experts, explanation of STEM literacy, and the group discussion session in creating STEM literacy assessment tools. The tools that were passed through the expert judgment process will be proceed to the validation process to find out its response from students. Data were analyzed using person separation method in Rasch model analysis. The result showed that the assessment tools are less sensitive in distinguishing students’ abilities. It indicated that teachers should create more sensitive tools in diagnostic students’ STEM literacy. It is also found that most of teachers face difficulties in creating technology and engineering literacy items.

1. Introduction

The reform in science education is triggered by the workforce demand. The World Economic Forum [1] reported that there are worker displacements because of the pandemic COVID-19 impact that create global recession in the world economic. It forces million workers to experience the transformation of work, lives, and productivity. World Economic Forum has identified the labor market impact of the fourth industrial revolution. The employer survey result indicated that there are some business adaptations in response to COVID-19. One of them is accelerate the digitalization of work processes along with accelerate digitalization upskilling/reskilling. Therefore, technological adaptation is also needed by the companies, such as cloud computing, big data analysis, cybersecurity, artificial intelligence, ML, NLP, and robots (non-humanoid).

Education in science, technology, engineering, and mathematics (STEM) fields become important to face future workforce demand. In many undergraduate institutions, number of graduates who works in STEM fields is a primarily tools for the assessment of teaching. Higher number of graduates describes higher quality of teaching.

Technology changed human life in one-way to another for many years. For instance, the mechanism of transportation that changed from using feet power and simple rides to sophisticated machines. These machines help humans to travel and discover the world. In other fields, technology creates easier communication internationally that impacting information transfer around the world and triggering the globalization. Jerald [2] noted that new technologies along with demographic, political, and economic changes have altered human work and social lives in ways of significant consequences for young people.

Those facts have triggered educators argue that traditional curriculum is not sufficient, schools must develop “something new” for students as a preparation to face the twenty-first century. Indonesia is facing this need, thus they are creating to new policies. New curriculum is created to emphasize it on creating productive, innovative, creative, good effective human through reinforcement of attitude, skill, and knowledge in order to face challenges in the twenty-first century. On the other hand, the developed countries decided to put more attention in emphasizing science and technology education through enhancing science and technology promotion in education to students by providing some grants for researchers and practitioners Ishikawa et al. [3]. However, this brings questions for researcher of what method and strategy is needed to fulfill new policies goals.

Most of STEM activities were conducted to improve students’ interest toward STEM disciplines and careers. It was believed by some researchers that the early interest in pursuing science and engineering is a better indicator whether the students will pursue a career in these fields [4].

The translation process in practice faced several difficulties because of its different perspectives [5]. However, STEM literacy should become the main consideration in translating the content curriculum into the learning practice, in instruction form. NRC report noticed that effective STEM learning engages students’ interests and experiences, identifies and builds on their knowledge, uses STEM practices, and provides experiences that sustain their interest [6]. The report also helpfully identified key elements that provided the foundation for effective STEM instruction. Those elements included a coherent set of standards and curriculum, teachers with high capacity to teach this discipline, supportive system of assessment and accountability, adequate instructional time, and equal access to high-quality STEM learning opportunities [6].

Analyzing the curriculum content is the first step that should be taken by teachers in starting the implementation. It is to find out whether STEM education could be embedded in it. Previous study teacher analysis results of the coherency of STEM education implementation to science content in the curriculum. The sample school is using two curriculums, 2006 and 2013 curriculum. A total of 13 elementary teachers and 5 junior high school teachers analyzed the science content. The result showed that the 2013 curriculum more coherent than 2006 curriculum [5]. Teachers analyzed the curriculum content and possibilities for integration, and they were asked to think of an example of two and full integration of STEM disciplines. The results showed lower-level teachers can identified full STEM integration by giving the examples, but the higher-level teachers were failed to find it.

The interview results showed that fourth-grade teachers found the representation of STEM in the fourth grade from its thematic concept that could be coordinated well with the subjects. They hoped higher-order thinking skills can be practiced to fourth-grade students, so they can be a problem solvers and speak communicatively in describing ideas. Furthermore, fifth- and sixth-grade teachers thought that STEM was represented by some of the subject matter, and they believed that STEM learning in these areas would motivate students in class. Moreover, they said that learning activities, teachers’ ability, and students’ discipline should be innovated in order to improve STEM education.

2. Teacher workshop of STEM literacy

STEM-based learning is in line with the breadth of the current science curriculum [7]. The 2013 curriculum places subject matter more as a vehicle to develop a variety of skills, creative thinking, awareness of technology, the environment, awareness of the existence of a masterful regulator other than humans, and the values of life where the students are. With another understanding, the 2013 curriculum of all subjects accommodates the achievement of national education goals in a holistic, non-fragmented, and proportional manner. For science education, this is of course very encouraging and is expected to be able to increase the relevance of science learning, bearing in mind the essence of science learning always places science as a vehicle for building students’ skills and attitudes/values. In addition, the new curriculum, if implemented properly, will increase the popularity of science [8].

However, we realized that we are facing challenges in preparing the teacher [5]. Teachers in Indonesia were produced at education-based universities such as UPI. Based on the new ministry policy, not only UPI that can produce teachers but also other educational college or universities. These facts triggered UPI to create professional and qualified teachers so that the graduate could compete well. On the other hand, Japan has different policy. Teachers in Japan were produced from two institutions for different levels of teachers [9]. Elementary teachers come from normal universities, while the secondary-level teachers come from higher normal universities and universities level. They were well prepared in their bachelor studies so that most of teachers in Japan have only bachelor degree. In fact, for the past several decades, Japan’s students have consistently ranked among the world’s top performing in science, mathematics, and reading. This achievement is one proof of the good quality of teachers in Japan.

The good quality of teacher leads a good quality of learning. They plan learning as a medium to transfer knowledge to students. They consider the main aspects of learning processes— students and instructions. Professional development program is needed to produce qualified teachers [10, 11, 12]. Therefore, we design workshop activities for teachers that focused on creating STEM literacy assessment.

Here are some considerations of workshop and the detailed activities.

What does teacher need?

• The explanation of STEM literacy

• The example of STEM literacy instruments

• Training session to develop STEM literacy instrument

• The evidence of STEM integration on science learning to students’ STEM literacy

What should researcher do?

• Share the definition and perspectives of STEM literacy

• Show the examples of STEM literacy instrument

• Conducting workshop

• Investigate the impact to STEM literacy

Workshop on STEM literacy introduction for middle school teachers will be conducted to limited participants. They come from three region: Bandung, Bogor, and Semarang. Each region will choose nine in-services science and math teachers. They got lectures from several keynote speakers, developed lesson plans, and STEM literacy instruments.

Figure 1 showed the lecture session from Prof. Yoshisuke Kumano, Ph.D. who shared STEM/STEAM education implementation in Japan. He talked about one of STEM activities that conducted as afterschool activities. It showed students’ activity in investigating the Earth’s climate using Dagik Earth. Students discuss issues that gave by instructors. They try to find the best solution. On the second day, teachers presented lesson plan that included the STEM education activities. They share ideas and discuss about the challenges. In the third day, they got lecture of STEM literacy assessment and created it based on the lesson plan topic.

Figure 1.

Lecture from STEM expert.

STEM literacy instruments were developed by considering four aspects of its disciplines. Bybee [13] defines STEM literacy, which refers to an individual’s STEM-related issues competencies, ability in identify STEM disciplines characteristics as forms of human knowledge, inquiry, and design, STEM awareness, and willingness to engage in STEM-related issues.

On the other hand, Tang & Williams, [14] defined STEM literacy based on its discipline literacy definition (Table 1).

Literacy	Organization	Definition
Science literacy	OECD [15]	An individual’s scientific knowledge and skills in identifying questions, explaining phenomena scientifically based on the evidence, understanding of the characteristic science, understanding how science and technology develop human life, and willingness to engage in science-related issues as a scientific society.
Technology literacy	ITEA [16]	Person who has technology literacy understands what technology is, how it is created, and how it shapes society and its changes by society. He or she eager to engage in technology issues.
Engineering literacy	OECD [15]	The understanding of how technologies are created through the engineering design process. Ability to combine the scientific and mathematical principles to practical ends systematically and creatively.
Mathematics literacy	OECD [15]	An individual’s capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgments, and to use and engage with mathematics in ways that meet the needs of that individual’s life as a constructive, concerned, and reflective citizen.

Table 1.

STEM literacy definition at each discipline [14].

From those definitions, we develop the indicators of STEM literacy as follows (Table 2).

Literacy	Aspects	Indicators
Science literacy	Content knowledge	explain phenomena scientifically
	Procedural knowledge	Interpreted data and evidence scientifically
Technology literacy	Technological principal	Understanding technological principal
Engineering literacy	Generating idea solution	Developing solutions and achieving goals
Mathematics literacy	Mathematics roles	Formulating situation mathematically

Table 2.

STEM literacy assessment aspects and indicators.

Table 3 shows the results of lesson plans and instruments that develop by teachers. Each region picked one level and content based on curriculum. They plan around 3–4 meetings of learning implementation and created 25 STEM literacy items. Some teachers from Semarang never have experience in developing or implementing STEM-based lesson plan and assessment. They face difficulty and asked more time to learn about STEM.

City	Lesson Plan Theme	Number of STEM Literacy Item
Bandung	Technology in reproductivity	25
Bogor	Temperature and changes	25
Semarang	Digestion	25

Table 3.

Workshop results.

3. STEM literacy assessment

3.1 Blueprint of STEM literacy assessment

During the workshop, teachers guided to build the items. Most of them face difficulties in creating technology, engineering, and mathematics literacy items. They said it is hard to find the technology that refers the concept of core discipline ideas. They thought that students will hard to understanding the technology principle. Therefore, it should be integrated in the learning processes. It would be easy for students if the technology is used as learning media (Meryl [17], Burnett [18], Grabe & Grabe [19]). They can understand the application of scientific concepts from the technology that brought to the class. Furthermore, they can understand how it is created by experiencing the engineering practices in the class [20, 21]. They triggered to find solution ideas and test those ideas, which are best to solve the problem by creating innovative technology.

Engineering literacy items are related to the solution ideas based on the problem. In this study, the problems are related to technology of reproductivity, temperature, and digestion system. Teachers started to find the related problems. They identified the global, regional, or local problems around students daily life. Contextual problems will help students to perceive the problems and generating solution ideas. Table 4 showed the example of engineering literacy items that created by teachers.

Aspect	Indicator	Items
Engineering literacy	Choosing appropriate solution to solve animal reproduction problems related to improving offspring in animals (C1)	Technology was created to help humans in dealing with problems that occur in everyday life, one of which is the provision of good quality beef for consumption. In your opinion, the right reproductive technology to produce good quality cattle in an efficient time is….; clone artificial insemination hybridization genetical manipulation
	Choosing appropriate solution to solve animal reproduction problems related to energy sources	In general, the source of electrical energy comes from PLN as the operator of the national electricity provider. However, some consumers, such as groups of breeders, require alternative energy sources to reduce operational costs. The energy source that breeders should use for their egg incubators is…; PLN electricity Solar heat Biogas PLN-solar panel hybrid system

Table 4.

STEM literacy in engineering literacy aspect.

Based on the interview, teachers face difficulties on creating engineering literacy items. It need more time than science literacy items. The number of engineering items is smaller than that science literacy items. Most of them created 10 science literacy items, 8 mathematics literacy, 4 technology literacy, and 3 engineering literacy. They said it is hard to find ideas of technology and its engineering processes that can be a stimulus in the items. It might be caused by the lack of teacher’s information literacy in finding out technological resources or engineering problem items. Margaret and Dave [22] wrote that information literacy is an important aspect in engineering programs. On the other side, Correia and Bozutti [23] study that an ideal teaching methodology for engineering nowadays is conducting practices, followed by the frequent use of technological resources. Kelley and Wicklein [24] also investigated teacher difficulties in engineering class assessment. He found that the teacher indicated difficulties in locating and integrating appropriate levels of mathematics and science for engineering design.

The mathematics literacy items that developed by teachers also reach smaller number than the science literacy items. Gattie & Wicklein [25] conducted research to find out the instructional needs of high school technology educators regarding engineering design instruction. It is noticed that more than 90% of the in-service teachers indicated that engineering design was a suitable focus to be integrated in the instructional activities. However, the teachers in this sample also indicated some demand to properly include engineering design in technology education. Several indications described that 93% of the teachers need to learn how to integrate the appropriate levels of mathematics and science into instructional content and 87% realized the need to develop additional analytical (mathematics) skills.

3.2 Validation of STEM literacy items

The results of measuring items and persons can be seen in Table 5. Person separation obtained a value of 0.96 or less than two. This means that the test instrument may be made less sensitive in distinguishing students with high abilities and students with low abilities. Based on these results, additional test items may be needed to increase the sensitivity of the test instrument. There are three difficulty levels of test items based on item separation, namely easy, medium, and difficult. The reliability of the person is 0.48, while the item reliability is 0.89. These results reflect that the consistency of student answers is weak. However, the overall quality of the items in the instrument’s reliability aspect is good.

Summary of 34 measured person
	TOTAL SCORE	COUNT	MEASURE	MODEL S.E.	INFIT		OUTFIT
					MNSQ	ZSTD	MNSQ	ZSTD
MEAN	12.9	24.0	52.06	4.94	1.02	.05	.94	−.13
SEM	.5	.0	1.26	.06	.04	.17	.04	.13
P.SD	2.9	.0	7.22	.34	.21	.97	.24	.77
S.SD	2.9	.0	7.33	.35	.21	.98	.25	.79
MAX.	20.0	24.0	71.24	6.20	1.34	1.56	1.36	1.09
MIN.	8.0	24.0	40.35	4.77	.61	−2.12	.50	−1.85
REAL RMSE	5.22	TRUE SD	4.99	SEPARATION	.96	Person RELIABILITY		.48
MODEL RMSE	4.95	TRUE SD	5.26	SEPARATION	1.06	Person RELIABILITY		.53
S.E. OF Person MEAN = 1.26
Person RAW SCORE-TO-MEASURE CORRELATION = 1.00
CRONBACH ALPHA (KR-20) Person RAW SCORE “TEST” RELIABILITY = .48 SEM = 2.09
STANDARDIZED (50 ITEM) RELIABILITY = .70
SUMMARY OF 24 MEASURED Item
	TOTAL SCORE	COUNT	MEASURE	MODEL S.E.	INFIT		OUTFIT
					MNSQ	ZSTD	MNSQ	ZSTD
MEAN	18.3	34.0	50.00	4.28	.99	.14	.94	.01
SEM	1.7	.0	2.77	.15	.04	.33	.07	.34
P.SD	8.3	.0	13.26	.74	.20	1.56	.32	1.62
S.SD	8.5	.0	13.55	.76	.20	1.60	.32	1.65
MAX.	30.0	34.0	77.71	6.25	1.48	4.17	1.64	4.05
MIN.	3.0	34.0	30.26	3.62	.72	−2.05	.44	−1.90
REAL RMSE	4.45	TRUE SD	12.49	SEPARATION	2.81	Item RELIABILITY		.89
MODEL RMSE	4.35	TRUE SD	12.53	SEPARATION	2.88	Item RELIABILITY		.89
S.E. OF Item MEAN = 2.77

Table 5.

Items analysis results.

There are several factors that affect person reliability. The first factor is the variance of the student’s ability level. A wider range of abilities equals a higher person’s reliability. The measurement results in Table 5 show low personal reliability caused by the low variance of students’ ability levels. In addition, the value of person separation is still less than two. The second factor is the length of the test or the length of the rating scale. A longer test is a test with a higher person reliability score. The next factor is the number of categories per item. Person reliability will be higher if each item has many categories. The last factor is the targeting of sample items. Better targeting will result in higher person reliability measurements.

The factors that affect item reliability are the variance of item difficulty, sample size, and do not depend on the length of the test. Higher item reliability will be obtained on condition that the item difficulty range is wider. In addition, the larger the sample used, the higher the reliability of the items obtained by the researcher. The item reliability factor that differs from person reliability is that item reliability does not depend on the length of the test and is also not influenced by model fit.

Cronbach’s alpha value (KR-20) in Table 6 shows that the results of the measurement of the interaction reliability between items and persons. The KR-20 value of 0.48 (weak) is a reliability value in classical theory measurements, which in the Rasch model is equivalent to person reliability. Therefore, the low KR-20 value is caused by the low variance of the student’s ability level. To improve this reliability value, instrument tests can be carried out on a number of samples with extreme variance in ability levels. Items are declared fit with the model when the more difficult items must be more difficult for students to answer. And conversely, the item is declared not fit with the model when the easy to answer item is easy for students. Both statements are true regardless of the student’s ability level.

t	Logit Average	Separation	Reliabilities	Alpha Cronbach (KR-20)
Person	52,06 (7,22)	0,96	0,48	0,48
Item	50,00 (13,26)	2,81	0,89

Table 6.

Cronbach alpha analysis results.

MNSQ infit-outfit value (mean square fit statistic) (Table 7) indicates the suitability of the data with the model. Based on the results of the analysis of the overall test, the infit-outfit value is in the excellent range, which indicates that the overall test instrument is fit with the model. Meanwhile, the infit-outfit ZTSD (z-score standardized fit statistic) shows the results of the t-test for the hypothesis of the suitability of the data with the model. The ZSTD infit-outfit value is in the range of −1.9 to +1.9, which indicates that the overall data can be estimated logically.

	MNSQ		ZSTD
	Infit	Outfit	Infit	Outfit
Person	1,02	0,94	0,05	−0,13
Item	0,99	0,94	0,14	0,01

Table 7.

Mean square fit statistic results.

4. Conclusion

Teachers experienced the workshop activities in developing STEM literacy assessment. They got knowledge of STEM education best practice in Japan from the expert. Most of the teachers have experienced in developing and implementing STEM-based lesson, but no one has it in the STEM literacy assessment. Teacher found difficulties in creating technology, engineering, and mathematics literacies. It indicated from the validation results, which showed low reliability. Person separation obtained a value of 0.96 or less than two. This means that the test instrument may be made less sensitive in distinguishing students with high abilities and students with low abilities. Based on these results, additional test items may be needed to increase the sensitivity of the test instrument. Therefore, it needs more practice for teachers in creating more sensitive items to distinguish students ability.