Open Access is an initiative that aims to make scientific research freely available to all. To date our community has made over 100 million downloads. It’s based on principles of collaboration, unobstructed discovery, and, most importantly, scientific progression. As PhD students, we found it difficult to access the research we needed, so we decided to create a new Open Access publisher that levels the playing field for scientists across the world. How? By making research easy to access, and puts the academic needs of the researchers before the business interests of publishers.
We are a community of more than 103,000 authors and editors from 3,291 institutions spanning 160 countries, including Nobel Prize winners and some of the world’s most-cited researchers. Publishing on IntechOpen allows authors to earn citations and find new collaborators, meaning more people see your work not only from your own field of study, but from other related fields too.
To purchase hard copies of this book, please contact the representative in India:
CBS Publishers & Distributors Pvt. Ltd.
www.cbspd.com
|
customercare@cbspd.com
Online or blended learning assessments through LMS-MOOCs carried out in the world of education today tend to be multiple-choice assessments that are only based on low-level cognitive. In fact, to measure the metacognitive of students is quite difficult, if only using the form of multiple choice questions. Therefore, it takes the form of questions and assessments that allow students to explore their reflective and metacognitive thinking according to the characteristics of the education they are attending. Vocational education tends to apply a project-based learning (PjBL) model that requires authentic and performance-based learning assessment methods. Therefore, it is necessary to develop an appropriate blended metacognitive skill assessment rubric instrument for vocational education. Metacognitive assessment was developed using research and development procedures, with students as subjects in vocational education in Makassar, Indonesia. The integration between elements of metacognitive skills: planning, monitoring, and evaluation with self-peer-teacher assessment can be an assessment method to measure students’ metacognitive thinking skills in PjBL. Especially metacognitive assessment through blended learning practice MOOCs that are in accordance with the characteristics of vocational education and can be adopted by general education.
*Address all correspondence to: ridwandm@unm.ac.id
1. Introduction
The world of work that is dynamic and develops in accordance with the direction of technological progress, requires workers not only to have hard skills in their respective fields, but also to have soft skills [1]. In the context of a dynamic and complex world of work, intelligence and soft skills are needed that are relevant to the world of work today and the world of work in the future [2]. These soft skills are of course in the form of adaptability, problem solving ability, analytical thinking, creativity, collaboration, and communication. The International Labor Organization (ILO) released several soft skills that are currently needed in the world of work such as analytical skills, creativity, problem solving, communication, collaboration, and entrepreneurship [3]. Some of these soft skills, such as analytical skills, creativity, and problem solving skills, are classified as critical thinking which is regulated by the ability to think reflectively or think metacognitively (metacognitive skills) [4]. Therefore, the ability to think metacognitively is very important for workers because it can help them maintain their work ethic in a very dynamic world of work with change and uncertainty.
Vocational education as an educational institution that aims to prepare a competent workforce is required to have an awareness of the demands of today’s world of work. UNESCO-UNEVOC has set one of the main priorities of vocational education in the world, namely to prepare a competent young workforce according to the demands of today’s global workforce. Vocational education or globally known as Technical and Vocational Education and Training (TVET) is required not only to equip students with hard skills but also to equip them with creativity skills, analytical thinking, problem solving, and leadership [5]. To support this, higher education should carry out various learning innovations, both in learning planning, learning processes, and learning evaluations. Digitization is one of the best choices because it is the demand of the current digital era that leads to learning 4.0 [1].
Learning 4.0 has now been promoted at various levels of education, not least at the higher education level. The use of online learning is one of the learning media used to assist the digital learning process [1]. The implications of online learning or e-learning in learning present new forms of learning and allow learners to collaborate and interact socially online [6]. In addition, online learning can increase the level of motivation of students [7] as well as helping students to access information and learning resources from anywhere and anytime [8]. The use of e-learning in vocational education has also been widely used and researched. The use of digital-based teaching materials that are integrated in e-learning can improve mathematics learning outcomes in vocational education [9]. From the aspect of users, teachers and students use mobile devices for vocational learning purposes [10]. Learning evaluation is of course also possible to do online with the help of e-learning. The advancement of internet technology and the increasing interest in online learning, issues around e-learning and its assessment methods are also getting more attention among educators [11].
Online learning evaluations carried out in the world of education today tend to measure using multiple choice-based questions [1]. Multiple choice-based questions are often used in summative and formative tests in education (online and offline or blended) [12]. In fact, to measure the higher-level cognitive of students is quite difficult, if only using the form of multiple choice questions. Because in practice, the use of multiple choice-based tests only touches low-level cognitive [13]. Therefore, it takes a form of evaluation and form of questions that allow students to explore their subjective and objective reflective and metacognitive thinking. Through this metacognitive thinking process, it is hoped that students will be able to reflect on their own learning and make adjustments so that students can achieve a deeper understanding [14]. In addition, a form of formative assessment that focuses on teaching students’ metacognitive processes is needed to evaluate their own learning and make adjustments to the learning process [15].
Several research results have examined the methods of assessing and measuring students’ metacognitive thinking through the online environment [1]. Online metacognitive thinking assessment in certain domain assignments and settings, can measure students’ metacognitive thinking abilities [16]. Researchers used measurement tools in the form of otter tasks, multiple choice questions, and open-ended questions. Another researcher conducted an assessment using an online-based group metacognitive scaffolding (GMS) to measure the metacognitive behavior of students in a small group in class [17]. The results show that GMS has a significant impact on changes in the metacognitive behavior of learners in a small group. Furthermore, Altıok et al. [18] measuring metacognitive thinking using an online environment integrated video portfolio and the results show an increase in the level of students’ metacognitive thinking in foreign language learning. The results of this study only focus on the measurement and assessment methods of learning in general education, not yet on learning in vocational education which has its own learning characteristics. In vocational education, appropriate learning models are used, namely work-based learning, project-based learning, or contextual teaching and learning so that authentic and performance-based learning assessment methods are needed [19]. The development of rubrics and assessment models uses a student-centered assessment approach where students are the subject and object of the assessment to reflect on their own learning, as well as peer assessment and teacher assessment approaches [20]. The results of the development of rubrics and metacognitive skills assessment models through blended learning MOOCs can be a reference for PjBL assessment methods that are in accordance with the characteristics of vocational education.
Technical and Vocational Education and Training includes theoretical and practical learning content developed in schools, training institutions, or companies. Based on this limitation, the knowledge and skills referred to here can be understood not only as technical knowledge and skills, but also knowledge of values and identity in a complex world of work [1]. This vocational education paradigm is also not only a learning process in the school environment, but can be carried out in non-formal training environments such as training institutions and in agencies or companies [21]. The main purpose of vocational education is to prepare graduates directly for work. Vocational education should provide specialized training that is reproductive in nature and based on teacher instruction, with an emphasis on knowledge of certain industrial sectors and includes specific skills or tricks of the trade. Vocational education has played a central role in supporting the transition from school to the world of work for youth. Vocational education for productive work is considered essential for economic and social development [22]. An important emphasis of vocational education is on developing specific work-related skills or skills to prepare students for entering the workforce, while general education emphasizes on equipping students with broad knowledge and basic skills in mathematics and communication [23]. Based on these theories, it can be concluded that in general, vocational education aims to prepare graduates to work in certain sectors. The function of this education is to carry out the process of transforming work competencies, knowledge of the world of work, as well as the ability to collaborate and interact between workers.
The study of metacognitive thinking has been widely associated with John Flavell as an expert in the field of cognitive development since the 1970s. The term metacognition as proposed by Flavell et al. [24] used to refer to awareness, monitoring and regulating of one’s cognitive processes. In line with this, Yusuf et al. [25] explained that metacognition refers to the principle of organizing thinking through the process of controlling one’s cognitive. The metacognitive component consists of self-awareness, as well as monitoring and evaluation. These components can improve students’ ability to solve problems.
Furthermore, Jacobs and Paris [26] explained that metacognition refers to thinking about thinking. Metacognition focuses on self-regulated thinking, namely what people know and how they apply that knowledge to certain tasks. Metacognitive theory as a systematic framework used to explain and direct cognitive processes, cognitive knowledge, and cognitive regulatory skills [27]. A fundamental distinction is made between metacognitive knowledge and metacognitive regulation. Knowledge of cognition refers to what individuals know about their own cognition or about cognition in general. It consists of declarative knowledge (knowing about things), procedural knowledge (knowing how to do things), and conditional knowledge (knowing why and when). Cognitive regulation refers to metacognitive activities that help control one’s thinking or learning. Three important skills that are widely recognized are planning (strategy selection and resource allocation), monitoring (awareness understanding and task performance), and evaluation (assessing the product and process of one’s learning arrangements) [1].
Schraw and Moshman [28] explain the classification of metacognitive knowledge and metacognitive regulation. In metacognitive knowledge, declarative knowledge includes knowledge about oneself as a learner and about what factors affect one’s performance. Procedural knowledge refers to knowledge about the implementation of procedural skills. Conditional knowledge refers to knowing when and why to apply various cognitive actions. Meanwhile, the regulation or metacognitive regulation is categorized into three domains, namely planning the cognitive process (planning), monitoring the cognitive process (monitoring), and evaluating the cognitive process (evaluation). Planning involves selecting the right strategy and allocation of resources that affect performance. Examples include making predictions before reading, sequencing strategies, and allocating time or attention selectively before starting a task. Monitoring refers to a person’s on-line awareness of task comprehension and performance. The ability to engage in periodic self-evaluations while studying is an example. Evaluation refers to the assessment of the product and process of setting one’s learning. Common examples include re-evaluating one’s goals and conclusions. In connection with some of the above definitions of metacognitive, [29] explained that metacognition can be divided into two components: metacognitive knowledge and metacognitive regulation. Metacognitive regulation is the monitoring of one’s cognition and includes planning activities, awareness of self-understanding and performance, and evaluation of the efficacy aspects of monitoring processes and strategies.
Based on the description of metacognitive above, it can be concluded that metacognitive or metacognitive thinking is an awareness of thinking about how we think, how we organize thinking strategies in order to complete certain tasks well. Metacognitive thinking can be categorized into 2 sub categories, namely metacognitive knowledge and metacognitive regulation. Metacognitive knowledge is further divided into declarative, procedural, and conditional thinking. Meanwhile, metacognitive regulation is divided into planning, monitoring, and evaluation processes.
In the context of learning in vocational education, these two categories allow to be measured and assessed. However, taking into account the performance-based and project- or product-based assessment methods in vocational education, the measurement of metacognitive regulation (planning, monitoring, and evaluation) is more likely to be measured [1]. As explained by Klerk et al. [30] that vocational education emphasizes performance-based assessment where students learn by doing. This is confirmed by Wimmers [19] that at the end of the vocational education program or professional education program, every student must achieve standardized work competence, so that in this educational program, performance-based assessment is a general method for assessing practical competence in an authentic context. Learners can measure their metacognitive thinking skills through the process of planning, monitoring, and evaluating their performance and the projects or products they make.
MOOCs cannot be separated from their early history in 2008 when George Siemens and Stephen Downes provided open enrollment for their Connectivism and Connective Knowledge course at the University of Manitoba. This course is designed as a liaison or cMOOC where students are expected to learn more about connecting with each other in online environments such as classroom learning. In 2012, prestigious educational institutions such as MIT, Harvard, and Stanford began experimenting with offering a MOOC model known as xMOOC, taking a more behavioristic approach to teaching [31]. Then in 2011, a professor of Computer Science at Stanford University, and Peter Norvig, Director of Research at Google, announced that they would offer an open online course in Artificial Intelligence. This course does not use a learning credit system, but students who complete this course will be given a certificate of acknowledgment that they have completed learning. As many as 160,000 people registered, so that the world’s attention was given to this phenomenal program and was given the term Massive Open Online Course/MOOCs [32].
MOOCs have attracted the attention of researchers, learning experts, and even governments who have raised various opinions and assumptions regarding the features offered and their advantages and disadvantages. Despite this heterogeneity, dozens of MOOC options emerge every day and thousands of people sign up for the courses available. Besides being free, their motivation is because the course content comes from prestigious universities including Harvard, Massachusetts Institute of Technology, Stanford, University of California, and so on. In addition, research teams from various scientific backgrounds from universities around the world focus daily on finding new alternatives in terms of content access and distribution in MOOCs. It is solely aimed at providing a more engaging learning experience for MOOCs users.
MOOCs have great potential in the world of educational technology so that their use becomes a challenge in itself from the massive aspect, open access, and connectivity which of course must be developed through a multidisciplinary approach. Cyber-socialecology MOOCs can provide a collaborative approach not only among students, but also between educational institutions so that students can adapt their learning models, preferences, and learning needs to MOOCs from different institutions. Another important thing in collaboration between educational institutions is the formalization or recognition of learning in MOOCs. In this way, formal MOOCs can become part of an educational institution’s curriculum or tailor a course of study to earn an online diploma. An example of this scenario has been done on the “Mobile Cloud Computing with Android” specialization provided by Coursera [31].
MOOCs can support competency-based education [33]. In addition, MOOCs need to improve the quality and personalization of the student learning experience to further increase the effectiveness of education in general. As well as, Rosé et al. [34] emphasized the need to explore the possibilities of new features, such as collaboration features that encourage collaborative online activities such as structured brainstorming, whole group feedback, group reflections, and other collaborative activities. This activity aims to foster and maintain connectivity support, direction, and a more positive experience for students.
One of the advantages of online courses such as MOOCs is that it is easy to be able to engage in classes from any geographic location at any time you want. Having students spread all over the world in different time zones does not pose much of a problem while studying. This is because it facilitates the delivery of learning asynchronously and synchronously. However, the lack of face-to-face engagement can lead to a sense of isolation and result in students feeling separated from their peers in the classroom [35]. As a result, students in online environments tend to feel like they are taking on independent study rather than being active members of a study group [36]. One way to minimize this sense of alienation is through the use of technology and more interactive content that can enhance collaboration and knowledge construction.
In the learning system through MOOCs, students learn content knowledge by utilizing information and multimedia systems based on the development of learning models and methods. In other words, through learning systems and technology MOOCs require students to use metacognitive skills to manage their own learning pace (metacognitive skills). Students must be active learners in encouraging and sustaining their own learning progress. For example, they should assess the extent to which their learning strategies are effective in facilitating their learning progress, and identify which content has been optimal in terms of helping them achieve their desired learning goals [37]. Tsai et al. [38] conducted research with the aim of proposing an integrated model that integrates aspects of metacognition and interest in learning to investigate student learning motivation through MOOCs. The results of this study revealed that the increase in metacognitive skills was also accompanied by an increase in student enjoyment and encouragement regarding learning in the setting and organization of MOOCs. The findings show that metacognitive aspects can explain whether learners are motivated to learn through MOOCs because of the consequences of cognitive aspects mediated by interest in learning. In the use of MOOCs, the term blended is known which combines learning in terms of the implementation of learning (online or face to face), the delivery of learning content (synchronous or asynchronous).
Blended learning is a combination of various modalities (on-site, self-directed and web-based learning), delivery media (internet, lectures, powerpoint presentations, textbooks); teaching methods (face-to-face or technology-based/online sessions) and web-based technologies (wikis, chat rooms, blogs, textbooks, online courses) [1]. The combination (hybrid) is carried out depending on criteria such as learning objectives, course content, lecturer experience and teaching style, student characteristics, and others [39]. While, Kaur [40] define blended/hybrid learning from multiple perspectives:
Holistic perspective: delivery of learning using various media formats, including integration of learning media into traditional classrooms (f2f) or into online learning environments regardless of the combination of synchronous or asynchronous media
Educational perspective: a lecture that integrates f2f lecture activities with online pedagogical content. Some f2f lectures are replaced by online activities, especially in terms of synchronous, and online-asynchronous classes.
Pragmatic perspective: lectures are taught both in the classroom and remotely using different pedagogic combination strategies.
Corporate training perspective: the use of various learning media formats to deliver a curriculum or course.
CLO-Chief learning officer perspective: a learning strategy that integrates several communication modalities (both synchronous and asynchronous).
The success of blended learning depends not only on the quality of the courses and the virtual/online environment, but also on the degree to which faculty and students are prepared to work in a virtual learning environment. It also really depends on the preparation of learning materials and activities by the lecturers and the technical abilities of lecturers and students [1]. In particular, to use all the tools/features offered by the Learning Management System (LMS)-MOOCs, such as related to metacognitive assignments and quiz-essays.
Project-based learning (PjBL) is an approach to teaching science and technology that focuses on investigating questions and problems that students find meaningful and interesting, as well as sparking curiosity about something [41]. By investigating these questions and problems, students are involved in understanding phenomena, recurring natural events, or finding solutions to problems using disciplinary core ideas, scientific and engineering practice, and cross-disciplinary concepts. PjBL involves students and lecturers in finding solutions to questions about the environment around them. Investigating real-world questions in which students investigate meaningfulness has long been touted as a viable learning method. Thus, PjBL triggers the curiosity and active involvement of students to find out what is going on in their environment [41]. The George Lucas Educational Foundation [42, 43], recommend 6 steps of PjBL, namely:
Essential questions: provide essential questions related to the focus or scope of the project that is related to the real world and is relevant to students.
Designing plans for projects: planning the rules of the game, tools, materials, and selection of activities that can support and answer the important questions of the project focus.
Create a schedule: create a timeline and determine project completion deadlines.
Monitor students and project progress: Monitor student progress and activities during the project completion process. Monitoring uses a rubric that can record all important activities.
Assessing the results: evaluating project progress, providing feedback on the level of understanding students have achieved, assisting lecturers in developing further learning strategies.
Evaluate the experience: Lecturers and students reflect on activities and results.
Figure 1 presents a procedural map of the use of metacognitive rubrics in PjBL.
Figure 1.
Procedural map of the use of metacognitive rubrics in PjBL.
PjBL is a model that organizes learning around projects. Projects are complex tasks, based on challenging questions or problems that involve students in design, problem solving, decision making, or investigative activities, and provide students with the opportunity to work independently guided over a long period of time, culminating in on the final product or presentation. As a learning strategy, PjBL involves students in authentic learning through working on a project. This approach varies greatly from the traditional teacher-centered classroom and provides an interdisciplinary, student-centered activity for students that is integrated with real-world problems and practices, and usually lasts over a long period of time [44].
PjBL, sometimes referred to as project work, can be seen as an extensive problem-based learning activity in which students need to find ways to verify a phenomenon or solve a problem. Thus, aspects of skills are determined to be relevant to aspects of attitudes and abilities needed by students, including abilities such as critical thinking, creative thinking, time management skills and the ability to work cooperatively with others [45]. PjBL is centered on generating questions or inquiries that lead students to see concepts and principles related to their learning. Work on this project requires a long period of time, involving students to generate new knowledge to build on the premise of student inquiry and understanding [46]. Furthermore, Netto-Shek explains that project work, when executed properly, gives students autonomy to make decisions and to work independently and collaboratively in producing solutions for situations that were not previously planned. Netto-Shek argues that in the process of project work by students, monitoring by lecturers provides guidance and advice if needed. As such, project work embeds authentic real-world challenges in student learning experiences.
PjBL allows students to hone and develop skills through knowledge reconstruction when students work together to develop their projects and overcome problems, thereby forcing them to maximize cognitive aspects and overall theoretical understanding and identify theoretical knowledge gaps [47]. This is a more authentic approach to the student learning experience compared to the traditional approach. In keeping with current trends, the Israel Institute of Technology, in 2014 launched an online course on nanotechnology and nanosensors in the MOOCs format, which continues to this day. This course was developed by Prof. Hossam Haick, from the School of Chemical Engineering, is a leading researcher in the field of nanotechnology. The nanotechnology and nanosensors course is the world’s first MOOCs in this field, and the first to be presented simultaneously in two languages: English and Arabic. Their purpose is twofold. First, it reaches everyone around the world, even those who live in countries that do not have diplomatic relations with Israel. Second, provide a model for promoting sociocultural learning in the context of technical education, by integrating project-based learning, multicultural teamwork, and peer assessment into a curriculum [48].
Following previous developments on project-based learning in higher education [49], assignments on nanotechnology and nanosensor learning in MOOCs involving features related to the use of authentic questions, inquiry communities, and the use of cognitive aspect support technologies. PjBL involves students in authentic inquiry directly [41, 49]. In order for an activity to be considered as PjBL, it is necessary to involve the construction of knowledge through the development of new ideas, understanding, and/or skills on the part of students. This raises questions about the role of project-based MOOCs in the process of knowledge construction and learning motivation among science and engineering students. More specifically, how to assess the appropriate knowledge construction project for blended MOOCs learning according to the characteristics of vocational education.
7. The development of PjBL metacognitive assessment models
The development of the metacognitive assessment model presented is the result of the author’s research, which uses Research and Development steps [1]. Metacognitive instruments and rubrics were developed based on the theory of [28] and Lai [29] previously reviewed, where metacognitive regulation covers 3 aspects, namely: the planning process, the monitoring process, and the evaluation process of the project undertaken.
7.1 Metacognitive scoring rubric for PjBL
Assessment models and rubrics are based on the previously studied metacognitive thinking theory. The results of this study use the theory of metacognitive regulation which is divided into planning, monitoring, and evaluation processes. Table 1 is a complete rubric that has been compiled based on the theoretical indicators that have been described previously.
Metacognitive aspect
Indicator
Score 1
Score 2
Score 3
Score 4
1. Planning (planning their learning activities according to their ability to understand the material)
The plan contains the ultimate learning objectives they expect in project work.
The plan contains indicators of the achievement of the final objectives of learning that can be measured well.
The plan contains the prerequisites for the initial knowledge needed in the project work.
The plan contains learning activities that will be carried out in completing the project.
The plan contains the ultimate learning goals they expect in project work.
The plan contains the ultimate learning objectives they expect in project work.
The plan contains indicators of the achievement of the final objectives of learning that can be measured well.
The plan contains the ultimate learning objectives they expect in project work.
The plan contains indicators of the achievement of the final objectives of learning that can be measured well.
The plan contains the prerequisites for the initial knowledge needed in project work.
The plan contains the ultimate learning objectives they expect in project work.
The plan contains indicators of the achievement of the final objectives of learning that can be measured well.
The plan contains the prerequisites for the initial knowledge needed in the project work.
The plan contains learning activities that will be carried out in completing the project.
2. Monitoring (self-monitoring of learning activities)
The learning monitoring report contains the completeness of learning activities in the form of checklist items.
The learning monitoring report contains the obstacles faced in carrying out learning activities during the project work process.
The learning monitoring report contains things that have helped them complete each learning activity that has been carried out.
The learning monitoring report contains strategies that will be carried out in minimizing the constraints that have been written previously.
The learning monitoring report contains the completeness of learning activities in the form of checklist items.
The learning monitoring report contains the completeness of learning activities in the form of checklist items.
The learning monitoring report contains the obstacles faced in carrying out learning activities during the project work process.
The learning monitoring report contains the completeness of learning activities in the form of checklist items.
The learning monitoring report contains the obstacles faced in carrying out learning activities during the project work process.
The learning monitoring report contains things that have helped them complete each learning activity that has been carried out.
The learning monitoring report contains the completeness of learning activities in the form of checklist items.
The learning monitoring report contains the obstacles faced in carrying out learning activities during the project work process.
The learning monitoring report contains things that have helped them complete each learning activity that has been carried out.
The learning monitoring report contains strategies that will be carried out in minimizing the constraints that have been written previously.
3. Evaluation (evaluating the results of his work)
The report contains clear, coherent, and complete information.
Reports are presented by including the appropriate images.
The report states that all project work processes are carried out in accordance with the stages.
The report writes a good conclusion that summarizes the final goal of working on a previously written project.
The report contains clear, coherent, and complete information.
The report contains clear, coherent, and complete information.
Reports are presented by including the appropriate images.
The report contains clear, coherent, and complete information.
Reports are presented by including the appropriate images.
The report states that all project work processes are carried out according to the stages.
The report contains clear, coherent, and complete information.
Reports are presented by including the appropriate images.
The report states that all project work processes are carried out in accordance with the stages.
The report writes a good conclusion that summarizes the final goal of working on a previously written project.
The rubrics in Table 1 are then integrated into the LMS-MOOCs. The following is a metacognitive rubric display on planning aspects that have been integrated into the LMS.
Figure 2 shows a metacognitive rubric consisting of 4 rating scales where each scale contains several assessment criteria from the planning aspect of project work which is one of the metacognitive aspects. Students and teaching staff directly choose one of the points in accordance with the contents of the student project project planning report being assessed.
Figure 2.
Metacognitive rubric integrated in LMS. (Source: [1]).
7.2 Implementation of PjBL metacognitive assessment rubric through blended learning MOOCs
This learning process applies the Blended learning method that combines two learning cycles, namely online-based and face-to-face. The online-based learning cycle is used to strengthen basic materials or theories before students work on projects directly in the laboratory [1]. In addition, online methods are also used to integrate metacognitive assessment instruments and rubrics. The following is a display of the results of Peer, Self, and Teacher Assessment from students (Figure 3).
Figure 3.
Peer, self, and teacher assessment.
The picture above shows the results of peer assessment (Grades received), self-assessment (Grades given), and teacher assessment (Grade for Submission and Grade for Assessment). Each student gave a score to 3 other students and received a score from 3 students based on the assessment rubric. After that, the teacher also gives a score based on the same assessment rubric. These scores are then downloaded in an excel file format for further processing by assigning a weight to each score. The score from the self-assessment is given a weight of 20, the score from the peer assessment is given a weight of 30, and the score from the teacher assessment is given a weight of 50 so that the maximum score is 100. The following is the final score from the results of measuring students’ overall metacognitive thinking in project work (Figure 4).
Figure 4.
Results of measuring students’ metacognitive thinking. (Source: [1]).
The graph above shows the scores of the three aspects of metacognitive thinking, namely planning, monitoring, and project evaluation. The score comes from three sources, namely peer, self, and teacher assessment. In the planning aspect, the peer assessment score (24.19) is categorized as high because it is close to the maximum score (30). Likewise, the self score (18.38) and teacher assessment (36.29) were also categorized as high because they were close to the maximum scores of 20 and 50. Scores on the metacognitive aspects of monitoring and evaluation also showed high scores from peer, self, and teacher assessment.
The scores in the graph can also be seen that the peer assessment scores from the planning, monitoring, and evaluation aspects are not much different. Likewise, the self and teacher assessment scores do not differ much from the three metacognitive aspects. This means that the metacognitive thinking rubric is effectively used as an assessment guide by students and lecturers in vocational education. These scores have shown the level of students’ metacognitive thinking that is in accordance with the characteristics of assessment methods in vocational education based on performance-based assessment and project-based learning models.
8. Metacognitive assessment model for PjBL through blended learning MOOCs
The results of the research on assessment models, instruments, and rubrics that have been integrated with LMS through blended learning practice MOOCs have successfully measured students’ metacognitive thinking skills [1]. This is because this assessment model provides opportunities for students to assess their own answers (self-assessment) and provides opportunities for students to assess the answers of their peers (peer-assessment). Students are involved in assessing and evaluating answers based on the assessment rubric given in BLEMS. This encourages students to be more proactive in evaluating their own metacognitive thinking skills so as to support the development of their metacognitive skills. This is in line with the results of Vaughan’s research that applies the Triad Approach Assessment (self, peer, teacher assessment) in blended learning where this assessment approach can support the development of students’ metacognitive skills [20]. In addition, the results of this study are also in accordance with the theory that has been described previously, namely metacognitive thinking or metacognitive thinking is an awareness of thinking about how we think, how we organize thinking strategies in order to complete certain tasks well. Metacognitive thinking can be categorized into 2 sub categories, namely metacognitive knowledge and metacognitive regulation. Metacognitive knowledge is further divided into declarative, procedural, and conditional thinking. Meanwhile, metacognitive regulation is divided into planning, monitoring, and evaluation processes.
In the context of learning in vocational education, these two categories allow to be measured and assessed. However, considering the performance-based and project- or product-based assessment methods in vocational education, the measurement of metacognitive regulation (planning, monitoring, and evaluation) is more likely to be measured. As explained by Klerk et al. [30] that the vocational education emphasizes performance-based assessment where students learn by doing. This is confirmed by Wimmers [19] that at the end of the vocational education program or professional education program, every student must achieve standardized work competence, so that in this educational program, performance-based assessment is a general method for assessing practical competence in an authentic context. In addition, PjBL is an alternative learning model in vocational education where students can plan, design, and reflect on their learning through projects [50]. PjBL is a student-centered learning model in which students work on a project, make a project report, and communicate the report to their peers and teaching staff [51]. Therefore, students can measure their metacognitive thinking skills through the process of planning, monitoring, and evaluating their performance and the projects or products they make.
In addition, the application of blended learning methods (online and face to face) in project-based learning is able to optimize the learning process carried out. The online method is carried out to strengthen basic theory before students work on projects directly (face to face) in the laboratory so that students are able to optimize the three metacognitive aspects, namely planning, monitoring, and evaluating project work. The following is a metacognitive assessment model that combines self, peer, and teacher assessment in a blended learning environment using a project-based learning model (Figure 5).
The picture above shows an assessment model that combines self, peer, and teacher assessment integrated in LMS-MOOCs with a project-based learning model. Blended learning consists of online learning and face to face in the laboratory. The online method is used to assess the three metacognitive aspects, namely planning, monitoring, and evaluation based on metacognitive rubrics that have been integrated in the online environment. While the face to face method is carried out in the laboratory for project work for students. In addition, face to face also allows educators to conduct authentic assessments of the three metacognitive aspects (planning, monitoring, evaluation). The final result of the implementation of this assessment model is the metacognitive thinking score of students in vocational education.
Learning evaluation methods are generally only teacher-centered, not involving students in assessing and reflecting on their own evaluation results. Their answers from carrying out activities at LMS-MOOCs were only judged by one side by the educator. Students only see the score or final score of each test they pass so they cannot see which aspect they lack. However, through this assessment model (self-assessment and peer-assessment), students are actively involved in assessing their higher order thinking skills, namely metacognitive thinking.
The assessment rubric aims to determine students’ metacognitive thinking skills in project-based learning in vocational education. The assessment rubric was developed for 3 activities, namely planning, monitoring, and evaluation, then integrated into the LMS-MOOCs blended learning practice method. This study also produces a metacognitive assessment model for blended learning models in vocational education. The resulting model is an integration of three activities with self-assessment, peer-assessment, and teacher-assessment assessments for the PjBL learning model [1]. The metacognitive assessment model can be an assessment method to measure students’ metacognitive thinking skills, especially in project/work-based learning in vocational education.
References
1.Mahande, R. D., Darmawan, F. A., and Malago, J. D. (2021). Metacognitive skill assessment model through the blended learning management system in vocational education. Jurnal Pendidikan Vokasi, 11(1), 1-13. DOI: 10.21831/jpv.v11i1.36912
2.Mane, F., and Corbella, T. (2017). Developing and running an establishment skills survey—Guide to anticipating and matching skills and jobs Vol. 5 Skills for Employment (Vol. 5). Publications Office of the European Union. DOI: 10.2816/413514
3.International Labour Office (2019). A Skilled Workforce for Strong, Sustainable and Balanced Growth (1st ed., Issue November). International Labour Organization.
4.Gotoh, Y. (2016). Development of Critical Thinking with Metacognitive Regulation. International Conference on Cognition and Exploratory Learning in Digital Age, Celda, 353-356.
5.Peter Ganter, J. (2019). The Trends Mapping Study on Innovation in Technical and Vocational Education and Training (TVET). the UNESCO-UNEVOC International Centre for Technical and Vocational Education and Training.
6.Mhouti, A. El, and Nasseh, A. (2017). Enhancing collaborative learning in Web 2.0-based e-learning systems: A design framework for building collaborative e-learning contents. Education and Information Technologies, 22, 2351-2364. DOI: 10.1007/s10639-016-9545-2
7.Kew, S. N., and Petsangsri, S. (2018). Examining the motivation level of students in e-learning in higher education institution in Thailand: A case study. Education and Information Technologies, 23, 2947-2967.
8.Al-araibi, A. A. M., and Naz, M. (2018). Technological aspect factors of E-learning readiness in higher education institutions: Delphi technique. Education and Information Technologies, 24, 567-590.
9.Zwart, D. P., Luit, J. E. H. Van, Noroozi, O., Goei, S. L., Zwart, D. P., Luit, J. E. H. Van, Noroozi, O., Goei, S. L., Zwart, D. P., Luit, J. E. H. Van, Noroozi, O., and Goei, S. L. (2017). The effects of digital learning material on students’ mathematics learning in vocational education The effects of digital learning material on students’ mathematics learning in vocational education. Cogent Education, 29(1). DOI: 10.1080/2331186X.2017.1313581
10.Mahande, R. D., Susanto, A., & Surjono, H. D. (2017). The Dynamics of Mobile Learning Utilization in Vocational Education: Frame Model Perspective Review. Turkish Online Journal of Educational Technology, 16(4), 65-76.
11.Hsu, C. M., Yeh, Y. C., and Yen, J. (2009). Development of design criteria and evaluation scale for web-based learning platforms. International Journal of Industrial Ergonomics, 39(1), 90-95. DOI: 10.1016/j.ergon.2008.08.006
12.Costello, E., Holland, J., and Kirwan, C. (2018). The future of online testing and assessment: Question quality in MOOCs. International Journal of Educational Technology in Higher Education, 15(42), 1-14.
13.Xiong, Y., and Suen, H. K. (2018). Possibilities, challenges and future directions. International Review of Education, 64, 241-263. DOI: 10.1007/s11159-018-9710-5
14.Earl, L., Katz, S., and WNCP. (2006). Rethinking Classroom Assessment with Purpose in Mind. In: Learning. DOI: 10.4135/9781446214695
15.Johnson, R. (2004). Peer Assessments in Physical Education. Journal of Physical Education, Recreation and Dance, 75(8), 33-40. DOI: 10.1080/07303084.2004.10607287
16.Veenman, M. V. J., Bavelaar, L., Wolf, L. De, and Haaren, M. G. P. Van. (2014). The on-line assessment of metacognitive skills in a computerized learning environment. Learning and Individual Differences, 29, 123-130. DOI: 10.1016/j.lindif.2013.01.003
17.Zheng, L., Li, X., Zhang, X., and Sun, W. (2019). The e ff ects of group metacognitive sca ff olding on group metacognitive behaviors, group performance, and cognitive load in computer-supported collaborative learning. The Internet and Higher Education, 42(19), 13-24. DOI: 10.1016/j.iheduc.2019.03.002
18.Altıok, S., Başer, Z., and Yükseltürk, E. (2019). Enhancing metacognitive awareness of undergraduates through using an e-educational video environment. Computers and Education, 139(May), 129-145. DOI: 10.1016/j.compedu.2019.05.010
19.Wimmers, P. F. (2016). Innovation and Change in Professional Education 13 Assessing Competence in Professional Performance across Disciplines and Professions (M. Mentkowski, Ed.; 1st ed.). Springer. DOI: 10.1007/978-3-319-30064-1
20.Koc, S., Liu, X., and Wachira, P. (2015). Assessment in Online and Blended Learning Environment. Information Age Publishing.
21.Sudira, P. (2017). TVET Abad XXI (Hartono, Ed.; 2nd ed.). UNY Press.
22.Jung, S., Chul, J., and Nam, S. (2019). Impact of vocational education and training on adult skills and employment: An applied multilevel analysis. International Journal of Educational Development, 66(September 2018), 129-138. DOI: 10.1016/j.ijedudev.2018.09.007
23.Hampf, F., and Woessmann, L. (2017). Vocational vs. General Education and Employment over the Life-Cycle: New Evidence from PIAAC. NBER Working Papers Series, 5(10298), 1-17.
24.Flavell, J. H., Miller, P. H., and Miller, S. A. (1979). Cognitive Development (L. Pearson, Ed.; 4th ed.). Prentice Hall.
25.Yusuf, Y., Rodding, R., Awang, H., and Mukhtar, I. (2017). Metacognitive Strategies in Promoting the Development of Generic Competences in High TVE in Malaysia. Pertanika Journal of Social Science and Humanities, 25, 247-256.
26.Jacobs, J. E., and Paris, S. G. (1987). Children’s Metacognition About Reading: Issues in Definition, Measurement, and Instruction Children’s Metacognition About Reading: Issues in Definition, Measurement, and Instruction. Educational Psychologist, 22(3), 37-41. DOI: 10.1080/00461520.1987.9653052
28.Schraw, G., and Moshman, D. (1995). Metacognitive Theories. 7(4), 351-371.
29.Lai, E. R. (2011). Metacognition: A Literature Review Research Report. Pearson’s Research Reports, April.
30.Klerk, S., De Veldkamp, B. P., and Eggen, T. J. H. M. (2018). A framework for designing and developing multimedia-based performance assessment in vocational education. Educational Technology Research and Development, 66(1), 147-171. DOI: 10.1007/s11423-017-9559-5
31.Mendoza, R., and González. (2016). User-centered Design Strategies for Massive Open Online Courses (MOOCs). In User-Centered Design Strategies for Massive Open Online Courses (MOOCs) (1st ed.). Information Science Reference. DOI: 10.4018/978-1-4666-9743-0
32.Haber, J. (2014). MOOCs. In Angewandte Chemie International Edition, 6(11), 951-952. (1st ed.). MIT Press.
33.Dasarathy, B., Sullivan, K., Schmidt, D., and Porter, A. (2018). P212-Dasarathy. Future of Software Engineering Proceedings, 212-224. DOI: 10.1145/2593882.2593897
34.Rosé, C. P., Ferschke, O., Tomar, G., Yang, D., Howley, I., Aleven, V., Siemens, G., Crosslin, M., Gasevic, D., and Baker, R. (2015). Challenges and opportunities of dual-layer MOOCs: Reflections from an edX deployment study. Computer-Supported Collaborative Learning Conference, CSCL, 2, 848-851.
35.Willging, P. A., and Johnson, S. D. (2019). Factors that influence students’ decision to dropout of online courses. Journal of Asynchronous Learning Network, 13(3), 115-127. DOI: 10.24059/olj.v8i4.1814
36.Lave, J., and Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation (E. Wenger, Ed.). Cambridge University Press.
37.Ackerman, R., Parush, A., Nassar, F., and Shtub, A. (2016). Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing. Computers in Human Behavior, 54, 101-113. DOI: 10.1016/j.chb.2015.07.041
38.Tsai Y., Hsun, Lin, C. Hung, Hong, J. Chao, and Tai K. Hsin. (2018). The Effects of Metacognition on Online Learning Interest and Continuance to Learn with MOOCs. Computers and Education, 121, 18-29. DOI: 10.1016/j.compedu.2018.02.011
39.Koſar, Gülten. 2016. “A Study of EFL Instructors’ Perceptions of Blended Learning.” Procedia - Social and Behavioral Sciences 232:736-44. DOI: 10.1016/j.sbspro.2016.10.100.
40.Kaur, Manjot. 2013. “Blended Learning-Its Challenges and Future.” Procedia - Social and Behavioral Sciences 93:612-17. DOI: 10.1016/j.sbspro.2013.09.248.
41.S. Krajcik and M. Czerniak (2014). Teaching Science in Elementary and Middle School: A Project-Based Approach. In Interdisciplinary Journal of Problem-Based Learning (5th ed., Vol. 8, Issue 1). Routledge. DOI: 10.7771/1541-5015.1489
42.Husamah, H., and Pantiwati, Y. (2014). Cooperative learning STAD-PjBL: Motivation, thinking skills, and learning outcomes of biology department students. International Journal of Education Learning and Development, 2(1), 68-85.
43.Sukamti, E. U., Putra, A. P., and Devi, A. C. (2019). Innovation of Project Base Learning (PjBL) on Outdoor Study for PGSD’s Student Activity on Education Diffusion. Int. J. Innov. Creat. Chang, 5(5), 546-561.
44.Joseph, C. T., and Chapman, A. (2016). Project-Based Learning for Academically-Able Student (1, Ed.). Sense Publisher.
45.Ngeow, K., and Kong, Y.-S. (2001). Learning To Learn: Preparing Teachers and Students for Learning To Learn: Preparing Teachers and Students for Problem-Based Learning. ERIC. ERIC Digest, 20, 1-6.
46.Netto-Shek, Ho, T. B., and Chang, A. S. (2014). Making Projects Work: Structuring Learning. Managing Project work in Schools: Issues and Innovative Practices. Singapore (1st ed.). Prentice Hall.
47.Helle, L., Tynjälä, P., and Olkinuora. (2016). Project-Based Learning in Post-Secondary Education – Theory, Practice and Rubber Sling Shots. Higher Education, 51, 287-314. DOI: 10.1007/s10734-004-6386-5
48.Barak, M., and Watted, A. (2017). Project-based MOOC: Enhancing knowledge construction and motivation to learn. In: B. Berenfeld and J. Ginestie (Eds.), Digital Tools and Solution for Inquiry-Based STEM Learning (p. 26). IGI Global. DOI: 10.4018/978-1-5225-2525-7
49.Barak, M., and Dori, Y. J. (2005). Enhancing undergraduate students’ chemistry understanding through project-based learning in an IT environment. Science Education, 89(1), 117-139. DOI: 10.1002/sce.20027
50.Doppelt, Y. (2005). Assessment of Project-Based Learning in a MECHATRONICS Context. 16(2).
51.Marx, R. W., Blumenfeld, P. C., Krajcik, J. S., and Soloway, E. (1997). Enacting Project-Based Science. The Elementary School Journal, 97(4), 342-358.
Written By
Ridwan Daud Mahande, Fitrah Asma Darmawan and Jasruddin Daud Malago
Submitted: 15 June 2021Reviewed: 21 June 2021Published: 17 September 2021
Open access peer-reviewed chapter
