Sample data to illustrate cue diagnosticity and cue utilization.
The accuracy of comprehension monitoring affects the effectiveness of rereading, which in turn affects comprehension. Thus, much research has focused on finding ways to improve monitoring accuracy. The cue-utilization framework of metacognitive monitoring provides a framework for understanding how to improve monitoring accuracy. It suggests that accuracy is driven by cues people use to judge comprehension. When people utilize cues that are highly diagnostic of performance on a test of comprehension, accuracy should improve. Many interventions that have been shown to improve monitoring accuracy have attributed the improved accuracy to increased access to highly diagnostic cues, but have failed to identify highly diagnostic cues. In our recent research, we found that instructing students to generate drawings before judging comprehension improved monitoring accuracy. Using graphic analyses protocol, we identified highly diagnostic cues. In this chapter, we will describe the procedure we used to identify these cues contained in drawings.
- metacomprehension accuracy
- cue diagnosticity
- cue utilization
In their seminal work, Nelson and Narens  described the theoretical relation between metacognitive monitoring and regulation of behavior. Building on this work, contemporary models of self-regulated learning describe learning as the interaction between metacognitive monitoring and regulation of study [2, 3, 4, 5, 6]. In particular, according to these models, as people study, they monitor their learning and use this information to guide subsequent study. Thus, accurate monitoring is required to effectively and efficiently manage one’s study [7, 8]. If people do not accurately differentiate well-learned materials from less-learned materials, they could waste time studying material that is already well learned or worse fail to restudy material that has not yet been adequately learned. Given the important role that monitoring plays in learning, it is important to find ways to improve the accuracy of metacognitive monitoring.
Accurate metacognitive monitoring is especially important in the area of reading . A number of interventions have been developed to improve the accuracy of comprehension monitoring (called metacomprehension accuracy ). However, only recently have researchers examined the effect of drawing on metacomprehension accuracy. The primary objective of this chapter is to present data that provide a potential explanation for the beneficial effect of drawing on metacomprehension accuracy.
To provide context for our study, we will first describe how metacomprehension has been measured. We will then present a theoretical framework that identifies key factors for improving monitoring accuracy and show how this framework can help explain why previous interventions have improved metacomprehension accuracy. Finally, we will present empirical evidence that suggests drawing improves metacomprehension accuracy by providing access to cues that are diagnostic of comprehension and facilitating the utilization of these cues when judging comprehension.
2. Measuring metacomprehension accuracy
Glenberg and Epstein  developed the now widely used procedure for studying metacomprehension accuracy. They had participants read a series of short texts, judge their understanding on each text, and then completed a test for each text.
Metacomprehension accuracy describes the relation between a person’s judgments of comprehension and actual test performance. Accuracy can be described in two distinct ways. One is the degree to which the magnitude of the judgments is related to the actual magnitude of test performance. This kind of accuracy has been called
These measures are theoretically orthogonal . That is, while absolute and relative accuracy could both be high for a person, absolute accuracy could be high while relative accuracy is low or vice versa. Moreover, variables that influence one kind of accuracy may not influence the other. For example, domain knowledge has been shown to influence absolute accuracy, but does not influence relative accuracy . Thus, to avoid confusion, it is important to be clear whether one is examining absolute or relative accuracy. We will focus on relative accuracy for the remainder of this chapter.
3. A framework for improving metacomprehension accuracy
Understanding approaches to improving metacomprehension accuracy requires theories of both metacognitive monitoring and comprehension . Rawson et al.  used the cue-utilization framework of metacognitive monitoring  and the construction-integration model of comprehension  to identify ways to improve metacomprehension accuracy. The cue-utilization framework suggests that people’s metacognitive judgments are not based on direct access to memory and comprehension processes; instead, judgments are based on cues people have available about the content of their memory and comprehension processes. The accuracy of metacomprehension judgments is then determined by the degree to which the cues used to judge comprehension are diagnostic (predictive) of performance on a test of comprehension.
Theories of comprehension, like the construction-integration model , help identify the cues that should be diagnostic of performance on tests of comprehension. According to this model, readers construct meaning from text at several levels: a lexical or surface level, a textbase level, and a situation model level. The lexical level is constructed as the words and phrases appearing in the text are encoded. The textbase level is constructed as segments of the surface text are parsed into propositions, and as links between text propositions are formed based on argument overlap and other text-explicit factors. Deeper understanding of the text is constructed at the level of the situation model, which involves connecting information from the text with the person’s prior knowledge and using it to generate inferences and implications from the text. A person’s situation model largely determines performance on tests of comprehension . Thus, getting people to base their metacomprehension judgments on cues related to their situation model rather than their textbase should increase the predictive accuracy of judgments [13, 17].
As noted by Thiede and de Bruin , interventions designed to improve metacomprehension accuracy have attempted to focus readers on cues related to the situation model when judging comprehension. Some have sought to increase the salience of diagnostic cues by instructing readers to encode texts in specific ways that facilitate construction of the situation model, while others have sought to increase the salience of diagnostic cues by instructing readers to retrieve information about the texts prior to judging comprehension. The different approaches are to improving metacomprehension accuracy alter the standard experimental procedure developed by Glenberg and Epstein , these changes are illustrated in Figure 1.
4. Encoding-based approaches to improving metacomprehension
One approach shown to improve metacomprehension accuracy is to provide instructions for reading texts that promote construction of the situation model—connecting ideas in a text to one another and to one’s prior knowledge. By promoting construction of the situation model during reading, cues associated with the situation model presumably become more salient at the time of judging comprehension. Given these cues should be predictive of test performance, using these cues for judgments should increase metacomprehension accuracy.
In the metacomprehension literature, two studies have examined how metacomprehension accuracy is affected by promoting construction of the situation model while reading. Specifically, Redford et al.  examined the effect of concept mapping on metacomprehension accuracy. A concept map is a graphic representation of the underlying structure of the meaning of a text. Constructing concept maps can be an effective organizational strategy, which helps readers connect ideas in a text . Thus, as concept mapping helps readers construct a situation model for a text, Redford et al.  hypothesized that generating concept maps would increase metacomprehension accuracy. In accord with this hypothesis, they showed that metacomprehension accuracy was greater for a group who generated concept maps than for a group presented concept maps during reading and a control group. Concept mapping has also been shown to improve metacomprehension accuracy for at-risk readers .
Another technique used to promote construction of the situation model is to have readers self-explain while reading. Chi  showed that self-explanation improved reading comprehension by helping them connect ideas into a more coherent representation of a text. Griffin et al.  hypothesized that self-explaining would help students connect ideas within a text and would focus students on cues related to the situation model when judging comprehension, thereby improving their metacomprehension accuracy. Consistent with this hypothesis, Griffin and colleagues showed accuracy was greater for a group of college students who self-explained as they read than for a group who did not self-explain while reading.
In sum, interventions that promote construction of a situation model for a text during reading improve metacomprehension accuracy. These interventions appear to focus readers on diagnostic cues for judging comprehension. The literature suggests the effects on metacomprehension are robust; interventions that promote development of a situation model have improved accuracy for typical, at-risk college students, and younger students.
5. Retrieval-based approaches to improving metacomprehension
Another approach to improving metacomprehension accuracy is to incorporate a retrieval attempt prior to judging comprehension into the standard procedure . According to the cue-utilization framework of metacognitive monitoring , as the person contemplates how well a text was understood, he or she may rely on a variety of cues to make this judgment. Retrieving information about texts may allow a reader to evaluate the quality of cues used to retrieve information about a text. That is, when judging comprehension, the person may reflect on how successfully he or she had retrieved information. Accordingly, a text may receive a high rating of comprehension if the person had been able to retrieve a great deal of information about the text during the retrieval attempt. By contrast, a text may receive a low rating of comprehension if the person struggled to retrieve information about the text. Assuming availability of information during the retrieval attempt is related to availability of information at the time of the test, then using the retrieval of information as a basis for metacomprehension judgments should improve metacomprehension accuracy because the basis of the judgments should be highly related to test performance.
Accuracy of metacomprehension judgments may be affected by the timing of the retrieval attempt. Activation theories  may help explain why. According to these theories, spreading activation occurs during reading and more information is active in working memory shortly after reading than after a delay—when information has decayed from memory. When retrieving information immediately after reading, a person may have access to a highly active mental network. This is, even for less-understood texts, the person may have access to information in short-term memory. Thus, the retrieval attempt for less-understood and more-understood texts may produce a set of homogeneous cues for judging comprehension that may not help discriminate less-understood texts from more-understood texts. By contrast, when the retrieval attempt occurs after a delay, activation of the mental network for a text may have decayed and a person may have access to only that information retrievable from long-term memory. Thus, for a less-understood text, the person may have little to draw on when retrieving information; whereas, for a more-understood text, the person may retrieve much more information. Retrieving information after a delay may produce a set of heterogeneous cues for judging comprehension that may highlight differences between more-understood texts and less-understood texts. Moreover, cues available in long-term memory are likely to be highly indicative of test performance because both cues attempts and tests occur after a delay and are based on retrieval of information from long-term memory. Thus, retrieval after a delay may produce higher levels of metacomprehension accuracy.
Researchers have examined the effect of different retrieval tasks on metacomprehension accuracy. For instance, Thiede and Anderson evaluated the effect of writing summaries on metacomprehension accuracy . They compared metacomprehension accuracy across three groups: a control group, an immediate-summary group, and a delayed-summary group. The control group read a set of texts, judged comprehension of each text, and then completed a test of each text. The immediate-summary group read a text then immediately wrote a summary for the text. After reading and summarizing each text, they made metacomprehension judgments and completed a test for each text. The delay-summary group read all the texts, they then wrote summaries for each text. After reading and summarizing all the texts, they made metacomprehension judgments and completed a test for each text. Consistent with the theory outlined above, metacomprehension accuracy was greater for the delayed-summary group than for the other groups. This effect holds for typical and at-risk college students .
Thiede and colleagues evaluated the efficacy of a less time consuming retrieval task on metacomprehension accuracy . They had students generate a list of five keywords that captured the essence of each text. Metacomprehension accuracy was greater for the delayed-keyword group than for the immediate-keyword group or the control group. The delayed-keyword effect has been replicated with college students [29, 30] and younger students [31, 32].
van Loon et al.  evaluated the effect of completing an informational diagram of cause-and-effect relations on metacomprehension accuracy. Students read short texts describing cause-and-effect relations. Then they were shown a diagram of the cause-and-effect relation described in a text with key information deleted from the diagram. Participants in diagramming groups were instructed to provide the missing information. Metacomprehension accuracy was greater for the group that completed diagrams after a delay than for the group that completed diagrams immediately after reading or for the group that did not complete diagrams.
In sum, retrieving information about texts prior to judging comprehension improves metacomprehension accuracy; however, only when the retrieval attempt is delayed. A variety of retrieval tasks have been shown to improve metacomprehension accuracy. The literature suggests the effects on metacomprehension are robust; retrieval tasks have improved accuracy for typical and at-risk college students, as well as for younger students.
6. Drawing to improve metacomprehension accuracy
Theoretically, drawing has promise as an intervention to improve metacomprehension because it has been shown to facilitate construction of the situation model. Although the results examining the effect of drawing on learning are mixed, with some studies showing drawing improves learning  and others showing no benefit to drawing . The results fairly consistently show that drawing improves conceptual understanding but not factual learning . Put differently, deep comprehension, which requires a complete mental model, benefits from drawing.
The generative theory of drawing construction  helps explain the benefit of drawing on conceptual understanding and comprehension. According to this theory, readers construct a verbal representation of written words and a visual representation when drawing. Constructing a mental model of the content involves (a) selecting key elements from the verbal and visual representations, (b) organizing the key elements and connecting them to prior knowledge, and (c) integrating the verbal and visual representations into a coherent mental model. Thus, a drawing generated while reading represents a reader’s integrated verbal and visual representations, which may provide a more coherent representation of a phenomenon that a representation based purely on verbal information (e.g., a summary of a text).
A high quality drawing connects key elements and illustrates how the system as a whole functions. If a person can create a high quality drawing, he or she should be able to perform well on a test of deeper comprehension because the drawing and the test both depend on a coherent mental model. If a person cannot generate a high quality drawing, he or she should not be able to perform well on a test of deeper comprehension. Therefore, the quality of a drawing should be predictive of performance on a test of comprehension—and using drawings as a cue for judging comprehension should promote high levels of metacomprehension accuracy. Thus, drawing while reading has potential as an encoding-based approach to improving metacomprehension accuracy.
Drawings have also been shown to provide valuable feedback regarding level of understanding . That is, drawings help students identify gaps in understand. Thus, drawing also has potential as a retrieval-based approach to improving metacomprehension accuracy.
Despite the theoretical appeal of using drawings to improve metacomprehension accuracy, only recently have researchers examined the effect of drawing on accuracy. In particular, drawing has been used as an encoding task [38, 39] and as a retrieval task . The results of these studies are mixed; however, methodological differences make it difficult to compare the results across studies.
Drawing had no effect on metacomprehension accuracy in two studies [38, 40]. In these studies, rather than read a set of different texts and generate a drawing for each, participants read contiguous texts and generate a
By contrast, Thiede et al.  had fifth grade students generate drawings for different science text while they read. Student then predicted their performance and completed a test for each text. This is the standard experimental procedure with the encoding-based approach to influence metacomprehension, as illustrated in Figure 1. A key finding of this study was that drawing dramatically improved metacomprehension accuracy when students received instruction on generating organizational drawing—drawings that connect ideas within a text to one another and to prior knowledge. By contrast, drawings had no effect on metacomprehension accuracy when students were not instructed to create organizational drawing. The current study builds on the study by Thiede et al.
6.1 Overview of study and study design
According to the cue-utilization framework, monitoring accuracy is dependent on cue diagnosticity (how predictive a cue is of test performance) and cue utilization (which cues a person uses for the metacognitive judgment). van Loon et al.  developed a procedure to decompose judgment accuracy into these two components. In particular, they examined the diagnosticity of a cue by computing the correlation between the cue and test performance across texts. Similarly, they examined cue utilization by computing the correlation between the cue and the metacomprehension judgment across the texts. As in Thiede et al. , we used an experimental design to examine the effect of drawing instruction on cue diagnosticity and cue utilization.
We evaluated the effect of two kinds of drawing instruction on cue diagnosticity and cue utilization. Ninety-two fifth grade students were randomly assigned to two instructional groups. Students in each group read five texts on different science topics and generated drawings as they read. They then predicted their performance, and completed a test for each text. The Organizational-Drawing group (
6.2 Potential cues for metacomprehension judgments
As noted above, theories of comprehension, like the construction integration model , define deeper comprehension as a representation of a text that includes connections of ideas contained in a text to each other and prior knowledge (the situation model). The metacomprehension literature suggests that metacomprehension accuracy improves when people base their metacomprehension judgments on cues related to their situation model. Moreover, studies of self-reported cue use provide evidence that accuracy is greater for people who report using cues related to the situation model (i.e., their ability to link ideas contained in a text) than for people who reported using other cues . Thus, cues that provide information related to connecting ideas and use of prior knowledge should be highly diagnostic.
To examine cue diagnosticity and cue utilization of drawings, we refined the graphic analysis protocol (GAP), which had been used to score graphics contained in science textbooks [41, 42], to score student drawings of scientific texts. The GAP-drawing provides a more fine-grained measure of drawing quality than the overall measure of quality typically used in drawing literature . The GAP-drawing provides scores on two broad dimensions: drawing content and drawing relations.
For each text, students generated a drawing as they read. Students also made a metacomprehension judgment (i.e., they predicted their performance on a five-item test of comprehension) and completed an inference test of reading comprehension for each text. Drawings were scored using the GAP-drawing. Cue diagnosticity was operationalized as the intra-individual correlation between drawing metrics and test performance. Cue utilization was operationalized as the intra-individual correlation between drawing metrics and metacomprehension judgments. To illustrate these measures and how cue diagnosticity and cue utilization influence metacomprehension accuracy, consider the example shown in Table 1.
|Text||Judgment||Performance||Number of elements||Number of big ideas||Connections|
|Metacomprehension accuracy = 0.78|
|Cue diagnosticity||Cue utilization|
|Number of elements||0.20||−0.11|
|Number of big ideas||0.40||0.33|
For the student below, the number of elements was fairly weakly correlated with test performance, which indicates this is not diagnostic of performance on the test of comprehension. The number of big ideas was more strongly correlated with test performance than was the number of elements, but the correlation is only moderate. By contrast, the connections are perfectly correlated with test performance—test performance was higher for texts with higher connections scores and lower for texts with lower connections scores. Connections are a highly diagnostic cue of comprehension. Regarding cue utilization, the number of elements is weakly and negatively correlated with metacomprehension judgments, the number of big ideas was moderately correlated with judgments, and connections was highly correlated with judgments. These correlations suggest that this student used connections as bases of metacomprehension judgments and relied less on the number of big ideas and the number of elements to judged comprehension.
Cue diagnosticity and cue utilization help explain the relative high level of metacomprehension accuracy for this student (metacomprehension accuracy = 0.78). For this student, connections were a highly diagnostic cue and the students used this cue for judging comprehension accuracy. Had this student relied heavily on the number of elements to judge comprehension, metacomprehension would have been reduced because the number of elements is not predictive of test performance.
6.3 Results of study
This chapter focuses on cue diagnosticity and cue utilization; however, it is important to note that metacomprehension accuracy was significantly greater for the Organizational-Drawing group (mean metacomprehension accuracy = 0.51) than for the Representational-Drawing group (mean metacomprehension accuracy = −0.03). Cue diagnosticity and cue utilization help explain the difference in accuracy across groups.
As shown in Table 2, several drawing metrics were predictive of performance on tests of comprehension for the Organizational-Drawing group. In particular, for this group, systematicity, semantic relations, connections and the number of big ideas were are significantly correlated with test performance. By contrast, for the Representational-Drawing group, none of the drawing metrics were predictive of comprehension test performance.
|Number of actions||−0.15 (0.13)||0.13 (0.11)|
|Number of related elements||−0.01 (0.11)||0.18 (0.11)|
|Number of novel elements||−0.12 (0.12)||0.13 (0.13)|
|Number of unrelated elements||−0.15 (0.14)||−0.01 (0.14)|
|Number of big ideas||−0.14 (0.18)||0.42 (0.14)*|
|Systematicity||0.22 (0.23)||0.24 (0.11)*|
|Semantic relations||0.03 (0.15)||0.22 (0.11)*|
|Connections||−0.09 (0.16)||0.66 (0.18)*|
|Number of captions||−0.02 (0.13)||0.15 (0.12)|
These results suggest that instruction on how to generate drawings significantly affects cue diagnosticity. That is, with instruction on how to generate organizational drawings, drawing metrics related to connecting ideas to one another and to prior knowledge are predictive of performance on a test of comprehension (see the rightmost column of Table 2). It is important to note that the cues identified as diagnostic for this group are those hypothesized to be predictive of comprehension by theories of comprehension. Without instruction on generating organizational drawings, drawings do not provide diagnostic cues. Thus, for this group, drawing does little to provide useful cues for judging comprehension.
To better understand how these cues might affect metacomprehension accuracy, we need to examine cue utilization. As shown in Table 3, for the Organizational-Drawing group, a variety of drawing metrics were correlated with metacomprehension judgments, which suggests students in this group utilized a number of different drawing metrics in making their judgments. Most importantly, this group utilized four of the cues that were highly diagnostic of performance on comprehension test (i.e., systematicity, semantic relations, connections and the number of big ideas). By contrast, for the Representational-Drawing group, only connections were correlated with metacomprehension judgments. However, for this group, connections were not correlated with test performance; therefore, utilizing this cue would not contribute to a high level of judgment accuracy.
|Number of actions||−0.16 (0.12)||0.24 (0.09)*|
|Number of related elements||0.14 (0.11)||0.07 (0.11)|
|Number of novel elements||0.40 (0.09)||0.10 (0.11)|
|Number of unrelated elements||0.07 (0.10)||−0.11 (0.13)|
|Number of big ideas||0.10 (0.13)||0.22 (0.10)*|
|Systematicity||0.21 (0.15)||0.27 (0.13)*|
|Semantic relations||0.16 (0.11)||0.24 (0.12)*|
|Connections||0.73 (0.17)*||0.60 (0.14)*|
|Number of captions||−0.02 (0.12)||0.07 (0.10)|
These results provide additional empirical evidence that metacomprehension accuracy is influenced by cue diagnosticity and cue utilization. Metacomprehension accuracy was greater for the Organizational-Drawing group than the Representational-Drawing group. Drawings provided diagnostic cues for the Organizational-Drawing group but not for the Representational-Drawing group. Moreover, diagnostic cues were utilized for metacomprehension judgments for the Organizational-Drawing group but not for the Representational-Drawing group.
Metacomprehension accuracy is important to reading comprehension because monitoring guides decisions about rereading [31, 44], which improves overall comprehension [32, 45]. Thus, it is important to find ways to improve metacomprehension accuracy.
The cue-utilization framework of metacognitive monitoring  suggests improving monitoring accuracy involves identifying cues that are highly diagnostic of test performance and then instructing people to use those cues when making judgments. Thus, as described above, researchers have employed a variety of techniques to help facilitate the construction of a situation model or access the situation model prior to judging comprehension because this arguably provides cues that are highly diagnostic of comprehension tests. Researchers have also employed other techniques to promote use of diagnostic cues when making metacomprehension judgments .
Recent research using drawings as an encoding task shows promise in improving metacomprehension accuracy. This research shows that drawings need to emphasize the underlying organization of the phenomenon described in the text to improve metacomprehension accuracy, which is consistent with research showing the effect of graphics on metacomprehension accuracy is determined by the nature of the graphics presented with texts [46, 47, 48]. Specifically, organizational graphics improved metacomprehension accuracy and other graphics have little or adverse effects on metacomprehension accuracy .
The GAP-drawing provides a scoring system to help identify specific attributes of drawings that could be diagnostic of comprehension and utilized as a basis for metacomprehension judgments. Our findings suggest that with instruction on generating organizational drawings while reading, metrics related to drawing relations are predictive of test performance (diagnostic). Moreover, the instruction promoted utilization of these cues when judging comprehension.
Instructions focused on generating organizational drawings improved metacomprehension accuracy and comprehension. Thus, drawing can influence learning. More research is needed to identify the most effective instruction for drawing. With attention to cue diagnosticity and cue utilization, this research could reshape the field of metacomprehension.