The Power of Peer Engagement: Exploring the Effects of Social Collaborative Annotation on Reading Comprehension of Primary Literature

1.1.

Overview and context

Ongoing and deep engagement with the primary literature (PL) as a reader, researcher, and writer are essential practices as doctoral students develop into scholars [1]. The primary research literature is a unique genre of writing, not meant to be read from start-to-finish like novels, and beginning graduate students tend to be unaware of this academic norm [2]. Reading is deeply intertwined with writing and research [3]. Despite this, many higher education programs do not explicitly teach critical reading of PL (CRPL) skills, which are generally assumed and unaided [2]. As Margolis noted, whenever skills in higher education are implicit yet essential, they form hidden expectations that advantage the most prepared students [4]. How this hidden curriculum works in graduate education (GE) can be understood through doctoral persistence. Who persists is predictable by the demographic factors of age, gender, ethnicity, and marital status [5, 6]. Men are more likely to finish than women, older White students are more likely to complete than older students of color [7], and married students are more likely to graduate than single students [8, 9]. Another way to explain disparities in doctoral persistence is that the mean time-to-attrition for doctoral students of color is 23 months, a full year earlier than their White counterparts [10]. A telling example of this disparity in GE in the United States is that only 32.5% of doctoral and 46% of master’s degrees were earned by students of color in 2018 [11]. What groups more likely to graduate have in common is greater social capital that provides access to essential resources and support [12, 13].

However, even the most advantaged doctoral students will struggle to persist without strong reading skills because “reading and writing do not stand in a functional relationship with inquiry [i.e., research] but are constitutive of it—essential elements of the whole” [14, p. 226]. Instructors often fail to take advantage of the interconnection between reading and writing [15] in terms of interrogating texts [16] and developing critical reading skills [17], which at the doctoral level include being able to evaluate arguments critically [2]. Instead, doctoral reading is done in isolation and without feedback, except for occasional journal clubs [18–20]. An essential aspect of literacies, whether reading, writing, or reflecting, is that they are “derived from social practices (…) are socially constructed, often with our friends, in specific contexts, for specific purposes (…) and grow (…) from schools, from work, from cultures, from knowledges, from technologies, and so on” [21, p. 70]. From this constructivist view of literacy, CRPL in isolation should hinder critical reading skill development in all students as well as the co-constructed skills of writing and research.

Previous scholarship details this hindrance. In a seminal article, Boote and Beile observed that the “dirty secret known by those who sit on dissertation committees is that most literature reviews are poorly conceptualized and written” [22, p. 4]. Problems with writing indicate that student authors are unfamiliar with the conventions that make for effective research writing, underpinned by what Matarese described as a need for “an effective approach to reading in their discipline” [23, p. 76]. In other words, weak reading leads to weak writing.

Social collaborative annotation (SCA) is a strategy for building a learning community that is impossible with individual reading activities and is exceptionally well suited for online learning environments [24, 25]. SCA is a tool for increasing learner engagement in online courses, defined as “psychological investment in and effort directed toward learning, understanding, or mastering” [26, pp. 205–206]. Despite generally positive outcomes, minimal scholarship addresses the usage of SCA in online GE [27], a context that grew 155% from 0.9 million students in 2012 to 2.3 million students in 2020, similar to online undergraduate education growth during the same period [28]. The present study addresses this gap by providing a rich qualitative explanation of how and why peer engagement with PL using SCA improved reading comprehension in an online critical reading skills intervention of reading PL in a new field [29]. This article aims to advance the understanding of online peer engagement in terms of reading and annotation skill development at doctoral level.

1.2.

Theoretical and conceptual framework

Sociocultural theory provides the theoretical framework to explain findings between language and learning in higher education [30]. Learning occurs primarily from participating in discursive activities between teacher and student or between students in a collaborative discussion [31], which culminates in a written thesis at the doctoral level. Students develop and refine their thinking in GE through reading, writing, and feedback from more knowledgeable peers, professors, and advisors [31]. In online learning environments, student engagement develops through social interactions, and centering interaction with peers, instructors, and content is essential for success [26].

The traditional method to advance doctoral scholarship is the pedagogy of feedback, a form of sociocultural scaffolding aligning with Vygotsky’s zone of proximal development [30]. Advisors, serving as the more capable peer, edit rounds of writing to help doctoral students define the research topic, scope, and design of the project, gather materials, write and work through manuscript drafts, and disseminate their findings [32, 33]. This model assumes students already have CRPL skills [1]. It also leaves many research advisors feeling drained and frustrated, especially when the dissertation lags or students quit [10, 33–36]. Consequently, many students experience challenges learning the co-constructed skillsets of reading, writing, and researching from ambiguous or avoidant feedback [37, 38], or worse, for raciolinguistically diverse students, lack of guidance or intentional mistreatment [39]. It is no surprise, then, that the pedagogy of feedback works for some, is neutral for many, and harms others [37–39].

Adequate support and mentoring to develop literacy skills are significant barriers to the progress of raciolinguistically diverse students in graduate education [38, 40]. Language is a primary way humans create racial and ethnic identities, and therefore, language is inherently related to race and vice versa, a connection highlighted in the field of raciolinguistics [41]. As a conceptual framework, a raciolinguistic perspective acknowledges the co-construction of race and linguistic varieties as part of colonialism, settler nation-state formation, and world history [42]. This perspective also acknowledges that online environments can unmask covert racism [43]. Thus, race and ethnicity data were collected in this study to contextualize participants’ raciolinguistic experiences within the sociocultural framework of peer-based discourse about PL.

3.1.

Context and participants

The context was an online eight-week professional development course offered by the National Science Policy Network (NSPN) from March 1 to April 25, 2022, as preparation for a national science policy brief writing competition. Study participants used a software ecosystem consisting of Canvas for the course and Hypothes.is for SCA. Learners had to create free accounts for each as part of the pre-session enrollment. Study participants were further instructed to choose usernames that did not reflect their names, an essential step to protect their privacy.

The study participants included only currently enrolled doctoral students in U.S. university programs. Individual doctoral students were the unit of analysis. The primary exclusion criteria were enrollment in a non-U.S. doctoral program and enrollment in programs where reading is a topic of study, such as English, Journalism, and Rhetoric. Participant recruitment occurred through convenience sampling [68]. The sponsoring organization, NSPN, contacted their nationwide membership (n = 1574) with an online invitation to participate. They sent invitations through their email newsletter, website posts, and social media channels. Of those who responded, 28 indicated an interest in the study, and 24 met the inclusion criteria, consented, and began the study. Four people who did not qualify were not currently enrolled doctoral students (one was an undergraduate, and three were post-baccalaureates). The completion rate for the study was 100%.

Participant profiles reflected various life and professional experiences, as shown in Table 1. Participants had doctoral experience ranging from one to six years of full-time equivalent study. Nearly half of the participants (46%) were in the pre-dissertation stage (i.e., “early” defined as years 1–3), and most studied Life Sciences (58%). Participants represented all geographic regions of the U.S., except the Northwest, and were attending research-intensive universities (R1 or R2). Most participants were ages 25–34 (88%) and identified as women (79%). No participants had formal preparation in science policy, the field that served as the intervention’s topic, or astrophysics, the field from which the pre-post reading assessments were drawn.

While participation by people identifying as BIPOC was not an explicit recruitment goal, the demographics were significant and reported herein for three reasons. First, the scholarship on raciolinguistics in higher education indicates significant barriers for students of color [41, 42]. Second, participants shared experiences related to race, ethnicity, and gender, and this data was significant to the findings. Third, the host organization, NSPN, tracks diversity, equity, and inclusion (DEI) outcomes in their science policy activities, where people of color are underrepresented. To these aims, participants who identified as Asian or Asian American (21%), Hispanic (17%), Black or African American (8%), and Two or More Races (4%) were more highly represented in the study than in NSPN’s membership. This data provided important feedback for NSPN about its outreach efforts.

Participants were randomly assigned unique study numbers, ranging from SCA01 to SCA24, to protect their identities. Quotations in the remaining discussion reflect only this anonymous identifier. Citing participants by study number, while nontraditional, is not proscribed [65, 66, 68] and allows for greater accuracy, synthesis, and transparency in reporting this rich and complex data set that would not be possible with traditional citation methods.

3.2.

Data collection

Data collection included six sources of qualitative data: software reports, participants’ SCA work products (i.e., SCA assignments with embedded prompts), written metacognitive reflections, researcher observations, and interviews. The collection tools and frequency varied by the data source. Software reports measured participation and were generated daily by Canvas and Hypothes.is. Participants' work products were generated weekly by participation in course assignments and submitted through Canvas. Metacognitive reflections were generated weekly and submitted through Canvas. Researcher observations were generated weekly using two rubrics, implementation and session observation. Interviews were completed weekly through open-ended questions and at the end of the course as an in-depth structured elicitation of participants’ experiences. For the follow-up interviews, data collection occurred through audio-only calls on Zoom. The audio files were transcribed using Otter.ai and checked for accuracy. Data collection was completed in April 2022.

3.3.

Data analysis

The qualitative data were analyzed using content and thematic analysis [69]. Content coding and thematic analysis of the participants’ interviews were appropriate to address qualitative research questions about participants’ lived experiences and understanding [70]. Specific qualitative analysis techniques included inductive coding, pattern analysis, thematic analysis, frequency (i.e., counting), and proximity analysis [70–74]. The initial analysis occurred in three rounds, content-coding, pattern analysis, and thematic analysis. Then frequency analysis regarding SCA patterns from the Hypothes.is data sets were visualized by Crowdlaaers using learning analytics and quantization [69]. Additional software support for qualitative analysis came from the software NVIVO (QSR International Inc., Melbourne, Australia), which the student researcher used to code all assets digitally.

Coding qualitative data, such as interview transcripts, began with a conventional, inductive coding process [65, 70]. The researcher read each transcript line-by-line and marked the margins with impressions, ideas, concepts, words, phrases, and perceptions of emotional affect [73]. During the first coding cycle, the researcher used a descriptive coding method to assign labels to data and then summarized those in a word or short phrase. This approach’s advantage was to gain information directly from the participants without imposing preconceived notions [73]. In addition, the researcher wrote summaries after each interview and memos while coding interviews, continuously recording interactions between data and theory [74].

The first analysis round included pattern analysis and thematic coding, highlighting patterns and themes within the same participant’s work over a few weeks and between participants. The second coding round was focused on coding, aiming for code saturation and testing emerging ideas with disconfirming information [73]. The final coding round was theoretical coding, building on emergent concepts to clarify relationships between codes and themes, comparing findings with theory, and generating new hypotheses.

3.4.

Researcher positionality

The research paradigm that supports this research is an indigenous insider with added expertise [74, 75]. Many past and present roles position the researcher as indigenous to the academic writing and publishing community, including scientific writing instructor, coach, manuscript editor, published writer, and former researcher in life science. The researcher understands the community’s perspectives, behaviors, beliefs, and knowledge through these roles. Likewise, insider status comes from holding past and present institutional affiliations, such as at Johns Hopkins University, Nature Publishing Group, Science, The New York Times, and others. Crucial for this study and collaboration with the context, NSPN, the researcher was a science policy reporter for two years with Nature Medicine. Thus, this community perceives the researcher as a legitimate member who can speak with authority about publishing, language, reading, writing, annotation, and science policy skills at the doctoral level.

A deeper reflection of this indigenous-insider paradigm reveals several unearned advantages [76]. First, the researcher grew up speaking “standard” English and, thus, had a raciolinguistic advantage when learning MAE. To conduct this research, the researcher did not have to unlearn, relearn, or assimilate into a new raciolinguistic identity [21, 42]. In addition, the researcher identifies as White and cis, the default norms of White Supremacy Culture (WSC) within the academy, with characteristics such as perfectionism, one right way to do things, objectivity, and obsession with the written word [77, 78]. Finally, the researcher identifies as a U.S. female teacher. The unexamined Whiteness of teaching is normalized and centered because most U.S. K-12 teachers are White women [79]. Thus, a central task of teacher preparation is to explore stereotypical racist attitudes towards students of color and minoritized groups, including assumptions about what students should know [80]. One way to challenge WSC within academia is to make expectations explicit and offer instruction in assumed skills, a central aim of this study..

This mix of positionality exposes the researcher to name, recognize, and act to change the racism and linguicism deep in the consciousness [21]. Applied to this research study, this work means that the researcher emphasizes qualitative data (i.e., interviews and observations) as the highest priority. Qualitative data focuses on the participants’ experiences and avoids the WSC trap of objectivity [65]. Also, the focus on narrative data allows for an epistemological perspective that centers on the participants’ experiences [66]. After all, the researcher is an indigenous insider who identifies as an expert in the dominant raciolinguistic group. However, the researcher does not assume to know already which constructs to test, which is why the study’s focus on qualitative data is essential for exploring, explaining, and generating additional hypotheses and implications for practice and research.

4.1.

Peer engagement supported reading comprehension

Participants reported engaging with their peers using SCA in at least two ways, a passive reading of other people’s annotations and an active peer response. Peers provided valuable, low-stakes checks for understanding the text (SCA09, SCA15, SSCA18, SCA21, SCA23). A participant summarized the process, “You can first annotate what you don’t understand and then compare to others” (SCA09). The online and collaborative nature of the SCA tool meant that feedback on ideas about the text was available to all participants as needed in real-time throughout the intervention, whenever participants were reading.

4.2.

SCA created a low-stakes and reassuring learning environment

SCA helped to create peer engagement around a shared text, but more than SCA was needed to guarantee a positive online learning environment. Other intervention components and activities combined with SCA created a low-stakes environment where people felt safe to ask questions. The combination created a reassuring and psychologically safe learning environment necessary for learning at any stage, especially with a new topic and group. Feelings of safety allowed some participants to open up and share many of the invisible experiences they faced as students of color.

4.3.

Annotation is a dynamic reading practice that improves with practice and feedback

Participation in the intervention produced various impacts on annotation practices. Several rich themes emerged from the data, including a range within the group, various experiences with annotation before the course and evolving annotation practices. These themes reveal that annotation is not a static or fixed skill but a dynamic practice that changes with instruction, experience, and feedback.

Overall, annotations varied widely per participant in quantity and quality, with differences attributable to their prior knowledge and experience with annotation. In total, participants produced 810 annotations, and of these, 141 (17.4%) were in threads, meaning they were responding to peers. Of the 24 participants, 12 produced the most threads, averaging responses to 10 threads each. These participants were the most engaged with their peers throughout the intervention. In addition, a moderate correlation between SCA frequency and quality was indicated by the Spearman correlation coefficient between these two variables, r (22) = 0.6, p < 0.05. This finding is nuanced because participants’ reported that two factors, competing work demands during the intervention and prior experience with annotation, affected their SCA quality and quantity.

Some participants had taken annotation courses as undergraduates. One participant reported many years of experience with annotating research papers. However, she produced high-quality, standalone annotations with low frequency, citing a hefty workload as a barrier to participation in the intervention (SCA09). Another participant reported some experience with annotation in college, and in contrast, he produced many annotations of low or moderate quality in many threads (SCA08). There was a range of annotation experiences in the group.

In addition to a range of experience, participants also reported a range of annotation practices. Reported methods include highlighting PDFs with Adobe Reader (SCA09, SCA11), taking notes on a tablet (SCA01), using comments in Google Docs (SCA08), and sending long email chains (SCA18, SCA04). Participants reported annotating primarily for reading clubs (SCA08, SCA18, SCA04), in preparation for qualifying exams (SCA18, SCA21), for paid work (SCA09), and teaching (SCA10). Finally, one participant reported never annotating articles independently (SCA08). Thus, the range of participants’ annotation experiences and practices was wide, despite all being enrolled in U.S. doctoral programs.

During this intervention, all participants reported gaining their first exposure to SCA and an online tool (i.e., Hypothes.is). Nearly all participants reported a positive first experience (83%), while some reported a neutral first experience (17%). None reported the experience as negative. There were many benefits for those with a positive first experience with SCA, such as seeing in one place “where all the ideas are happening, and what each person is thinking” (SCA08). Others liked it because “you can first annotate yourself, mark what you don’t understand, and then compare to other people were saying” (SCA09) or “just have little conversations in the margins” (SCA04). About half of the participants reported using the “I” icon at the top of the annotation tool to block annotations, so they could first read an unmarked text before seeing others’ ideas.

In contrast, the other half reported reading the annotations while they read the text. Some compared SCA for reading collaboratively to the same benefits as Google Docs for writing collaboratively (SCA08, SCA04, SCA09). Another participant described having difficulty transitioning from reading articles on paper to digital articles, despite hating the paper waste. She explained how the course “introduced me to social annotations, which is very good and close to annotations on hard copies” (SCA11). For those with a neutral experience, some found that SCA was slower than their current method (SCA17, SCA19) or already had an effective annotation method (SCA06, SCA10). Nonetheless, many liked the SCA activity more than they thought, and several planned to use it again immediately (SCA17, SCA09, SCA19).

Another central theme is that annotation practices evolve with practice, instruction, modeling, and feedback. A participant explained that during her first exposure to PL as an undergraduate, she received no formal instruction and tried to “remember every detail and to understand everything” (SCA15). Then she realized that was impossible. She learned in a literature course that “annotations help you understand better, so I started annotating basic stuff that I didn’t know about the topic, like clarifying concepts” (SCA15). Now that she is in a doctoral program, her annotation practice has evolved further, such that her annotations are “mostly about doubts that I have, or how one piece of information links to another” (SCA15). Similarly, other participants’ annotation practices began with highlighting everything (SCA01, SCA04, SCA09, SCA11) and evolved to marking up texts with essential categories, such as ``motivation, RQ (for research question), methods, and importance” (SCA11).

After the intervention, the participant reported that her annotation practice became more strategic. She is looking for “the paper’s main argument and any holes or gaps in the evidence” (SCA15). Likewise, other participants no longer believed that a paper’s conclusions were automatically valid because they passed peer review. They now felt they must actively analyze those claims using the evidence presented (SCA04, SCA08, SCA11, SCA17, SCA18, SCA23, SCA24). Other participants explained how the course’s repeated annotation activities helped them develop a regular annotation habit in their doctoral reading (SCA01, SCA04, SCA23). These participants’ experience conveys that annotation practices change with experience and feedback, including feedback from peers and seeing peers’ annotations. In summary, the intervention had positive impacts on participants' diverse annotation practices, illuminating that annotation is a dynamic, evolving skill.

4.4.

Many valuable applications of SCA within higher education

Most participants liked SCA and volunteered many potentially valuable applications of SCA within HE. Suggestions included communicating with advisors in real-time about papers, various types of group work, and preparing undergraduate and graduate students to pass major exams. One participant explained that she and her advisor share key papers by sending a lot of PDFs back and forth through Slack, noting that “we don’t annotate them, but make chat comments, like ‘this is the paper and here’s what they found.’ It’s not really a conversation. I think SCA could really improve our communication” (SCA04). Others echoed this notion of SCA improving communication with advisors using SCA (SCA18, SCA14). Thus, participants suggested that SCA could improve communication about research articles in doctoral programs.

Other suggestions for SCA applications focused on improving group work where PL is central. Recommendations included coursework (SCA01, SCA17, SCA09, SCA15, SCA13), studying for qualifying exams (SCA18, SCA14), group writing projects (SCA18, SCA23, SCA06), and journal or reading clubs (SCA08, SCA18, SCA04). One of the problems with journal clubs is that “everyone just sits there and hopes that they don’t get called on with a question or ask questions” (SCA18). However, SCA overcomes reticence to share ideas by creating a forum for conversation in the margins with low social barriers to entry (SCA08). Another problem with reading clubs is long chains emailing comments on papers. “With SCA, we could go through papers together, and then if we were confused about something, we can ask in real-time rather than just chatting [texting] about it or sending a long chain of emails back and forth” (SCA04). These participants felt that SCA would encourage more participation, which is also efficient.

SCA could also replace traditional annotation for coursework and study for qualifying exams. Both situations involved writing summaries of a set of papers as a group. One participant suggested SCA instead of traditional annotation because “it’s so much easier to use, and you can see what people are thinking in each part” (SCA03). The same participant also explained how her policy writing group continued using SCA in the second part of the intervention course, even though it was not required. Her group continued using SCA “because it’s helpful, especially when you’re learning a field, to talk about and share thoughts on what we were reading” (SCA18). Thus, many participants felt that SCA would improve many aspects of studying with others using the PL.

The final suggested application of SCA was to prepare undergraduate and graduate students to read PL better and pass major exams, such as qualifying exams. Participants suggested preparing students in PL annotation as early as the second semester of undergraduate study (SCA01, SCA18, SCA15), during upper-division undergraduate courses (SCA04, SCA09, SCA23), or the first or second year of graduate school (SCA17, SCA23). Focusing on graduate school, one participant described a situation where she thought SCA would be most helpful:

There are no undergraduate standards for reading PL. The GRE doesn’t even test it. So, we bring first- and second-year doctoral students into the lab and do literature reviews every other month. Students are required to turn in a literature review about a paper, and then we [the senior graduate students and post-docs] review it and analyze their critiques. We try to help them think critically about papers. There would be a lot of value for the new students to see some social annotation about a new paper and the important things we pull out. (SCA17)

This participant recognized no undergraduate preparation standards for PL, yet new graduate students much write literature reviews. Making reading and critical thinking about the primary literature explicit through SCA would add transparency and value to an incoming graduate student’s educational experience.

Further, the underlying rationale for preparation to read primary literature reflects a reality summarized as “most people I’ve met [in graduate school] never learned how to do it, even though the professors expect us to” (SCA23). Another participant elaborated on this experience:

After a bachelor’s, master’s, and now a Ph.D., nobody ever taught me how to do reading for writing. I think that people feel, with reading, [that] you learn by doing, but it’s not true. You can learn some reading from writing, but what helps most is being taught to read critically, like this [intervention] did. (SCA18)

This participant explained how she had never been taught to read critically, benefitted from explicit instruction, and suggested SCA as a method to better prepare students for advancement in HE. In summary, participants appreciated SCA and identified many potential applications within higher education. 

6.1.

Implications for future practice

Overall, the study illustrated the effective use of SCA in facilitating cognitive connections and group learning, indicating the necessity to adapt instructional practices in online learning environments. Key elements of this adaptation include direct instruction, group practice, real-time peer feedback, and CRPL. These elements point to changes in instructional design that anticipate learners' challenges, particularly when understanding primary literature. This can be achieved by incorporating numerous worked examples and encouraging real-time SCA discussions to support peer-based discourse and critical thinking.

SCA's implementation revealed several teaching practice implications. The study's findings underscored the need for explicit instruction in reading primary literature to address the challenges posed by a lack of formal training. Second, it demonstrated the positive impact of SCA and peer engagement in teaching, particularly in enhancing reading comprehension and self-monitoring. Third, it emphasized the importance of early integration of instruction in critical reading of primary literature was emphasized, suggesting it should be part of both undergraduate and graduate teaching. Courses at these levels include group SCA discussions and practices that analyze research articles' methods and key concepts, aiding students in comprehending essential disciplinary ideas and practices.

The study further highlighted SCA's advantages, including its transparent approach to evaluating research articles, benefiting both expert and novice readers. The collaborative nature of SCA promotes collective learning, making it a valuable tool for reading beyond one's primary field and understanding broad social issues. SCA's utility was also demonstrated in a hybrid learning approach combined with CRPL, which could find applications in academic settings such as journal clubs, article comparison exercises, and disucssing seminal articles.

Critical reflection on the peer-engagement pedagogical design was another highlighted aspect, with tools like SCA assisting in identifying areas requiring more explicit instruction. However, the study also urged caution regarding potential pitfalls of SCA, such as distraction from comprehensive text processing and ensuring active student involvement was another focus, with suggestions of using tiered prompts for peer engagement to address varied participation levels. Future implementations should consider these challenges.

Finally, the study recognized the negative impacts of competitive academic environments on marginalized students and emphasized the necessity of a positive peer-based learning environment to counter such experiences. These findings underscore the importance of building a supportive learning environment that fosters peer-engaged cooperative learning, enabling students to express their uncertainties and participate in meaningful discourse. This includes addressing challenges faced by students from marginalized groups to ensure a more inclusive academic environment. The findings urge efforts to eliminate invisible barriers faced by students from marginalized backgrounds, enhancing the inclusivity and enrichment of academic experiences. Further, research on SCA in teaching practices should explore its full potential across various academic contexts, focusing on its long-term efficacy and adaptability among diverse groups of learners.

6.2.

Implications for future research

SCA adds another layer of socially constructed learning by making other people’s ideas visible and creating a low-stakes forum for analytic discourse. Such discourse is an essential aspect of critical reading and writing in a discipline. Future research should create an explicit conceptual bridge using SCA from reading analysis to writing synthesis, focusing on crucial doctoral outputs, such as literature reviews, presentations, posters, papers, and dissertations. Synthesis differs from summarizing in that synthesis is produced with analysis, while summaries can be made with skimming. The goal of synthesis is a deeper, more advanced understanding of texts that evolves the reader’s understanding. Thus, SCA could serve as a peer-reviewed bridge between reading and writing by extending the learning from the reading intervention, focusing on analysis, to additional practices that prepare learners for synthesis.

For example, using an SCA discourse practice environment could serve as the basis for developing and revising a written paragraph synthesizing critical elements of several articles. SCA discourse practice in this context would allow learners to see other people’s ideas, check their understanding with peers, and reduce apprehension. This line of future research could be maintained over a long term beyond a single course, such as an academic year or longer, to model disciplinary literacy spanning multiple courses.

Any future research with SCA should consider participants’ prior experience with annotation, as it varied widely in this study. Assessment of prior experience could include a skills assessment or survey. Developing a validated instrument, for instance, to assess annotation experience, would be very useful for developing a field of research that includes quantitative data. Finally, future research should consider SCA plus generative artificial intelligence systems (GAIs), such as GPT and Bard, as these systems can substantially alter learners' interactions with academic texts. 

6.3.

Limitations of SCA

Learning complex skills—such as reading, writing, and research via PL at the doctoral level—requires high levels of self-efficacy, self-regulation, and other mature learner characteristics [46]. These characteristics are more likely present collectively within working groups. However, to the extent that these properties of group learning benefit individuals, the nature of doctoral assessment through an individual thesis means that individuals also need to develop the group attributes to call on them when the group is not present. Thus, a limitation of this approach is that group benefits of collaborative social annotation at the doctoral level are insufficient unless the individual can use the experience to develop core skills further and make degree progress.

SCA has several other limitations. Kalir (2020) provides an example of discourse analysis but does not address doctoral students’ specific learning needs [2]. Likewise, a pilot project by Chen combined social and collaborative annotation (SCA) in a graduate course using Slack and Hypothes.is provide an implementation model relevant to course instruction [57]. However, the course took much work to develop and implement. Like Kalir, Chen did not focus on working with the primary research literature [25, 57]. These studies suggest an opportunity to investigate SCA for developing critical reading skills using the PL at the doctoral level, which this study did. Nonetheless, the body of research at the graduate level is limited and warrants further investigation, including combining SCA with direct instruction in other critical reading methods, such as the CERIC method [83] or the Toulmin method [84].

6.4.

Limitations of the study

The major limitations of this study were time-consuming data collection and subsequent time-intensive qualitative analysis. These limitations are known in qualitative and mixed-methods research [65]. Also, the qualitative arm study was not powered for more complex statistical analyses, such as ANOVA, and thus, generalizing the findings was not a goal [85].

Other limitations reflect the nature of a voluntary professional learning setting. These limitations include participants who volunteered because they were highly motivated to learn to read in a new field (i.e., science policy research) with attendant career goals. Thus, there were no participants with low motivation for reading. Future studies should include readers with more varied motivation levels. Also, participants were heterogeneous in terms of field and year of study, and it is possible that a more homogenous sample would produce different findings. For instance, a group of less experienced readers may have improved more or revealed essential features of becoming an expert reader not reported by a more advanced academic population. Finally, the study was not longitudinal by design, and thus, it needs to be clarified how participants will use SCA over time.