Measurement of Participation: The Role Checklist Version 3: Satisfaction and Performance

Abstract

Participation in society is an area of interest to both clinicians and population researchers. Measurement of participation is therefore important, yet differences in definition, in terms of both content and scope, have made general agreement on one instrument tool elusive. What is recognized is the need for a theoretically based tool that captures both the insider and the outsider perspective. The outsider perspective, inclusive of the generally held views of a society, supports the utility for aggregating population data, whereas the insider perspective provides the internally held views of an individual needed for client-centered treatment planning. The Role Checklist Version 3 modifies one of the most commonly used assessment tools in occupational therapy practice, has good preliminary psychometric properties, and is theoretically consistent with both the ICF and the Model of Human Occupation. The Model of Human Occupation is the most widely used theoretical model in occupational therapy. This chapter provides an overview of the theoretical development, empirical testing, and implications for use of this participation measure by occupational therapists along with implications for population researchers.

1. Introduction

Humans interact with each other in consistent and scripted ways. This interaction is known as participation and takes the form of roles. These roles have specific meaning both to the person performing: the insider, and to those around them: the outsiders. A person’s identity and sense of competency are tied inextricably to this role participation. Role participation is both persistent and changes over time, as people go through both planned and unplanned life transitions. Clinically, a person’s role participation becomes the focus of the occupational therapist when disability-related limitations affect a person's capacity to participate in desired and meaningful roles.

Occupational therapists aim to assure that persons with disabilities have the motivation, opportunities, and capacity to overcome disability-related limitations and participate in social life. The International Classification of Health, Disability and Function (ICF) seeks to establish uniform definitions worldwide [1]. The American Occupational Therapy Association includes the ICF definition of participation in their practice framework [2] (see Box 1). The ICF is a classification system that “conceptualizes a person’s level of functioning as a dynamic interaction between her or his health conditions, environment, and personal factors” [1]. Haglund et al. [3] found that the ICF classification aides occupational therapists in their practice; however, it alone does not encompass all that is needed for good practice. Therefore, it was found to be important to discover a tool consistent with the ICF that also is grounded in the theory of occupational therapy.

BOX 1

“Achieving health, well-being, and participation in

life through engagement in occupation is the overarching

statement that describes the domain and process of occupational

therapy in its fullest sense.”

“Participation—“involvement in a life situation” (WHO, 2001, p. 10). Participation naturally occurs when clients are actively involved in carrying out occupations or daily life activities they find purposeful and meaningful. More specific outcomes of occupational therapy intervention are multidimensional and support the end result of participation.”

AOTA Occupational Therapy Framework(March/April 2014, Volume 68 Supplement 1, p. S4)

The ICF defines participation as “involvement in life situation” different than it defines activities which are “the execution of a task or action by an individual” [1] (p. 10). Despite the difference in definition, the ICF places both activities and participation together in one chapter and specifies four ways for ICF users to distinguish between the two. This ambiguity has resulted in an entire thread of literature as rehabilitation researchers seek to identify ways to approach measurement of participation. For if there is no agreement around the definition and scope, how can there be agreement on how to measure? This difficulty does not, however, make the task any less important.

The Model of Human Occupation (MOHO) is the theoretical approach used most commonly commonly worldwide [4–6]. Developed by Kielhofner [7] and colleagues, Kielhofner’s vision for MOHO is to support practice that is occupation focused, client centered, holistic, evidence based, and complementary to practice based on other occupational therapy models and interdisciplinary theories [8]. In this chapter, therefore we use the approach of differentiating between activities and participation by using the theory of MOHO to provide a framework that explains how participation in occupation is achieved.

The Role Checklist, theoretically grounded in MOHO, was developed by Frances Oakley in 1981 to capture occupational performance by measuring role incumbency and value [8]. However, over time the concepts of MOHO have evolved. In 2002, Kielhofner introduced the dimensions of doing [7]. Scott responded to this new concept by revisiting the Role Checklist, and began the process of revising the Role Checklist (see Figure 1). The latest MOHO text, the 5th edition, places the Role Checklist Version 3: Participation and Satisfaction (RCv3) among measures of occupational participation [8]. This chapter describes the process of revision and establishment of the psychometric properties needed to prepare the RCv3 as a cross-culturally valid measure of participation.

Figure 1.

Underlying concepts and their influence on Role Checklist revisions.

1.1. Role Checklist revisions: history and timelines

The Role Checklist, developed in 1981, is a short, two-part assessment tool that captures a person’s perceived incumbency and role value in relation to the following 10 roles: student, worker, volunteer, caregiver, home maintainer, friend, family member, religious participant, hobbyist/amateur, and participant in organizations. Each role is provided with a brief definition followed by examples. The definitions contained a criterion of at least weekly involvement as occupational therapists who employ MOHO are interested in how these roles do or do not structure the respondent’s occupational participation. For each of these 10 roles, Part 1 of the assessments asks respondents to indicate whether they have previously held the role, are currently in the role, and/or expect to be in the role in the future. More specifically, Part 1 is assessing perceived incumbency, defined as the respondent’s belief that he or she occupies a role [9]. Once Part 1 is completed, Part 2 asks respondents to indicate how much they value the role. To determine role value, the degree of importance the role has to the respondent, he or she must rank each of the 10 roles as “very valuable,” “somewhat valuable,” or “not at all valuable.” The Role Checklist is available now in 20 languages [10], and available at http://www.cade.uic.edu/moho.

1.1.1. Role Checklist Version 2: quality of performance

The Role Checklist, as it was created in 1981, was developed around current MOHO concepts of the time. However, as MOHO has articulated the dimensions of doing, revisions were needed to respond to these changes. In a first step, Patricia Scott, in coordination with Frances Oakley, responded to both clinical experience with the instrument and the established standards in the field of measurement of participation [11]. The result was the establishment of the Role Checklist Version 2: quality of performance (RCv2:QP). This revision retains the same 10 roles as the original Role Checklist and prompts respondents to rank their satisfaction with quality of performance on a scale from “very dissatisfied” to “very satisfied.” In a 2014 study, Scott and colleagues found the RCv2:QP to have high levels of test-retest reliability and consistency between paper and electronic administration [12]. As described above, Scott added a Part 3 to the Role Checklist to enable respondents to rate their perception of the quality of their performance. This addition made the Role Checklist more sensitive to change and added a component to enable the “insider view,” an important aspect of the person’s self-assessment of adequacy or acceptability. Part 3 brought the Role Checklist closer to meeting the criteria for measures of participation [11].

1.1.2. Role Checklist Version 3: performance and satisfaction

While creating a scoring system, Scott recognized a need for a reflection of desired participation. Scoring had been elusive in the past, in great part due to the lack of accepted standards for patterns of role participation. With the client at the focal center, and role incumbency being an internalized concept, occupational therapists do not endorse the adoption of a universal standard for desired roles. The focus on individualizing contextual performance is based on current role satisfaction and the desirability of delaying engagement (or not) in desired future roles. This effort led to the reconceptualization of Part 3 and synthesis into a one-page document. This version is named Role Checklist Version 3: Satisfaction and Performance.

This one-page assessment tool is written as follows. Each role is first defined; the definitions have remained the same for the roles of student, worker, caregiver, and home maintainer. However, the time specifications for volunteer, friend, family member, hobbyist, and participant in organizations have changed from a weekly basis to “on a regular basis.” Lastly, to identify as a religious participant, involvement is no longer required to be “at least once a week,” and instead has no time frame. To complete the assessment, for each role, the respondent is asked to indicate if he or she is currently performing the role. If the respondent indicates “Yes” to currently performing the given role, he or she then is prompted to indicate his or her level of satisfaction with their role performance. If the respondent indicates “No” to currently performing the given role, he or she is then asked to indicate his or her interest in role participation in the future by selecting “I would like to do this NOW,” “I would like to do this IN THE FUTURE,” or “I am NOT INTERESTED in doing this.”

This new, revised assessment tool is no longer concerned with past role incumbency or role value. Instead, the RCv3 elicits information that allows clinicians to provide a client-centered plan of care and monitor client progress, and researches a measure to collect outcome data on an individual or population level.

1.2. Connection to international audiences

In a 2014 publication, Scott [13] presented the case for the Role Checklist Version 2: Quality of Performance as valid measure consistent with the ICF definition of participation. Her work drew the attention of International colleagues who joined Scott and formed the International Role Alliance for the study of Participation (IRAP) in October 2013.

The International Role Alliance for the Study of Participation (IRAP) maintains the mission to promote participation in society for all persons with disabling conditions. Specifically, this group seeks to establish the revised Role Checklist as a cross-culturally valid method of measuring participation, actual participation, and desired future participation, contextualized by the value and satisfaction a person associates with that participation. IRAP core members are academic clinical scholars from universities in Switzerland (German), Sweden, Japan, USA, UK, and Norway. Each has completed translations and contributed data to establish the initial psychometric properties of the Role Checklist Version 2: Quality of Performance. Initial investigations of the utility of the tool and its clinical applicability took place in Sweden and in Norway.

Among one of the first agendas of this group was to establish valid cross-cultural guidelines for translation. These guidelines would, to quote Dr. Lena Haglund from Sweden, a founding IRAP member, “place a fence around the translation process to keep the MOHO concepts in.” These guidelines have been tested for feasibility and implementation. Collaborators from Iceland and China have completed valid translations of the RCv3, and Spanish and Norwegian translations are in progress. Now the translation guidelines are shown to produce culturally equivalent versions, more than a dozen collaborators are on board to complete further translations.

IRAP members work through a worldwide network of occupational therapists who provide services to persons with disabilities across the globe. This network is enabled on two fronts: first, the MOHO Clearing House, which is the source for a dozen measurements, instruments in 20 languages and second, the World Federation of Occupational Therapy with 77 member organizations, which represent over 350,000 occupational therapists internationally.

2. Concepts of concern: ICF and MOHO levels of doing

In this section, it is useful to better understand the differences between activities and participation or, as understood in MOHO, occupational participation and occupational performance [3]. In Section 1, we offered the definitions from the ICF of activities and participation. They are worth repeating here:

Participation: “involvement in life situation”

Activities which is “the execution of a task or action by an individual”

(WHO [1], p. 10)

Occupational therapists are inherently attuned to and concerned with the things people do and how they do them. Doing is described in MOHO at three levels: occupational skill, occupational performance, and occupational participation. Kielhofner [7] refers to this hierarchy as the “dimensions of doing.” Occupational skill can be simply described as purposeful actions needed to carry out a task. They are the motor skills, process skills, and communication and interaction skills that come together to make up occupational performance [14]. Occupational performance is the act of utilizing these skills to carry out a task. These acts of occupational performance comprise occupational participation; however, there is more to participation than performance alone [15]. Kielhofner [14] explains that participation is composed of occupations that are, “part of one's sociocultural context and that are desired and/or necessary to one's well-being.” They are linked to a sense of belonging, value, and meaning [15]. This is the dimension of doing in which roles reside. These levels of doing can be best understood through examples. The ability to push, grasp, and categorize would be considered occupational skill. Occupational performance would include such activities as vacuuming, scrubbing a kitchen counter, or sorting laundry. Partaking in and identifying with the role of home maintainer would be considered as occupational participation.

Occupational role participation is more than partaking in an activity [15]. Kielhofner [7] explained that it “refers to engaging in work, play, or activities of daily living that are part of one’s sociocultural context and that are desired and/or necessary to one’s well-being” (p. 101). Dijkers [15] notes that there are many aspects of participation, including, but not limited to frequency of activities, value and meaning, responsibility, autonomy, reciprocity, location, and the company of others. Driving a car is an activity performed by many throughout the course of a day; however, it is not always perceived the same way. A teenage boy, whom may not value his education and finds the daily commute to school quite cumbersome, may view it as nothing more than a daily task. However, later that day he may take that same route during his work hours as a pizza delivery boy. He finds the task of driving to be a necessary aspect of his valued occupation of work. Even further, the addition of friends as passengers on the way to a Friday night football game can be considered engagement in social participation. The line between participation and common performance lies within the individual's perception. Bonsaksen and colleagues analyzed over 7000 role examples and associated them with these levels of doing.

3. Reliability and validity psychometrics of RCv3

Although several studies have supported the psychometric properties of the Role Checklist, it has not yet achieved the level of psychometric evidence to be included among other MOHO assessments. Table 1 charts the history of studies of different versions of the Role Checklist. It is important to note that caution must be used in the assignment of psychometric properties found for one version to another. The only consistent feature across all three versions is the 10 roles. The Role Checklist, original and versions 2 and 3, have excellent validity and cross-cultural reproducibility; however, there is still a need for feasibility, utility, and reliability testing of the RCv3.

Properties verified	Title	Authors	Year	Comments
The Role Checklist
Reliability	The Role Checklist: Development and empirical assessment of reliability	Oakley, F., Kielhofner, G., Barris, R., & Reichler, R. K.	1986	Construct
Cross-cultural validity & reliability	Spanish Translation of the Role Checklist	Colón, H., & Haertlein, C.	2002	No application to RCv3
Cross-cultural validity & test-retest reliability	Cross-Cultural Reproducibility of the Brazilian Portuguese Version of the Role Checklist for Persons With Chronic Obstructive Pulmonary Disease	Cordeiro, J. R., Camelier, A., Oakley, F., & Jardim, J. R.	2007	No application to RCv3
The Role Checklist Version 2: Quality of Performance
Electronic administration & test-retest reliability	The Role Checklist Version 2: Quality of Performance: Reliability and validation of electronic administration	Scott PJ, McFadden, R, Yates, K, Baker, S. & McSoley, S.	2014	Supports electronic administration & test-test reliability
Feasibility	Using the Role Checklist Version 2: Quality of Performance in the Occupational Therapy Process in a Psychiatric Hospital	Aslaksen, M., Scott, P., Haglund, L., Ellingham, B., & Bonsaksen, T.	2014	Informs responsiveness and utility
Construct validity with ICF	Measuring participation outcomes following life-saving medical interventions: The Role Checklist Version 2: Quality of Performance	Scott, P.J.	2014	Same roles are used in all versions
Construct validity with MOHO levels of doing	Does the Role Checklist Measure Occupational Participation?	Bonsaksen, T., Meidert, U., Schuman, D., Kvarsnes, H., Haglund, L., Prior, S., Forsyth, K., Yamada, T. & Scott, P.J.	2015	Role examples reflect mixed classification into occupational performance and occupational participation
Concurrent validity	Establishing Concurrent Validity of the Role Checklist version 2 with the OCAIRS in Post Liver transplant patients	Scott, PJ, Cacich D., Fulk, M., Michel, K., & Whiffen, K	2016	Both assessments measure the construct of participation
Content validity	Measuring Participation According to the ICF With the Modified Role Checklist	Meidert, U., Bonsaksen, T., & Scott, P.	n.d.	97% of role examples classified as consistent with ICF construct of participation
Role Checklist Version 3: Performance and Satisfaction
Cross-cultural validity	Translation Guidelines for the revised Role Checklist	Van Antwerp, L., Haglund, L., Fenger, K., & Scott, P.	2016	Translation guidelines are feasible
Discriminate validity	Measurement of Participation: The Role Checklist Version 3.	Scott, P., & Latham, K.	n.d.	Discriminates between persons identifying disability-related limitations and those without.

Table 1.

Psychometric properties of the various versions of the Role Checklist.

3.1. Feasibility

In 2014, Aslasken and colleagues [16] completed a pilot study to verify subject feasibility and to illustrate how an occupational therapist used the translated RCv2: QP to direct a clinical intervention. Aslaksen reported on the feedback from four Swedish therapists and one case report from Norway. The therapists each reported that a revised tool would be helpful if it were (1) provided on a single page layout, (2) had opportunity for comments, and (3) gathered information for each role one at a time [16]. The respondent who took the RCv2: QP reported frustration with the definitions provided for each role. Despite feeling as though he identified with select roles, he did not qualify as a participant in such roles according to the provided definitions [16]. In creating the RCv3, Scott took note of these recommendations and included changes addressing several of these concerns. The new RCv3 presents on a single page and prompts answers for each role one at a time. In addition, the time specifications used to define volunteer, friend, family member, hobbyist, participant in organizations, and religious participant have changed. As Aslasken’s study did, this study aims to show subject feasibility, among other psychometric properties, through employing therapist and client thought on the updates when compared to the original Role Checklist, developed in 1981 by Fran Oakley.

3.2. Utility

Utility is being the degree to which the treatment outcome is positively influenced by an assessment, in this case, the RCv3. Hayes et al. [17] provide a functional approach to evaluating an assessment’s quality. The authors justify that treatment utility is not a matter of cost-benefit ratio, but instead it is the “demonstration of a particular type of benefit” ([17], p. 964). Nelson-Gray [18] provided examples of typical treatment utility questions: “(a) Does treatment selection that is based on a particular assessment result in a more successful client outcome? And (b) Does supplying outcome data to therapists result in a more successful outcome?” These are the questions Aslasken et al. [16] provides only a partial answer to—these two questions remain largely unanswered for the RCv3.

3.3. Test-retest reliability

Establishing reliability is crucial to substantiate an assessment. Kerlinger [19] supported this notion when stating, “concern for reliability comes from the necessity for dependability in measurement” (p. 442). There are three definitions of reliability [19]. The first is characterized by the question of accuracy and stability. Does this instrument truly measure the outcome measure it sets out to find? The second looks to inquire the instrument’s error of measurement. This refers to its precision; how far it is from “hitting the bullseye” [20]. The third focuses on the stability, dependability, and predictability through multiple administrations. The first definition, hitting the mark, is appropriate for the RCv3. The 10 roles listed in the Role Checklist, all versions, are mutually independent, that is, no scaling is possible. For example, being in the role of a home maintainer is not necessarily associated with being a friend, any more than being a volunteer is associated with being a religious participant. Therefore, it is not surprising that the test-retest reliability performed on the 1981 version is associated with that performed in 2014 on the RCv2:QP.

3.4. Flexible administration

Previous studies have found that assessments administered electronically are consistent with paper and pencil administrations [21]. Using electronic means to administer the assessment allows for flexibility versus being limited to paper and pencil format. Because technology is advancing and becoming more prevalent in our society, clients benefit from having the option of taking an assessment in an electronic format. The RCv3 can be administered as a self-report, completed on a variety of electronic devices such as a smartphone, a tablet, or a laptop, or as an interview with the answers being submitted by the therapist. Studies have shown that clients often feel more at ease when taking surveys online versus on paper due to feeling less concerned with societal norms and how others might perceive them especially when it comes to personal and sensitive topics such as role incumbency and satisfaction [21, 22]. Additionally, using electronic means to gather data eliminates human error when transferring data into a database for analysis and is time and cost-effective [22, 23].

Although using electronic means to implement an assessment has many benefits, there are some shortcomings as well. Based on a study conducted by Gwaltney et al. [21], there are two reasons why administering an assessment electronically may not be equivalent to paper and pencil administration. First, the assessment tool presents itself different on paper versus electronically. Aspects such as letter size, spacing, or how many items per screen can vary between the two and ultimately alter the way a client would respond. Second, some clients may not feel comfortable using a computer or other electronic devise such as an iPad or a smartphone. This is especially true for the older population in which paper and pencil are most familiar. For these reasons, it is important to have an administrator around to address confusion or questions the client might have, as well as aid clients who may have a cognitive impairment, low vision, physical challenges, or lack of experience using technology.

3.5. Summary

As seen in Table 1, the RCv3 has established validity and cross-cultural standards; however, there is currently an obvious need for feasibility, utility, and replication of reliability. Establishing test-retest reliability for the RCV3 will improve the psychometrics that assessment tools needed to be considered a valid standardized instrument for occupational therapists to confidently use in treatment planning, goal progression, and outcome measurement. Verifying the RCv3 as a psychometric tool with feasibility and treatment utility through therapist perceptions, as well as replicating reliability will validate revisions to the assessment tool.

4. Use as a population-based measure

Thus far, we have addressed the use of the RCv3 as a client-centered measure of participation theoretically grounded in MOHO. It also has implications for a population-based population measure of interest to policy makers, as well as health and disability scholars.

In the 2011 World Report on Disability, the World Health Organization claims that improvement within a person’s social participation may be made when the health-care professional addresses barriers, which hinder their everyday activities (WHO [24], p. 4). These barriers must be identified through reliable measures, which measure participation. The CRPD, Convention on the Rights of Person’s with Disabilities, specifies that there is a need and an obligation for development of assessments, which promote participation (WHO [24], p. 11). Recommendation 8 describes the need to develop methodologies for data collection on persons with disabilities, which are tested cross-culturally and applied consistently.

5. Chapter summary

There are currently no general appropriate assessments, which may be reliably scored among health-care professionals on a multidisciplinary team, are cross-culturally consistent, and are both cost-effective and efficient. We have presented the Role Checklist Version 3: Quality and Performance as a way to fill this void. In this chapter, we have defined participation as defined by international standards, the ICF, and theoretically according to the most widely used model in occupational therapy, MOHO. We have substantiated conceptual consistency with the ICF participation domains. In addition, cross-cultural translational guidelines have been developed and internationally established as both feasible and valid [25]. Therefore, the revised Role Checklist that meets criteria for a balanced measure of participation [26] is conceptually consistent with the ICF [13], has cross-culturally valid translation guidelines [25], construct validity as a MOHO-based measure of participation [27] and concurrent validity with an established measure of participation [12].