A Review of Research on Gamification Approach in Education

Abstract

Gamification has become the most popular topic of the last few years. Studies in gamification area are examined based on certain different criteria in this study and content analysis method was used in order to identify trends in this area. Web of Science were scanned through using gamification as keyword without year restriction. A total number of 313 studies were regarded as appropriate for the aim of the study and examined. It is seen that research in this area have begun in 2011 and increased every year. It is also seen that motivational theories are mostly preferred in the studies conducted in gamification area. It was determined that goal-duty, reward and progression sticks are the mostly used components as game components. It is seen that gamification applications are frequently preferred in virtual environment, simulation and augmented reality learning environments after mobile environments and in parallel with these, they are also preferred in learning areas such as public, service, food and health. Therefore, identifying different activities which could affect success in online environments, integrating these into education environment and provide these activities with theories appropriate for students’ ages for them not to lose their motivation are essential.

1. Introduction

It is known that people have a tendency to play game [1]. It is essential to prepare enjoyable learning environments for people who are in need of new search for playing game and having fun. Gamification concept which appeals to users from every age has arised based on the idea of integrating structure of the games into education [2]. Although concept of game is too old, concept of gamification is a new concept. Gamification has become the most popular topic of the last few years [3, 4, 5].

Gamification approach which emerges as the new face of education in the information age is defined as the process of including users into game [6]. According to Bunchball, gamification refers to make the components and mechanics of game more enjoyable through non-play activities in order to generate behavior change among individuals [7]. While gamification approach is an appropriate method for obtaining positive outcomes in education and it leads students to adapt new behaviors through motivating them, making them to study more or learn new things; it also aims to keep people’s motivation high and facilitate attachment and certain behaviors [8]. Level completion, awarding, specialization, professional development which lead to addiction among individuals playing game are used for improvement in education. Therefore, using the system in games for instructional aims would both make instruction more enjoyable and allow individuals to change their behaviors [9].

It is necessary to look at the statistical data to understand why the concept of playing spreads so fast. The market size of the global gaming industry is 99.6 billion dollars by 2016. When the game revenues at the country level are examined by 2016, it is seen that China ranked first with 24.2 billion dollars, followed by the USA with 23.4 billion dollars and Japan with 12.4 billion dollars. Total Western Europe (Germany, France, England, Spain, Italy) totaling $ 17.3 billion in game revenues. The top five countries in the industry dominate about 70% of total gaming revenues. There is an annual income of $ 685 million to 16th in Turkey [10]. In Turkey there are more than 20 thousand internet cafe and is visited by 7.5 million active players each month here [11].

According to Game Designers, Developers, Producer and Publishers (OYUNDER), the average age of gamers in our country is 31 and there are more than 25 million active gamers. Every day, the average number of people playing on a daily basis is more than 10 million, and the average playing time is increasing all over the world [12]. about 30 million people in Turkey are active on the computer, playing digital games in the mobile phone or game console. Experts, an annual turnover of 600 million dollars in the sector in Turkey, said that the turnover of the world reached $ 100 billion [13].

Turkey Games Market Report 2016; Turkey 14.5 million young people (above the EU average) and 46 million online user than 80 million people is a developing country with a young population and it is one of the most valuable gaming market potential in EMEA [14]. In a study conducted in the United States, the average age of those playing in the last 12 years is 37 years. This rate shows that excitement about playing games on the computer is at an advanced level. In addition, 61% of senior executives who participated in the study stated that they were playing during working hours. Lastly, an international survey shows that the ratio of the total population of regular electronic game players is 66% in Germany, 57% in Mexico, 53% in Russia and 52% in England [15].

These figures and studies show us that people are conscious and willing to play. In fact, this enthusiasm is growing day by day and the new generation is a full-fledged player profile. From this point of view, the excitement and motivation resources that people show to play games can be put into life. These motivators, dynamics and processes can be used to generate behavioral changes in order to increase productivity in real life [16].

In recent years, it has been shown as a potential mechanism to enhance participatory participation through online work and practices that play games [17, 18, 19]. Firstly; it is done with points, graphics, level, competition etc. features. By adding features to otherwise ordinary tasks, it can create a more enjoyable and engaging experience for the user [20, 21, 22]. By using the features of the games, it is possible to make the testing experience less burdensome, thus reducing wear. In previous surveys, participants’ personal report surveys have found that playing experiences are usually more fun than their non-play counterparts [23, 24, 25, 26, 27].

In the cases where the method of play is preferred in education, game elements such as rewards, points, rosettes come first in the design phase. Therefore, there are arguments that playing is a system that is rewarded only for learning, and that this is a negative effect on the motivation of students who are engaged in such a process [28]. Two systematic investigations have recently been conducted, with gaming playing an impact on ‘online programs’ (mostly e-learning) [29] and web-based mental health interventions. Looyestyn and his colleagues have found that playing games on objective measures of events such as the use of the program, number of visits to the website and contributions is a big influence. In contrast, Brown and colleagues assessed the effect of gaming on loyalty to online mental health intervention, and they have found that most games play only one game item, rather than just light application of games, but have little effect on effectiveness [30].

According to Yilmaz and O’Connor, gamification studies aim to carry out human-human or human-computer interaction actively [31]. Accordingly, it is aimed to determine what to award at first, what kind of behavior is expected from individuals and more importantly what kind of options should be offered to maintain the system in a healthy way.

According to Gartner Hype cycle, although gamification has become popular in recent years and it has begun to be used for educational purposes, it shows a rapid growth in order to meet the requirements in this context [32]. When topics related with gamification keywords are examined, it is seen that search on examples related with gamification and gamification education are at a high level [33]. When trends in gamification concept are compared based on regions, it was figured out that Singapore and South Africa were emerged as the first countries [33]. Gamification is commonly used in these countries since it support occupational training toward adults. It is known that gamification method is used by institutional organizations in order to increase satisfaction and efficacy among workers and increase product and service quality in the organization [9].

When the literature is examined, it seen that gamification is used in many different areas including marketing [34, 35], health [3, 36], sustainability, journalism, entertainment [3] and education [3, 35, 36, 37, 38]. Gamification is taken really seriously in countries which frequently use technology. Professor Kevin Werbach from Pennsylvania University gives online education on gamification. This is lasting 10 weeks and it can be followed with Turkish subtitles. Certificate is also given at the end of education [39]. There are organizations in Turkey which realized the effect of gamification approach as well. The first gamification attempt named as “Gameatwork” was successful with the prepared web site. Gamification will be more prevalent when it is used in different areas and there will be various attempts when organizations continue to realize its effectiveness [40].

On the other hand, in education there are already processes like playground; students gain points when they achieve the desired learning goal, scores that they earn become grades, and at the end of the academic term they pass to the next level [28, 38]. Playing will allow students to have fun in the learning process [41, 42] or by giving the user an advantage in time management [43] motivation affects the positive direction. Although Samur is a new and popular method with a similar approach, when a limited number of studies conducted in the field of education are examined, it is generally stated that positive results are obtained in the processes involved in this method [44]. With a similar approach Buckley and Doyle are evaluating the fact that it provides individuals with the opportunity to experiment, make mistakes, gain experience and make sure that failure is not an end and that the individual can achieve his goal [28].

According to Landers and Callan, gamification application for the education of students or individuals who wants to develop themselves has not yet become prevalent enough [45]. Findings in the literature also support this notion. It is seen that there are limited number of studies examining achievement, motivation and views of students in gamification approach. Examining studies related with gamification approach and identifying trends in these studies constitute the problem of this study. Therefore, this study is expected to contribute to researchers for future research.

Main aim of this study is to examine the studies related with gamification in Web of Science database and identify trends in the area of gamification based on these studies.

2. Method

Content analysis method was used in this study to identify the trends in gamification research. According to Yildirim and Simsek, content analysis is conceptualizing the collected data at first, then organizing the concepts through using themes and interpretting themes [46]. Content analysis is commonly used with frequency analysis technic through digitizing the data. Content analysis is a scientific approach that enables to examine verbal, written and other materials in an objective and systematic way [47]. According to Cohen et al., content analysis is also described as the process of summarizing and describing the basic content of written information and the messages it contains [48]. Content analysis is a reusable, objective and systematic technique in which some words or chapters of a text are summarized under categories, depending on the rules [49].

This review was undertaken and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (S1 Appendix) and Meta-analysis of Observational Studies in Epidemiology (MOOSE) guidelines for observational studies (S2 Appendix) [50]. In analyzing the research included in the study, PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyzes), which is both a critical and quality-guided guide, was used in combining the results of the evidence-based research. Coded information;

• Distribution of Studies based on Years

• Distribution of Publications based on Number of Authors

• Distribution of Studies Based on Type of Publication

• Distribution of Studies Based on Paradigms

• Distribution of the Studies Based on Research Sample

• Distribution of the Studies Based on the Environments

• Distribution of the Studies Based on Theory/Model/Strategy

• Distribution of Game Components, Dynamics and Mechanics

• Distribution Based on Learning Area

It was decided that the researcher should read all the researchers found in the result of the research. Data collected in the research were first evaluated in MS Excel by entering into meta-analysis programs. Gamification keyword was scanned in full-texts in Web of Science database without year restriction and 313 studies were obtained by November 2015. Web of Science Categories; Computer Science Theory Methods (n = 101), Education Educational Research (n = 83), Engineering Electrical Electronic (n = 82), Computer Science Information Systems (n = 79), Computer Science Interdisciplinary Applications (n = 48). Letter to editor, book reviews and meeting abstract were not included in the research (papers excluded n = 20). Since the number of studies on gamification in Web of Science database increase day by day, the present study included studies published before December 2015 and therefore, 313 studies in total were examined. The flow diagram depicts the flow of information through an up-down approach throughout the different stages of the process. The system is treated first in general terms. Then, the processes in the internal structure of the system shape up to the relationship between the number of defined, included and excluded entries and the reasons for exclusion (refer to S1 Appendix) [50]. Analysis of studies was carried out based on the common views of researchers in order to ensure reliability and validity. In digitization process of the data, if a study included two or more dimensions, frequencies were calculated through covering every dimension separately. Data were interpreted after constituting tables.

Cohen’s kappa statistical technic was used to calculate the relationship between category classification carried out by researchers and high level of inter-rater reliability was obtained (.96). Distribution of the articles based on years is provided in Graph 1.

Graph 1.

Distribution of studies based on years.

As it can be seen in Graph 1, research in gamification area have begun in 2011 and increased through the years. Therefore, it is not possible to observe academic studies on gamification before 2011. Eighty-two studies were obtained in 2015 since studies including December 2015 might not be added into the system.

Number of authors in the studies is demonstrated in Table 1.

Author	n
3 authors	125
2 authors	79
1 author	71
4 authors	18
5 authors and above	20
Total	313

Table 1.

Distribution of publications based on number of authors.

As it can be seen in Table 1, articles have been mostly written by 3 authors (n = 125), 2 authors (n = 76) and 1 author (n = 71). It is seen that number of studies conducted by 4 or 5 authors are lower. Distribution of studies on gamification based on countries is provided in Table 2. Countries with less than 6 publications are not included in Table 2. All of them were shown under “other” category.

Country	n	Country	n
USA	51	Portugal	9
Spain	29	Japan	9
Germany	25	Finland	9
England	22	Romania	7
Korea	18	Austria	6
Australia	15	Holland	6
Canada	13	Greece	6
Brazil	11	Other	77
Total			313

Table 2.

Distribution of studies based on countries.

When studies in Web of Science are examined based on countries, it is seen that there are 39 different countries in which the studies were carried out. It was figured out that 51 of the studies were conducted in USA, 29 of them in Spain, 25 of them in Germany, 22 of them in England, 18 of them in Korea, 15 of them in Australia, 13 of them in Canada and 11 of them in Brazil.

It was determined that gamification is actively used in Australia, USA, India, Canada and Holland in 2013. Brazil and France followed this rapid development [51]. According to the results of Google Trends (2016) since January, it is seen that Singapore, South Africa, Holland, Denmark, Australia, Sweden and India are interested in gamification approach. This shows that studies might change based on countries throughout the years [33].

Distribution of studies based on type of publication is provided in Table 3.

Type of publication	n	%
Proceeding	164	52.4
Article	149	47.61
Total	313	100

Table 3.

Distribution of studies based on type of publication.

According to Table 3, number of proceedings (n = 164) presented in conferences with a rate of 52.4% is really high. Researchers explained that gamification concept has been the discussion topic of conferences since 2010 and this might be the reason for this result [52]. Karatas examined studies covering gamification approach for education and indicated that there are few number of master and doctorate thesis since this area is newly recognized [53] .

Distribution of studies based on paradigms is shown in Graph 2.

Graph 2.

Distribution of studies based on paradigms.

According to Graph 2, 163 studies used quantitative method. Studies using mixed methods are the second with 59 number of studies. Since it is a trend topic, it is seen that there are 28 review studies. In pedagogical terms, most of the researchers examined gamification approach especially on ensuring integration [54, 55]. This might be the reason for high number of review studies. Other studies also discussed environments in which gamification system could be integrated into a certain learning [56, 57, 58, 59].

3. Results

In this section, results related with studies on gamification area examined in line with the aims of the present research are provided with tables.

3.1. Distribution of the studies based on research sample

Distribution of the studies based on research sample is provided in Table 4.

Research sample	n
Adults	152
Students	95
Voluntary participants	21
Children	15
Customers	5
Writers/players	4
Patients	4
Civil servants	4
Library users	3
Teachers	3
Sportsmen	3
Twitter community	2
Managers	2
Total	313

Table 4.

Distribution of the studies based on research sample.

According to the results provided in Table 4, it is seen that studies related with gamification have mostly been conducted with adults. This result might be related with the fact that playing games is prevalent among adult population. Following the category of adults, students were the second category of research sample frequently used in the studies related with gamification. This result is consistent with other studies including undergraduate students as participants [60, 61]. Researchers need to have technical knowledge and knowledge on software and hardware for gamification and this might be the reason for including university students in the studies [53]. It is seen that voluntary participants (n = 21), customers (n = 5), writers and players (n = 4), patients (n = 4) and civil cervants (n = 4) constitute the research samples of the studies as well. Schouten et al. examined the users’ experiences and motivations toward gamification among individuals with low levels of literacy [62]. This study showed that individuals with low educational level are also included in the studies on gamification approach as participants.

3.2. Distribution of the studies based on the environments

Distribution of the studies based on the environment in which the studies were carried out is provided in Table 5.

Environments	n
Mobile environment	126
Online environment	91
Internet-based environment	32
Social media platform	25
Virtual environment	11
Game-based system	8
Online and offline platform	4
Real environment	3
Sustainable environments	3
Advertisements	2
Other	8
Total	313

Table 5.

Distribution of the studies based on the environments.

It is seen that mobile environments are the most frequently used environments (n = 126). Following mobile environments, online environments (n = 91), internet-based environments (n = 91) and social media platform (n = 25) are the most frequently used environments.

Educations including Microsoft Ribbon Hero which aims to teach office program in a funny way, ClassDojo which is game-supported class management application that can be used by parents and students [63], Duolingo and Lingualeo which is used for foreign language teaching [64], Kahoot which is used in question-answer activities in class through mobile devices [65], web design with Khan Academy which provides free learning materials throughout the world and programming languages continue to provide education through mobile devices. Almost all of these applications allow online access. In addition, these environments are preferred since they might ensure positive learning outcomes through selecting appropriate social gamification tools for contents which could be provided in social learning platforms [8].

3.3. Distribution of the studies based on theory/model/strategy

In this study, approaches, theories or models used in the studies are also examined and shown in Table 6.

Theoretical approaches	n	%
Motivation theory	131	41.86
Game design factors	86	27.48
Unspecified	44	14.06
Main characteristics of game learning	27	8.63
Learning theory	9	2.88
Flow theory	8	2.56
Technology acceptance model	3	0.96
Mechanics, dynamics, esthetics	3	0.96
Alternative learning-model	1	0.32
Self-determination theories	1	0.32
Total	313	100

Table 6.

Theoretical distribution of the studies.

It is seen that motivational theories are the most frequently used approach in the studies on gamification (n = 131; 41.86%). Other studies also frequently included motivational theories [66, 67, 68]. It was also revealed that there are many studies emphasizing game design factors (n = 86; 27.48%). There are also studies in which their theoretical structures are unspecified (n = 44; 14.06%). This might be because of the fact that this is a recent area. It is determined that motivational theories are important indicators for internal and external motivations of students in a course about gamification system provided to engineering students in order to reinforce their entrepreneurship [69].

3.4. Distribution of game components, dynamics and mechanics

Distribution of game components, dynamics and mechanics is provided in Table 7.

Game components	n	%
Goal-duty	105	25.06
Reward	101	24.11
Progression stick	52	12.42
Cup	35	8.36
Feedback	32	7.64
Success	19	4.54
Badge	12	2.87
Virtual goods	9	2.15
Experience	9	2.15
Leader board	8	1.91
Promotion	6	1.44
Score	6	1.44
Quiz	5	1.2
Cooperation	5	1.2
Sustainability	4	0.96
Difficulty	3	0.72
Event detection	3	0.72
Fantasy and control	2	0.48
Entertainment	2	0.48
Richness of data	1	0.24
Total	419	100

Table 7.

Distribution of game components, dynamics and mechanics.

As it can be seen in Table 7, the most frequently used game components used in the studies are goal-duty (n = 105; 25.06%), reward (n = 101; 24.11%) and progression sticks (n = 52; 12.42%). O’Donovan investigated distribution of individuals motivated through gamification based on game components. Results showed that individuals were mostly motivated through progression sticks [55]. Sari and Altun figured out that badges and cups earned by students motivate them to earn more rewards. It is stated that although students compete with each other in order to earn these components, in fact, they compete with themselves [70].

When positive outcomes of using gamification in learning environments are considered, it has been concluded that integration of game components into these designs will positively affect learners’ motivation [71].

3.5. Distribution based on learning area

Learning areas in which the studies have been conducted are also examined in this study and results are provided in Table 8.

Learning areas	n	%
Mobile learning	91	29.08
Virtual environments/simulations/augmented reality	59	18.85
Public/service/food/transportation/health	56	17.9
Web-based gamification education	43	13.74
Computer games	17	5.44
Game design/creativity	15	4.8
Computer assisted cooperative learning	14	4.48
Digital environments/games	9	2.88
Social network	3	0.96
Sustainability	3	0.96
Educational sciences	3	0.96
Total	313	100

Table 8.

Distribution based on learning area.

According to the results, it is seen that learning areas are activated in mobile learning area (n = 91, 29.08%). These findings are expected since games can be used through mobile devices and adults prefer this area. Mobile learning which is used as a support for learning processes among contemporary methods arises as the first step of every learning approach today [72]. Results showed that there are varieties in terms of learning areas used in the studies. It is also seen that gamification applications are preferred in virtual environment, simulation and augmented reality learning areas (n = 59; 18.85%), public, service, food, transportation and health sector (n = 56; 17.9%). Different learning areas show that gamification concept could be used in all learning areas. Since gamification concept might differ from game to game, learning areas will also shape in every application.

4. Discussion and conclusion

Designers could view users’ experience in a wider context through gamification concept and constitute an effective learning environment with using motivational components in environments in which there is a competition, cooperation and social interaction [73]. Gamification will become more prevalent when it is begun to be used in different areas and there would be various attempts in this area when the potential of gamification is realized.

The present research is aimed to examine the studies conducted in the area of gamification. It was revealed that studies in the area of gamification have begun in 2011 and increased through the years. When studies in Web of Science database are examined based on countries, it is seen that studies have been conducted in 39 different countries. These countries were identified as USA, Spain, Germany, England and Korea. It was also indicated that gamification are actively used in Australia, USA, India, Canada and Holland. Brazil and France follow this rapid development [51]. When trends in gamification search in 2016 are examined, it was revealed that Singapore and South Africa are the first two countries [33]. This also shows that gamification approach would be a trend topic in different regions over time and different studies would be conducted in this area.

Furthermore, it was figured out that number of proceedings presented in conferences are higher when compared to other types of publication. Researchers indicated that gamification concept has been the discussion topic of conferences since 2010 and this might be the reason for this result [52]. It is expected that number of different types of publication for educational purposes would increase. It was revealed that there are 8 master thesis and 1 doctorate thesis in the area of gamification until January 2016 in YOK thesis center of Turkey [74]. It is considered that this rate will increase at the end of 2016.

According to the results, it is seen that quantitative studies are more preferred when compared to other type of research methods. Karatas indicated that there are insufficient number of quantitative and qualitative studies in the area of gamification [53]. It is expected that paradigms of the studies will become more salient. It is considered that studies examining game designs appropriate for gamification approach would be conducted more frequently in the future. In addition, it is expected that number of studies including achievement tests as quantitative studies will also increase.

When distribution based on research sample is examined, it is seen that studies related with gamification have mostly been conducted with adults. Since playing games is prevalent among adults, the accuracy of this result is accepted. Gokkaya revealed that since gamification supports learning at work as a practical solution for qualified personnel requirement, it is an ideal method for obtaining positive outcomes in especially adult education [9]. Following adults, students are also frequently included in the studies. This result is consistent with other studies including undergraduate students as participants [60, 61]. Diversity in research sample shows that gamification might be used in every age group and area.

Results showed that mobile environments are the most frequently used environments in the studies related with gamification. Gamification applications are supported by mobile devices and this might be the reason for these results. It is also seen that online, internet-based environments and social media platforms are also used in the studies. Educations including Microsoft Ribbon Hero which aims to teach office program in a funny way, ClassDojo which is game-supported class management application that can be used by parents and students [63], Duolingo and Lingualeo which is used for foreign language teaching [64], Kahoot which is used in question-answer activities in class through mobile devices [65], web design with Khan Academy which provides free learning materials throughout the world and programing languages continue to provide education through mobile devices. Almost all of these applications allow online access. These applications are commonly preferred since they allow users to share their status or scores in social media after activity. Therefore, it is essential to determine different activities which will affect success in online environments and integrate them into education environment.

It is seen that motivational theories are the most frequently used approach in the studies on gamification. Although it is known that motivational theories are mostly preferred, it must be also noted that gamification could include many different disciplines. Karatas indicated that future research would not only benefit from motivational theories, they might include different theories related with game, learning and behavioral sciences [53]. It can be said that gamification should be provided to students appropriate for their levels so they would not lose their motivations.

It is determined that the most frequently used game components in the studies are goal-duty, reward and progression sticks. O’Donovan investigated distribution of individuals motivated through gamification based on game components [55]. Results showed that individuals were mostly motivated through progression sticks. Reinforcing structure of game components might be the reason for this result. When positive outcomes of using gamification in learning environments are considered, it has been concluded that integration of game components into these designs would positively affect learners’ motivation [71]. When studies in the area of gamification increase, usage rates among game components would change. It is considered that different game components will be integrated into education based on learning environments and areas in future research. Applications might be prepared for lessons with advanced level of difficulty through providing rewards and scores.

Furthermore, results showed that learning area concentrates on mobile learning area. Results showed that there are varieties in terms of learning areas used in the studies. It is seen that gamification applications are also preferred in virtual environment, simulation and augmented reality learning areas, public, service, food, transportation and health sector. Although the number of studies are low, there are studies in learning areas such as science, food, interactive teaching, marketing, simulations, medicine and transportation. This shows that area of gamification is not limited to technology. Different learning areas show that gamification concept could be used in all learning areas. Karatas stated that gamification might be used in different learning areas. Since gamification concept might differ from game to game, learning areas would also be shaped in every application [53]. It is expected that appropriate designs for different learning areas would increase in future research.

This study will be a guide for researchers to integrate different game components into education environments, constitute independent learning areas and learning environments including different theories and conduct research in this area.

Supporting information

S1 Appendix. This is the completed PRISMA flow diagram.

S1 Appendix. Completed PRISMA Flow Diagram.

S2 Appendix. This is the completed PRISMA checklist.

S2 Appendix. Completed PRISMA checklist

Section/topic	#	Checklist item	Reported on page #
TITLE
Title	1	Identify the report as a systematic review, meta-analysis, or both.	1
ABSTRACT
Structured summary	2	Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number.	1
INTRODUCTION
Rationale	3	Describe the rationale for the review in the context of what is already known.	1, 2
Objectives	4	Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS).	2, 3
METHODS
Protocol and registration	5	Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number.	NA
Eligibility criteria	6	Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale.	3
Information sources	7	Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched.	3, 4
Search	8	Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.	3, 4
Study selection	9	State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis).	3, 4
Data collection process	10	Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators.	3, 4
Data items	11	List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made.	3, 4
Risk of bias in individual studies	12	Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis.	NR
Summary measures	13	State the principal summary measures (e.g., risk ratio, difference in means).	NR
Synthesis of results	14	Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I^2) for each meta-analysis.	4
Risk of bias across studies	15	Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).	4
Additional analyses	16	Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified.	4
RESULTS
Study selection	17	Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.	10, S1 Appendix
Study characteristics	18	For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations.	4, 5
Risk of bias within studies	19	Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12).	4, 5
Results of individual studies	20	For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot.	4, 5
Synthesis of results	21	Present results of each meta-analysis done, including confidence intervals and measures of consistency.	4, 5
Risk of bias across studies	22	Present results of any assessment of risk of bias across studies (see Item 15).	NR
Additional analysis	23	Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]).	5–8
DISCUSSION
Summary of evidence	24	Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers).	5–8
Limitations	25	Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete retrieval of identified research, reporting bias).	4, 10 S1 Appendix
Conclusions	26	Provide a general interpretation of the results in the context of other evidence, and implications for future research.	8–10
FUNDING
Funding	27	Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review.	NA

From: Moher et al. [50]. For more information, visit: www.prisma-statement.org. NR: not reported. NA: not applicable.