Differences in Perceived Benefit, Use, and Learner Satisfaction between Open Source LMS and Proprietary LMS

Currently, many institutions are using expensive proprietary learning management systems (LMSs). Open source LMSs have been introduced to offer affordable solutions; however, these solutions have not been fully embraced. The researchers undertook a study to examine the differences between these two types of LMSs. This study used a survey to collect data pertaining to perceived benefit, LMS use, and learner satisfaction. The survey sample comprised 608 information technology (IT) major undergraduates from two Malaysian universities. Two groups were set up based on the LMSs used, where the first group ( n = 290) and the second group ( n = 318) used the proprietary and open source solutions, respectively. Students were asked to fill out a questionnaire to elicit their opinions concerning constructs perceived benefit , use , and learner satisfaction , and data were analyzed using SPSS (ver. 19). Independent sample t-tests were performed, indicating that there were significant differences in the three constructs, which favored the open source solution. Correla‐ tional analysis showed that each construct correlated significantly with each other, suggesting that each contributes to the overall effectiveness of the system. These findings reinforce the imperative of deploying open source learning solutions that are not only affordable but also effective to support students’ needs for effective online learning.


Introduction
Today's educational realm is witnessing an ongoing tremendous transformation in the teaching and learning process as the result of the continual advancement of technology. For decades, learners were immersed in learning settings that were dictated by physical learning tools (which was naturally cumbersome), confined learning spaces, and the presence of the teacher. Then, this learning landscape changed to a better setting, with the introduction of the then technology -the monochrome television (TV). The teacher, on certain occasions, showed students the scheduled broadcast over the educational channel containing pre-recorded teaching topics relevant to the current learning objectives. In the same period, many public universities that offered distance-learning courses began to conduct such learning classes by broadcasting live lectures to students in their designated classes, which were geographically spread across the country. Even though this type of learning environment was not exactly perfect, many students managed to learn quite effectively with minimum cost. Later, this learning setting morphed into a revolutionized teaching and learning environment in response to the advancement of the personal computing platform together with the introduction of the Internet, especially the World Wide Web. Hence, the birth of electronic learning (e-learning) was inevitable, bringing in tremendous benefit to the educational, social, and economical spheres. From the educational perspective, students' independent learning becomes more intense with more online materials and contents being delivered over the Internet and Intranet. This intensification of independent learning has shifted the role of instructors -from being the teacher to the facilitator, especially in collaborative learning classes.
Given the enormous economic and educational potential of e-learning, many solutions have been introduced since the late 1990s. These solutions assume many different terms or names, such as course management system (CMS), learning content management system (LCMS), virtual learning environment (VLE), virtual learning system (VLS), learning portal, or elearning platform, which reflects the many flavors of their functionalities. Among these, LMS is the dominant term commonly used in the educational sphere that focuses on learners rather than learning contents. The literature is quite replete with many definitions of LMS. For example, an LMS is "[a] comprehensive, integrated software [application] that the development, delivery, assessment, and administration of courses in traditional face-to-face, blended, or online learning environments" [1]. In a similar tone, Ref. [2] defined an LMS as "... as a software application for the administration, documentation, tracking, reporting and delivery of e-learning education courses or training programs." Many learning management system (LMS) companies have entered the market to provide online learning solutions to many institutes of higher learning (IHLs). Invariably, these proprietary LMSs were, and still are, prohibitively expensive to other branches of educational sphere, such as public schools, colleges, and training institutions. The licenses of the LMSs are notoriously exorbitant, ranging from tens of thousands to hundreds of thousands (depending on the scale of users). In fact, the costing of LMS covers not only the cost of acquisition, but installation, customization, and maintenance costs as well. To highlight the impact of the preceding factors, the finding of a survey by eLearning Guild survey [3] involving 909 of its members serves as a guideline for any prospective organizations that decide to implement these learning solutions. Depending on the scale and needs of an organization, the cost of acquisition, installation, and customization can range from as low as $10,000 to more than $1 million. On top of this cost, the maintenance of such a system will incur additional cost, ranging from $10,000 to more than $250,000 annually. Clearly, the overall cost of running these learning management systems is quite staggering, especially for small organizations. Despite these cost constraints, many institutions still prefer to use proprietary LMSs because of several factors, such as ease of upgrades, security, downtime, and support, which are relatively better handled by proprietary systems [4].
In view of the high initial cost of implementation, many non-profit organizations, such as the open source software (OSS) community (which consists of dedicated individuals or teams) have begun developing their own version of LMS, with considerable degree of success. According to a white paper by Ref. [5], "[o]pen-source solutions are software for which the source code is provided under a license that permits users to access, change, and improve it." Likewise, Ref. [6] defined open source software solutions as "... computer solutions or applications that are developed, tested, updated, and distributed among the community members." The development of open source LMSs entails the utilization of open source platforms, such as PHP/MySQL, Java, Python, Ruby on Rails, or on open source content management systems (CMS), such as Joomla and Drupal [7]. In addition, open source LMSs, such as Moodle, Sakai, and Wordpress, are built on content management systems, such as Joomla and Drupal. Initially, open source systems were built for education, but now they have been adopted by both educational organizations and some companies as well [7].
From the initial outlay perspective, "several OSS systems can help mitigate the ever-increasing licensing fee of commercial providers" [8]. In fact, in certain functionalities, they may have surpassed certain performances of the proprietary LMSs. For example, better customization, intuitive navigation, "simple chat tool" [9] and "highly interactive" [10] are some of the features of the OSS systems that users found to be appealing. "Ample evidence can be gleaned from the relevant literature that supports the use of affordable OSS systems to help improve student leaning" [9,11,12]. Then again, the superiority of one system over the others may no longer hold true when the latter may have made further improvements, far exceeding the former. Nonetheless, there are bound to be intrinsic differences between OSS and licensed LMSs, which cover a range of features, functionalities, and characteristics. These differences in functionalities or features could make -depending on the background of a range of stakeholders, such as the end user (e.g., students and lecturers) and the system administrator -certain LMS systems more preferable compared to others. For example, the end user would naturally prefer an LMS system that is easy to use, while the system administrator would desire an LMS system that is easy to maintain. Irrespective of the types of LMSs, these learning solutions should be able to perform the following core functions for educational purposes as follows [2]: • Centralize and automate administrative functions • Use self-service and self-guided services • Personalize content and enable knowledge reuse Figure 1 shows a snapshot of the learning materials interface of an LMS system indicating available lecture and presentation notes to registered students of a particular course, serving as the third core function (i.e., Assemble and deliver learning content rapidly) of any LMS systems as mentioned above. Implementing LMSs for learning purposes that involves audiences consisting of students, teachers, and administrators would entail the following features [13]: • Registration and Enrollment options to teachers and students.
• Adding/Deleting Courses by the University/Educational Bodies.
• Setting the different User Roles and user account.
• Setting the course calendar.

• Uploading and Retrieving Assignment and Resources
• Forum module Figure 2 shows a snapshot of the group forum interface of an LMS system that can setup to facilitate discussion among a group of students involved in a project or an assignment. Through this online forum, students will be able to discuss their ongoing work without the usual constraints faced by face-to-face discussion, namely, time and place. At any time, at any place every member of the group can compose and post comments to collaborate on that work.

Figure 2. A snapshot of a group forum interface
In general, there are two main categories of learning management system, namely Education LMS and Corporate LMS [7]. The former primarily focuses on learners and learning facilities, launching and monitoring of online learning, and keeping record of learning activities. On the other hand, even though the latter shares similar functionalities as the former, corporate LMS is also equipped with e-Commerce capability, regulatory compliance, competency, performance, human capital, and talent management [7]. Claroline represent some of the latter systems. Undisputedly, deciding on which one of the two will rely on many aspects, such as user acceptance, technical support, maintenance, training, servicing, and cost of ownership, which will have an overall significant impact on the use of the system. For any technology implementations, the ultimate aim is to ensure that the intended users (e.g., students, executives, trainees, or personnel) will be able to improve their knowledge and skills after using such systems. More importantly, users must be made to realize that the systems are indispensable to the efforts to make them more competent. From the managerial perspective, it becomes the imperative of the instructors, teachers, or administrators to select and implement the right system in their organization lest the implementation will run into problems, which could be costly and damaging. To achieve a successful implementation of any LMS system will entail conditions that help users to appreciate the full potential of the solution. In other words, they should perceive the system to be highly beneficial to their training or leaning. Of course, there are interrelated factors that come into play in shaping the perceived usefulness or perceived benefit of such systems.

No. Commercial Education LMS Open source Education LMSs
To explain the factors and their relations, several researchers have formulated a few models such as Technology Acceptance Model (TAM) [14], Unified Theory of Acceptance and Use of Technology (UTAUT) [15], DeLone and McLean model (DL&ML) [16], and Educational Technology Model (ETM) [17]. Invariably, some of the newer models were formulated based on older models, thus some having the same underlying constructs, such as system quality, service quality, course quality, learner satisfaction, LMS use, and perceived benefit [18]. In this paper, the constructs that were examined were learner satisfaction, LMS use, and perceived benefit given that the remaining constructs mainly deal with the technical aspects of the systems. According to Ref. [19], user (learner) satisfaction, which measures learner's attitude toward the system, is " the extent to which users believe the information system available to them meets their information requirements." Thus, if the user perceives the system to be poor, the system is rendered inferior. In contrast, higher learner satisfaction of the system will lead to higher "intention to use," which in turn improves usage [16].
Based on these interrelations, satisfied learners will perceive the system to be beneficial to their learning and will most likely use the system more persistently. "The construct LMS use measures the extent to which learners use the LMS, which in effect serves as a barometer that shows the success (or failure) of such a system implementation" [16]. With frequent use of the system, learners will be more likely to improve their knowledge and skills -the positive impact of which will resonate throughout the organization. Accordingly, measuring the net benefit of the system entails the evaluation of the system along with the purpose of the system. "One of the practical ways to measure the perceived net benefit is through eliciting learners' perception on the benefit of the system" [18]. In unison, all these factors will have a serious impact of the selection and use of such a LMS. Furthermore, the use of such a system will also be influenced by several mediating, notably demographic factors, which need to be considered when implementing online learning for students.
As with other computer-based solutions, both proprietary and non-proprietary systems are readily available. Given the availability of both types of the systems, undertaking a comparative study of these two systems will not only be interesting but purposeful, as the lessons to be learned will help universities' administrators to make an informed decision on the final choice of a particular system type to be used in their organizations. Such a choice will have an overwhelming impact on the learning and teaching process in the long term. Thus, three research questions that focus on perceived benefit, LMS use, and learner satisfaction were formulated to guide the study as follows:

Research method
In this study, the researchers used a quantitative research method based on a survey to collect the required data from a group of students. Using this approach helped the researchers to test the preceding research hypotheses by employing relevant participants, research instruments, and procedure. The details of the research method are as follows.

Participants
The sample of the survey comprised a group of 608 undergraduates, who were majoring in information technology at two institutions of higher learning in Malaysia. In terms of gender composition, this sample consisted of 401 female undergraduates and 207 male undergraduates. Their mean age was 21.5 years, and, on average, they had been using the LMSs for more than 2 years. Their participation in this survey was based on voluntary basis.

Research instruments
The research instrument used in this study was mostly based on the questionnaire used by Ref. [18] to collect data pertaining to the constructs LMS use, learner satisfaction, and perceived benefit. There were 11 items in the questionnaire, which were split into three categories. The first category comprised four items to measure LMS use, the second category comprised three items to measure learner satisfaction, and the third category consisted of four items to measure perceived benefit. The participants were asked to state their opinions with regard to the three constructs along 5-Likert-type scales, ranging from "1" (strongly disagree) to "5" (strongly agree). Cronbach's alpha coefficients for construct reliability measurement of LMS use, learner satisfaction, and perceived benefit were 0.89, 0.88, and 0.91, respectively. These coefficients suggest that the internal consistency of the items are good, exceeding the acceptable value of 0.7 [20]. Table 2 summarizes the 11 items, constructs, and internal consistencies as reported in Ref. [18].

Procedure
Two groups of participants were formed based on their locations of study. The first group comprised 290 undergraduates (204 females, 86 males) of a private university, who used a proprietary (licensed) LMS. The second group comprised 318 undergraduates (197 females, 121 males) of a public university, who used use a non-proprietary (open source) LMS. The participation of the undergraduates was secured through personal contact of the researchers to allow them to send an online survey questionnaire to the students. This questionnaire also contained a brief introduction of the purpose of the survey and an assurance that their answers would remain confidential. Collected questionnaires were analyzed using Statistical Software for Social Science (Ver. 19.) The statistical procedures to address the research questions were a series of independent t-tests and Pearson correlation. The former procedure was used to examine any significant differences in the perceived constructs. The latter procedure was employed to examine the relations among the constructs. "Pearson correlation is commonly used in social science studies to examine the size and direction of the linear relationship between two continuous variables" [21].

Findings
Participants' responses to the questionnaire items were processed to produce the required descriptive statistics, namely, the mean scores, maximum scores, minimum scores, and standard deviations. The overall mean scores (standard deviations) of LMS use, learner satisfaction, and perceived benefit were 3.95 (.68), 3.97 (.69), and 3.78 (.65), respectively, as shown in Table 3.  Bivariate correlations between pairs of constructs were also computed using the Pearson correlation procedure. Perceived benefit and LMS use were significantly correlated, r(606) = 0.11, p < 0.05. Likewise, perceived benefit and learner satisfaction were also significantly correlated, r(606) = 0.12, p < 0.05. For constructs LMS use and learner satisfaction, their correlation was found to be strong and highly significant, r(606) = 0.95, p < 0.001. Table 4 summarizes the correlations among perceived benefit, LMS use, and learner satisfaction.

Discussion and conclusion
One of the major findings of the independent-samples t-test indicates that the participants who used the open source LMS rated the perceived benefit of their LMS significantly higher than their counterpart who used the proprietary LMS. In other words, the difference in perceived benefit between the two types of LMSs as reported by the participants was significant. Thus, this finding supports the first research hypothesis of the study. Similarly, the LMS use of the open source LMS was observed to differ significantly from the LMS use of the proprietary LMS, which lends support for the second research hypothesis of the study. Likewise, another finding of the independent-samples t-test indicates that the participants who used the open source LMS tended to rate Learner satisfaction significantly higher than those who used the proprietary LMS. There was a significant difference between the two groups in this measure, thus supporting the third research hypothesis of the study.
Given the support of all research hypotheses, there is growing evidence to suggest that learning management solutions developed by the open source community might have matured to a level that is on par with licensed solutions -or, as demonstrated in this case, the former might have surpassed the latter in terms of perceived benefit, LMS use, and learner satisfaction. Clearly, when these measures are perceived to be important by users, it can be inferred that the implementation of such a leaning management system is successful to a certain extent. Undisputedly, "there are numerous technical and socio-psychological factors" [22,23,24] "involving all the stakeholders that can determine the success (or failure) of LMSs" [25]. In this study, the perceived benefit (perceived usefulness), learner satisfaction, and LMS use of the open source LMS were highly rated. In addition, based on the correlational analysis, all the three factors were also significantly correlated with each other. More notably, the significant positive association between learner satisfaction and perceived benefit is consistent with earlier findings. This finding reinforces the contention that users will accept an LMS to be beneficial when they "are satisfied with the performance of such a system" [26]. This significant positive relation will in turn make users utilize the system more often and readily to support their learning process. Continued use of such systems will not only benefit students and instructors but administrators as well. In view of this revelation, it is important for both teaching staff and academic administrative personnel to institute several initiatives to highlight the benefits of LMS to their students. Through such initiatives, students will be able to recognize and appreciate the immense potential of an LMS. With the right frame of mind, students, especially the freshmen, will be primed to adapt to new, novel learning environment.
Interestingly, in this study, perceived benefit, LMS use, and learner satisfaction of an open source LMS were rated higher than the proprietary LMS. This finding suggests that solutions developed by the open source organizations or individuals have a promising future in providing online learning opportunities to a wide spectrum of users. Though the proprietary LMS entered the educational landscape much earlier than the open source LMS and dominated the online learning environment, the ongoing and relentless efforts by the open source community have made the latter system a formidable solution on par with the former system. Given its relatively inexpensive outlay, many institutions, especially cash-strapped schools, can now afford to implement open source, non-proprietary learning solutions. As demonstrated in this study, the open source LMS was rated high by students who used it to support their learning, especially outside the classroom. The high ratings of the perceived benefit, LMS use, and learner satisfaction strongly suggest that "features and functionalities of open source LMSs to support online independent learning have improved over the years to provide the essential teaching and learning needs" [27].
Despite the many advantages of open source LMSs, some factors may hinder successful implementation of such systems. Even though the acquisition of open source LMSs are almost without cost, a highly trained personnel (e.g., a system analyst) is required to manage the systems, which encompasses a wide range of technicalities. Without proper system management, the solution put in place will ultimately become inefficient and ineffective. Like any other database systems, LMSs require constant monitoring, updating, and maintenance over time as the number of users is growing. In addition, there is concern that some of the open source LMSs do not provide the level of security that is needed by certain educational institutions. Hence, important information could be comprised, which is detrimental to the organizations' reputation. Of course, there are other factors as well that can make an open source implementation challenging, but these two factors represent the major concern that can make such adoption either a success or a failure. Given these issues, adopting an open source solution entails appropriate planning that holistically focuses on all aspects pertaining to technical, training, and cost considerations.
In this study, even though the open source Education LMS was highly rated higher than the proprietary Education LMS in terms of the three constructs, the researchers strongly believe that the success of any LMS system -irrespective of being either proprietary or open sourcewill rely on technical, managerial, institutional, and attitudinal aspects. For example, at the institutional level, universities should not view LMS as a mere technological tool, but more importantly, this system should be implemented with careful, comprehensive planning. To plan such an implementation would entail a rigorous review of existing infrastructure, current teaching and learning practices, and policies. Failure to factor in these aspects in the planning process could make the system underutilized. The researchers opine that for any universities to embark on an LMS project, a dedicated team drawn from various and relevant background should be set up to plan all the necessary details to help guide the selection, installation, testing, and full deployment of a proper LMS system. Ideally, these team members should work in a unit, which may be called (as an example) an e-learning unit under the ambit of the academic affairs department.
This team should be given sufficient empowerment to study, formulate, and deploy strategies for effective LMS deployment. This team should examine existing infrastructure to help determine the capabilities of available hardware to support the proposed learning management system. Having the knowledge of the performance of this hardware would help IT personnel to carry out appropriate upgrading or retrofitting work to run the proposed learning management system. It is worthy to note that implementing learning management system is not only about the software per se, but the hardware to run the system is equally important. In other words, the importance of the symbiotic relation between software and hardware to operate such a learning management should not be downplayed. Thus, a thorough review of existing equipment and IT tools is not only important but also mandatory. Likewise, a review of existing teaching and learning practice of a university should be performed with utmost importance. After all, the main aim of deploying LMS is to improve the existing process of teaching and learning, thus this review would help identify weak spots or areas that require redress. Problems associated with teaching and learning in many universities worldwide have increased without respite given the ever-increasing number of students enrolling in diverse academic programs. Face-toface lectures have become a serious problem to many educational institutions as infrastructure is stretched to its limits, putting great constraint on it capability. Naturally, lectures have to be complemented by other means, notably online learning to cater vast student populations. Nowadays, many universities have adopted blended learning as a solution to handle the teaching needs and learning needs of lecturers and students, respectively.
In addition, the introduction of learning management systems in institutes of higher learning would entail a sound, appropriate policy to make their implementations a success. Ultimately, these systems would be deemed worthwhile and beneficial if all the stakeholders (students, lecturers, and administrators) could fully utilize these solutions. For example, a part of the policy may contain provisions to necessitate (or to enforce) all lecturers to use the LMS in the following teaching activities: a) uploading lecture notes on LMS, b) making online announcements, c) setting up online discussion groups, d) conducting online quizzes and tests, e) providing online feedback of students performance, and f) posting online information and news. With all these teaching activities performed on the learning management system, students would be more prone to partake in online learning activities to complement their face-to-face learning. Hence, the use of the system would be more intense, leading students to perceive the system to be highly beneficial. Sustaining this level of teaching and learning activities would in the end make the adoption of the LMS a success.
Overall, the findings of this study provide some assurances that open source education learning management systems are on par with their proprietary counterparts for the constructs learner satisfaction, LMS use, and perceived benefit. In fact, the latter system has been demonstrated to be rated higher than the former system for the three constructs. However, this finding is informative in terms of the continually increasing capability of open source education learning management systems, but not conclusive to stake claim that these open source systems are better than proprietary systems. Many factors are involved in making the adoption of learning management systems a success; thus, the interpretation of this finding should be embraced with caution as this study was based on students' opinions. Other stakeholders' opinions and feedback are needed to ascertain the performance of any learning management systems with some degree of certainty.
In summary, open source education learning management systems are beginning to be seriously viewed as an effective, efficient learning solution from the student perspective. Now, more learning opportunities will be made available to a greater pool of learners across the nation to help them pursue their academic programs in an environment that suits today's educational landscape -more precisely, digital landscape in which more and more contents and materials are in electronic form. Therefore, it is incumbent on the management of universities, training centers, and institutions that are currently using expensive learning solutions to seek affordable solutions, which are equally effective, to further enhance their students' online learning experiences. More importantly, schools, especially the public ones, which do not have such systems in place, should try to learn from others that have successfully implemented the open source education learning management systems so that their pupils can experience online learning at the early age.