Applying a Usability Technique in the Open Source Software Development Process: Experiences from the Trenches

Abstract

The growth in the number of non-developer open source software (OSS) application users has drawn attention to usability in the OSS community. OSS communities do not generally know how to apply usability techniques and are unclear about which techniques to use in each activity of the development process. The aim of our research is to determine the feasibility of applying the focus groups technique in the OSS ERMaster project. To do this, we participated as project volunteers. We used the case study research method to investigate technique application and OSS community participation. As a result, we identified adverse conditions that were an obstacle to the application of the original technique. We then adapted the technique to make it applicable in an OSS project. We can conclude that was not easy to recruit OSS users and developers to participate in technique application.

1. Introduction

Open source software (OSS) has spread so swiftly that it now rivals commercial software systems in terms of deployment [1]. Some OSS communities nowadays do not have processes in place to guarantee that, taking into account the features of this community as a whole, the developed software is good [2]. Shortcomings with respect to process, activity, task and technique definition in the field of OSS development has led researchers from different fields to take an interest in this field of research and try to remedy the failings [3, 4, 5]. The growth in the number of non-developer OSS application users and the escalating use of these applications have created a need for and interest in developing usable OSS [6, 7, 8, 9, 10]. However, several authors have acknowledged that the usability of OSS is poor [6, 11, 12]. In this respect, the empirical study conducted by Raza et al. [7] reports that 60% of respondents (non-developer users) stated that poor usability is the main obstacle to be overcome by OSS applications if users are to migrate away from commercial software. On this ground, OSS projects must tackle their level of usability and usability-related problems more conscientiously [12].

On one hand, the human-computer interaction (HCI) field offers usability techniques whose key aim is to build usable software. However, they are applied as part of HCI methods and not within the OSS development process. On the other hand, the OSS development process focuses on source code and thus on feature development. The OSS development process has a number of characteristics (e.g., OSS community developers and users are geographically distributed, code-focused world view). This prevents many of the HCI usability techniques from being adopted directly [13]. Furthermore, these characteristics are not unique to OSS projects, they are also shared by projects carried out in distributed environments. However, our research’s interest is focused on the OSS development process.

Even so, the OSS community has now started to adopt some usability techniques. Most of the techniques that the OSS community has taken on board are for evaluating usability [13]. Some usability techniques have been adapted ad hoc for adoption in OSS development projects [13]. This paper addresses the research problem of how to adopt the focus groups usability technique for requirements engineering activities as part of the development process of a real OSS project known as ERMaster¹.

Our research spans two areas: SE and HCI. We use usability techniques as a bridge to communicate these two areas, where our aim is to deploy HCI knowledge in the SE field and especially in the OSS development process. If adapted, usability techniques can be adopted in the OSS development process [13]. Therefore, this paper has two goals. Firstly, we intend to adapt the focus groups usability technique [14] for adoption in the OSS development process. Secondly, we aim to determine the feasibility of adopting this usability technique in a real OSS project.

Requirements engineering activities play a very important role in the success or failure of an OSS project. However, they are sometimes extremely hard to perform because there is no definition of OSS user segments before the software is developed. Also, it is far from straightforward to address all the requirements analysis activities due to the particular characteristics of OSS development groups. On this ground, this paper considers just the product concept development activity. Additionally, OSS projects have not adopted many usability techniques related to the requirements engineering and product concept development activities [13]. The next step after selecting the activity is to pick a related usability technique for adoption in the OSS development process. Our choice consisted of the focus groups usability technique to be incorporated to the OSS development process. It is important to mention that this research is part of a study where we are applying usability techniques related to the activities of Requirements Engineering in OSS projects, so for the case at hand we report the incorporation of the focus groups technique. This technique has been used to identify functionalities the users need and to improve interaction with the tool. With this technique we are not evaluating the usability of the User Interface because it is not the aim of our research.

The main reasons for the generally poor usability of OSS developments are: OSS developers have tended to develop software for themselves [4, 10] and the developer community is very much in the dark about who its users are [9, 11]. The aim of the focus groups technique is to gather information related to user opinions, problems and concerns at meetings scheduled for this purpose [13, 14, 15]. This technique helps to focus product concept design on its hypothetical functionality [14]. The focus groups technique requires a small research sample for the purposes of product evaluation. Consequently, the participation of just a few users is sufficient to represent the product concept model, that is, developers use this technique to discover a user’s mental model of the product. On this ground, we selected the focus groups technique for adoption in an OSS project.

This paper makes a significant contribution to the field of SE and particularly to OSS development projects because we have not been able to identify papers reporting the use of the focus groups technique and detailing how it has been applied in OSS development projects [16, 17]. We used a case study as the research method to test the feasibility of our proposal for adopting usability techniques in OSS projects [18]. Consequently, we had to volunteer for the selected OSS project and join the community.

This paper is organised as follows. Section 2 outlines the state of the art. Section 3 illustrates the research method followed to apply the usability technique in an OSS project. Section 4 reports the proposed solution. Section 5 discusses the results. Section 6 reports the lessons learned. Finally, Section 7 outlines the conclusions and future research.

2. State of the art

There are papers in the literature reporting the usability evaluation of some OSS applications [19, 20, 21]. Assa et al. [19] studied the usability issues facing software developers using code analysers by evaluating one of the popular open-source static-code analysis tools. Al-Odan and Al-Daraiseh [20] conducted a thorough study, placing five of the most popular free and open source software tools side by side for comparison with respect to both user acceptance and technical specifications. Ternauciuc and Vasiu [21] tried to inventory existing methods for testing and improving usability, with a particular focus on e-learning platforms. However, usability technique definition and integration into OSS projects is a complicated process, which has not been researched at length [6, 22, 23, 24, 25]. Existing papers suggest that usability techniques should be reconceptualized. However, they do not explain how the OSS community should go about adaptation. Nichols and Twidale [4] and Ternauciuc and Vasiu [21] are the only authors to put forward some general ideas for improving usability. However, the issues to be taken into account in order to adopt such techniques in OSS developments are unclear.

Usability technique definition and integration into OSS projects is a complicated process, about which there are few papers [6, 23, 24, 25]. These papers suggest that usability techniques should be reconceptualized, but they do not explain how the OSS community should go about adaptation. Nichols and Twidale [4] are the only authors to put forward some general ideas for improving usability. However, the issues to be taken into account to adopt such techniques in OSS developments are unclear.

In particular, very few studies have reported the application of the focus groups technique in OSS projects [23, 26, 27] In the study by Terry et al. [23], the focus groups technique was adopted with adaptations in various OSS projects (for example, in a desktop windows environment and in a desktop operating system). For the focus groups technique a usability expert meets the developers either in person or through Internet Relay Chat. These meetings are held periodically (weekly, monthly or annually) and their aim is that the applications or the designs proposed for a new functionality are evaluated by an expert in usability [23]. In this case, the OSS developers are the ones that participate in the Focus Groups, rather than the final users. In Semedo and colleagues’ study [26], the participants replied to a questionnaire beforehand to assess their previous experience using the CLASS tool. This OSS application permits the recording, analysis and interpretation of respiratory sounds. The participants received a training session for the application. Two days later, they participated in a focus group session directed by the researchers to better understand their experience when interacting with the application. This focus group lasted approximately 20 minutes and was recorded on video. In Kolagani and colleagues’ study [27] the authors developed the Watershed GIS OSS for the management of geographical information. In this study the focus groups technique is adopted with the aim of obtaining the requirements for both types of users (experts and common) of the tool. In the last two studies [26, 27] it is stated that the final users participated in the Focus Group sessions conducted by the researchers themselves.

Although research examining usability in OSS has been published [9, 23, 28, 29], there is no standardised procedure for determining how to adopt usability in OSS development. It appears to be less straightforward to integrate usability into the OSS development process than into commercial development projects due to some of the characteristics of the OSS community, like: (i) feature-centred development, (ii) worldwide geographical distribution, (iii) limited resources, and (iv) a culture that may be alien to interaction designers. Consequently, usability technique adoption is a demanding task because most HCI techniques are not designed for the type of environment in which OSS is developed [13].

In the wake of the literature review, we can say that only one of the research papers reports a general and systematic proposal for integrating usability techniques into the OSS development process [13]. To do this, it considers the particular characteristics, philosophy and idiosyncrasy of the OSS development process, without forfeiting the essence of usability techniques. Two systematic mapping studies (SMS) related to usability in OSS were conducted in advance of our research. A SMS reviews the literature on a particular field of interest [29]. The first SMS was conducted by Castro [13] reviewing papers published up until 30 July 2013. The second SMS was conducted with a search range from 1 August 2013 to 30 April 2015 [30] and later updated considering the 30 July 2017 as the final date.

Castro’s work proposes an integration framework that can incorporate most usability techniques in OSS developments. It is important to clarify that this framework only proposes the general adaptations that must be made to the techniques. These adaptations depend on the requirements of the technique that cannot be satisfied by the way the OSS community works. Castro’s research [13] was validated on only two OSS projects (OpenOffice Writer and FreeMind) and for three usability techniques (user profiles, direct observation and post-test observation). Therefore, Castro’s proposal [13] requires further validation by adapting new usability techniques and participating in more OSS projects.

3. Research method

We used a case study as the qualitative research method to validate our research [31]. From a case study, we learn about the experiences of applying usability techniques adapted to OSS projects. This research method is used when the phenomenon under investigation (in this case, the adoption of an adapted usability technique) is studied within its real setting (in this case, an OSS project). OSS projects are the perfect setting for the case study reported here because OSS communities are generally uninformed about usability techniques, do not have the resources to test usability and cannot usually count on usability expert involvement [4, 9, 11]. Small project teams in particular have little information about what techniques are at their disposal for improving usability [6, 32].

The case study addresses the following research question (RQ): How to incorporate the focus groups technique in a real OSS project?

ERMaster, a graphical editing tool for entity-relation diagrams (ERD), was selected as the OSS project in which to adopt the focus groups technique. In this research, we first identified the obstacles to applying the focus groups technique in the ERMaster project. We then decided how to deal with the obstacles. Finally, we proposed the adaptations necessary to adopt the focus groups technique in this project.

We created web artefacts to improve communication with OSS community members and efficiently synchronise the necessary activities to apply the focus groups usability technique. The web artefact used to test the feasibility of the proposed technique was a forum. Forums are used in the focus groups technique to gather information and compile sketches related to the application user interface. Thanks to this web artefact, we were able to set up a virtual meeting point with OSS users who are geographically distributed all over the world.

4. Proposed solution

In this section, we describe the focus groups usability technique applied in an OSS project. Firstly, we describe the case study design. Secondly, we specify the characteristics of the selected OSS project. Thirdly, we describe the selected usability technique as prescribed by HCI. We then introduce the adaptations made to the focus groups technique for application in an OSS project. Finally, we report the results of applying this usability technique.

4.1. Case study design

Case studies are one of the most popular forms of qualitative empirical research [33]. A case study investigates the phenomenon of interest in its real-world context. The phenomenon of interest for this research is the adoption of usability techniques with adaptations, whereas the real-world context is OSS projects. It is not easy to run controlled experiments in the field of OSS because the characteristics of OSS communities (for example, age, availability, expertise, experience, etc.) are unmanageable. Since not all OSS project team members have the same characteristics, it is impossible to minimise the effects of external factors (for example, geographic distribution and time differences). This rules out evaluation by means of an experiment. On this ground, we selected the case study methodology to validate the feasibility of our proposal for adopting a usability technique in an OSS project. We describe the case study following the guidelines set out by Runeson and Host [18]. According to these guidelines, we divide our research into two parts: an exploratory part and a descriptive part. We start by looking at what happens in a real-world scenario and then we describe what happens when we apply the adapted techniques to improve application usability [18].

4.2. ERMaster OSS project characteristics

We selected ERMaster as the OSS project in which to adopt the focus groups technique. ERMaster is an Eclipse plug-in and is very useful for novice or expert database (DB) designers. There are several OSS project repositories. One of the most popular is Source-Forge.net. This repository classifies OSS projects by categories. Since this technique is related to requirements engineering for product concept development, we looked at projects with a low level of coding (that is, projects where key features were still being added) that were not overly ambitious and were at the very early development stages (alpha version) in order to select a suitable OSS project in which to adopt the selected usability technique. Considering the above, we selected the ERMaster OSS project. Thanks to the characteristics of this project, we can adopt a usability technique related to a requirements activity (product concept development). Therefore, the benefits of applying the technique will have a bigger impact on the development process and software system usability.

4.3. Focus groups usability technique description

The focus groups technique is a useful tool for evaluating user needs and feelings about a product expressed at group sessions [34]. More formalised definitions in the field of HCI describe the focus groups technique as a qualitative technique whose aim is to gather information about user opinions, problems and concerns at meetings planned for the purpose [13, 14, 15]. According to the literature, several authors [13, 15, 35] neither consider the planning required before and after applying the focus groups technique nor propose definite steps for applying the technique either. By contrast, Mayhew proposes a number of specific steps for applying this technique [14]. According to Mayhew [14], the focus groups technique is composed of the five steps described below.

Step 1 (Design the focus groups format) involves designing a script for the purpose of implementing a planned sequence of activities to be performed before, during and after conducting the focus group in order to achieve the goals set out in this study. Step 2 (Design data collection forms) involves designing a data input form (e.g., to note down the opinions, problems and comments raised by focus group participants). Additionally, a list of specific questions (related, for example, to the user interface and the work environment) has to be compiled and addressed and discussed by the focus group. Step 3 (Conduct the focus group) should take about 1–2 hours. According to Mayhew, a good sized focus group would have between six and eight members. Additionally, she believes that the moderator and note-taker play a very important role with respect to the key information stated by participants [14]. Steps 4 and 5 of the focus groups technique (Analyse data and Draw/document conclusions) address the transcription, analysis and summary of the results to draw and document the focus group conclusions.

4.4. Adaptations of the focus groups usability technique

The focus groups usability technique cannot be applied directly in the OSS development process because this community has features that do not conform to the HCI world, like, for example: the worldwide geographical distribution of its members, a code-centred world view, a shortage of resources and a culture that can be somewhat alien to interaction developers. Even though usability techniques demand conditions that, as a rule, OSS projects cannot meet, the techniques can be adapted to bring them into line with the idiosyncrasy of the OSS world. In the following, we describe the adaptations of the focus groups usability technique for application in OSS projects.

A usability expert is indispensable for applying Step 1 (Design focus group format) of the technique [14]. This expert is needed to structure the scripted objectives, topics and questions to be analysed when the focus group is conducted. We propose to substitute this expert with the principal developer, an experienced OSS project user or a HCI student under the supervision of a mentor to guide the focus groups format development. With regard to Step 2 (Design a data collection form), we found that the topics to be dealt with in the focus group cannot be physically handed out to participants because they are distributed all over the world. On this ground, we suggest that remote participation in the OSS community should be logged (online forum). Additionally, the outline of the topics should be posted on the same online forum so that users can recall their experiences with the software system interface under study.

In Step 3 (Conduct focus groups), we found that users are required to meet face to face to participate in technique application. Additionally, we found that a moderator and a note-taker had to be there in person to guide discussions and take notes during focus group application, respectively. This condition cannot be met due to the characteristics of OSS projects. On this ground, we suggest the following adaptations: (i) users will participate remotely in virtual meetings via the online forum; (ii) the moderator will be replaced by the principal developer, an expert OSS project user or a HCI student under the supervision of a mentor, (iii) there will be no note-taker during the conduct of the focus group because the online forum will be logged automatically.

In Step 4 (Analyse data), the information should be organised and then grouped by characteristics (such as age range, gender, occupation, etc.) before analysis. This simplifies the process of data analysis for the purpose of comparing and correctly interpreting the information gathered in the focus group [36]. Finally, Step 5 (Report conclusions) draws the conclusions with respect to the opinions expressed by users. We did not identify any adverse conditions for the last two steps, and therefore no adaptations had to be made. Table 1 summarises the steps, identified adverse conditions and suggested adaptations for the focus groups technique [14]. There are mainly two adaptations. First, users participate online via a forum. Secondly, the usability expert is replaced by a developer, expert user or a HCI student under the supervision of a mentor. In this particular case, a HCI student under the supervision of a mentor substituted the expert.

Steps	Adverse conditions	Proposed adaptations
1. Design the focus group script format.	Usability expert participation is required.	The expert may be a developer, an expert user or a HCI student (supervised by a mentor).
2. Design the data collection form.	Usability expert participation in the project is required. The list of topics to be discussed in the focus groups cannot be handed out as printed matter because the users do not attend in person.	The expert may be a developer, an expert user or a HCI student (supervised by a mentor). The outlines and the list of topics to be discussed are published on the online forum.
3. Conduct the focus group.	Users are required to participate in person to apply this technique. A moderator, which could be the principal developer or an experienced user, and a note-taker are required to attend in person. The number of participants is limited (4–9) The time taken to conduct a focus group is limited (1–4 hours). There is no assistant to take notes.	Users participate remotely via the online forum. The moderator may be an expert OSS project user or a HCI student (supervised by a mentor). The number of participants in the online forum is unlimited. The time available for submitting opinions to the online forum will depend on each participant. The log of each participant in the community is recorded in the online forum (i.e., the forum will play the role of a virtual assistant to take notes).
4. Analyse data.	No adverse conditions were identified.	N/A
5. Report conclusions.	(There are no adverse conditions. The adaptation is the result of the OSS community work method and is not a response to an adverse condition.)	The focus group conclusions and recommendations are published in forums and distributed by electronic mailing lists to the OSS community.

Table 1.

Steps, adverse conditions and proposed adaptations for the focus groups technique.

According to HCI prescriptions, design tips for a new application feature output by the focus groups technique are appraised by a usability expert [15]. In the adapted focus groups technique, the end users submitted their designs and opinions via web artefacts (like forums and emails) and not at face-to-face meetings due to the characteristics of the OSS communities. Table 2 presents the steps and tasks of the adapted focus groups technique that is proposed as a result of this study for its application to an OSS project.

Steps	Tasks
1. Design the format of the focus group Script.	Develop step by step the Focus Group script to apply it to the chosen OSS project.
2. Determine who will be the participating users of the focus group.	Ask the project administrator for permission to act with the community of real users that are working with the chosen OSS project.
3. Define the topics/questions for the focus group.	Decide what format of focus group to use when the technique is applied.
4. Design the data collection form.	Design a form on a spreadsheet to register data.
5. Conduct the focus group.	Conduct the focus group through the online forum
6. Analyse and interpret the data obtained during the focus group.	Register the results in the previously designed form.
7. Produce a report on the conclusions and recommendations.	Produce a report with the conclusions and recommendations that resulted from the data analysis of the focus group.
8. Present the results.	Present the focus group results in the forum to inform the OSS community.

Table 2.

Steps and tasks of the adapted focus groups technique.

4.5. Case study results

We applied the focus groups technique in the ERMaster project. We had trouble recruiting real users because it took a long time to get permission from the principal developer. We had to contact the principal developer by means of several media (email and personal wiki) before he gave us his consent. Six ERMaster users participated in the focus groups technique application. After designing the focus group format taking into account the stated topics and objectives, we proceeded to phrase the questions in order to apply the focus groups technique. Table 3 shows the (unstructured) format design.

Activities	Scenarios	Actors
1. Determine the focus groups objectives.	Emails	Principal developer, expert user belonging to the ERMaster community or a HCI student (under the supervision of a mentor)
2. Encourage the OSS community to participate in the forum, considering its importance.	SourceForge web site online forum	ERMaster principal developer I2-TIC master student
3. Briefly explain the aim and benefits of applying the technique in the OSS project	SourceForge web site online forum	ERMaster principal developer I2-TIC master student
4. Determine the topics to be addressed (with regard to the user interface and work environment).	Focus group format	ERMaster principal developer I2-TIC master student
5. Design the questions in line with the focus group topics.	Question format design	I2-TIC master student
6. Conduct the online forum.	SourceForge forum	OSS community
7. Review the focus group participant responses (forum/email). Email was an easy option for users due to time constraints. The responses to the questions were sent to one of the researchers.	Emails and SourceForge forum	I2-TIC master student
8. Compile the data and enter in data collection form (using an Excel spreadsheet designed for the purpose).	Focus groups application data collection form (Excel)	I2-TIC master student
9. Analyse and interpret the collected data.	Results reporting	I2-TIC master student
10. Submit a report with the conclusions.	Report containing the conclusions and recommendations of the focus groups data analysis	ERMaster principal developer I2-TIC master student

Table 3.

Focus groups technique format.

The questions should be aligned with the objectives addressed in the focus groups and are related to the ERMaster application work environment. The focus groups questions are designed to evaluate usability issues such as ease of learning, efficiency of use, memorability, errors and satisfaction [35]. By studying these factors, we focus on user-centred design, an issue neglected in OSS development projects. Previously, we published the call for participation on the ERMaster forum. Figure 1 shows an example of a response to the questionnaire, given by one used from the official ERMaster² forum. Table 4 presents the questions and the summary of results obtained from the focus groups questionnaire given by the users on the official ERMaster forum.

Figure 1.

Focus groups online forum executed on SourceForge.

Points to be considered		Summary of results
1.	E-mail address:
2.	Age	The ages of participants in the focus group were in the range of 36 to 40 years old.
3.	Gender	Participants were predominantly male (83%)
4.	Occupation	The majority of users work in Information Technology related areas.
5.	What experience do you have with ERMaster	17% of participants consider they have an intermediate level of experience using ERMaster and 83% state they have an advanced level using this tool.
6.	How long have you been using ERMaster?	The majority of participants have used this tool between 1 and 3 years.
7.	What do you like about working with the ERMaster environment?	They all expressed that the graphic environment allows them to design DER with ease and that it reduces the amount of coding work due to it being a good multiuse tool to work with different Database motors.
8.	Do you think the ERMaster menus are suitable for purpose?	83% considered menus were complete.
9.	What utilities, menus, options, etc., would you like to change or add to in ERMaster?	83% believed they should only need one or two shortcuts to perform certain specific activities (for example, DER design and DER data dictionary obtainment).
10.	How good do you think the environment is for entity-relationship diagram design?	The majority of participants believed ERMaster’s graphic environment did not need any improvements. However, they do mention the tool should be installed independently, i.e. without needing the prior installation of Java and Eclipse.
11.	Is the ERMaster query environment pleasant to work with?	All participants mention ERMaster’s query environment is pleasant to work with; the majority consider the toolbars, options and colours are sober and adequate for the work done with the application.
12.	Are the ERMaster icons, menus, options, etc., easy to understand?	All users considerate easy to learn how to use and to use the icons, menus and options.
13.	What problems have you often come across as an ERMaster user?	83% of users consider it is an easy to use tool, once it is installed. However, they consider that only expert users or those with an advanced level of average knowledge could easily install the tool due to the need of installing first Java and then Eclipse. Certain novel users complain about not having access to this tool due to the difficulty of installing it.
14.	Do you think that the ERMaster interface is easy to remember?	All users consider ERMaster’s interface is easy to remember even through some time may have passed since they last used it.
15.	How do you think the ERMaster interface should be changed or added to?	Menus, action bars and icons are easily accessible, but some users asked for the addition of an extra 10% of options of interest (for example, template editing, edit tracing, etc.)

Table 4.

Summary of the results obtained from the questionnaire.

We expected a higher rate of participation from the ERMaster community. Since it took a long time to get the principal developer’s permission and users did not show much interest in participating in our research, we only managed to recruit six participants. The focus group was moderated by the principal developer. However, he did not comment on the opinions of the users posted on the online forum. We believe that the principal developer did not get involved in the open online forum because he was not unduly concerned about improving the usability of the OSS application. The presence of a note-taker was unnecessary in order to conduct the focus group, as, on one hand, the comments posted on the open online forum were logged automatically and, on the other, the researchers kept all the emails that they received.

The principal developer sent an introduction and formal invitation to the ERMaster project community to participate in the open online forum. For many application users, however, their forum registration is the only record available as they did not post any opinions. Some users answered the questions and submitted their responses by email instead of publishing them on the forum. Other users eventually responded to one or two questions related to their major field of interest but failed to complete the entire questionnaire. A few other users stated that they were happy with the tool and did not answer any of the questions. We then screened all the feedback, and selected the contributors who answered all or most of the questions. As a result of this screening, we got a sample of six users for our research.

The noteworthy results of the application of the focus groups technique considering the data gathered include: (i) novice users had problems with installation (because it is an Eclipse plug-in), (ii) expert users regard ERMaster is being a tool that is easy to learn, easy to use and easy to understand and had no trouble remembering how to use it, (iii) ERMaster is designed ergonomically using menus, action bars and easy access icons, but some users requested the addition of options of interest (for example, export DBs to Excel), and (iv) users consider the ERMaster work environment to be adequate, as there are Help and Query tools.

5. Discussion of results

In this section we discuss and answer the research question of this study.

RQ: How to incorporate the focus groups technique in a real OSS project?

The usability techniques have been created for another type of software developments, i.e. they have not been conceived with the specific characteristics of the OSS development process in mind. For this reason, it is necessary to adapt these techniques. These adaptations are based on the adverse conditions these techniques present. Some adverse conditions can be overcome using certain web artefacts (for instance, wikis, forums, blogs, etc.), which are known by the OSS community. As a result, many of these adaptations will be familiar to the members of this community, which favours to a certain extent the application of these usability techniques.

The adaptations of the focus groups technique are mainly two. Firstly, users participate online though a web artefact: a forum. Secondly, the usability expert is replaced by a developer, an experienced user or an HCI student under supervision of a mentor. In our particular case, the expert was replaced by an HCI student under the supervision of a mentor.

After applying the focus groups technique to the ERMaster project, we were able to confirm that it is very hard to get a representative set of users. We believe that the main reason for this is that users are unmotivated. We had to be persistent and use different communication mechanisms (for example, personal wikis and electronic mails) to get the consent of the principal developers (only one out of five principal developers responded). The biggest problem with applying the focus groups technique was user availability: most users are volunteers and had very little spare time. In fact, the participants did not have the time to enter their comments in the online forum and ended up emailing their opinions to one of the researchers. Since the focus groups participants had a medium level of experience with respect to both the ERMaster tool and the field of computing, they did not pinpoint any major problems which novice users may have had.

The adoption of the adapted focus groups technique was acceptable as this technique requires a small number of participants to get reliable results. With regard to our proposal of substituting a developer, expert user or HCI student under the supervision of a mentor for the usability expert, the expert was replaced in this case by a HCI student supervised by a mentor. Note that this student was in his final year of the Master of Information and Communication Technologies Research and Innovation at the Autonomous University of Madrid and was taking two HCI courses. Additionally, the student was supervised by two usability experts. On this ground, there is no risk of the proposed adaptation for the selected technique having a negative impact on the quality of the software.

We can conclude that the results of the adoption of the focus groups usability technique were not what we expected. Firstly, we banked on the participation of a large number of users based on the statistics provided on the application web site. Secondly, it was hard to contact and recruit users to participate in the research. Note that OSS community members are all volunteers, and they participate in their spare time. Despite all these problems, however, the adaptation of the focus groups technique was reliable for adoption in the ERMaster project, as it does not take many users to get a reliable result.

The main limitation of our research is the number of case studies (only one). This is preliminary research. Therefore, more cases studies are required to validate the proposed adaptations. Note that there are other usability techniques (for example, user profiles, heuristic evaluation) that might benefit from the proposed adaptations (e.g., HCI students supervised by a mentor standing in for experts) to enhance technique adoption in the OSS development process. Briefly, the results of our research are not very generalizable because we conducted only one case study. Therefore, the focus groups technique needs to be applied to other OSS projects. However, the preliminary results provide a basis on which we can build to improve the performance of other case studies.

6. Lessons learned

The lessons that we have learned from applying the focus groups technique in the OSS project are as follows:

• OSS project administrators (particularly of small projects) must start to attach importance and become aware of the impact of the issues dealt with by HCI in the development of usable software. We believe that this could motivate users to participate in the application usability techniques. This finding is consistent with the proposals of other authors [23, 37].

• As the developer team of a small OSS project is likely to have limited knowledge of usability techniques and interaction design, we suggest recruiting a community user who has some knowledge of or is enthusiastic about these issues to contribute in the early stages of the OSS project development.

• OSS projects that want to apply the steps to incorporate techniques and need to find an expert in HCI or an HCI student guided by an expert mentor can do so by publishing advertisements in the webpage, forum, wiki and blogs of the project. Furthermore, the administrator of the OSS project can get in touch with universities to encourage expert usability students to collaborate in the application of techniques to improve the usability of an OSS application.

• One of the underlying principles of the OSS community is collaboration [23, 38]. However, we did not get much collaboration during the application of the technique because real users (i.e., users registered at the project Source-Forge website) are perhaps short of time or unaware of the importance of usability. On this ground, we suggest that technique application should be publicised through social networks to recruit as many participants as possible.

• The OSS community should (with the administrator’s permission) provide incentives to encourage users to participate in this type of initiatives (i.e. participate in the application of usability techniques). A possible example of one such incentive would be public acknowledgement of users that have made contributions towards improving the application interaction design in a section of the project website.

• By adapting this technique in particular, we were able to determine that it is possible to incorporate it into small OSS projects. Because these techniques demand conditions that OSS projects generally cannot meet, it is necessary to make adaptations to bring them closer to the idiosyncrasy of OSS projects. This result reinforces the theory that it is possible to incorporate adapted usability techniques in OSS projects considering Usability Framework proposed by Castro.

7. Conclusions and future research

The goal of this research was to evaluate the feasibility of adopting HCI usability techniques in OSS projects. We adapted the focus groups technique for adoption. Through adaptation, we were able to account for some OSS development characteristics that pose an obstacle to the application of the technique as per HCI recommendations (for example, OSS developers and users are geographically distributed). In particular, we adapted the focus groups usability technique for application in the ERMaster OSS project.

It is not easy to recruit volunteer users to participate in OSS usability projects. As already mentioned, users often do not have much time, and it is hard to get them to take part without an incentive. With the focus groups technique, although we did not get much collaboration from users or even the principal developers, we did manage to apply the technique because it requires only a small number of participants to get a reliable result [14, 15, 34, 35, 36]. Author opinions differ as to the number of users to be taken into account for the focus groups technique to be successful [14, 15, 34, 35, 36]. Most of these authors agree that a focus group should include from six to nine users if it is to work. Fewer than six participants would not generate enough ideas for discussion. In this research, however, any users that are willing to collaborate are allowed to, that is, there is no limit on the number of users because this would go against the working philosophy of the OSS community where anyone who wants to is welcome to participate. In sum, our proposed adaptation does not place any constraints on the number of technique participants. This adaptation is a response to the OSS community working philosophy rather than to an adverse condition posing an obstacle to technique application.

The focus groups technique is useful for gathering opinions and suggestions from participant users for the product concept development activity and its results are descriptive. After analysing and applying the focus groups usability technique in requirements engineering activities in OSS developments, we found that there are adverse conditions that are an obstacle to its application like, for example, the shortage of OSS users interested in applying the technique, community geographical and temporal distribution and OSS community motivation.

We believe that, in order to improve the integration of usability techniques in OSS projects, the OSS community has to start attaching importance to and raising awareness about the repercussions that the issues addressed by the HCI field have on software development. Additionally, as HCI techniques need to be adapted to overcome the adverse conditions for adoption in OSS development projects, the OSS community also has to broaden its view of software development in order to consider usability and not focus exclusively on feature development. In the future, we aim to conduct further case studies to adapt and apply other usability techniques in OSS projects. We will analyse other web artefacts that can be adapted to improve communication in OSS communities (e.g., social networks) and gradually raise the awareness of OSS developers about the benefits of applying HCI usability techniques.

Acknowledgments

This research was funded by the SENESCYT-Ecuador, and Quevedo State Technical University. Also this research was funded by the Spanish Ministry of Education, Culture and Sports FLEXOR (TIN2014-52129-R) and TIN2014-60490-P projects and the eMadrid-CM project (S2013/ICE-2715). Finally, this research received funding from the “DIUDA 22316 Project” of the University of Atacama.