Open access peer-reviewed chapter

Learning from Online Voices: A Mixed Methods Approach to Explore Patient Online Reviews of Hospital Care in Portugal

Written By

Carla Marisa Ferreira Gomes, Marlene Paula Castro Amorim and Mário Jorge Ferreira Rodrigues

Submitted: December 20th, 2019 Reviewed: April 20th, 2021 Published: May 21st, 2021

DOI: 10.5772/intechopen.97793

Chapter metrics overview

269 Chapter Downloads

View Full Metrics


Online patient reviews can offer a rich information source to users of healthcare services, as well as for hospital management and quality monitoring. Whereas in recent years the volume of online patient reviews has been consistently growing, organizations still lack standardized approaches and tools to allow for the systematic monitoring of users’ online comments. Therefore, managers are lagging in the ability to make use of such data from patients’ voices for improving the quality of the services provided. If organizations fail to develop the right capabilities to consider users’ online reviews and feedback, they risk not only to miss important quality failure alerts, as wells as to frustrate their customers’ expectations for service and attention. In this chapter, we present a qualitative analysis of patients’ reviews for healthcare services in Portugal, building on a sample of data extracted from Google for the year of 2019. The chapter reports the major quality management themes addressed by hospital users in their online expressions and offers some guidelines to support a structured analysis and visualization of results from online users’ word of mouth data.


  • Online Ratings
  • electronic Word-of-Mouth
  • Quality management
  • Hospital
  • Quality in Health
  • User Generated Content

1. Introduction

The Quality of Health Services influences the dynamics of Health Institutions and determines the care provided to users, which are co-producers in a logic of interactivity. The collection of citizens’ opinions about the Quality of Health Services, allows for a subjective report distinct from other quality measures (e.g., quality indicators). It is usually carried out in a structured, offline, and conventional way (e.g., satisfaction surveys) [1, 2], but globalization and the internet have brought a novel feedback source called analysis of online comments and classifications (OCC) [3, 4].

OCC are considered a form of electronic-word-of-mouth as they capture the essence of consumers’ direct sharing of experiences just as the traditional word-of-mouth. Word of mouth is one of the most pervasive forms of disseminating information about customer experiences. As a form of interpersonal influence is acknowledged as one of the more important drivers that influence customer purchase decisions, notably when the services that are being evaluated for purchase involve intangible attributes that cannot be experimented and assessed a priori. Online media has created an enormous arena of opportunity for companies and customers to share information. As such we have witnessed in recent years an explosion of voluntary testimonials from customers about their service encounters in a mode of expression that has been labeled as electronic word of mouth that are usually shared together with service ratings and classifications. Altogether electronic word of mouth and customer ratings altogether designed as OCC, offer a rich source of information for customer preservice evaluations. Moreover, they are also an enormous source of data and insights to inform the quality management function in the company. The remaining challenge is to devise structured methods to make sense of all the available – often unstructured – information. OCC are being produced at increasing rates and provide a different understanding of user satisfaction compared to traditional measures [5]. For many services, and particularly for the context of the health sector, the current debate concerns how to make this information relevant to support the decisions of users and hospital managers, notably by allowing for the development of methods of analysis that help in the identification of quality gaps [1, 2, 6, 7].

Studies have already been developed within the scope of OCC analysis in other areas of the industry (e.g., research goods, restaurants, hotels), but transposing these methodologies to health services is not straightforward. The investigations carried out in this area are international and are directed only at the user’s decision and do not explore methods that assist managers in the continuous improvement of the quality of health services. Having such methods and tools available, health service managers would have a better understanding of the quality of the service provided, and thus have more informed decisions about how service quality can be improved. Moreover, its existence in the Portuguese context is unknown.

This investigative gap, added by factors such as the increase in the volume of this information and the apparent lack of structured methods or tools to systematically extract useful information, determined the development of this study and the identification of the following general objectives: (1) to explore ways to analyze and summarize large volumes of information available online generated by users in the context of health; (2) to devise ways for summarizing information and to propose methods to support the decision making of health institution’s managers.

The analysis was conducted using a mixed method approach that involves bringing together qualitative and qualitative sources of data in a meaningful manner. Research about the quality of services has been characterized by the prevalence of quantitative approaches, building on data from questionnaires that aimed to capture customer perceptions about the various service attributes. OCC put on the table a mix of customer ratings and customer generated (narrative) content that were addressed in this work: ratings attributed by service consumers were mixed with qualitative analysis from the narrative content. The quantitative analysis, by itself, was also made mixing well known computer algorithms with a thorough manual content analysis. This chapter has five sections. The first section was the introduction to the topic and elaborated on the motivation and relevance of the topic; section two provides background and related work information; section three explains how the sample data was specified and collected; the fourth section presents the data extraction results and respective analysis. The chapter ends in Section 5 offering some discussion and conclusions.


2. Background and related work

In health, consumers do not have a passive role, but rather actors whose opinion about service experience can be decisive to support the performance of managers, notably for the continuous improvement of the service quality. It is important to create mechanisms that facilitate users’ feedback, as it is difficult to give a voice to the user, but even more difficult is to incorporate the voice of the users in the decision-making processes of the management teams [8]. Thus, users’ OCC can be a vehicle that facilitates communication between users and health institutions.

The concept of quality in health has evolved over time. Health services are considered credible or high-risk services and have characteristics and particularities that differentiate them from other services or products. They are considered heterogeneous, intangible and require a large citizen participation [5, 9, 10, 11]. It is not generally possible to assess its quality before experiencing it, fand the possibility of returns is also limited. It is only possible to correct surgical complications or reduce consequences that can be harmful for the user himself (e.g., incapacity at work), for the family and for society itself (e.g., increased health spending). The same does not happen with research goods or products (e.g., purchase of a smartphone) or in experience services (e.g., restaurant). In health services there is an information asymmetry between the user and the superior care provider compared to other services [11, 12].

Donabedian [13] proposed a model in which quality in health depends on interventions aimed at the three pillars: Structures of care provision, that is, attributes of the environment such as material and human resources, facilities and organization; Processes arising from the provision of care such as technical and interpersonal skills of health professionals; Results, understood as the reflection of health care in the user, which include rehabilitation and recovery of users, control of chronic illness, change in behaviors and lifestyles, and satisfaction with the care provided [14, 15, 16]. The Donabedian model is analogous to the division suggested by Rothenfluh and Schulz [16], which categorizes health care into characteristics of a research, experience, and credibility nature. For WHO, OECD & World Bank [17] Quality in Health is a continuous and dynamic process where care is sought at the right time, in a coordinated manner, responding to the needs and preferences of the citizen, minimizing damage, and wasting resources, seeking to increase the probability desired health outcomes. It is a complex process [6] and its measurement is a great challenge [16].

There are several models proposed for the evaluation of the quality of health services, namely the theoretical model of Donabedian [13], the model of Grönroos [18] and the model of Parasuraman [19, 20, 21, 22]. The model by Grönroos and Parasuraman et al. they were initially developed for most services, and the Grönroos model is not as perfected as that of Parasuraman, whereas the Donabedian model was developed specifically for health services.

The abandonment of the paternalistic and hierarchical health model and the adoption of the model based on shared decision, fostered the centrality of the citizen. The growing awareness of the citizen forced organizations to take an interest and to promote Quality and continuous improvement policies in the pursuit of organizational excellence [23]. Thus, users’ perceptions of service quality have become a critical component for measuring the quality of health care and services [24]. The authors describe two ways of assessing Quality in the health Services: health outcomes or Quality indicators (Order No. 5739/2015, May 29) [25]; and user experience assessment instruments [26], which subjectively reflect the quality of health care, in a unique and individual perspective, containing valuable information on distinct aspects of services.

There are several ways to listen to the user, namely classic or conventional methods and more recently, OCC have emerged. Classical methods are usually constituted by a standardized set of questions about the care received from a specific provider (health unit or professional), such as face-to-face, digitized or telephone satisfaction surveys [27]. This form of feedback implies the non-voluntary participation of the user, whose collaboration initiative comes from the health institution, which also has the responsibility of selecting the questions, mostly consisting of closed questions, and translating into a numerical result, as well as it is also the institution that determines the frequency with which it is applied [28].

OCC can represent a new way of assessing Quality in Health, understood as positive, negative, or neutral online publications and reviews, carried out by real or potential users, about a service or product [6]. They are an unsolicited way, therefore, voluntary for users to write opinions or perceptions about aspects related to health services online. Usually, this information is not standardized [26] and can only be translated into a punctuation (e.g., number of stars) and/or a free text, the wording of which does not obey any prerequisite in terms of structuring [11, 27, 29].

The use of OCC has advantages compared to traditional measures, particularly: websites are easy to access and use [1, 30] they are a form of information that promotes transparency, since data are made available online and are accessible to most people [28, 31]; they are a real-time barometer of public opinion, in a context of rapid and constant change, and allow the identification of prominent issues [11, 32]. They can be a substitute, instantaneous or almost in real time, for the analysis of the users’ experience, with the possibility of using automated methodologies, which facilitate their analysis [31, 33, 34] and it is in this sense that the present study will seek to make a major contribution; they offer a convenient, safe, low-cost mechanism for organizations to hear users’ voices [31, 34], being an important means of alarm to signal the deterioration in the quality of care [16] or, on the other hand, to identify successful practices [31].

On the other hand, there are several limitations that are mentioned in the literature to the use of OCC, which include: influence of other factors, in addition to the quality of care; anonymity and vulnerability to fraud [7]; the risk of not being representative of the general population [30, 31], noting that the literature refers that they are the youngest female users, with a higher degree of academic training, living in metropolitan areas [3, 35] and those who use health care (e.g. chronically ill) most frequently write this information.

Although consumer opinion websites have been in existence for more than two decades, the first study on health OCC was published in 2009 [36], and most studies were published after 2010 [33]. Studies published before 2010 used content analysis with small samples, whereas more recent studies recovered and studied a larger data set using automated technologies [33].

Most of the studies turn out to be descriptive, where the numerical classifications are analyzed with the determination of their frequency. There are other descriptive studies that analyze the narrative comments of OCC, understood as content analysis. If at an early stage, they used traditional qualitative methods to find the main categories of these comments [33], more recently used advanced techniques, such as Natural Language Processing (NLP) (“natural language processing”). They are, therefore, advanced analytical methods that allow the content analysis of thousands of narrative comments.

Research authors use diversified indicators to collect their data, such as the number of stars [7], the number of words [37], gender [38] or the total score. The studies that carry out content analysis may suggest a great diversity of categories within the scope of Quality in Health to group the collected data. It is also common for authors to perform sentiment analysis [39] and to classify textual data according to positive, negative, neutral and/or mixed perspectives.

OCC are increasingly important sources of information for making diverse types of decisions [40]. Understanding and systematizing them is urgent and essential for the citizen and the manager [2]. The following model (Figure 1) shows that a systematic collection and analysis of this information by the Manager can feed and have a direct impact on the continuous improvement of the Quality of Health Services.

Figure 1.

Collection cycle and analysis of OCCs and their importance in the Manager’s intervention.


3. Sample definition and data collection

The main purpose of this work is to develop and discuss methods to process the diverse and rich information that is available in the online platforms, and that is offered predominately in an unstructured format. Despite the richness, diversity and volume of this information, managers still lack structured approaches to deal with such large volumes of data, and to make sense of it for supporting decisions and initiatives for quality management. In this work we offer a contribution for the advancement in the knowledge about how to extract and categorize existing online customer generated content offering a structured approach to make sense of the data. Specifically, the research work builds on prevalent service quality models, that offer an established multidimensional approach that has validated over the year a range of distinct service quality constructs (e.g. reliability, empathy, etc.) that frame the domains of assessment that customers consider when evaluation service experiences. The study builds on such service quality conceptualization to classify the customer reviews content and develop summary metrics to inform quality management though in the context of health services. The study offers a stepwise view of the content analysis deriving highlights about critical aspects in the extraction of data, as well as in the process of cleaning the extracted data for the purpose of delimiting a meaningful sample of customer comments. The advancements in such data extraction and analysis are of critical importance, particularly for medium sized organizations, who might lack the resources to devote specific capacity to make use of the available information. Moreover, as the pace of expansion of the available customer content continues to accelerate, the need to deploy methods to analyze content that are aligned with the prevalent quality models, that are embedded in current managerial knowledge is an urgent matter.

An exploratory-descriptive, cross-sectional, and qualitative study was developed using content analysis after selecting a rational and criterial non-probabilistic sample. It seeks to respond to the following specific objectives: to identify relevant variables for the segmentation of content generated by users; identify relevant indicators that allow monitoring the behavior of this information and signal the moment when the manager must extract and analyze the content generated by the users; and to identify categories that classify the content contained in the online comments generated by citizens in the context of health, in line with the quality models in health services.

The data was extracted from Google’s online platform and is freely accessible to anyone. The choice of this platform is due to accessibility, reach and the fact that it covers all services and national territory. Ethical considerations were safeguarded since the data that could identify the population or sample studied were hidden and submitted to codification, preserving anonymity and confidentiality.

The determination of the sample that was intended to be representative of the comments, was a complex and time-consuming process associated with the dispersion and volume of information, as well as the diversity and form of organization of the Portuguese Health Institutions, as shown in Table 1, implying continuous methodological adjustments. It included two stages, namely the delimitation of the Institutions to be included in the sampling process and the second stage, which focused on extracting comments and selecting valid comments, that is, on determining the units of analysis.

Management modelHospitals (Hs)Typology (portaria n° 82/2014)N° of Hs TotalN° of OCC

Table 1.

Distribution of the number of OCC in the different hospital institutions.

Entities not discriminated by portaria n° 82/2014 (10 abril).

EPE: Entidades Públicas Empresariais (public business entity); SPA: Sociedade Pública Administrativa (administrative public society); PPP: Parceria Público-Privada (public-private partnership).

In the first stage, we applied preliminary delimitation criteria (typology of Hospital Institutions (IH’s), total number of OCC and IH management models) and all SPEs except Braga Hospital and group IV were selected. Of the initial 8866 OCC, 7188 OCC remained, so it was necessary to carry out an additional delimitation, using the formation of clusters according to additional and previously defined criteria (Type of HIs; Population covered (year 2018); No. of consultations (year 2018); number of urgencies (year 2018); number of surgeries (year 2018); total number of OCC), to form homogeneous groups. In this sub step, 4 Type IHs (ULS in the Northeast, the West CH, the Baixo Vouga, the ULS of the Alentejo Coast) were selected; 1 Type IHs (CHU of the Algarve); 1 Typology III HIs (CHU S. João) revealing a set of HIs that are representative of the country’s diversity and accounted for a total of 1088 OCC.

In the second stage, the extraction of the OCC was done automatically, using the research team’s software, to a database elaborated using Microsoft® Excel® Office 365MSO, which is organized based on previously defined indicators. The definition of the indicators to be included in the study were based on two sources: indicators used in previous investigations (narrative commentary, number of stars, gender, number of words) [7, 38] and indicators defined in the scope of this study (date, location, language, total score of the institution and number of Likes).

After a posterior extraction of 1179 OCC, which included the previous 1088 and increased due to the dynamic character of these data and platforms, a preliminary selection of comments was made eliminating obsolete data, specifically the 437 OCC that did not present characters and the 150 irrelevant OCC (with meaningless criticism or with a meaningless argument) (Table 2). Subsequently, for the 592 OCC included in the study, the units of analysis were defined, fragmenting the comments into different thematic categories. The thematic categories resulted from the joining of the perspective of two authors: the model of Parasuraman et al. [20] and Gillespie and Reader [41]. This combination made it possible to define four categories to be used in the present study: “Tangibility”, “Response Capacity”, “Empathy” and “Reliability/Guarantee”. The models selected for the definition of the categories to be used in the present study are Quality models, which, although one of them is old, still presents current dimensions for the existing services, namely in the scope of health services. Thus, in the 592 OCC selected, 898 sub-comments were identified, which constituted the units of analysis of the present study.

Total of extracted OCCs: 1179
Typology/ClusterHsNo textIrrelevantRelevantAfter Categorization
I / 1ULSN2141524
I / 3CHBV732099111
I / 3CHO762574140
I / 4ULSLA2391936

Table 2.

Distribuition of OCC.


4. Results and analysis

The present study combined computerized automatic analysis with qualitative techniques for the analysis of feelings and the categorization of comments in the domains of Quality in Health. The relevant comments considered in the analysis were categorized as shown in Table 2.

The comments were mostly written in Portuguese, but 15 other languages were also found (e.g., Korean, Finnish, Slovak, Turkish, Indonesian, etc.). The translation was automatic, using Google translator, a methodological option justified by the need to facilitate the replicability of the present study. The HIs with the highest percentage (37%) of foreign comments was type II, possibly associated with the tourist flow.

The OCC were performed mainly by men, except for the I/1 ULSN typology and III CHSJ typology. The genre was obtained from the nickname associated with the comment, but some publications were made by foreign citizens with characters related to their language (e.g., Korean), which makes it impossible to automatically classify them. The remaining results will be presented according to each of the three specific objectives outlined previously in point 2.

The information contained in data was quite disorganized and unstructured, which made it difficult or impossible to analyze. It was noticed that, similarly to what happens in the platforms of other industries (e.g., TripAdvisor), its pre-segmentation would facilitate the extraction and analysis of data, the specification of which could be previously defined in pre-filled fields outlined in the online writing platforms. Available to users using health services. The sentiment analysis (negative, neutral, and positive) of the comment was obtained through a state-of-the-art platform for content analysis in Portuguese called Linguakit [42]. A prior automatic classification was carried out for each of the comments. However, the automatic analysis of the feeling of the text, in the negative, neutral, and positive categories, has not proved to be very robust because it often has wrong syntactic constructions, resulting from the context of free and informal writing in which they originate. This forced this automatic classification to be compared with a second manual classification, to guarantee the reliability and validity of the results. In this manual classification of feeling, a new dimension was introduced, called “mixed,” when the comment refers to various aspects of Quality of Service or to several episodes experienced in health care, where the positive and negative feeling coexist simultaneously.

A comment was considered positive (when the editor mentions aspects of health quality that made him happy or pleased with health care), negative (when the editor mentions unpleasant aspects related to health services), neutral (when the comment has neither a positive nor a negative feeling) and a mixed feeling (when in the same comment the writer simultaneously mentions positive aspects, but also negative experiences related to health care). Subsequently, the feeling and the number of stars were related, where a negative feeling in the user’s textual account, would demonstrate from the outset that he would be dissatisfied with his experience in health care and would score it with 1 or 2 stars, if the comment were neutral, would give 3 stars and if it were positive, it would give the value of 4 or 5 stars.

In summary, from the results presented, it is concluded that, it is extremely important that online publishing platforms find mechanisms that predispose the user to explain the feeling affection to the report about their experience in health care, since, this care prior facilitates the visualization, as well as the collection and treatment of the data made available about the user’s opinion and therefore facilitates the management of information and the intervention of the manager, allowing him to provide himself with easily accessible information that can be useful for the improving the performance of the health organization. Since, and as our results demonstrate, treating and classifying a posteriori the feeling of a large volume of data published by users about health services can be a challenging task, where there is a considerable risk of bias in the data, in addition to predisposing to the use of complex methods, supplementary surveillance, as well as spending unnecessary resources and time. Therefore, the need to requalify the comments considered neutral, reinforces that, in fact, it is necessary that the platforms specify fields that fragment the information from the start so that it can be easily managed (Table 3). These results add to the body of knowledge in service quality as well as to the domain of information science, in particular for studies addressing electronic word of mouth, and they bring forward specific challenges in the analysis of content and advance highlights on how to design adequate platforms for the collection for customer inputs in a manner that meets the requirements of subsequent phases, − i.e. content analysis.

ClassificationFeeling1 star2 stars3 stars4 stars5 stars
Manual – AutomaticNegative1932−2-2

Table 3.

Process used to determine the differential between automatic classification and manual classification for type 1, cluster 3 CHBV.

Typology/ClusterIHn%Average PEn%Average PEn%Average PEn%Average PE
I / 1ULSN417%3,0625%1,8625%2,3833%3,0
I / 3CHBV118%1,86043%1,43223%2,43726%2,1
I / 3CHO2018%2,34339%1,92623%2,52220%2,8
I / 4ULSLA411%1,8822%1,5822%2,61644%2,8

Table 4.

Distribution of comments by average star ratings.

Subtitles: IH = Hospital; PE = Star Rating; n = total ORCs.

From the previous results, one of the difficulties that the manager may face when dealing with this data is related to the question of the reliability of the automatic analysis of the content. On the other hand, greater reliability will require manual analysis, and this involves spending additional resources (e.g., time, human resources) for management. The results of the present study suggest that instead of an automatic classification followed by a manual classification, a viable alternative would be to do an automatic analysis, to check the consistency between the number of stars and the sentiment and finally, manually analyze those comments where the sentiment is the inverse of the score. In other words, the alternative is to carry out automatic classification first, and then, only in case of incongruities, a subsequent manual classification.

4.1 Categories for the classification of content generated online by users, in line with the principles of quality in health services

One of the purposes of the present investigation was to identify categories to classify the online content generated by users. This was carried out in two ways: automatic and manual analysis of feelings as developed in point I. and the categorization of comments in the domains related to Quality in Health, previously defined: “Tangibility”, “Response Capacity”, “Empathy” and “Reliability” /Guarantee”. For this last categorization, an automatic classification was attempted, which proved to be inefficient, due to the characteristics of the text, since it presented some peculiarities, namely sarcasm, irony, spelling errors, acronyms, abbreviations, and stories. Therefore, the author classified manually in the four categories and one of the elements of the research team made an independent review, and in the case of disagreement, the categorization was discussed and resolved between the two elements.

4.1.1 Categorization of sentiment analysis

The results show that in relation to the sentiment, in the total of the 6 IHs it was found that 65% of the comments were negative (383/592), 1% neutral (6/592), 8% mixed (48/592) and 26% positive (155/592). In literature was found that there is no consensus, and the present investigation is in line with the results found in the study by Emmert et al. [39].

Emmert et al. [38] refer that 80% of all comments (average length of 45.3 words ±42.8) were classified as positive, 4% as neutral and 16% as negative. And that longer narrative comments were more likely to be negative, while shorter comments were more likely to be positive. When an association was made between the sentiment and the size of the OCC (measured by automatically counting the number of words in each comment) it was found that the average dimension (in number of words) of the positive comments (43.98 words) differs from the negative ones (61.69 words) for the analyzed sample (t test, Sig. 0.000). Thus, we verified through the t test that the negative comments were longer than the positive ones, in accordance with the literature and similarly to the results obtained in the studies by Rastegar-Mojarad, et al. [37] and Emmert et al. [39]. Information-rich analyzes tend to be longer, with a consequent increase in utility for readers [43] and even though longer narratives were more likely to be negative [28]. In this way, it is understood that the longer comments say more and tend to be mostly negative. Thus, the results of the present investigation show that the manager must be aware of the behavior of this indicator when analyzing online data.

It was also found that users put more likes in negative comments. The results show that there are differences between the number of likes of positive comments (average 0.95) and negative ones (average 1.41) (t test, Sig. 0.000). When the association between sentiment and the average score in stars was made, it is not possible to infer that the number of stars in the positive comments differed from the negative ones (t-test, Sig. 0.144). In particular, the average number of stars for positive comments was 4.60, for the sample analyzed, and the number of negative comments was 1.31.

4.1.2 Rating of comments regarding the quality of health services

Understanding what topics are most often spoken by the user’s “voice” can help caregivers, managers, and administrators of health institutions to improve the user-centered health system. This type of content analysis provides the manager with richer information that can support the development of improvement actions.

All sub comments were liable to be classified under an attribute of Quality in Health, as can be seen in Table 4, with none remaining unclassified. But there is a wide diversity of results, an irregular pattern that made it difficult to appreciate and, therefore, the establishment of associations.

The most debated topics in users’ online newsrooms are about the “Response Capacity” dimensions (for 3 of the HIs - I / 3 CHBV typology, I /3 CHO typology and III CHUSJ typology) and “Reliability /guarantee” in the remaining 3 IHs (typology I /1 ULSN, typology I/4 ULSLA and typology II CHUA). If, on the one hand, the literature stresses that, the user may not have the competence or knowledge capable of evaluating the technical or clinical aspects of care, that is, aspects related to the “Reliability / Guarantee” dimension and that its evaluation may be inaccurate [16, 21], on the other hand, in this study it appears to be an aspect in which users focus their attention, as can be seen for the values related to the dimension “Reliability/Guarantee”.

“Tangibility” was the least mentioned dimension for all the HIs in the studied sample, being, therefore, the subject least mentioned by users (Table 4). It was observed that the aspects of “Tangibility” are the least highlighted in the comments available to all IHs by users, compared to the other dimensions of QeS, so that a change in this proportion, that is, the increase in the percentage of tangible aspects in view of the other domains, it could be an alarm signal that is the target of investigation and intervention by the manager of the health organization. Thus, and given the national context where most public health institutions were built several years ago, resulting from the triggering of the NHS, and that the Portuguese historical and economic context, as well as the financial crisis that hit Europe and that had repercussions in the divestment in the facilities and material resources of public services, may be an explanatory factor for users to have tolerance regarding the tangible aspects existing in the services of the HIs. Not least because, much of the investment made in the health sector was directed to flagrant problems such as the increase in the prevalence of chronic diseases resulting from the increase in average life expectancy [44]. It is a fact that, it seems natural that users have elevated expectations about the functioning of public health services, as these are financed by their taxes, but their attention is focused on aspects related to “Capacity response” and “Reliability/Guarantee” and not so much for the characteristics related to the appearance of the facilities, professionals, and equipment. Health services are a type of service that is not visible to everyone because they are characterized by an asymmetry of information between the user and health professionals [11]. In addition, these services are produced while they are consumed, and therefore have an intangible and heterogeneous character [5, 10, 11].

The “Response Capacity” seems to be the dimension to which users are less satisfied in this investigation since there is a lower average score value in stars for all the target HIs studied. And there was no average score in the number of stars above 3.2, it appears that the degree of user satisfaction with the selected HIs will not be high.


5. Conclusions and future work

The volume and diversity of information available online about health services implies challenges in the analysis, scarce methods or tools that facilitate its use by the manager in the continuous improvement of the Quality of Health Services. Health systems must, increasingly, be attentive to the opinions of the users who are the ones who experience the care, arranging economic tools that allow to listen to their perception and to effectively profit from the preciousness of these data. Since the standard tools for extracting and analyzing this information are not adapted to the syntactic characteristics of users’ newsrooms.

The present study intended, on the one hand, to contribute to fill the investigative gap present in the national community, as well as to find effective and facilitating ways of making the most of these data, in a health system marked by the scarcity of resources. This scarcity of resources ended up being exponentiated by the current context of the pandemic, and the real impact of the pandemic on the health system, on public administration, on society in general and on each one of us has yet to be investigated.

The study allowed the identification of relevant variables for segmenting the content generated by users: hospital episode, editor, service, professional and the feeling of the comment (e.g., positive, negative), as the results showed that the automatic classification (positive/negative) differs significantly manual classification, demonstrating the complexity of its a posteriori classification implying that it is important not to compromise the voluntary character and the trust of the user. And finally, it allowed to classify the content according to attributes of Quality in Health: the comments were mostly negative (65%) and the average dimension of the positive ones (43.98) differs from the negative ones (61.69), resembling the results of previous studies [38, 39]. There are significant differences between the number of likes of positive (0.95) and negative (1.41) comments and the dimensions with the most comments were “Responsiveness” and “Reliability/Guarantee;” the least commented dimension was “Tangibility” for all HIs. Overall, one of the key contributions of the study is to advance in the classification of available customer generated content in a manner that is aligned with prevalent service quality models, i.e. establishing a correspondence between customer reviews and prevalent service quality attributes. Such approach offers manager an expedite way to make sense of the volumes of voluntary customer generated content, building on existing approaches to display service quality data. A ability to develop capabilities to make sense of the existing customer narratives is very important in a context where the volume of available context grows every day, and the risks of obsolescence of such knowledge are real. By using the service quality models as a steppingstone, this study proposes an approach to foster the development of rapid content analysis routines in service contexts. The methodology adopted in this study is replicable and able to be extended to other domains of service management where the volume of customer generated content is also gaining space, including hospitality services, education, public services to name a few. By refining the classification grid with the prevalent service quality attributes for each of these sectors, that are also already explored in the literature, the process of data extraction and classification is of straightforward application.

It is possible to identify some limitations in the present study, some of which have already been described by other researchers, namely, the fact that it contemplates data from only one platform (Google) [43, 45], with the risk of non-representativeness of the general population. Moreover, the study is focused on the exploration of customer content for the specific domain of health services, therefore leaving out particularities of other service sectors (e.g. hospitality, education, etc.). Nevertheless, and as explained in the results section, the method applied can be replicated in such sectors given that there is a necessary and preliminary revision of the service attributes used for the classification of customer reviews and content to match those that in the literature, have been identified as pertinent for each case.

In terms of future research, the following fields of intervention are suggested: development of software capable of processing and classifying data produced by users, adapted to their style of writing; explore, develop and adapt methodologies to other health institutions (e.g. health centers, long-term care units and private health units); and elaborate a project applicable to hospital units, allowing the results of the study to be transferred from the paper to measures and proposals for effective and practical improvement in health institutions. As mentioned by Hong et al. [33] the results of the investigations should go beyond simple descriptive analysis and theory-based hypothesis testing to provide more clinical and political implications.



This work was financially supported by the research unit on Governance, Competitiveness and Public Policy (UIDB/04058/2020) + (UIDP/04058/2020), funded by national funds through FCT - Fundação para a Ciência e a Tecnologia.


  1. 1. Huete-Alcocer N. A Literature Review of Word of Mouth and Electronic Word of Mouth: Implications for Consumer Behavior. Frontiers in Psychology. 2017;8(1256):1-4
  2. 2. Amorim M, Rodrigues M. Building on eWOM to Understand Service Quality in Hotel Services. Open access peer-reviewed chapter: Book Citation Index. 2017
  3. 3. Boylan AM, Williams V, Powell J. Online patient feedback: a scoping review and stakeholder consultation to guide health policy. Journal of Health Services Research and Policy. 2019;25(2):1-10
  4. 4. Greaves F, Millett C, Nuki P. England’s Experience Incorporating “anecdotal” reports from consumers into their National Reporting System: Lessons for the United States of what to do or not to do? Medical Care Research and Review Supplement. 2014;71(5):65S–80S
  5. 5. Gu D, Yang X, Li X, Jain H, Liang C. Understanding the role of Mobile internet-based health services on patient satisfaction and word-of-mouth. International Journal of Environmental Research and Public Health. 2018;15:1-23
  6. 6. Filieri R. What makes online reviews helpful? A diagnosticity-adoption framework to explain informational and normative influences in e-WOM. Journal of Business Research. 2015;68(6):1261–1270
  7. 7. Kilaru A, Meisel Z, Paciotti B, Ha Y, Smith R, Ranard B & Merchant R. What do patients say about emergency departments in online reviews? A qualitative study. BMJ Qual Saf. 2016;25:14–24
  8. 8. Donabedian A. Evaluating the Quality of Medical Care. The Milbank Quarterly. 2005:83(4);691–729. Reprinted from The Milbank Memorial Fund Quarterly (1966), 44 (3), 166–203
  9. 9. Argan M. Investigating word-of-mouth (WOM) factors influencing patients’ physician choice and satisfaction. International Journal of Medical Research & Health Sciences. 2016;5(1):191-198
  10. 10. Martin S. Word-of-mouth in the health care sector: a literature analysis of the current state of research and future perspectives. Int Rev Public Nonprofit Mark. 2017;14:35–56
  11. 11. Shah AM, Yan X, Shah SAA, Shah SJ, Mamirkulova G. Exploring the impact of online information signals in leveraging the economic returns of physicians. Journal of Biomedical Informatics. 2019;98:103-272
  12. 12. Lu N, Wu H. Exploring the impact of word-of-mouth about Physicians’ service quality on patient choice based on online health communities. BMC Medical Informatics and Decision Making. 2016;16(1):1–10
  13. 13. Donabedian A. The Quality of Care. JAMA. 1988;260(12):1743-1748. doi:10.1001/jama.1988.03410120089033
  14. 14. Direção Geral de Saúde (2013). Plano Nacional da Saúde 2012-2016. Maio 2013, 1-110
  15. 15. Donabedian A. Evaluating the Quality of Medical Care. The Milbank Quarterly. 2005:83(4);691–729. Reprinted from The Milbank Memorial Fund Quarterly (1966), 44 (3), 166–203
  16. 16. Rothenfluh F, Schulz PJ. Physician rating websites: What Aspects are important to identify a good doctor, and are patients capable of assessing them? A mixed-methods approach including physicians’ and health care consumers’ perspectives. Journal of Medical Internet Research. 2017;19(5):1–13
  17. 17. World Health Organization, OCDE & World Bank (2018). Delivering quality health services: a global imperative for universal health coverage
  18. 18. Grönroos C. A service quality model and its marketing implications. European Journal of Marketing. 1984;18(4):36–44
  19. 19. Parasuraman A, Zeithaml VA, Berry LL. A Conceptual Model of Service Quality and Its Implications for Future Research. Journal of Marketing. 1985;49(4):41-50
  20. 20. Parasuraman A, Zeithaml VA, Berry LL. SERVQUAL: A Multiple-Item Scale for Measuring Consumer Perceptions of Service Quality. Journal of Retailing. 1988;64(1):12-40
  21. 21. Tinoco PCR, Cândido CJF, Feliciano IMPL. A importância da avaliação da qualidade percebida em radiologia e o impacto dos fatores sociodemográficos num hospital público português. Port J Public Health. 2018;1-12
  22. 22. Cao X, Liu Y, Zhu Z, Hu J, Chen X. Online selection of a physician by patients: Empirical study from elaboration likelihood perspective, Computers in Human Behavior. Computers in Human Behavior. 2017;37 (C):403-412
  23. 23. Silva SC. Avaliação da qualidade de serviço numa organização de saúde privada em Lisboa [thesis]. Instituto Universitário de Lisboa; 2014
  24. 24. Akdere M, Top M, Tekingündüz S. Examining patient perceptions of service quality in Turkish hospitals: The SERVPERF model. Total Quality Management and Business Excellence. 2020;31(3–4):342–352
  25. 25. Despacho n.º 5739/2015 Diário da República, 2.ª série — N.º 104 — 29 de maio de 2015 Retrieved from:
  26. 26. Greaves F, Ramirez-Cano D, Millett C, Darzi A, Donaldson L. Use of Sentiment Analysis for Capturing Patient Experience From Free-Text Comments Posted Online. Journal of Medicine Internet Research. 2013a;15(11):e239:1-9
  27. 27. James T, Calderon E, Cook D. Exploring patient perceptions of healthcare service quality through analysis of unstructured feedback. Expert Systems with Applications. 2017;71:479-492
  28. 28. Boylan AM, Turk A, van Velthoven MH, et al. Online patient feedback as a measure of quality in primary care: a multimethod study using correlation and qualitative analysis. BMJ Open. 2020;10:1-10
  29. 29. Emmert M, Meszmer N, Sander U. Do health care providers use online patient ratings to improve the quality of care? Results from an online-based cross-sectional study. Journal of Medical Internet Research, 2016;18(9):1–14
  30. 30. Emmert M, Halling F, Meier F. Evaluations of Dentists on a German Physician Rating Website: An Analysis of the Ratings. J Med Internet Res. 2015;17(1):1-10
  31. 31. Kowalski R. Patients’ written reviews as a resource for public healthcare management in England. Procedia Computer Science. 2017;113:545–550
  32. 32. Wong C, Sap M, Schwartz A, Town R, Baker T, Ungar L, Merchant R. Twitter Sentiment Predicts Affordable Care Act Marketplace Enrollment. J Med Internet Res. 2015;17(2):1-8
  33. 33. Hong Y, Liang C, Radcliff T, Wigfall L, Street R. What Do Patients Say About Doctors Online? A Systematic Review of Studies on Patient Online Reviews. Journal of Medical Internet Research. 2019; 21(4):1-14
  34. 34. Yang H, Guo X, Wu T, Ju X. Exploring the effects of patient-generated and system-generated information on patients’ online search, evaluation and decision. Electronic Commerce Research & Applications. 2015;14(3):192-203
  35. 35. Patel S, Cain R, Neailey K, Hooberman L. General practitioners’ concerns about online patient feedback: Findings from a descriptive exploratory qualitative study in England. Journal of Medical Internet Research. 2015;17(12):1-15
  36. 36. Black E, Thompson L, Saliba H, Dawson K, Black N. An analysis of healthcare providers’ online Ratings. Informatics in Primary Care. 2009;17:249–253
  37. 37. Rastegar-Mojarad M, Ye Z, Wall D, Murali N, Lin S. Collecting and Analyzing Patient Experiences of Health Care From Social Media. JMIR Res Protoc. 2015;4(3):1-19
  38. 38. Emmert M, Meier F, Heider A, Dürr C, Sander U. What do patients say about their physicians? An analysis of3000 narrative comments posted on a German physician rating website. Health Policy. 2014;118:66–73
  39. 39. Emmert M, Meszmer N, Schlesinge M. A cross-sectional study assessing the association between online ratings and clinical quality of care measures for US hospitals: results from an observational study. BMC Health Services Research. 2018;18(82)
  40. 40. Grabner-Kräuter S, Waiguny M. Insights into the impact of online physician reviews on patients’ decision making: Randomized experiment. Journal of Medicine Internet Research. 2015;17(4):1-16. DOI: 10.2196/jmir.3991
  41. 41. Gillespie A, Reader TW. The healthcare complaints analysis tool: Development and reliability testing of a method for service monitoring and organisational learning. BMJ Qual Saf. 2016;25:937–946
  42. 42. Multilingual Open Information Extraction 17th Portuguese Conference on Artificial Intelligence, EPIA 2015, Coimbra, Portugal, September 8-11, 2015. Proceedings, 2015. Retrieved from:
  43. 43. Lockie M, Waiguny MKJ, Grabner-Kräuter S. How style, information depth and textual characteristics influence the usefulness of general practitioners’ reviews. Australasian Marketing Journal. 2015;23(3): 168–178
  44. 44. Bernardino M. Gestão em Saúde – Organização Interna dos Serviços. Coimbra: Edições Almedina; 2019, 243 p
  45. 45. Kirkpatrick W, Abboudi J, Kim N, Medina J, Maltenfor M, Seigerman D, et al. Beredjiklian, MD. An assessment of online reviews of hand surgeons. Arch Bone Jt Surg. 2017;5(3):139-144

Written By

Carla Marisa Ferreira Gomes, Marlene Paula Castro Amorim and Mário Jorge Ferreira Rodrigues

Submitted: December 20th, 2019 Reviewed: April 20th, 2021 Published: May 21st, 2021