Framework to Evaluate Level of Good Faith in Implementations of Public Dashboards

Abstract

To hold governments accountable to open government data (GD) standards, public dashboards need to be evaluated in terms of how well they meet public needs. To assist with that effort, this chapter presents a framework and rubric by which public dashboards can be evaluated for their level of good faith implementation. It starts by reviewing challenges to governments sharing data in good faith despite increasing open government data (OGD) policies and laws being put in place globally. Next, it presents a use-case in which the authors explain how they examined a public dashboard in their local context that appeared to be following OGD, but not in good faith, and developed an alternative implementation that appeared to increase the level of good faith. The framework and rubric proposed were used to successfully compare and contrast the level of good faith of both implementations, as well as another public dashboard described in the scientific literature, and to generate recommendations to increase the level of good faith. In conclusion, the utility of this framework and rubric for evaluating and comparing good faith in public implementations of dashboards was demonstrated, and researchers are encouraged to build upon this research to quantify the level of good faith in public dashboards as a way of increasing oversight of OGD compliance.

1. Introduction

There has been a global trend for populations to increasingly hold governments accountable to open government data (OGD) standards [1]. Because of this, governments have undertaken open data projects, such as providing public access to government data through publicly-accessible dashboards [2, 3]. However, government actors also may have an incentive to hide or obscure data, so there are barriers to accessing data for public dashboards [1]. This chapter focuses on the specific problem where governments attempt to demonstrate compliance with OGD standards through the presentation of a public dashboard, while at the same time, appearing to hide or obscure the data it is supposed to represent through poor dashboard design.

Our motivation to tackle this topic comes from our own disappointing experience trying to use a public dashboard implemented as part of OGD standards established where we live, in Massachusetts in the United States (US). Currently, in general, no standard guidance or recommendations are in place as a process to follow for the development of OGD public dashboards, and no framework or rubric has been proposed to evaluate them. These challenges are barriers to assessing how well public dashboards meet public need, and holding governments accountable for this. The significance of our contribution is that we propose a framework and rubric on which to base the evaluation of how well these public dashboards meet public need. The implication is that the application of this framework and rubric can be further researched in terms of utility in evaluating public dashboards. From this starting point, globally, we can begin to develop scientific consensus on what attributes in evaluate to a good-faith public dashboard implementation, and what the public should rightfully expect from the implementation of an OGD public dashboard.

2. Guidance for the design of public dashboards

The COVID-19 pandemic brought to attention a longstanding need for well-designed dashboards in public health and medicine [4]. It also brought to light that there are no uniform guiding principles behind developing publicly-facing dashboards intended to serve public interests. As a prime example, a recent review of United States (US) government public dashboards for COVID-19 found that “states engaged in dashboard practices that generally aligned with many of the goals set forth by the Centers for Disease Control and Prevention, Essential Public Health Services” (from abstract) [4]. However, the results of this review do not address whether the public was adequately served by any of these dashboards that were funded with the public’s money. Important questions not answered were: Did these dashboards meet the public’s information needs? Did they meet the information needs of public health practitioners? Or more importantly – whose information needs were these dashboards supposed to meet, and what were these needs?

2.1 Philosophies behind public dashboard design

At present, there is no overarching philosophy behind public dashboard design, for public health or other topics [2]. Although individual projects will publish use-cases where they discuss their design philosophy [2, 5, 6], there has not been an overall effort by the professional informatics societies or other academic groups to assemble principles behind the design for dashboards intended to serve the public. This may be because such an effort would be daunting, and would require a relatively narrow scope. The scope should be aimed at addressing high-level requirements focused on ensuring that the public’s needs are met by whatever dashboard solution is developed, regardless of the topic.

This chapter will attempt to summarize the literature into a framework that provides a general, generic rubric by which to evaluate how well a dashboard design for the public ensures that the public’s needs are met through measuring their adherence to high-level requirements. The framework will also put forth a method by which to compare alternative dashboard solutions aimed at meeting similar public needs as to how consistent the solution is with the public’s dashboard requirements.

The framework and rubric are intended to evaluate outcomes. Logically, a design process that adequately includes the public that the dashboard is intended to serve will inevitably produce a dashboard solution that meets these outcomes. Hence, there is no need to invest public funding in bloated efforts such as the Rapid Cycle Quality Improvement (RCQI) model, which is promoted by many health departments and organizations, and is extremely paperwork intensive [7, 8]. Part of what causes the RCQI model to be so effort-intensive is that it measures process outcomes. By contrast, the evaluation framework for public dashboards recommended in this chapter is streamlined, and focused on achieving a design solution, not a process solution.

Nevertheless, an optimal design solution will not be achieved without an adequate design process. Therefore, it is important to consider how the public should be involved in the process of designing public dashboards – especially those that are publicly-funded, and therefore have obligations to respond to the public’s needs.

2.2 Dashboard design process

As stated previously, there is currently no agreed-upon best-practices design process for dashboards in general, and public dashboards specifically [2]. Each time a dashboard is developed, a different design process is used. But a generic, logical process can be summarized in Figure 1.

Figure 1.

Generic logical dashboard design process. This design process produces an alpha prototype for initial testing, and a beta prototype for widespread field testing.

As shown in Figure 1, typically, before the dashboard is designed, some sort of design process is chosen, and this design process is followed to develop an “alpha prototype”. The alpha prototype represents a working mock-up that exists for the purposes of getting feedback and working out an initial design. Next, the alpha prototype undergoes a testing process to inform developers as to modifications that are necessary before widespread testing is done. Once those modifications are made, a beta prototype exists, and can be launched for field testing.

As described in Figure 1, depending upon the project, there can be different components included in the design process for the alpha prototype. First, there will be iterative design processes as part of designing the alpha prototype, as well as the development of design documentation and the actual creation of the prototype. The details behind each of these components will vary by project. Once the alpha prototype exists, the process to convert it to the beta prototype involves some sort of user testing, and some sort of evaluation for adherence to standards. Granted, an alpha prototype may be released into the field without having undergone the beta prototype process, but that means it has not been user-tested or evaluated for adhering to standards.

This logical process can apply to any dashboard development effort. As one example, researchers aimed to design a dashboard for clinicians [9]. They wrote requirements and developed an alpha prototype, then worked with clinicians to gain feedback to guide the development of a beta prototype (which would presumably be developed in the future and field-tested) [9]. This article focused mainly on the feedback process to improving the alpha prototype, but the focus of articles can be on any part of the dashboard design process. Another article focused on the development of a beta prototype aimed at both the public and leaders for real-time decision-making related to traffic flow [2]. While the beta prototype was developed and appeared ready for testing, the article did not report any results, so the current final stage of this project was not evident in the article [2].

Although, this logical design process should theoretically involve the intended users of the dashboard, and prototypes should undergo iterative testing, this is not always the case with public dashboards. Because public dashboards often involve government agencies and leaders at some level, whether as data sources or as intended audiences, these forces can have unintended impacts on the dashboard design and quality.

2.3 Governmental data suppression and misrepresentation

As a general trend, consumers are demanding more data transparency, and calls are being made for governments to make data available for public oversight [1]. Likewise, there is an increasing trend toward using dashboards for empowering the public [2, 3]. Not only do dashboards of public data provide a mechanism for public oversight of leaders, but they also reduce information asymmetry, which refers to the circumstance in which one party (the government) has more information than another party (the public), thus disempowering them [2, 10, 11].

However, governments are not always keen to share the data for various reasons. It has been argued that government agencies will be more likely to comply with open government data (OGD) practices if they see it as an opportunity to showcase their agency’s success [1]. However, if the agency believes the data will cast the agency in a negative light, the agency may be less likely to be inclined toward OGD practices. Ruijer and colleagues recommend that institutional incentives and pressure be created for OGD, because governments have a natural interest in suppressing data they think may be harmful to them in some way if analyzed [1].

However, data suppression is not the only method governments employ to prevent data use and interpretation. One limitation of legal requirements for OGD is that the agency may comply with the requirements in bad faith. During the COVID-19 outbreak in early 2020, a state epidemiologist in Florida said she was fired for refusing to manually falsify data behind a state dashboard [12]. Simply reviewing the limitations of big data can reveal ways to share big data in bad faith in a dashboard, such as visualizing too much data, visualizing incomprehensible or inappropriate data, and not visualizing needed data [13].

For this reason, in addition to holding governments to OGD standards, government efforts need to be evaluated as to whether or not they meet OGD standards in good faith. The framework presented here guides as to how to evaluate good vs. bad faith implementations of a public dashboard.

2.4 Dashboard requirements

The evaluation framework presented has six principles on which to judge the level of good or bad faith in a public dashboard: 1) ease of access to the underlying data, 2) the transparency of the underlying data, 3) approach to data classification, 4) utility of comparison functions, 5) utility of navigation functions, and 6) utility of metrics presented. These principles will be described below.

2.4.1 Access to underlying data

A dashboard is essentially a front-end, with data behind it being visualized [14]. Hence, once a dashboard is published, members of the public may want to access the underlying data for various reasons, including oversight of the dashboard. But governments resistant to data-sharing may use the dashboard in bad faith as a firewall between the public and the underlying data to prevent data access [1]. Hence, good faith OGD principles hold that public dashboards should not serve as barriers, but instead serve as facilitators to access the underlying data being visualized in the dashboard.

2.4.2 Transparency of underlying data

Although raw data are used for the dashboard, in the dashboarding process, they undergo many transformations to be properly visually displayed [9, 14]. The processing of the data can develop calculations that are then displayed in the dashboard. Therefore, to be transparent, the dashboard must not only facilitate access to the underlying raw data, but also to the transformations the data underwent in being displayed. A simple way to accomplish this kind of transparency is to use open source tools and publish the code, along with documentation [14]. This allows citizen data scientists an opportunity to review and evaluate the decisions made in the dashboard display.

2.4.3 Data classification

How data are classified in a dashboard can greatly impact the utility of the dashboard. As an example, developers of an emergency department (ED) dashboard that was in use for five years under beta testing found that after the ED experienced an outbreak of Middle East Respiratory Syndrome (MERS), major structural changes were needed to the dashboard [15]. Another paper about developing a visualization of patient histories for clinicians described in detail how each entity being displayed on the dashboard would be classified [16]. Hence, inappropriate classifications or ones deliberately made in bad faith can negatively impact data interpretation to the point that the dashboard could be incomprehensible to its users.

2.4.4 Comparison functions

Dashboards are used to inform decision-making, and therefore, being able to make needed comparisons is an important factor in a dashboard’s usability [14, 17]. As an example, the public traffic dashboard described earlier presented visualizations of the ten most congested areas of the city, as well as textual feedback on the two most suitable routes between downtown and outlying areas, to provide optimal comparators to allow the public to make the most-informed route decision [2]. While ultimately, optimal design choices could be debated, it is easy to conceive of how agencies looking to maintain opacity could obscure data interpretation in a dashboard in bad faith by deliberately limiting the ability to make useful comparisons.

2.4.5 Navigation functions

Dashboards are typically at least somewhat interactive, providing the user the ability to navigate through the data display, which responds to actions by the user [14, 18, 19]. When operating in good faith, developers often conduct extensive usability testing to ensure that the dashboard is intuitive to use in terms of navigating through the data display, and that any interactivity is useful [15]. But when implemented in bad faith, a dashboard could be designed to deliberately confuse the user as to how to navigate and interpret the data in the dashboard.

2.4.6 Metrics presented

One of the main purposes of dashboards is to present metrics that represent statistics or visualizations meant to summarize a particular state of the data [14, 15]. For example, in the traffic dashboard, the metrics presented are intended to communicate traffic congestion to the users, while the metrics presented in the clinical dashboard are intended for healthcare workers to use in clinical decision-making [2, 16]. In a good-faith effort, developers may conduct extensive user-testing to ensure that the metrics presented are communicative to dashboard users, as is often done with dashboards developed to serve workers in healthcare [9, 20]. However, in a bad faith effort, the metrics presented could be deliberately confusing to the user, and serve merely to hide ugly truths in the underlying data.

3. Use-case: dashboard for hospital-acquired infection rates at Massachusetts hospitals

In the US, the Commonwealth of Massachusetts (MA) Department of Public Health (DPH) posts annual healthcare-associated infection (HAI) reports about MA hospitals on their web page [21]. The purpose of the reports is public data transparency by law [21]. Briefly, HAI, such as catheter-acquired urinary tract infection (CAUTI), central line-associated bloodstream infections (CLABSI), and surgical site infections (SSI), are serious issues because they are the fault of the hospital, and can lead to sepsis, which is a systemic infection that can end in death [22, 23]. Since catheterization happens in the intensive care unit (ICU) setting, typically hospitals track CAUTI and CLABSI as part of healthcare quality activities centered around ICUs [14]. By contrast, SSIs are tracked in association with specific operative procedures (e.g., colon surgery) [24].

In the US compared to other countries, rates of HAI are relatively high, likely because they are not required to be tracked at the federal level [14, 25]. Hospitals can opt into the federal voluntary tracking system called the National Health Safety Network (NHSN), but the NHSN does not have a publicly-facing dashboard, and the data are inaccurate, especially in undercounting severe events [14, 24, 26]. As HAI is a serious public health issue, there has been a call for greater data transparency, so the reports posted on the MA DPH web site represent MA’s attempt to comply with state-level mandates for OGD.

Although summary reports are available for download on the MA DPH web site, it is not possible to access hospital-level reports directly from the web site. To download hospital-level reports, the user must access a dashboard presented on the web page in a link (Figure 2).

Figure 2.

MA HAI public dashboard landing page. Note: “A” labels a menu of tabs that can be used for navigation to view metrics on the various hospital-acquired infections (HAIs). In panel labeled “B”, tabs can be used to toggle between viewing state-level metrics and hospital-level metrics. Hospitals can be selected for display using a map labeled “C”.

As per Figure 2, once inside the dashboard, individual PDF-style reports can be found through navigation to the hospital in question, and the reports appear to present formatted output from a database. One way to navigate to the hospital record is to locate it on the map (“C” in Figure 2) and click on its icon, causing the panel “B” to display hospital-level metrics and a link to the hospital’s PDF report.

Each PDF report has a header displaying attribute data about the hospital (e.g., number of beds), followed by a series of ICU-, procedure-, and infection-level output. This mirrors the structure seen in the dashboard tabs and reports (Figure 2, “A” and “B”). For the ICUs at each hospital, the report displays a set of tables summarizing CAUTI and CLABSI rates, followed by time-series graphs. For a set of high-risk surgical procedures, SSI rates and graphs for the hospital are displayed. Medication-resistant staphylococcus aureus (MRSA) and C. Difficile infections are serious HAIs that can be acquired in any part of the hospital and are diagnosed using laboratory tests [27]. Rates and graphs of MRSA and C. Difficile infections are also displayed on the report.

The underlying data come from the NHSN. This is not stated on the dashboard. Instead, there is a summary report and presentation posted alongside the dashboard on the web site, and the analyses in these files are based on NHSN data [21]. It seems that the DPH is using this NHSN data using as a back-end to the dashboard, and the dashboard is an attempt to comply with OGD laws.

Because the authors are aware of the high rates of HAI in the US, and because we both live in MA and we both are women who are cognizant that sexism in US healthcare adds additional layers of risk to women [28], we identified that we were in a state of information asymmetry. Specifically, we had the information need to compare MA hospitals to choose the least risky or “lethal” one for elective surgery or childbearing (planned procedures), but we felt this need was not met by this OGD implementation.

In this section, we start by evaluating the existing MA DPH HAI dashboard against our good vs. bad faith framework. Next, we propose an alternative dashboard solution that improves the good vs. bad faith features of the implementation.

3.1 Considering existing dashboard: design process and requirements

Figure 3 provides a logical entity-relationship diagram (ERD) for the data behind the dashboard.

Figure 3.

Logical entity-relationship diagram for data behind dashboard. Note: The schema presented assumes four entities: The hospital entity (primary key [PK]: HospRowID), each intensive care unit (ICU) attached to a hospital which contains the frequency of infection and catheter days attributes to allow rate calculation (PK: ICURowID), each procedure type attached to a hospital (to support the analysis of surgical site infection [SSI], with PK: ProcRowID), and each other infection type at the hospital not tracked with ICUs (PK: LabID).

As described earlier, the landing page (Figure 2) provides a map by which users can select a hospital, causing the metrics for the hospital to appear in a panel. The user chooses which measurement to view (e.g., CAUTI) through navigation using the tabs. This suggests the dashboard is aimed at individuals with a working knowledge of MA geography who are intending on comparing and selecting hospitals least likely to cause HAI for an elective procedure (e.g., childbearing), or to establish as their top choice of the local hospital should they ever need to be admitted. This interface makes it difficult to compare HAI at different hospitals, because metrics from more than one hospital cannot be viewed at the same time. Further, metrics about different HAIs at the same hospital are on different panels, so within-hospital comparisons cannot be facilitated. There appears to be no overall metric to use by which to compare hospitals in terms of their HAI rates.

Figure 4 shows an example of the metrics reported by each hospital on the dashboard reporting panel (“B” in Figure 2). The figure also shows one of the two tables and one of the two figures displayed on the CAUTI tab for the selected hospital. In all, two tables and two figures are displayed in portrait style in panel “B” (Figure 2), and Figure 4 shows the top table and figure displayed. In the table displayed (labeled “1” in Figure 4), the metrics presented are the number of infections, predicted infections, standard infection ratios (SIRS), a confidence interval for the SIR, and an interpretation of the level. The figure (labeled “2” in Figure 4) displays a time-series graph of SIRs for the past five years. In the other table on the panel (not shown in Figure 4), ICU-level metrics are provided about catheter-days, predicted catheter-days, Standard Utilization Ratios (SURs) and their confidence interval, and an interpretation, and an analogous time-series graph of five years of SURs is presented (also not shown in Figure 4).

Figure 4.

Dashboard metric display for each hospital. Note: To view hospital-acquired infection (HAI) rates at hospitals, a hospital is selected (Figure 2, panel “C”), then the user selects the tab for the HAI of interest. In Figure 4, a hospital has been identified, and a tab for catheter-associated urinary tract infection (CAUTI) has been selected (see circle). Two tables and two figures are presented in portrait format on the reporting panel for each hospital (Figure 2, panel “B”). Figure 4 shows the first table and figure presented (“1” and “2”); the table reflects stratified metrics for CAUTI at each ICU at the hospital, and the graph reflects a time series of these metrics stratified by hospital vs. state levels, and intensive care unit (ICU) vs. ward (“ward” is not defined in the dashboard). The metrics provided in “1” are a number of infections, predicted infections, standard infection ratios (SIRs), a confidence interval for the SIR, and an interpretation of the level. In “B”, the SIR is graphed.

SIRs and SURs are not metrics used typically by the public to understand rates of HAI in hospitals. Risk communication about rates to the public is typically done in the format of n per 10,000 or 100,000, depending upon the magnitude of the rate [29]. Further, stratifying rates by ICU is confusing, as prospective patients may not know what ICU in which they will be placed. Because the hospital environment confers the strongest risk factors for HAI (e.g., worker burnout), HAI rates will be intra-correlated within each hospital [30]. Therefore, it is confusing to present all these rates and stratify them by ICU. Figure 4 only displays 50% of the information available about CAUTI at one hospital. With each tab displaying similar metrics about SSI and other infections, the experimental unit being used is so small, it obfuscates any summary statistics or comparisons. Also, it is unclear how the “predicted” metrics presented were calculated.

Ultimately, the design process and requirements behind this dashboard are not known. There is no documentation as to how this dashboard was designed, and what it is supposed to do. It appears to be an alpha prototype that was launched without a stated a priori design process, and without any user testing or formal evaluation.

3.2 Alternative design: design process and requirements

We chose to redesign the dashboard into a new alpha prototype that met requirements that we, as members of the public, delineated. Consistent with the good faith principles proposed, our requirements included the following: 1) the dataset we use should be easily downloadable by anyone using the dashboard, 2) the documentation of how the dashboard was developed should be easy to access, 3) hospitals should present summary metrics rather than stratified ones, 4) different HAI metrics for the same hospital should be presented together, and 5) there needs to be a way to easily compare hospitals and choose the least risky hospital. To do this, we first obtained the data underlying the original dashboard. Next, we analyzed it to determine better metrics to present. We also selected open-source software to use to redeploy an alpha prototype of a new dashboard. Finally, we conducted informal user testing on this alpha prototype.

3.2.1 Obtaining the data from the original dashboard

Scraping was done in open-source RStudio and predominantly used packages pdftools, and pdftables [31, 32]. All the PDF reports from each hospital were downloaded and placed in one directory. As a first step, a loop was used with the pdftools package which crawled through each report extracting the data into memory. This was done in conjunction with the pdftables package, which is essentially an online application that applies structure to the unstructured tabular data placed in memory from pdftools. To use this online application, an application programming interface (API) key is issued from the PDF Tables web site, and is used in the RStudio programming to pass the data to the online application [33]. The code resulted in the data being processed into a series of unstructured *.xlsx files and downloaded locally. Then, in a final data cleaning step, data were transformed into the tables in the format specified in Figure 3.

3.2.2 Determining metrics to present

We chose to focus our inquiry on the data from the hospital and ICU tables, as CAUTI and CLABSI are by far the most prevalent and deadly HAIs [23]. Therefore, we scoped our alpha prototype to only display data from the ICU and the hospital tables (although we make all the data we scraped available in the downloadable dataset). This limited us to basing the dashboard on hospital- and ICU-level metrics only.

Next, we intended to present CAUTI and CLABSI frequencies and rates, whereby the numerator would be the number of infections, and the denominator would be the “number of patients catheterized”. We felt that the dashboard’s use of catheter-days as the rate denominator was confusing to the public, and appeared to attenuate the prevalence of patients having experienced a CAUTI or CLABSI. Although, “number of patients catheterized” was not available in the data, “annual admissions” was. Since the proportion of patients admitted annually who are catheterized probably does not vary much from hospital to hospital, we chose to use the number of admissions as the denominator and a proxy measurement.

Third, we needed to develop a way of sorting hospitals as to their likelihood of causing an HAI to allow easy comparisons by public users, so we decided to develop an equation to predict the likelihood of an HAI at the hospital. We did this by developing a linear regression model with hospital-level attributes as independent variables (IVs), and CAUTI rate in 2019 as the dependent variable (DV). We chose CAUTI over CLABSI after observing the two rates were highly correlated, and CAUTI was more prevalent.

Table 1 describes the candidate IVs for the linear regression model. The table also includes the source of external data that were added to the hospital data. We studied our IVs, and found serious collinearity among several variables, so we used principal component analysis (PCA) to help us make informed choices about parsimony [37]. The data predominantly loaded on three factors (not shown). The first factor included all the size and utilization variables for the hospital; these were summed into a Factor 1 score. The second-factor loadings included the proportion of those aged 65 and older and the non-urban flag (Table 1), so those were summed as Factor 2. Proportion non-White was strongly inversely correlated with Factor 2, so it was kept for the model, and county population did not load, so it was removed from the analysis. Factor 3 loadings included teaching status, for-profit status, and Medicare Performance Score (MPS). Rather than create a score, we simply chose to include the variable from Factor 3 that led to the best model fit to represent the factor, which was MPS. Then we finalized our linear regression model, and developed a predicted CAUTI rate (ŷ) using our model that included the following IVs: MPS, Factor 1 score, Factor 2 score, and proportion of non-White residents in hospital county.

Variable	Source	Role	Source
Teaching hospital status	Original dashboard	Exposure	Scraped data from dashboard
Hospital profit status	Original dashboard	Confounder	Scraped data from dashboard
Measurements of hospital size (number of beds)	Original dashboard	Confounder	Scraped data from dashboard
Measurements of hospital utilization (number of admissions, number of patient days)	Original dashboard	Confounder	Scraped data from dashboard
Medicare Performance Score	Medicare performance score dataset [34]	Confounder	Medicare performance score dataset
Non-urban county	Rural health information hub [35], United States census [36]	Confounder	Scraped data from dashboard to determine county, then application of hospital rurality flag developed from public data based on county
Hospital county population size	United States census [36]	Confounder	Census measurements
Hospital county population proportion below the poverty level	United States Census [36]	Confounder	Census measurements
Hospital county population proportion below the poverty level	United States Census [36]	Confounder	Census measurements
Hospital county population proportion of non-White residents	United States Census [36]	Confounder	Census measurements
Numerators for rates – infection frequencies	Original dashboard	Create outcome	Scraped data from dashboard
Infection rates	Original dashboard	Outcome	Calculated as the numerator for rates divided by admissions

Table 1.

Conceptual model specification.

Next, we used the regression equation to calculate ŷ as a “lethality score” for each hospital. Of the 71 hospitals in the dataset, 21 were missing MPS and 8 were missing other data in the model. Therefore, only the 42 with complete data (IVs and DVs) were used to develop the regression model. As a result, the lethality score was non-sensical for some hospitals; where the residual was large, the lethality score was replaced with the 2019 CAUTI rate. If CAUTI data were missing, it was assumed that the hospital had no CAUTI cases, and therefore was scored as 0.

Once the lethality score was calculated, we chose to sort the hospitals by score, and divide them into four categories: least probable (color-coded green), somewhat probable (color-coded yellow), more probable (color-coded red), and most probable (color-coded dark gray). Due to missing CAUTI information and many hospitals having zero CAUTI cases, our data were severely skewed left, so making quartiles of the lethality score to divide the hospitals into four categories was not meaningful. To compensate, we sorted the data by lethality score and placed the first 23 hospitals (32%), which included all the hospitals with zero cases, in the least probable category. We placed the next 16 (23%) in somewhat probable, the next 16 (23%) in more probable, and the final 16 (23%) are most probable. We chose to use this classification in data display on the dashboard to allow for easy comparison between hospitals of risk of a patient contracting HAI.

3.2.3 Choice of software and display

R is an open-source analytics software that allows for user-developed “packages” to be added a la carte to the main application [38]. RStudio was developed to be an integrated development environment (IDE) for R that allows for advanced visualization capabilities that support dashboard development [39]. In RStudio, web applications like dashboards can be placed on a host server and deployed on the internet such that as users interact with the web front-end, it can query and display data from the back-end hosted on the server.

The package Rshiny [40] was developed to support dashboarding in RStudio, and can work with other visualization packages depending upon the design goals of the dashboard. In our newly designed dashboard, the package leaflet was used for a base map on which we placed the hospital icons (like the original dashboard), and add-ons were made to display other items. These add-ons were adapted from other published codes [41]. JavaScript with wrapper DT (data table) was used to display stratified ICU rates, and CSS was used for formatting.

The dashboard we developed was deployed on a server (https://natasha-dukach.shinyapps.io/healthcare-associated-infections-reports/) and code for the dashboard was published (https://github.com/NatashaDukach/HAI-MA-2019). When accessing the link to the dashboard, the user initially sees a map with icons (in the form of dots) on it indicating hospitals. The icons are color-coded according to the lethality score described previously. Clicking on an icon will expand a bubble reporting information about the hospital (Figure 5).

Figure 5.

Alternative dashboard solution. Note: In our new version, two tabs are created (see “A”). The figure shows the first tab titled “ICU Rate Explorer”. The second tab, titled “data collection”, has information about the design of the dashboard and links to the original code. Each of the hospitals is indicated on the map by a color-coded icon that can be clicked on to display a bubble. The legend by “B” displays our color-coding scheme. When clicking on a hospital icon, hospital-level metrics are shown in a bubble, and there is a link that leads to the display of intensive care unit (ICU)-level metrics (see “C”).

As shown in Figure 5, like the original dashboard, this one has a map for navigation. Unlike the original, it only has two tabs: “ICU Rate Explorer” (the one shown in Figure 5), and “Data Collection”, which provides documentation and links to original data and code (see “A” in Figure 5). The hospital icons are placed on the map and coded according to our color scheme (see legend in Figure 5 by “B”). This allows for easy comparison between hospitals. When clicking on an icon for a hospital, a bubble appears that contains the following hospital metrics: Number of admissions, number of ICU beds, overall CAUTI rate, and overall CLABSI rate. There is also a link on the bubble where the user can click to open a new box that provides CAUTI and CLABSI rates stratified by ICU. Future development plans include adding other overall rates (e.g., for SSI), and adding in data from previous years to allow for the evaluation of trends.

3.2.4 User testing

Informally, members of two potential user bases were queried as to their reactions to the differences between the two dashboards: members of the academic public health space, and members of the MA public. When the dashboard redesign was pitched as a project to public health academics, it was dismissed as an unimportant escapade for various reasons. Some reasons cited were lack of agreement on terminology and patient safety priorities, the challenges with undercount of HAIs in NHSN data, and differential reporting accuracy in teaching vs. non-teaching hospitals. Academics also acknowledged that the system for tracking, addressing, and preventing HAIs is hopelessly broken in the US, and therefore it seems a waste of time to prop up such a system when it produces inaccurate data.

A few members of the MA public who are familiar with technology also provided informal feedback about the utility of the dashboard from a patient standpoint. They reported that the alternative solution was more intuitive than the original, and did a better job of representing the highly limited data from the NHSN.

These differences in reactions underscore the challenge of OGD and ensuring that public dashboards are developed and deployed in good faith. Those from the public health field expressed that since the system is broken and the data are inaccurate, they should be dismissed, while those in the public felt that since the data existed, it should be accessible, even if it was not completely accurate. It not only highlights the differing perspectives of those on either side of information asymmetry, it glaringly illustrates how those who are being held accountable by the usage of the data see dashboarding differently than those who are using the data to do oversight and accountability.

4. Application

We wanted to compare the original HAI dashboard with the one we developed based on the good faith principles described earlier. We started by creating the framework presented in Table 2, which guides as to the good faith and bad faith characteristics of public dashboards.

Dashboard function/characteristic	Good faith	Bad faith
Access to underlying data	Easy to download analytic dataset on which dashboard rides in *.csv format (e.g., through report export function) Easy to access or download component datasets that went into the analytic dataset on which the dashboard rides	Lack of downloading functions, or downloading functions that provide only reports in non-tabular and non-data format Lack of transparency on how the analytic dataset was developed Lack of transparency about source datasets
Transparency of underlying data	For each native variable used in the dashboard, its source dataset is specified, and a link is given if available. For each calculated variable used in the dashboard, clear documentation is available. It is clear which data reflect real measurements, and which reflect simulations, imputations, or predictions	Source datasets may be specified, but little information about the use of their variables in the dashboard is provided It is not made clear which dashboard variables are calculated, and how they are calculated is also not made clear It is not clear which data reflect real measurements, and which data have been simulated, imputed, or otherwise fabricated
Data classification	Data are classified in ways that are intuitive to the consumer, and results are presented according to those classifications	Data are classified in ways that either make the development work easier for the analyst, or serve to mask negative indicators Data are not grouped into classifications consumers use, making it impossible to obtain summary statistics for these classification levels
Comparison functions	Dashboard allows for comparisons that provide useful consumer decision-support	Dashboard prevents comparisons that would provide consumer decision-support, or makes them very difficult to make using the dashboard functions
Navigation functions	Navigation functions reflect how users conceive of accessing the entities in the dashboard Specifically, map navigation reflects how users conceive of their geographic locale when searching for information This allows consumers to intuitively ingest and assimilate information as they interact with the dashboard	Navigation functions reflect how public officials want consumers to navigate the entities in the dashboard This forces consumers to think differently about the topic, and disrupts their ability to ingest and assimilate information Map functions force the consumer to conceive of their geographic locale in an unintuitive way, making map navigation confusing
Metrics presented	Metrics are intuitive to consumers Metrics are presented in such a way that they are intuitive to ingest and assimilate Metrics that are presented to make comparisons between entities intuitive to support decision-making	Metrics reflect jargon, and are unintuitive to consumers Metrics require consumers to read documentation to understand Metrics are presented in such a way as to be confusing, making them impossible to be used for decision-support

Table 2.

Proposed framework for evaluation of good faith and bad faith public dashboards.

Using this framework, we applied a rating system. We chose zero to represent “neither bad faith nor good faith”, −5 to represent “mostly bad faith” and + 5 to represent “mostly good faith”. Then, based on our experience and available information, we rated the original MA HAI dashboard and our alternative dashboard solution to compare the ratings. To experiment with applying our framework to another public dashboard, we used the information published in the article described earlier to rate the traffic dashboard which was in Rio de Janeiro [2]. Our ratings appear in Table 3.

Dashboard function/characteristic	MA Hosp	Alt. MA Hosp	Comment: MA Hosp vs. Alt. MA Hosp	Rio	Comment: MA Hosp vs. Rio
ACCESS TO UNDERLYING DATA	−5	5	The original solution had no access to underlying data (except by way of PDF-style reports). Alternative solution posts data publicly for download.	5	Rio dashboard uses open data from City Hall with user-generated content collected through Waze
Transparency of Underlying Data	−5	3	It was difficult to identify the source of the data in the original solution. The alternative solution uses the same data, which is from NHSN. Because NHSN itself is somewhat opaque, the final solution lacks transparency.	0	Unclear from the article, but it appears that it is possible to audit Rio dashboard design if a member of a certain role (e.g., public data scientist). Not all tools used were open source.
Data Classification	0	3	In informal user testing, public users found the data classifications much more intuitive in the alternative compared to the original solution. However, formal user testing was not conducted.	5	Much effort was made to classify data in Rio dashboard to make it useful for the public to make route decisions.
Comparison Functions	−5	5	In informal user testing, users found the comparison function in the alternative solution useful for decision-making, and could not find a comparison function in the original solution.	5	Rio dashboard was designed to allow the public to make comparisons about potential traffic routes.
Navigation Functions	0	5	In informal user testing, users reported being able to easily navigate the data and dashboard in the alternative solution, but having extreme difficulty in navigating the original solution.	5	Rio dashboard for the public had a very simple, intuitive interface with images and only a few metrics critical to decision-making. This made it possible to easily navigate the dashboard display.
Metrics Presented	−5	5	In informal testing, users indicated that they did not understand the metrics presented on the original solution but found the color-coding of the alternative solution intuitive for decision-making.	3	The few metrics presented on the Rio dashboard were geared specifically to helping the public make route decisions based on traffic metrics. However, no formal user testing is presented.
Total	−20	26		23

Table 3.

Application of rating system.

Note: MA Hosp = original hospital-acquired infection (HAI) dashboard from the Commonwealth of Massachusetts (MA), MA Hosp Alt. = alternative solution, Rio = Rio traffic dashboard [2], and NHSN = National Healthcare Safety Network.

As shown in Table 3, using Table 2 as a rubric and our rating scale, we were able to rate each dashboard and assign a score. We were also able to define in the comments in the table the evidence on which we based our score. Table 3 demonstrates that this framework can be used to compare two different alternatives of a public dashboard displaying the same data, as well as two completely different public dashboards. The total scores show that while our redesigned prototype of the HAI dashboard had a similar level of good faith implementation compared to the Rio traffic dashboard (scores 26 vs. 23, respectively), the original HAI dashboard had a very low level of good faith implementation compared to the other two (score − 20).

5. Discussion

As is consistent with the global trend, the state of MA implemented an OGD requirement to share HAI data with the public through posting a public dashboard on a web page. However, as residents of MA, the authors found that this dashboard did not serve our information needs, and essentially obscured the data it was supposed to present. To address this challenge, we not only redesigned the dashboard into a new prototype, but we also tested our proposed framework for evaluating the level of good faith in public dashboards by applying it. Using our proposed framework and rubric, we evaluated the original HAI dashboard, our redesigned prototype, and a public dashboard on another topic presented in the scientific literature on the level of good faith implementation. Through this exercise, we demonstrated that the proposed framework is reasonable to use when evaluating the level of good faith in a public dashboard.

The next step in the pursuit of holding governments accountable for meeting OGD standards in public dashboards is to improve upon this framework and rubric through rigorous research. As part of this research, entire groups of individuals could be asked to score dashboards on each of these characteristics, and the results could easily be summarized to allow an evidence-based comparison between dashboards. Results can be easily visualized in a dumbbell plot (using packages ggplot2, ggalt, and tidyverse [42, 43, 44]), which we have done with our individual scores, but could be done with summary scores (Figure 6).

Figure 6.

Example of visualization of framework score comparison. Note: MA Hosp = original hospital-acquired infection (HAI) dashboard from the Commonwealth of Massachusetts (MA), MA Hosp Alt. = alternative solution, and Rio = Rio traffic dashboard [2].

As visualized in Figure 5 and summed in Table 3, our scoring system suggested that the alternative HAI dashboard we developed was done in a level of good faith (score = 26) similar to that of the Rio traffic dashboard (score = 23), and that the original HAI dashboard appears to not have been done in good faith (score = −20), and may serve as a governmental attempt to hide or obscure uncomfortable data. This exercise shows that the framework and rubric developed can be used to compare the level of good faith in public dashboards, and to provide evidence-based recommendations on how governments can improve them so they meet both the spirit and the letter of OGD requirements.

6. Conclusion

In conclusion, in this chapter, we describe the challenge of holding governments accountable for developing public dashboards to meet OGD requirements in a way that also serves the public’s information needs. To address this challenge, we propose a framework of six principles of good faith OGD by which public dashboards could be evaluated to ensure data shared by the government under OGD policies and laws are done so in good faith. We also demonstrate applying this framework to the use-case of a public dashboard intended for residents of MA in the US to use to compare and select hospitals based on their HAI rates. As a demonstration, we present our redesign of the dashboard, then use a rubric based on the framework to score and compare the original dashboard and our alternative in terms of levels of good faith OGD. We also demonstrate using the rubric on a published use-case in the literature. As our framework and rubric provide a reasonable starting point as a method for evaluating and comparing the level of good faith in public dashboards, we strongly recommend that future research into this topic consider our framework and rubric, and build upon it through gathering evidence in the field.