Evaluating the Efficiency of a Collaborative Learning Network in Supporting Third Sector Organisations in the UK

Abstract

Third-sector organisations are a collective term for voluntary and community services, charities, and social enterprises. Within the UK and internationally, a subset provides a crucial and ever-expanding role in mental health care provision, delivering valuable client and community-led services. However, in the UK these organisations are under increased pressure to demonstrate their value, and many are constrained by scarce resources and lack of expertise. The Service Improvement Learning Collaborative was conceived as an innovative model for shared learning to enhance the value of this sector in mental health care support, generating a valuable resource of practice-based learning and promoting the implementation of effective practices. The initiative combines a collaborative learning model with mentorship support and in-depth, data analytical profiling. This collaborative involved a network of six organisations focused on exploring the maximisation of data quality, the minimisation of client attrition, and the optimisation of clinical outcomes. Evaluating the collated data helped identify the many unique challenges facing the sector and evidenced the model as a pragmatic solution for service quality improvement. This chapter provides an overview of the project’s methodology, including its underlying rationale, first year of operation, and value of experiential learning for the field.

1. Introduction

In the UK, Third-Sector Organisations (TSOs) are a collective term for voluntary and community agencies, charities, and social enterprises, of which a sub-section provides health and social care via independent and value-driven services [1]. Recent audits of the whole sector reveal a notable presence, with over 160,000 organisations and nearly 1-million employees and volunteers operating in the UK [1]. Across many high-income countries, it is an area which is growing rapidly as governments seek to harness their innovation and local capabilities [1, 2]. Given their nature, TSOs tend to be highly regarded for their proximity to the community, welcoming facilities, and the ability to engage those with complex and chronic needs [1, 2, 3, 4].

Despite the potential benefits of TSOs, little research has been undertaken to evidence their impact and effectiveness [2, 3]. Research applicable to many mental health care TSOs in the UK, including systematic reviews [2], national audits [1] and interviews with mental health charities [3], highlight the clinical and economic barriers affecting the production and utilisation of practice-based evidence (PBE). Many are constrained by tight budgets and scarce resources and often exist as ‘micro-entities’ making bidding processes and research prohibitively expensive [1, 4]. The evidence that has been produced has been characterised as low in quality, lacking methodological rigour, theoretical modelling, and reliance on non-representative stakeholder feedback [2, 3]. Access to learning is equally challenging with constraints on resources to review the latest research literature [3, 4].

For TSOs to overcome these challenges, there must be greater alignment of needs and priorities between providers, commissioners, policymakers and academic institutions. One approach to optimising the production and sharing of knowledge has been to form collaborative learning networks (CLNs) of services using a similar treatment model or methodology for generating evidence [5]. By partnering with similar providers, these networks enable organisations to explore, share and integrate learning across a network, maximising the potential for practice-based learning. CLNs have demonstrable potential within the UK mental health care sector, having reported success in the Improving Access to Psychological Therapy (IAPT) programme [6] and Children and Young People’s [5] services. The IAPT programme, which is a national government-funded initiative for English primary mental health services, has been an influential driver in generating public domain service performance data. Having mandated sessional measurement across all services over a decade ago, it has recently achieved pre-and-post outcomes completion rates of 98% for clients completing therapy [7]. These high levels of data completeness are essential for supporting CLNs [6].

The quality implementation framework (QIF) [8] has been previously used as a schematic structure to introduce practice changes, including routine outcome monitoring (ROM), within mental health care services [9]. This model synthesises 25 implementation methods from almost 2000 evaluation reports, comprising 4 action phases and 14 critical steps [8]. Combined with research on the value of CLNs, an initiative was undertaken to bring together multiple TSOs delivering mental health care to enhance service quality. This chapter describes the rationale, process, and outcome of this initiative across its initial start-up and first year of operation using a traditional storytelling structure, with reference to the QIF [8] and other implementation frameworks [10, 11, 12, 13].

1.1 Telling stories

Implementation science is the scientific study of techniques to enhance the quality and effectiveness of health services by advancing the systematic uptake of evidence-based practice (EBP) in routine clinical settings [14]. The learning from the field demonstrates the gap between what is shown to be effective to what is implemented in practice [14]. According to the QIF, in preparation for implementing practice change, agents must assess the host setting and build capacity, meeting with the service, analysing its infrastructure, surveying and training practitioners, and securing buy-in [8, 9]. Regardless of how well-founded and robust the evidence may be, it is no guarantee it will be accepted and readily adopted by stakeholders [9, 15]. Persuasive communication is therefore critical for framing research findings for specific contexts to enhance their uptake and impact [16]. The power of storytelling is increasingly recognised as an effective technique for transforming attitudes, perceptions and behaviours as they summarise concepts simply, quickly and effectively, appealing directly to a stakeholder’s values and interest [16]. For instance, within UK mental health care services, storytelling as a technique has been associated with rapid improvements in data quality [9]. It is for this reason, our chapter aims to share the experiential learning and evaluation of this CLN for mental health care TSOs using a traditional storytelling outline, describing its setting, characters, plot, and themes.

2. The SILC story

2.1 Setting

To overcome the challenges of effective service development, a CLN was devised to support TSOs in the collection and use of data to inform the future development of operational practice. Inspired by the Institute for Healthcare Improvement’s (IHI) [12] ‘Breakthrough Series’ Collaborative Model and implementation science research [11, 12, 13, 14], this initiative intended to break new ground by working in close partnership with TSOs to generate evidence and inform quality improvement. The framework integrated implementation techniques using plan, do, study, act (PDSA) cycles [10] focusing on specific areas of service delivery and, as modelled by the QIF, create a structure for implementation [8, 9]. This would become known as the service improvement learning collaborative (SILC).

Working in partnership, TSOs were invited to upgrade their measurement system to a more sophisticated software platform providing additional reporting features relevant for service operation and development [17]. Services were required to verify their commitment and autonomy at a managerial, board and trustee level to commence on a year-long journey to profile and engage with subject-relevant resources and attend monthly mentorship sessions and quarterly overnight residentials. A memorandum of understanding was devised to emphasise that membership was contingent on full-service participation and this was incorporated into the development of an implementation plan [8, 9].

This project took place over the course of a year, focusing on a different challenge each quarter, including a focus on data collection, session attendance, endings, and clinical outcomes. The project commenced with a planning meeting involving introductions, training and attitudinal surveys. With reference to the QIF, these steps were undertaken to assess the fit between the organisation’s aspirations and readiness for change, allowing for open discussion and early feedback [8, 9]. Across the project, there were monthly supportive calls with an assigned mentor from the research team, and quarterly in-person residential meetings with fellow TSOs, each supported by in-depth data profiling throughout. The purpose of the mentorship and residential sessions were to support participants in monitoring aspects of service quality and provide supportive feedback mechanisms which, according to the QIF, are critical post-implementation support strategies [8]. To improve future applications, the end of the year culminated in a summative conference with fellow mental health services to share the findings from the project’s first year in operation [8, 9, 10]. A diagram of the SILC CLN model, including the induction, mentorship, residentials and summative conference, is outlined in Figure 1.

Figure 1.

The SILC CLN model, adapted from the IHI [10] ‘breakthrough series’ collaborative model.

2.2 Characters

The QIF emphasises the criticality in creating an implementation team to oversee its rollout and set targets and agree off-track remedial action [8, 9]. The SILC project team was assembled in 2016, consisting of academics and clinicians with extensive experience in the field of talking therapies and service design [9]. This team was responsible for developing learning resources, providing mentorship support and tracking data through the relevant quarterly themes of service development. The team also worked directly with individual service leads to cascade learning and implement practice change, compiling routine reflective case notes and disseminating learning throughout the network.

A series of prospective pilot services were approached and recruited in early 2017, subject to expressions of interest and eligibility criteria. The SILC initiative was specifically aimed at mental health care TSOs using CORE IMS computerised quality evaluation systems [17] to obtain evidence on their delivery and strengthen their position for funding and benchmarking. Those eligible had been using CORE outcome measurement systems for over 5 years, primarily as an administrative tool to log clinical activity. Within all but one TSO expressing interest, there was little analysis of the data being undertaken, and no indication of it being used clinically or to enhance service quality. Prospective services were using traditional pre and post-therapy measurement approaches, acquiring outcomes data for around 40–50% of clients; a rate which is representative of the field and this methodology generally [18]. Many were also experiencing high rates of non-attendance and attrition, plus modest clinical outcomes for those with outcomes data.

The exploration phase of Aarons, Hurlburt and Horwitz [11] conceptual model for implementation identifies the importance of inner and outer contexts. In this project, it seems early withdrawal during the recruitment stages was due to a combination of socio-political factors and lack of absorptive capacity which impeded progress [11]. What had started as 12 prospective members soon halved to only six. Various reasons were given but discontinuation was mostly cited as being due to managerial turnover, lack of capacity for change, and workforce restructuring, or resistance. By contrast, the remaining TSOs demonstrated their levels of commitment via an initial attitudinal survey which, when disseminated to all practitioners (n = 49), achieved a high response rate of around 80%.

The six services joining the project ranged in size, geographical location and clinical specialism. Annual throughput ranged from around 80–300 clients per organisation. Clinical support specialisms included psychological support for female victims of domestic abuse; women on low incomes; parenting; unpaid carers; and general counselling support. Informed by QIF support strategies, each service was assigned a mentor from the SILC project team using a consultation and matching process [8, 9]. Members received regular updates via a monthly blog post on the project’s website (www.silcuk.org) and a quarterly newsletter via email. Resources were shared via the website and there were opportunities to contribute in online discussion forums. The combination of online meeting platforms and email correspondence enhanced the sharing of stories, communicating learning and progress, and helped to sustain the network.

2.3 Plot

Expanding on the story structure framework, this section will incorporate a generic narrative mountain structure, breaking down the plot by its background, rising action, climax, falling action, and resolution.

2.3.1 Background

During each quarter, the project team worked with each TSO to produce an implementation plan including a set of targets, infographics, quality checklists, report templates and mentorship support, with PDSA cycles to structure the process [8, 9, 10]. Many of these tools required regular, in-depth auditing of data recorded during assessment, treatment, and discharge. Analyses were complemented by attitudinal surveys to front-line practitioners focusing on their perceptions and experiences across each quarter. Services were encouraged to reflect and communicate their learning at the quarterly residential meetings, while critically appraising fellow member’s contributions.

2.3.2 Rising action: Events leading up to the main challenge(s)

Throughout the project, it became clear that an organisation’s success in addressing the challenges depended on their relationship with the process of using measurement questionnaires and how deeply practitioners and clients were engaged in responding to feedback. The team later conceptualised this as a development cycle with four distinct evolutionary stages that described the operational depth of practitioners’ relationship with measurement: Pre and post-therapy measurement using paper forms; measurement at every session using paper forms; digital measurement at every session using tablets or computers; and digital measurement at every session tracking and sharing outcome progress directly with clients throughout the entire therapeutic encounter. It was recognised that services which were further along in this cycle had an inverse relationship with measurement in terms of its input and value towards stakeholders. Those in the later stages were able to maximise the value for clients that in turn benefitted other groups including practitioners, service management, and boards/funders. Conversely, those operating in the earlier stages were limited in their value to certain groups, typically to the boards/funders. Figure 2 shows a conceptual model of this, including the resulting value for stakeholders.

Figure 2.

The evolutionary stages of measurement within SILC TSOs illustrating the development cycle and value to stakeholders.

Conceptual implementation models highlight how the structures and processes that exist within organisations have an influence on the adoption of practice changes during the active implementation phases [8, 10, 11]. Within the SILC project, it was observed that completing paper forms, particularly at every session, generated huge administrative and inefficient burdens for members. This created barriers for practitioners looking to use data as feedback to enhance client outcomes and develop their clinical skills. During the year, most organisations evolved their administrative processes by replacing paper with digital methods, recording via electronic tablets. The services most successful in achieving the optimal rates for each quarterly challenge described understanding measurement as a construct and extension of the client. By focusing on creating the maximum value of measurement for clients, a myriad of other benefits at different stakeholder levels was also reported [19]. Naturally, some services were more equipped than others in accessing the appropriate technologies.

2.3.3 Climax: The main challenge(s) reach a high point

During the project, one of the participating TSOs withdrew due to a turnover in management and evolving financial pressures. Two other services experienced management turnover during the project which, although not impacting on their participation, did require additional input and training from the SILC project team. Practitioner turnover was understood to be common in TSOs [2, 3, 4], however, the rate of turnover concentrated at a managerial level had not been anticipated. For services with a complex management structure, this too complicated the sharing of learning and addressing each quarterly challenge. It was discovered that when managers with an on-hand leadership style were absent, this would impact on key aspects of their service operation, including the collection of high-quality data.

Another key challenge regarded the issue of session attendance and unplanned endings. A list of categorical reasons for why a session was not attended was compiled to record each time this occurred. Although the reasons recorded for cancellations were high, this was not the case for those who did not attend (DNA) (no advanced warning given) despite subsequent sessions being attended in approximately half of all instances. The most common reason for cancellations during the second quarter (n = 482) was ‘Health Problems’ (40%) while for DNAs (n = 160) it was ‘Unknown’ or ‘Not Recorded’ (76%). The absence of reasons recorded despite sessions being subsequently attended suggests practitioners either forgot or did not feel comfortable exploring why a session had been missed. This is concerning as DNAs were found to be indicative of an unplanned ending.

Definitions are important and have shown to vary the reported unplanned ending rate [20]. During the project, the unplanned ending rate reduced from 32% at baseline to 27% at the end of the third quarter, however defining and interpreting these rates revealed notable issues. Among the participating members, there were multiple interpretations about what constituted a planned versus unplanned ending. Given its inherently subjective nature and potentially negative connotations, this limited the analysis somewhat. However, the links between session non-attendance and unplanned endings were consistent across all services and tended to occur early in treatment, as described in the next section.

2.3.4 Falling action: The characters work towards resolving the main challenge(s)

One of the aims of the SILC project was to provide services with regular analyses to inform delivery and operation. This section reports on some of the headline findings along with extract quotes from two of the SILC TSOs. Systems-level modelling demonstrates the importance of considering the interrelationships between individual practice elements as opposed to solely focusing on each in isolation [11, 21]. Although the challenges during each quarter were distinct, the areas of overlap were noteworthy. Not only was session non-attendance linked with unplanned endings, but those TSOs with the longest standing commitment to high-quality data also reported the highest rates of clinical improvement.

2.3.4.1 Data quality

One major shift during the first quarter was to adopt sessional ROM, moving from traditional pre and post-therapy measurement approaches. This process was supported by a dedicated project member auditing and feeding back information to services. By the end of the first quarter, pre-and-post outcome completion rates increased from an average of 65% at baseline to 98%, while by the end of the year, this was 97%, with all TSOs achieving above 90% and half achieving 100% completion rates (Figure 3). These values were almost identical to the IAPT programme’s recent achievement of 98%, a decade after its first site implementation [7].

Figure 3.

Improvement of pre-and-post outcome measures completion rates for all SILC TSOs, 1 year before-and-after the project.

2.3.4.2 Session non-attendance

At the start of the second quarter, members began to record session non-attendance, including when an appointment was cancelled (by client) or the client DNA (no advanced warning given). One of the primary areas of interest was understood when sessions being missed were most likely to occur. Aggregating each service’s datasets, the total number of appointments per sequential session number was tallied to assess what proportion was recorded as either cancelled or DNA. Including only session numbers with over 10 appointments each, it was possible to chart this data (Figure 4). It was identified that cancellations as a proportion tended to increase the longer therapy progressed; although this might be due to a lower number of appointments at these stages. DNAs as a proportion did not exceed 10% for any session number although they did tend to occur earlier in therapy, with sessions 2–5 reporting the highest rates of 7–8%. The occurrence of DNAs declined somewhat as therapy progressed, possibly due to contracting which discharged clients after missed appointments without prior notice. Focusing on session non-attendance helped determine the scale of the challenge and how the pattern of cancellations and DNAs differed, prompting two participating services to a revise their policy in the interests of equitable access and service efficiency.

Figure 4.

The rate of appointment non-attendance per session number showing a higher proportion of DNAs earlier and cancellations later in therapy, across all SILC TSOs.

2.3.4.3 Unplanned endings

For the third quarter, the focus shifted to exploring the nature of unplanned endings. An analysis was undertaken to explore the potential associations between unplanned endings and the rate of non-attendance during therapy. This analysis found that, across all services, there was a link between session absence and ultimate attrition, especially regarding DNAs. For all TSOs, the DNA rate for clients with an unplanned (13%) versus planned (2%) ending was around 6½ times difference, ranging from 2 to 18 times across providers (Figure 5). By the end of the third quarter, those with planned endings attended almost 3 times more sessions (11) than those with unplanned endings (4) and were more likely to report reliable improvement for planned (62%, n = 226) versus unplanned (36%, n = 70) endings.

Figure 5.

A comparison of session non-attendance reporting a higher rate for unplanned versus planned endings across all SILC TSOs.

To assess how the pattern of non-attendance varied during therapy per ending type, session numbers and total appointments recorded were banded across all services (Figure 6). This analysis found that again, non-attendance was indicative of an unplanned ending, with higher rates of cancellations and DNAs. For those with an unplanned ending, it also revealed that while DNAs as a proportion were reduced in the lower session number bandings (2–4; 5%), they remained consistent at around 17–21%, excluding the 14–16 banding which reported a rate of 30%. Similar to the overall patterns of attendance, cancellations as a proportion of all appointments tended to increase the longer therapy progressed but again, this could be explained by a decrease in appointments recorded during these later subgroup stages.

Figure 6.

A comparison of session non-attendance bandings showing a steady DNA rate and increasing cancellations for unplanned versus planned endings, across all SILC TSOs.

2.3.4.4 Clinical outcomes

In the final quarter, the project focused on clinical outcomes and understanding therapist variation and trajectories of change. To identify a possible dose-effect, an analysis was undertaken to assess the rates of change across individual domains of the CORE-OM (wellbeing, problems, functioning, and risk) within the one service using the full 34-item measure, as opposed to the shorter CORE-10 which does not record all domains [17]. A pattern of average scores were mapped relative to individual session numbers up to the 10th session (for clients having 10+ appointments each) for those who reported reliable improvement (n = 130; 891 sessions) versus those who reported no reliable change (n = 39 clients; 243 sessions) or reliable deterioration (n = 7 clients; 53 sessions) (Figure 7). Based on this analysis, most of the score changes tended to occur early in treatment for those reporting reliable improvement, with an average decrease in scores of −6.1 across the first four sessions, remaining steady between sessions four to seven (−0.5), and then decreasing steadily from sessions seven to 10 (−2.3). For those reporting no reliable change or reliable deterioration, scores generally remained steady, with average changes ranging from 0.2 to 1.7. This suggests the first four sessions were important for identifying clients who were likely to improve or not. This triggered the integration of a flag feature to remind practitioners to review progress early in therapy to identify those at-risk of showing no change to provide additional support.

Figure 7.

A pattern-of-change comparison across the CORE-OM per session number illustrating early improvements for clients reporting reliable improvement compared with no reliable change or reliable deterioration.

2.3.4.5 The lived experiences of two TSOs engaging in the SILC project

Informed by the QIF, improvement for future applications requires learning from experience [8]. To gauge the experiences of those participating in the project, a brief semi-structured interview was conducted at the end of the year to explore what service managers thought of the initiative, and how they might improve it for future services embarking on a similar journey of collaborative learning. The boxes below contain extracts from these interviews with two self-selected TSOs.

Service A: Interview Extracts

Our first question was how is it going to work for our clients? Building that value for them, and the practitioners, giving them a value to the work. This is not a measurement, it’s not an outcome, it’s an aide to the process, something that helps the work with clients. And then, once we all understood that, we could have an open conversation about why we might want something like this. You really need that opportunity to embed it early on though.; It completely allowed us to cement and consolidate how we work. I mean the data the project provided, really cemented what we were doing, how we were doing, we were using data in the right way, but it also gave us ways to look at data differently, what we could do, so it was an enhancing experience. That allowed us to feel quite proud of what we do, and have it validated, which for us a charity tucked away from others, that was a nice thing to have it validated on that level.; I did that like kind of cyclical journey, that it’s not linear, we’ve got new practitioners all the time, we’ve just got 8 new practitioners in now, and they’re going back through that loop. They’re doing their first data clean this week where I’m just putting them through all the information, right we need to go through and see, right this is done, this is done, and you keep on embedding it, keeping the data quality up really. Constant, it must be really because when I’ve dipped out of the service, it went a little bit, my practitioners got a little bit complacent.; I think one of the biggest things for us, the 4-session thing, spotting that. We actively use that in supervision now, so it’s really looking at, from that first session, you can see it quite clearly. So, there’s more focus in those first 4 sessions, really looking at what the client needs, with a view to contracting through goals, further through that process. So that we’re really meeting those needs, making that environment that’s conducive then to achieving good outcomes.; We’re about sharing good practice, we’re about empowerment, we’re about creating choice and all those things. Being part of SILC fitted with part of the ethos so it was nice to go and be there in that capacity with other services. There’s something about talking to someone who’s been through it, we’re just through it. It’s that kind of picking their brains and have you thought this?; For me it’s about credentialing the sector, it’s about professionalism, it’s about best practice, it’s about evidence base, not being afraid to strive, to get to those levels, and get good outcomes and be accountable for that. I don’t think therapy is any different from if you go to a shop to buy something you expect it to be good quality. I don’t see why in therapy, clients shouldn’t expect it to be any different.

Service E: Interview Extracts

Having the support from the team that was specific to our service, having experts on hand when you needed them. Keeping on top of the data quality is not as easy without the help of the project team, and our monthly calls and the little tool pointing out the problems… Whereas sitting down and finding the problems myself is another matter.; I personally enjoy getting involved in things like this. I find it very stimulating. It ticked a lot of boxes for me, in terms of what we wanted for the service, but also for me personally, it was an interest. You couldn’t have designed it better for me… So, I think without that personal interest and enthusiasm it wouldn’t have happened.; I think I’m very fortunate in that I’ve got a very good group of people, I think credit needs to go where it's due, they are a group of people who are motivated and supportive and I think all we did was talk about, well this is going to be a benefit to the service, and they’re all very committed to the service and they came, I suppose, with open minds.; That’s been one of the key things for me, has been the experience of being part of the learning collaborative. And I think that is so valuable, personally and also for the service, because you’re going through a journey with other services, their journey’s different but there are similar issues. It’s just that ability to share learning and connect with people who have a similar job, are having similar issues. When you have something, and they say, yeah that’s happened to me. And for me, it takes away that sense of being in your own little bubble, in your own little service, which I wouldn’t say is isolating but that you’re not part of anything else. The learning collaborative made you feel part of something bigger with some connections, and yeah, doing the same thing you’re doing, I thought it’s fabulous, it’s brilliant.; It’s the practical stuff, we’ve become a service that does sessional measurement using tablets, that’s the way we do things now. We’ve changed the way we manage DNAs, we have an appreciation of data quality, and that’s not just me, the team come along and say why haven’t I got 100%? Why is this only saying 90%? Can we have a look where that 10% has gone? So, there is an appreciation now of the importance of good data. In fact, the things that SILC was meant to address, are the things that have changed in our service.; It’s a no-brainer. Why wouldn’t you? I can’t see any reason why you wouldn’t, unless you haven’t got the support to see it through. Know your organisation, know that you’ve got that support, to be able to put the time into it, those are the two caveats, otherwise, it’s a no-brainer.

2.3.5 Resolution: How things have ended up in this story

In keeping with the IHI’s [10] collaborative learning model framework, the first year of the SILC project culminated in a summative conference. Nearly 100 delegates were in attendance, each representing a range of different sectors within the field of talking therapies. Both the project team and self-selected SILC TSOs held a discussion regarding their experiential learning during the first year of the project. There was a consensus at the event about the operational challenges facing modern-day talking therapy services. While systems were becoming increasingly sophisticated, the training and support necessary to build in-house expertise were reportedly difficult to access due to time and resource constraints, a saturated and uncertain field, and isolated working practices. Providers, particularly in the third-sector, desired the opportunity to work in partnership with others to share learning and enhance theirs and the sector’s organisational and therapeutic models further.

With the first stage complete, the SILC project has amassed a wealth of learning which will be converted into a modular learning programme, providing a resource for future applications of the network [8, 9]. This will replicate the CLN model and invite existing SILC members to act as guest speakers and offer unique support and valuable insights to newly recruited collaborative members. There are three existing SILC TSOs who have declared their interest and commitment to continuing with the project. Due to a turnover in management and decrease in contribution, two members have since withdrawn. The next phase of the initiative will focus on expanding the network, building on the existing knowledge and aggregate data to support ongoing analyses and resource development.

2.4 Themes

Themes are the essence of a story, the central constructs which reflect the actions, perceptions and experiences of the characters in their situational contexts. They represent the underlying ‘big ideas’ which transcend the distinctions between settings and circumstances and help conceptualise elements and links between them. This is important given the lack of guiding conceptual models for the sustainment phase of implementation [11]. Listed below is a discussion on some of the key themes both the participating services and project team uncovered during this stage of the project.

2.4.1 The possibilities of CLNs in mental health TSOs

The unintegrated nature of TSOs in the UK means there can be obstructions to developing and integrating EBP [2, 3, 4]. Within the field of talking therapies, determining what constitutes as EBP has been criticised for its reliance on controlled study methodologies which, due to their somewhat artificial nature, are considered detached from the clinical realities of routine practice settings [22, 23]. Certain advocates support a PBE approach to complement and address these limitations [24]. However, PBE relies on the collection of robust, aggregate datasets across multiple organisations sharing a common system or model.

Fragmentation, isolated working practices, and resource constraints can limit TSOs generating the PBE necessary to support their delivery [2, 3, 4]. Indeed, the primary interest from prospective members in this project was overcoming these barriers and demonstrating they were treating clients effectively. By pooling experience, resources and expertise around a central, unifying theme, TSOs were able to systematically explore, assess, understand and reflect upon key aspects of service quality development. Through iterative cycles, strategic improvement models and coordinated and collaborative dialogue [10], services were able to generate timely and actionable insights that were relevant to their unique circumstances. Testing practice changes on small scales, using focused inquiry and PDSA cycles, helped achieve small wins which, according to evaluation theories, can be an effective strategy for boosting perceived capabilities [6, 10].

Replicating previous research findings [2, 3], access to a supportive academic project team was deemed invaluable for producing, mentoring and synthesising analyses and learning across the network. However, liaising with several TSOs proved to be a lengthier and more complicated process than first envisaged; an experience which is echoed elsewhere [6]. This identifies an important obstacle for sustaining CLNs, particularly those undertaking continuous analyses. It might be that by offsetting resources to a project team, this creates a more efficient process within individual services as it shares the expertise around a common need. If this were true, then it could prove more efficient and cost-effective for TSOs overall.

Given the central communicative nature of CLNs, it is important these channels are equitable. Within the third-sector, organisations tend to differ in size and can be equally varied in their operational modelling [2, 3]. This inequity in size and visibility could feasibly leverage greater influence over smaller providers to work towards their agenda. To overcome the challenges of distinct delivery models within CLNs, a central governing platform using cooperative representation could therefore be valuable for identifying topics of interests and establishing a dictionary of terms. Similarly, these communication channels ought to use terminology that is consistent and agreed upon, particularly around subjective concepts such as ending types as doing so would ensure greater validity and reliability in data analytics [20].

2.4.2 Management and leadership

Many implementation frameworks emphasise the planning stages as critical to successfully embedding innovation [8, 11, 12, 13]. Because implementation can be a complex process involving integrating existing practices with new, it typically requires a well-planned, structured and iterative process, addressing the various philosophical and practical barriers that can occur regularly [9, 15, 21]. It is within these contexts that supportive leadership can be a facilitating factor [2, 11, 15, 21, 25]. Without effective leadership to track, monitor and effectively champion the merging of practices, any expended effort can unravel [9, 15, 25]. Those in leadership positions need to be present and well-respected, retaining a detailed awareness and understanding of delivery and operation [15]. Service quality development through CLNs therefore appears to be reliant on management structures and local leadership.

In considering the scale of change and level of turnover in TSOs, particularly at a managerial level, the reliance on leadership highlights a notable barrier. Given the project team tended to work exclusively through managers brokering knowledge and training, their absence ultimately affected their organisation’s participation and operational processes. It could be argued this was a side-effect of the chosen methodology which may have benefitted from a broader involvement and contribution among the workforce. Advocates across the field recommend ensuring a local champion is permanently in place, advising that those departing a service provide sufficient training to those replacing them [9, 15, 26]. While this recommendation is practical, how it applies to TSOs is perhaps more complicated.

Continually nurturing the operational climate through sustained involvement and being present can help resolve the functional mechanisms of feedback systems [15, 25, 26, 27, 28]. A perceived lack of presence in the project among some practitioners served to undermine the initial enthusiasm and positive ethos established at the project’s outset. Services which thrived tended to dedicate additional time and resources to sharing information in an open and accessible manner. This actively engaged the workforce in the minutiae of feedback informed treatment (FIT) [28] and encouraged more open dialogue. The literature on FIT teaches the value of routinely soliciting responses from clients about treatment progress, aiding practitioners at a therapeutic level [28, 29, 30]. However, there is an additional service-level which could also help inform practitioners and other stakeholders about enhancing client engagement and outcomes. By combining a FIT model with a feedback informed service, practitioners could have timely access to relevant learning. With reference to the QIF [8], supportive feedback mechanisms will be relevant to all stakeholder levels and through aggregate data, the client voice can be made accessible to all, helping sustain innovation.

2.4.3 The resource challenge of TSOs

Based on the learning from this initiative and relevant national and international research [2, 3, 4], there appears to be a significant resource challenge facing TSOs. Although many report having an interest in quality improvement [3], the constraints on providers including turnover, financial pressures and limited budgets, appear to greatly impact their ability to generate data and engage in practice development [2, 3, 4]. For a sector that relies heavily on volunteers, some of whom are in trainee positions [1], preserving a level of local expertise represents a continual challenge, particularly as systems become more expansive, specialised and costly. Although the CLN was a means to pool and share resources, supporting the implementation phase [11], external pressures had a notable influence on its integration, process and overall output. The level of attrition at the beginning and eventual withdrawal of others highlights the scale of this challenge. Consequently, this further demonstrates the criticality of the QIF phases in thoroughly assessing the fit between the host setting’s aspirations and readiness for change [8, 9].

Given the sheer scale of change and advancing pace of new technologies, feedback systems and innovations are becoming increasingly sophisticated while at the same time, access to training and support might not be keeping pace [3, 31]. For many, including attendees at the summative conference and across the wider literature [3], allocating resources to this endeavour might be considered non-feasible as few can afford or justify it economically. This issue is further compounded by the fluctuating and isolated nature of services as well as barriers in accessing the literature due to subscription paywalls [2, 3]. Accordingly, this highlights the need to consider the additional training and support required when adopting new innovations.

Despite its limitations, a CLN could address some of the resource challenges identified, increasing the opportunities for learning. Disseminating feedback throughout a network might help overcome some of the barriers to accessing research and forming partnerships [5, 6, 10]. Shared learning across all levels of the network, could foster a broader culture of openness and training, supporting collaboration across multiple platforms, while also generating an asset for feeding back insights across the sector. Undoubtedly, this would rely on the aggregation of robust datasets and communication platform to support this process [5].

2.4.4 Designing the infrastructure

The experiences from this project revealed the influence of organisational factors and infrastructure on the uptake of practice changes. Although research on the integration of feedback systems and ROM have identified numerous practical barriers, much of the emphasis has focused on practitioners [9, 15, 31, 32, 33, 34, 35, 36, 37]. Indeed, positive attitudes towards feedback have been shown to facilitate the effect on clinical outcomes improvement, while resistance can have the opposite effect [33, 38, 39, 40]. Resistance reportedly stems underlying performance anxiety or negativity about the relevance and utility of the practice [9, 15]. However, the learning from this project highlights how positivity and motivation might not be sufficient in isolation.

Despite the generally positive attitudes from the survey and among the management mentees, itself likely a result of the selection process, many TSOs still encountered challenges, many of which appeared to be due to limitations in the infrastructure and frustrations with the technology. This, in turn, affected their capacity to use the system, something which is shown to be a facilitator in implementing EBP [25, 27, 31]. Restrictive and frustrated working practices can lead to negative perceptions forming [25, 27, 36, 41], suggesting attitudes might be mediated by how user-friendly and engaging a system is. For TSOs facing time and resource constraints, the simplicity of a feedback system is perhaps more pivotal. In these circumstances, systems may benefit from a uniform, standardised approach so that training and support can be refined and accessible via fully integrated and self-led instructional packages [32]. In terms of the QIF [8], the critical steps for assessing the needs and resources, capacity, and pre-implementation training would benefit from accessible resources which are intuitive and easy to understand.

2.4.5 Refocusing measurement to respond and maximise the value to clients

Traditionally, measurement in TSOs have been undertaken to satisfy the needs of boards and funders and to a lesser extent, service managers [3, 4]. The pressures on services have meant that pre and post-measurement approaches have dominated, with its purpose serving mainly administrative rather than clinical needs [3, 9]. ROM established a method for improving data quality and representativeness, although the emphasis regarding its clinical utility or use in service development has only recently been advanced [7]. This illustrates how the focus and value of measurement have been positioned to satisfy a broader sector-level drive. However, by framing measurement in a way to maximise the value for clients, as observed in this project, there appear to be many cumulative gains for all stakeholders, including practitioners, service managers and boards/funders.

Across each of the common challenges, there seemed to be a critical period, usually within the first four to six sessions, which correlated with eventual outcome. For instance, a large proportion of DNAs tended to occur early in treatment which were a useful indicator of an unplanned ending, and by extension, a reduced chance of reliable improvement [20]. For clients reporting reliable improvement in one TSO, most change seemed to occur during the first four sessions, while those reporting no reliable change or reliable deterioration showed little change across a 10-session period. This emulates the wider literature which identifies the initial stages as being a useful indicator for a client’s subsequent engagement and outcome [42, 43, 44, 45]. Accordingly, this trend highlights the criticality of early engagement and warrants a further discussion about the implications of keeping clients involved in therapy who report no change or attend infrequently. Evidence has shown that decisions to prolong or conclude therapy despite a lack of positive therapeutic change can be influenced by subjective beliefs, norms and attitudes, sometimes superseding what feedback monitoring and practice guidelines recommend [45].

According to the literature, the clinical benefit of measurement can be mediated by a practitioner’s engagement and attitude towards outcomes monitoring [33, 38, 39]. Moreover, timely access to feedback has been shown to be a critical factor in the use of data among practitioners [27, 34, 36, 46]. TSOs which encourage open dialogue and pay greater attention to this information could produce cumulative benefits in each of the quarterly themes identified [10, 30, 47]. An organisational culture of openness and commitment to learning was important and replicates findings reported elsewhere [15, 46]. Additionally, giving practitioners access to service-level data might assist them in overcoming residual ambivalence because its application to service quality development is readily observable.

2.4.6 Recommendations

For those interested in implementing a CLN to support TSOs, there are several recommendations based on this project’s findings. Firstly, recording high-quality data is crucial to this model. Securing high-quality data helps support the network and aggregate learning by effectively threading the client voice throughout all stakeholder levels. Promoting client engagement in the process of measurement is an effective strategy for enhancing data quality and building the opportunities for clinical application [9, 31, 47]. Because of this, it is important that implementation teams do not underestimate the infrastructure necessary to support practitioners working to deliver these innovations [15, 32, 35, 46]. While pooling resources can help overcome challenges relating to cost and access to expertise, without a shared framework and understanding of the key concepts, a CLN and its associated analyses are likely to be impacted. In keeping with the wider literature, access to expertise and committed project team can be beneficial for supporting the network [2, 3, 5, 6, 9]. Focusing on distinct areas of service delivery through iterative improvement cycles and acknowledging their interdependency can help achieve cumulative benefits through the combination of smaller gains [6, 21, 25]. For TSOs, the role of leadership and effects of turnover cannot be understated. While it might not be feasible in TSOs to ensure a local champion is always in place, it is valuable to build a system that enables receptiveness towards continual practice innovation. A broader involvement and contribution among the workforce through wider supportive feedback mechanisms represents one effective strategy to overcome this.

3. Conclusion

TSOs represent a valuable and growing player in the provision of mental health care, yet many are constrained by limited budgets, isolated working practices, and a constantly shifting workforce. Together, these make producing and accessing evidence difficult, further limiting the sector from credentialing their impact and engaging in service development. To overcome these challenges, a CLN was implemented involving six TSOs and a dedicated project team to share learning and resources with the aim of improving delivery and operation in the areas of data quality, session attendance, unplanned endings and clinical outcomes. The CLN was inspired by the IHI collaborative model [10] framework for integrating and testing improvements using PDSA cycles and the implementation process was guided by the QIF [8]. It was found that introducing ROM substantially improved data quality which acted as the bedrock for all subsequent analyses and discussion. There appeared to be strong links between each of the common challenges, including increased non-attendance being associated with the occurrence of an unplanned ending, itself linked with a lower chance of reliable improvement. Overall, this approach to generating timely and relevant practice-based insight through partnership working and mentorship support proved to be effective for stimulating service quality enhancement. Although TSOs face many unique challenges, including high staff turnover and strained budgets, those with on-hand and inspirational leadership and commitment towards maximising the value of measurement for clients reported most success.

Acknowledgments

This work was supported by the Artemis Trust (No grant number). The funder had no further role in the design, collection, analysis, interpretation and compilation of this paper and no financial interests or benefits have arisen from the direct applications of this research.

Conflict of interest

Scott Steen and John Mellor-Clark declare they have no conflicts of interest.