Open access peer-reviewed chapter

A Framework to Manage Quality of Enterprise Content Management Systems

Written By

José González Enríquez, Francisco José Domínguez Mayo, Julián Alberto García García, María José Escalona Cuaresma and Manuel Mejías Risoto

Submitted: August 18th, 2016 Reviewed: October 6th, 2016 Published: February 22nd, 2017

DOI: 10.5772/66199

Chapter metrics overview

1,608 Chapter Downloads

View Full Metrics


There is a wide range of enterprise content management (ECM) systems which supports, among other things, document management processes, records management and Web content management. However, each of these systems has many features and some of them can meet organizational needs depending on the scale, sector and workflow of the organization. In addition, it is very common that organizations are unaware of what ECM system best fits their needs, since each company has its particular scope and strategic objectives. This chapter is contextualized within the real project called THOT designed for the Andalusian Public Administration in Spain. The aim of this project is to study in detail ECM systems and propose an objective method to compare them for the specific scope and strategic objective of organizations. Quality evaluation framework (QuEF) has been adapted for this purpose.


  • software engineering
  • standards
  • methodologies
  • enterprise content management
  • quality analysis and evaluation

1. Introduction

Today's world economic situation is dominated by concepts such as globalization, which involves the relocation of companies, the constant search for lower costs to maximize profits by continuous mergers and acquisitions, and a constant motivation for improving and optimizing all processes. Defining business models to manage organizations effectively is essential and nowadays, it has become a common practice followed by a large number of companies in all areas of business, especially within the information and communications technologies (ICT) context [1]. This can be extrapolated to the context of document processes, as these can help sales and product development organizations capitalize on new business opportunities [2]. For example, this implementation would permit a faster commercialization of new offerings, a flexible response to customers’ needs, a quicker reply to changing market dynamics and business competitiveness improvement.

Every day, organizations have to handle a lot of different type of information such as general documents, audio‐visual files/resources, management reports, customer cards or invoices, among others. In addition, this information is not disconnected, but integrated within the organization's business process. The ever‐growing use of this kind of information within ICT organizations confirms this tendency for the years to come. There are two main alternatives to manage efficiently and effectively these organizations’ information resources. The first one consists in specifying, designing and implementing an ad‐hoc system for this goal within organizations. This alternative is not usually feasible in most of them due to the high cost of maintenance, development and future updates.

The second option deals with using proprietary or open source enterprise content management (ECM) solutions. Performed literature review indicates that this option best fits companies’ budgets.

A variety of ECM definitions can be found in the literature, according to the Association for Information and Image Management (AIIM) [3]. The ECM consists of strategies, methods and systems used to capture, manage, store, preserve and deliver content and documents related to organizational processes. ECM systems (ECMSs) and strategies facilitate an organization's unstructured information management. However, there currently is a wide range of document management solutions with different cost and functional scope (such as interoperability, security, usability, efficiency, customization among others). Therefore, to choose the one which best meet the organization's necessities is not a simple task. Smith and McKeen [4] define ECM solutions as an integrated approach to manage all organization's information including paper documents, data, reports, Web pages and digital assets. In addition, Tyrväinen et al. [5] provide a framework to stimulate and guide future research as well as point out research issues specific to the ECM field. The aforementioned authors argue that ECM offers an important and complex subfield of information systems. As regards the term ECM, these authors add that it has been widely adopted by software product vendors and practitioners in order to refer to technologies used to manage the content of assets like documents, Web sites, intranets and extranets in organizational or inter‐organizational contexts. Preliminary findings suggest that ECM has not managed to attract adequate attention.

Nordheim et al. [6] also examine the topic of ECM solutions, but under a strategic development and implementation process for a large oil company. The authors represent a case of a hybrid development approach to ECM that involves the lifecycle, teleological and dialectical engines of development. They state that this is opposite to the evolutionary development motor, which has prevailed in the hitherto reported content management research. Nordheim et al. also show a case study which complements process‐based research on enterprise system implementations in general. In addition, they suggest that research and practice on large‐scale ECM implementations should acknowledge all the four motors of change.

Furthermore, Munkvoldet et al. [7] explain that the concept of ECM represents integrated enterprise‐wide management of the lifecycles of all forms of recorded information content and their metadata, organized according to corporate taxonomies and supported by appropriate technological and administrative infrastructures. This publication is based on a case study about a Norwegian oil company (Statoil), where they identify a wide range of issues related to content, infrastructure and change management. As ECM perspective is concerned, Munkvoldet et al. argue that ECM perspective is found to integrate and extend the existing research areas of information resource and document management, as well as the repository model of knowledge management. Thus, they show how ECM deserves further attention beyond its current market hype, as a potential area of Information Systems (IS) research crossing several previously separate areas of information management from the enterprise viewpoint.

All ECM definitions show that there is a wide range of ECM descriptions and solutions in literature which support, among other things, document management processes, record management and Web content management. The problem is that each of these solutions has many features and some of them can meet organizational needs depending on the scale, sector and workflow of the organization. In addition, it is very common that organizations are unaware of what ECM system best fits their needs because each company has its particular scope and strategic objectives.

This chapter is contextualized within the THOT project, which is a real project designed for the Andalusian Public Administration in Spain. Document processes management is essential and critical in this context, since e‐Government is taking a key role in setting the strategic plans of the Public Administration. Therefore, the AOPJA (Public Agency of Contracting Services for Transport and Infrastructure Constructions) and the University of Seville are executing the THOT project as a result of these needs. THOT is an innovative project with high costs (621,250 Euros) focused on document management applied to service agreement records and transport infrastructure projects. Consequently, deciding on the most suitable ECM solution for the AOPJA context poses considerable responsibility. An ECM system that meets all requirements must be developed once the most suitable ECM solution is characterized and defined for the AOPJA context.

Therefore, THOT project aims to analyse in detail strategies and document management systems to investigate and define an innovative solution, so that records management can improve. Then, the objective is to study ECM systems in detail and propose an objective method to compare them for the specific scope and strategic objective of the AOPJA.

For this purpose, a quality evaluation framework (named QuEF) is proposed to analyse and evaluate ECM solutions. QuEF is a work‐in‐progress framework that has been used in a case of study (Public Administration of the Regional Government of Andalusia, Spain) in order to validate it. Moreover, QuEF provides an agile, flexible and efficient solution based on a Web environment, allowing organizations to choose the best ECM system for their necessities and ensure the continuous quality improvement of these systems in the organization.

For these reasons, the objective of this chapter is twofold: to identify assessment criteria for ECM solutions, focused on the AOPJA context, by means of the proposed framework and to demonstrate the value QuEF offers to support technology acquisition.

As such, this work is structured as follows: Section 2 summarizes some of the most recent work related to ECM systems and reference standards on this topic.

Section 3 describes QuEF and its theoretical foundations and Section 4 presents the THOT project.

Then, Section 5 explains how the framework has been applied to the project and introduces each phase of the framework and its execution process along the project.

Finally, Section 6 states learned lessons and on‐going work.


2. Related work

There are different versions of ECM systems found in literature; Scott [8] evaluates the factors that lead to users choosing an ECM system. The results show the importance of cognitive involvement with technology highlighting the importance of including cognitive participation in building acceptance studies.

Alalwan and Weistroffer [9] performed a comprehensive literature review of ECM. A conceptual framework of the areas of interest in relation to ECM as well as an agenda for future research on this topic were proposed.

Ninety‐one ECM publications were reviewed in this work. The authors concluded that the ECM systems involve interacting technical, social, organizational and business aspects. Also, the authors suggested that the current ECM literature could be grouped into three main research pylons:

  • The first one consists of the four ECM component dimensions.

  • The second one deals with the enterprise system lifecycle.

  • The final one constitutes the strategic managerial aspect. An agenda for future based on the review and the suggested conceptual framework is also suggested.

Smith and McKeen include the strategies, tools, processes and skills an organization needs to manage its information assets on ECM solutions along its lifecycle. The authors explained that an effective ECM strategy should address each of the four lifecycles stages:

  1. Capture: gathering all activities associated with collecting content.

  2. Organize: indexing, classifying and linking content and databases together to provide access within and across business units and functions.

  3. Process: shifting and analysing content in such a way that may facilitate.

  4. Maintain: ensuring that content is regularly updated.

The authors pointed out that most ECM initiatives take a bottom‐up approach that focuses on delivering immediate benefits through projects such as intranet portals, information searching and Web content management, while the top‐down vision for ECM includes improved decision‐making, better utilization of information and collection of competitive intelligence. However, performed research indicated that managers also acknowledge the fact that greater value can be gained from taking a more strategic approach to ECM. The authors showed that those organizations that can effectively “hand‐shake” content stewardship practices with appropriate information behaviours, and values and information technology on a broader scale, can have a significant effect on their performance.

Grahlmann et al. [10] explain that ECM centers on managing all types of content used in organizations. The authors present an overview of previous research explaining that scientific literature on ECM is limited and no consensus on the definition of ECM is reached. Therefore, the literature review surfaces several ECM definitions that are merged herein into a more consistent and comprehensive definition of ECM. Indeed, the authors mentioned above provide the functional ECM framework (FEF) which is an overview of the potential functionalities of ECMSs. They applied FEF to three case studies to communicate on ECMSs, to familiarize oneself with and to direct future research. It may also form the basis for more formal reference architecture, and practitioners may use it as an assessment tool for comparing the functionalities provided by existing ECMSs.

Herbst et al. [11] showed that ECM is an important enabler of information management, as it supports the creation, storage, retrieval and retention processes of organizational documents together with their content. They identified a set of critical success factors for ECM and develop a framework that helps organizations assess their readiness for ECM. In Herbst et al., this framework was developed following the data collected in workshops held between ECM project leaders and members of five companies. The authors argued that expert's opinions and experiences are combined with research results from the academic literature, and two illustrative cases showed how the framework has been put into practice.

In addition, Rickenberg et al. [12] explained that ECM can be considered an integrated approach to information management. They exposed that ECM research is still an emerging field of IS research, even though practitioners pay much attention to this concept. Furthermore, they provided a detailed review of the body of academic research: the ECM domain, its evolution and the characterization of the main topics. In Rickenberg et al., an established ECM research framework is adopted, refined and explained together with its associated elements and working definitions. On this basis, 68 publications were reviewed and classified, and concepts were derived. Prior research was synthesized and findings were integrated in a concept‐centric way. Finally, the authors exposed implications for research and practice, including future trends.

Two works are related to the ECM implementations. Haug [13] included a definition of a process model for ECM implementation in SMEs. This author proposed a new pattern definition for ECM technology development. In [14], Van Rooij explained that legal issues generated by ERP could be similar to those generated by ECM systems. Therefore, it is advised, with appropriate adaptation, to take these issues into account when developing strategies for implementing the ECMs.

ISO 2709:2008 [15] specifies the requirements for a generalized exchange format containing records describing all forms of material capable of bibliographic description as well as other types of records. ISO 15836: 2009 [16] establishes a standard for describing resources across domains known as metadata elements Dublin Core Set. This standard defines the elements that are commonly used in the context of an application profile, limiting their use in accordance with the policies of a particular community and it does not define the implementation details.

ISO 10244:2010 [17] provides businesses with the tools to identify the relevant aspects of the business work processes and document them in a standardized format.


3. The THOT project scenario

The THOT project is an e‐Government project with the objective to implement an ECM system in the Public Administration of Andalusian region of Spain granted for 621,250.00Euros. This chapter is focused on the first phase of the THOT project and explains how the technological and functional status of existing ECM systems has been studied. It is very important to evaluate all existing alternatives in the market in order to align the scope of the organization with its purpose. It is also relevant not to reject the decision because it is difficult to change the chosen system due to the cost, once the development of the solution has started. As the evaluation process concerns, a static evaluation or characterization of an ECM solution is not enough, since new improvements of ECM systems are continuously appearing and one has to compare alternatives dynamically and objectively. In addition, this work considers the different preferences of the elements containing an ECM system in terms of given specific context. This study discusses in detail existing ECM systems in the market and proposes an objective method to compare them within the specific scope and strategic objective of organizations.

Nowadays, the Andalusian Public Administration is driving the need for a change in the following document management systems:

  • JUPITER [18], although this situation will change, because this information system it is going to be replaced by an ERP technology platform

  • TheERIS‐G3 [19], which is an e‐Government system to manage the electronic procedures to process public records in each public agency

  • @rchivA [20], which facilitates the Patrimony Documentation management of Andalusia

This project aims to cover different disciplines of research and innovation as document management, electronic government, dissemination and Web services integration policies, enabling organizations to provide a common framework for document management. To achieve the aforementioned objectives, the project is being carried out along the following stages and activities:

  1. Studying technological and functional status

    • Analysing the current situation

    • Benchmarking existing tools and systems and selecting the most appropriate option

    • Defining new document management functionalities and fitness for certification as a document management system

  2. Setting the context of research results

    • Adjusting Andalusian horizontal documentary series

    • Developing policies to preserve digital documents

  3. Defining the solution

    • Defining the document management solution

    • Developing and implementing a basic functional system

    • Defining a dissemination system

  4. Disseminating results

    • Defining the project dissemination plan

    • Defining project dissemination indicators

    • Executing the dissemination plan

In this context, QuEF is proposed as a work‐in‐progress framework to be validated and used for the evaluation of existing ECM systems. This framework offers a suitable methodology to analyse and evaluate ECM solutions dynamically and objectively. In addition, it includes methods to calculate preferences of ECM features and it defines a lifecycle and tool support to enforce the quality continual improvement of an ECM solution.


4. The QuEF framework methodology

QuEF [21] is a work‐in‐progress framework that has been used in this case study (Public Administration of the Regional Government of Andalusia, Spain) in order to validate it. QuEF is a framework to manage quality of entities (products, processes, services or organizations, among others) in any context and domain. In previous work, this framework was used to manage quality in model‐driven Web development methodologies. In addition, this framework can also be used for consumers to identify the most suitable product or process for them and decide accordingly.

This framework describes templates to define a specific Quality Model for the domain under study. It also offers a method to customize the Quality Model, evaluate it and calculate the preferences of its elements. Besides, the framework includes the definition of a set of phases to enforce continuous quality improvements in the Quality Model. The most important aspect is that quality management is the central quality entity in the Quality Model. Furthermore, a tool support is also implemented in order to promote this solution in real environments. Therefore, users may indeed obtain quality management in an automatic way using QuEF, and hence, computerise the quality management of entities (products, processes, services or organizations, among others) with the final aim to reduce cost and time and improve overall quality. As such, this framework provides

  • A set of phases to enforce the continuous quality improvement

  • Quality standards and international best practices

  • Methods for each phase and templates to customize the Quality Model

  • Multi‐criteria methods to calculate the elements preference value of the Quality Model

  • A tool to support the quality management lifecycle process

Table 1 shows the relationship among other standards. It also represents the relation of these standards as well as the best practices and approaches that have been applied to the QuEF framework both to define QuEF itself and to apply it to a specific domain.

Standards, best practices and approaches Work application
ISO/IEC 20000, ITIL ISO/IEC 2000 standard and ITIL best practices deal with improving service quality based on a quality continual improvement of the service lifecycle. For instance, ITIL defines Strategy phase, Design phase, Operation phase, Transition phase and quality continual improvement phase. QuEF covers the same idea with a different goal since QuEF framework manages quality‐based on a quality continual improvement of the Quality Model lifecycle.
TQM, Six Sigma, CMMI Planguage,  C‐INCAMI or CTQ, among others The QuEF framework defines different phases with artifacts, methods and tools for each phase. Most of these approaches could be adapted and applied to some phases of QuEF. They cover the similar aspects between QuEF and quality management Strategy and Operation phase. For instance:
  • TQM is a management integrative philosophy aims at continuously improving the quality of products and processes. It could be applied to Strategy and QCI phases in QuEF.

  • Six Sigma is a business process management strategy very similar to TQM working with many established quality‐management tools. Most of them could be used in Strategy phase and Operation phase of QuEF.

  • CMMI is a process improvement approach that intends to help organizations improve their performance. Therefore, it could be applied to Strategy and quality continual improvement phases.

  • Planguage could be applied to the Strategy phase of QuEF for specifying quality.

  • C‐INCAMI provides a domain (ontological) model defining all the concepts and relationships needed to design and implement processes. Hence, it could be used in a Strategy and Operation phases.

  • CTQ could also be applied to the Strategy phase, the Design phase and the Operation phase of QuEF for specifying project context, nonfunctional requirements, measurement, evaluation and analysis.

Table 1.

Standards, best practices and approaches related to the QuEF framework.

As Figure 1 shows, the framework can be used from two points of view: providers’, who need to analyse, control, evaluate and improve entities and consumers, who need to compare entities (depending on their context) to decide on the most suitable one for them.

Figure 1.

Conceptual scheme representing the goals to be achieved with QuEF.

It mainly differs from other frameworks in that it focuses on the Quality Model as well as defines a lifecycle where all phases turn around that Quality Model, as shown in Figure 2.

Figure 2.

Quality management based on the Quality Model lifecycle.

Moreover, QuEF provides an agile, flexible and efficient solution based on a Web environment, so that organizations can choose the most suitable ECM system for their purposes as well as enforce the continuous quality improvement of these systems within the organization. It is based on ITIL v3, but with a difference; it does not focus on services, but on a Quality Model. Similarly to ITIL v3, it comprises of five phases to ensure the continuous quality improvement of the Quality Model. The aim is to centralize all quality management efforts on the Quality Model. This means that it incorporates several phases including different objectives and artefacts. The aforementioned phases are

  • Quality Model Strategy (QMS) phase: This phase is strategic active that focuses on the definition of a quality management strategy. The past, present and future view elements of the Quality Model in the domain under study are essential to achieve an effective and efficient quality management process.

  • Quality Model Design (QMD) phase: This is the phase where the Quality Model is finally designed depending on the requirements from the previous phase. It is the model used in the next phase for the quality management performance.

  • Quality Model Operation (QMO) phase: In this phase, the Quality Model is used to carry out the quality management process. Consequently, the analysis and evaluation management processes are performed within this phase.

  • Quality Model Transition (QMT) phase: This phase describes the processes that execute changes in the Quality Model, in cases where the domain or context changes due to the appearance of new trends, but without affecting the Operation phase.

  • Continuous Quality Improvement (CQI) phase: This phase performs all mechanisms to improve quality in all processes in the lifecycle and the Quality Model.

An effective and efficient quality management essentially demands to define the domain under study. Thus, it is important to consider what type of ECM system is concerned. It is not the same to develop an ECM system for a bank, where security stands as a more important quality characteristic, as opposed to the development of an ECM system as presented herein, where usability, functionality and performance are crucial. The purpose of QuEF is not only to assure a clear strategy for quality management but also automatically facilitates a continuous quality improvement by means of generating checklists and documentation, as well as automatic evaluations and plans which control and improve quality and thus, automatically, reduce effort and time.

Figure 3 shows the specific proposed metamodel for QuEF. There are many definitions in literature trying to clarify what a Quality Model is. In QuEF, it means a set of characteristics and its relationships, which constitutes the base to specify quality requirements and evaluate them. The Quality Model represents its core with quality management revolving around it. This work proposes a Quality Model metamodel consisting of a simplification and adaptation of the ISO/IEC 15939, so that the model implementation can be more flexible and practical. The Quality Model contains Features, Sub‐Feature and Property.

  • Feature (FT‐<Level 1>): It is a general concept that involves a set of higher‐level concept of ECM system properties that describes it. It includes a set of Sub‐Features.

  • Sub‐Feature (SF‐<Level 0>): It includes a specific concept of an entity. It is a set of lower‐level concept properties of an entity. It is also utilized to categorize ECM systems in two levels (Feature and Sub‐Feature).

  • Property: It indicates the degree to which a Sub‐Feature is measured by the use of a Metric. Particularly, a property is used for describing and analysing the Sub‐Features of an entity. It is an element of an ECM system. In other words, a property is used for describing and analysing Sub‐Features.

Figure 3.

Quality metamodel on QuEF.

4.1. Case study: applying the QuEF framework to the THOT project scenario

As previously mentioned, the THOT project is a transfer project carried out in collaboration with the Regional Government of Andalusia (Junta de Andalucía) in Spain. It has two main objectives:

  1. to obtain a detailed analysis and evaluation of ECM systems applied to contracting records for infrastructure projects and, to find out and

  2. to define an innovative solution that may improve procedural records management.

Currently, there are several solutions for this type of systems in the market, although herein, the most appropriate ones have been selected in relation to the scope of the project. Thus, the QuEF framework has been used to obtain the detailed analysis and evaluation of different ECM alternatives. As Figure 4 shows, two points of view have been identified in the THOT project scenario: ECM system providers who need to analyse, control, evaluate and improve ECM systems and the Public Administration of the Regional Government of Andalusia, which needs to compare ECM systems (depending on the THOT project scenario) to make a decision on the most suitable one to apply.

Figure 4.

Conceptual scheme representing the application of QuEF framework to the THOT project scenario.

In order to apply the QuEF framework, IWT2 research group (Web Engineering & Early Testing) is developing QuEF‐TS with the aim of automating all processes and artefacts that QuEF defines for each phase. In this particular case, an Enterprise Architect (EA) modelling environment with UML 2.2 is provided.

Figure 5.

Activity diagram for the tool support use.

Activity diagrams are used to describe the business and operational step‐by‐step workflows of components within the system. Figure 5 shows the overall flow of control between ECM system providers and Public Administration may be observed (not necessarily registered users) as well as the system administrator (in this case, the IWT2 research group).

Then, the Quality Model lifecycle has been applied to obtain a characterization model of ECM systems. Firstly, the Strategy phase has been established for an effective quality management of ECM systems. In this case, the quality management problem is, in fact, a multiple‐criteria decision‐making or multiple‐criteria decision‐analysis (MCDA) problem, because multiple criteria are used to calculate the preferences regarding the Quality Model elements demand. Besides, it is a multi‐objective optimization problem, as it is important not only to implement the most valuable ECM system but also to reduce cost, risk and uncertainty.

Secondly, the Design phase has been applied to fix the Quality Model that was used to analyse and evaluate quality. The transition phase is also important, since new systems or trends regarding the insertion of new characteristics in the Quality Model can appear in future iterations and these changes must be controlled.

Thirdly, this study will explain how the ECM systems have been analysed and evaluated in the Operation phase. Hence, a tool support has been implemented to automate the generation of all artefacts.

Finally, the CQI phase is shown to clarify how this in conjunction with its phases have been set in order to perform an improved quality cycle encapsulating all phases within the framework.

4.2. The Quality Model Strategy phase

In this phase, it is important for the THOT project purposes to explain that the characterization model must be based on concrete solutions for ECM systems and not in work associated with theoretical proposals in the ECM systems context. The main motivation of this decision is the fact that a new ECM system must be developed, which will be based on an existing ECM system. In addition, the high cost this type of projects entails makes it very risky to implement a solution from the ground up. Thus, it is better that all expected requirements and functionalities have already been validated in the market.

Then, a characterization model has to be defined in order to analyse and evaluate the systems with QuEF. Carrying out this task demands that the main Features and Sub‐Features of all these systems are known. A systematic literature review (SLR) was used to analyse the current situation. A SLR is a means of identifying, evaluating and interpreting all available documents related to a particular thesis in a specific investigation area.

To perform the SLR, the protocol defined by Kitchenmham [22, 23] was chosen. It is one of the most acknowledged in software engineering. This model establishes the necessity of specifying some research questions (RQs) that will guide the work. For this work, the following RQs were proposed:

  • RQ1 ‐ What ECM systems currently exist in the market and what do they offer?

  • RQ2 ‐ How can ECM systems be adapted to the general guidelines of the Andalusian Public Administration?

  • RQ3 ‐ What is the most appropriate ECM system that the Andalusian Public Administration, and more specifically, the contracting services for transport and infrastructure constructions must use?

  • RQ4 ‐ What areas of improvement are needed for the selected ECM system?

The databases considered for this SLR were ACM Digital Library, EiCompendex, IEEE Xplore, ISI Web of Knowledge, Science Direct, SCOPUS, Springer Link and Wiley InterScience Journal Finder.

Once the method was applied, the results showed that the following tools needed to be analysed: Alfresco [24], Documentum [25], Nuexo/Athento [26], IBM FileNet [27] and OpenText [28]. Then, the concept mapping method [29, 30] was executed in order to obtain the characterization model. This method involves all stakeholders in the project. Several meetings were organized with all stakeholders and system providers to discuss requirements in their systems and all this information was used to build the characterization model by the concept mapping method. Concept mapping is a general method that can be used to help any individual or group describe their ideas about some topics in a pictorial form.

In accordance with all these analysed systems and the strategy followed herein, a set of preferences for each element of the Quality Model was defined for adapting it to the project scope.

4.3. The Quality Model Design phase

The QMD phase in QuEF is understood to encompass all the relevant elements to design the Quality Model. As such, the QMD phase defines the necessary basic characteristics to be analysed using the defined templates. Defining these characteristics, it is possible to asses each solution uniformly.

The characterization scheme is composed of 10 features, which respond to the questions identified in the QMS phase. This priority is contextualized within the needs of this project: to define an innovative solution for document management applied to procurement of services and transport infrastructure projects within the Regional Government of Andalusia, Spain. However, the software allows any potential user utilizing QuEF to set priorities based on his/her own needs. These basic characteristics are illustrated below:

FT01: Functional modules. The results obtained in the QMS phase point out that a valid ECM system must include natively and minimally the following functionalities.

FT02: User orientation. Although ECM systems offer standard solutions on its orientation towards the end user, many companies need to use easy and versatile systems because not all their employees have the same user profile to handle computer tools.

FT03: Functionality to capture, access, retrieve and view documents. The ability that lets anyone transform the system depending on the organization preferences or user profile.

FT04: Documental lifecycle. This feature enables the user to assess the level or degree of support the system offers to the document cycle. The following Sub‐Features of this Feature are described as follows:

FT05: Workflows. This Feature assesses whether the tool supports management with business processes.

FT06: eGovernment. These Features measure the degree of support offered in the context of access and use digital documents.

FT07: Interoperability compliance. A specific section dealing with interoperability has been included: Integration with tools. This Sub‐Feature evaluates whether the ECM system provides mechanisms (e.g. APIs) to integrate with third‐party tools.

FT08: Security and control. One of the major objectives of document management solutions is to ensure information security. This is facilitated by controlling access to the system from inside and outside the organization and managing the relevant documents in such a way that they are either archived or destroyed. Consequently, these solutions must provide services that ensure that the information stored is secure. It evaluates whether the system is functional enough to analyze data, or otherwise, whether the system allows using third‐party tools.

FT09: Architecture. It evaluates whether the system has an open or closed architecture.

FT10: Cost. Cost (both initial and long‐term by maintenance) is one of the most important factors any organization must take into account when choosing an ECM solution.

FT11: Assistance and RM (Roadmap) support. This last Feature listed in the latter group includes aspects for the evaluation of the characteristics support, assistance and roadmap provided by the ECM solution.

4.4. The Quality Model Transition phase

The QMT phase provides guidance to undergo changes in the Quality Model without affecting the QMO phase. It helps to know how to handle changes in the Quality Model.

Along this first iteration of the framework, lots of new trends were considered likely to be included in the Quality Model. This study considered in the beginning Alfresco, Documentum, IBM FileNet and OpenText. Then, in a second iteration of the framework, Nuxeo/Athento and KM (SAP 2013) were included. Nevertheless, KM was rejected in the QMS phase because this system did not comply with the outlined project's scope, thus Nuxeo/Athento was finally considered. New Features and Sub‐Features were included in the Quality Model and fixed in the QMD phase.

4.5. The Quality Model Operation phase

This phase provides guidance to analyse, evaluate and plan the CQI of ECM systems. In the THOT project, the decision concerning the suitability of the most appropriate ECM system was taken that was deemed most suitable for it. As a result thereof, the Quality Model and each set of preferences pertaining thereto have been defined. As such, this weighted Quality Model need to be used so as to analyse and evaluate the different systems. Thus, in this phase, the model was implemented to manage quality in ECM systems, which were analysed by means of checklists. These checklists are artefacts that contain all Features, Sub‐Features and Properties that have been defined to analyse an entity. Hence, checklists are used in order to know the current state of an ECM system.

In addition, QuEF factory has been developed to automate all tasks in this phase. It allows the user to generate all set of artefacts in each phase of QuEF.

Henceforth, the QuEF factory is being developed as a plug‐in of EA. It means that one has to define the Quality Model in EA and explain in which directory the generation of the QuEF‐O Web application ought to be created (Figure 6).

Figure 6.

QuEF factory.

Figure 7.

Checklist menu in the QuEF‐O Web application lifecycle.

Then, the QuEF‐O Web application is generated and the user can use all necessary artefacts to analyse and evaluate all ECM systems found. For instance, Figure 7 shows all checklists that have been generated in terms of the Quality Model that have been defined.

Figure 8.

Functional modules checklist in the QuEF‐O Web application.

Figure 9.

Defined preferences for the sub‐features of Functional modules of the QuEF‐O Web application.

Figure 8 represents the checklist to analyse the functional modules Feature and Sub‐Features. This set of checklist is used in order to study all ECM systems.

As far as preferences are concerned, these have to beset for each element of the Quality Model. For instance, Figure 9 shows the preferences defined in the QMS phase for functional modules Feature.

Finally, each ECM system can be evaluated in terms of the project's scope (Figure 10), taking into consideration the preferences of each element. Consequently, the user can decide what ECM system is the most suitable for his/her needs. Thus, if new elements have to be included in the evaluation, then new iterations of the framework have to be carried out and all artefacts are generated automatically using the QuEF factory. It is the author's view that this way may lead to cost reduction, effort and time associated with and may even improve quality in the quality management processes.

Figure 10.

QuEF‐O Web application evaluation interface.

4.6. Quality Continual Improvement phase

It is very important to consider the continuous quality improvement in a quality management process based on the Quality Model lifecycle. Thus, this objective can be achieved only through a constant monitoring and measurement of all activities and processes involved in quality management. The main goals of this phase are summarized as follows:

  • To use methods from quality management in order to learn from past successes and failures

  • To recommend improvements to all processes and activities involved in the Quality Model management

  • To control and analyse the basic characteristics as well as monitor and validate them in real‐life cases (environments)

  • To suggest improvements to increase Return on Investment (ROI) and Value on Investment (VOI) associated with properties

  • To support the Strategy and Design phases for the definition of new needs and basic characteristics or processes/activities associated with them

  • The results of this phase of the lifecycle must incorporate all the necessary information to

    • Improve the quality of the Quality Model provided

    • Add new properties and basic characteristics that best fit users’ properties and the market

    • Improve and streamline internal processes of the approaches

Therefore, it is the authors’ view that all these practices along the project ought to be considered so as to achieve a continuous quality improvement encompassing all phases of the framework.


5. Conclusions and future work

This chapter shows the results of a research project that aim to implement an ECM system in the Public Administration and has been carried out in a real environment.

The SLR carried out focused on Features and Sub‐Features of ECM solutions that were implemented in existing systems. This is because in the THOT project, the conceptualization of ECM systems is not as relevant as the definition of a set of Features and Sub‐Features to compare existing ECM systems in the market. The conceptualization of ECM systems constitutes a detailed research to guide providers of these systems. Therefore, getting a common Quality Model including the requirements that all these systems must fulfil in the future is essential.

Nevertheless, this project offers providers and users the opportunity to decide what existing ECM system in the market is the most appropriate for their purpose. This decision is of paramount importance, as it is going to be used to implement all business requirements in Public Administration. In consequence, the high cost prevents changing the development project of the new solution, once started. In other words, the project does not focus on the development of a new ECM system, but on deciding which is the most appropriate one in the market.

This study has proposed an evaluation framework concerning different alternatives of ECM systems, and choosing the most suitable one for the scope and context thereof, with the final aim of warranting continuous quality improvement.

QuEF is a framework to manage quality in any product or process, so it can be applied to any entity. It is based on ITIL v3 with the difference that it does not focus on services, but on a Quality Model. Similarly to ITIL v3, it comprises of five phases to ensure the continuous quality improvement of the Quality Model. The aim is to centralize all efforts of quality management on the Quality Model. In addition, the framework also defines protocols and methods to perform each phase, so that all protocols and methods can be systematized. Besides, QuEF‐Factory is a tool support that can generate a Web application for each phase in terms of a Quality Model (QuEF‐S, QuEF‐D, QuEF‐T, QuEF‐O and QuEF‐QCI). QuEF and its tools improve quality management effectiveness and efficiency, since it clarifies the purposes and objectives of management.

Moreover, a set of tools have been evaluated regarding different aspects. Studying the results, they show that Alfresco System and Nuxeo/Athento should be discarded because of the cost and the functionality offered, respectively. The final user should choose between IBM FileNet and Documentum ECM because both are very similar in terms of cost/value.



This research has been partially supported by the NDTQ‐Framework project (TIC‐5789) of Junta de Andalucí́a, by the TEMPROS project of the Spain Ministry (TIN2010‐20057‐C03‐02), by the SoftPLM network, TIN2015‐71938‐REDT, the FEDER Funds and Fujitsu Laboratories of Europe (FLE). We also thank all the staff and researches of the Agency of Public Works of the Regional Government of Andalusia, Spain, for their help, support and professional attitude.


  1. 1. Mohamed Z. (1997). Business process management: A boundaryless approach to modern competitiveness. Business Process Management Journal, 3(1), 64–80.
  2. 2. IDC White Paper sponsored by Ricoh. (2012). Organizational Blind Spot: The Role of Document‐Driven Business Processes in Driving Top‐Line Growth.
  3. 3. AIIM. (2013). The Global Community of Information Professionals,, Accessed 4 May 2016.
  4. 4. Smith, H. A., & McKeen, J. D. (2003). Developments in practice VIII: Enterprise content management. The Communications of the Association for Information Systems, 11(1), 41.
  5. 5. Tyrväinen, P., Päivärinta, T., Salminen, A., & Iivari, J. (2006). Characterizing the evolving research on enterprise content management. European Journal of Information Systems, 15(6), 627–634.
  6. 6. Nordheim, S., & Päivärinta, T. (2006). Implementing enterprise content management: from evolution through strategy to contradictions out‐of‐the‐box. European Journal of Information Systems, 15(6), 648–662.
  7. 7. Munkvold, B. E., Päivärinta, T., Hodne, A. K., & Stangeland, E. (2006). Contemporary issues of enterprise content management. Scandinavian Journal of Information Systems, 18(2), 4.
  8. 8. Scott, J. E. (2011). User Perceptions of an Enterprise Content Management System, 44th Hawaii International Conference on System Sciences (HICSS), p.1,9, 4–7 doi: 10.1109/HICSS.2011.473
  9. 9. Alalwan, J. A., & Weistroffer, H. R. (2012). Enterprise content management research: a comprehensive review. Journal of Enterprise Information Management, 25(5), 441–461.
  10. 10. Grahlmann, K. R., Helms, R. W., Hilhorst, C., Brinkkemper, S., & van Amerongen, S. (2012). Reviewing enterprise content management: A functional framework. European Journal of Information Systems, 21(3), 268–286.
  11. 11. Herbst, A., Simons, A., vomBrocke, J., & Derungs, R. (2014). Critical Success Factors in Enterprise Content Management: Toward a Framework for Readiness Assessment. In Enterprise Content Management in Information Systems Research (pp. 109–124). Springer Berlin Heidelberg.
  12. 12. Rickenberg T. A., Neumann M., Hohler B., & Breitner M., Enterprise Content Management ‐ A Literature Review. AMCIS 2012, available online at, Accessed 29 July 2012.
  13. 13. Haug A., (2012). The implementation of enterprise content management systems in SMEs. Journal of Enterprise Information Management, 25(4), 349–372.
  14. 14. Van Rooij, J. C. (2013). Legacy Issues in the Implementation of Enterprise Content Management (ECM). International Journal of Information, 3(3), 120–123.
  15. 15. ISO2709:2008, Information and documentation—format for information exchange,, Accessed 4 July 2016.
  16. 16. ISO15836:2009, Information and documentation—the Dublin Core metadata element set,, Accessed 4 July 2016.
  17. 17. ISO10244:2010, Document management—business process baselining and analysis,, Accessed 4 July 2016.
  18. 18. JUPITER Project, Government of Andalucia, usuario/listado/fichacompleta.jsf?idProyecto=19, Accessed 15 July 2016.
  19. 19. ERIS‐G3 Project, Government of Andalucia, usuario/listado/fichacompleta.jsf?idProyecto=680, Accessed 15 July 2016.
  20. 20. Archiva (@rchiva), Government of Andalucia,‐lec/areatecnica/archiva, Accessed 4 July 2013.
  21. 21. Domínguez‐Mayo F.J., Escalona M. J., Mejías M., Ross M., & Staples G. 2012. A Quality Management Based on the Quality Model Lifecycle. Computer Standards & Interfaces. 34(4), 396–412.
  22. 22. Kitchenham et al. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. Version 2.3. Department of Computer Science, University of Durham, Durham, UK. EBSE‐2007‐01.
  23. 23. Kitchenham B., Brereton O. P., Budgen D., Turner M., Bailey J., & Linkman S. (2009). Systematic literature reviews in software engineering—A systematic literature review, Information and Software Technology, 51(1), 7–15, ISSN 0950‐5849,
  24. 24. Alfresco, available online at, Accessed 19 July 2016.
  25. 25. Documentum, available online at, Accessed 19 July 2016.
  26. 26. Nuxeo/Athento, available online at, Accessed 19 July 2016.
  27. 27. IBM FileNet, available online at http://www‐, Accessed 19 July 2016.
  28. 28. OpenText, available online at, Accessed 19 July 2016.
  29. 29. Social research methods,, Accessed 17 November 2014.
  30. 30. Trochim, W.M.K. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12, 1–16.

Written By

José González Enríquez, Francisco José Domínguez Mayo, Julián Alberto García García, María José Escalona Cuaresma and Manuel Mejías Risoto

Submitted: August 18th, 2016 Reviewed: October 6th, 2016 Published: February 22nd, 2017