Summary of tools and practices to perform each methodological stage.
To excel in the overall business performance, daily processes and activities connected to produce a good or service need to be outperformed. Even though there is extensive literature on performance management and performance management systems, there is still no consensus over the conceptual model of such systems, in what is designated as Operational Performance Management Systems (OPMS). This chapter proposes a new approach to conceive feasible and desirable OPMS tools to assist managers on controlling and responding to operational needs, by combining Design Thinking (DT) and Data Analytics (DA), that provide holistic and deep business knowledge, as well as a data-driven based management. The authors conduct an empirical application through a case study within the context of a European airport’s Baggage Handling System (BHS). The case study procedure follows the proposed methodology’s stages, where the authors construct the problem space with a wide array of collected data, along with the solution exploration and refinement.
- operational performance management
- operational performance management systems
- design thinking
- data analytics
With the rise of the modern society and its socio-economic structure based on companies (public or private, profit or non-profit based) and its interdependences, the need for performance excellence has become of the utmost importance. The twenty-first century marked the beginning of a new millennium where uncertainty is at its high and companies are constantly trying to outstand their business performance to thrive and gain competitive advantage.
To excel on their overall business performance, the same must apply to the operational performance. Thus, operational performance management systems must reflect the problems and impediments hindering the company’s competitiveness. The trends show they are trying to gain knowledge about themselves and the market by leveraging ICT and Data Analytics (DA) capabilities to transform data into information [1, 2]. At the same time, they are focused on understanding the user’s (internal or external) pains and needs and come up with a way of meeting them to improve business performance [3, 4].
This double-sided story shows us both a hard part (described from an objective transformation of the real world into data) and soft part (where humans and their different perspectives of the real world are considered). However, there is little research combining ‘soft’ and ‘hard’ methodologies to tackle problems or develop new products or services which gather the emphatic human-centric approach with the rigorous and objective perspective of data science.
Thus, the author’s objective is to propose a new hybrid methodology which uses a Design Thinking approach to an unknown problem environment and develops analytic-based solutions to create OPMS tools. This is intended to provide:
the full picture of the operational context within a multi-stakeholder environment;
a broad but clear definition of the problem-space to have a common basis for ideation within a collaborative environment;
digital solutions for managers to control effectively their operations;
confirm DT outputs which might seem too intuitive and subjective to traditional companies.
The proposed methodology was tested under real-life conditions by performing a case study investigation methodology on a Baggage Handling System (BHS) company, within a European airport. This allowed to test various tools and the overall procedure and is able to critically assess the advantages and limitations of this methodology for the intended purposes.
2. Methodology foundations
This chapter draws upon three main research topics: Operational Performance Management, Design Thinking and Data Analytics. These topics are briefly addressed hereinafter, as they lay the foundation for the proposed methodology.
2.1 Operational performance management
Drawing upon different Performance Management (PM) literature reviews [5, 6, 7, 8], this subject area has been progressively developed throughout the twentieth and twenty-first centuries. Its evolution can be summarized into four major periods. The progressive phases had different objectives and perspectives over what issues should PM address. They constructed the current understanding of an ever-changing and multi-dimensional performance management discipline, with applications on a wider variety of contexts (e.g. small vs. large organizational dimensions, fixed-teams vs. collaborative environments, operational vs. marketing performance) and with a broader scope of objectives (e.g. flexibility, agility, sustainability) . Subsequently, most PM’s researchers recognize the lack of consensus around PM systems’ designing and applicability [5, 6, 9, 10]. Nevertheless, they adopt open and suggestive frameworks for PM’s characteristics and future considerations.
Altogether, these researchers point future directions to contemplate network performance along internal structure (i.e. collaborative teams) and external environment (i.e. supply chain); dynamic and learning systems for performance management to comply with the disruptive and transformational change on business trends; and much focus on measuring the intangible and social aspects as organizations are progressively depending on knowledge and multi-department teaming for thriving on the competitive business world.
Focusing on the operational environment, Operational Performance Management Systems (OPMS) need to follow these trends in order to create effective and agile management systems which can cope with the dynamicity of the real world and help on enhancing the daily operational performance.
Traditionally, operation managers relied on report-centric performance management in which, within a regular periodicity (e.g. monthly, weekly, daily), managers analyzed the compliance of performance indicators meeting their targets for the various operation’s key functions and proposed changes based on the perceived causes for the target deviation . Conversely, on the more recent years, more dynamic and informative methods have been utilized to provide a faster, simpler and more helpful way of portraying the correct information for a proper operational performance management . These methods comprise the use of dashboards and interactive Business Intelligence tools which take advantage of the current ICT developments to portray the information needed, at the right time, for managers to analyze it.
Additionally, in order to know the crucial information that needs to be portrayed to operation managers, there must be a deep knowledge about the specific operational environment and how it connects with the rest of the organization’s efforts, so the overall business performance can be optimized. Subsequently,  took a design-led approach to link strategic and operational activities to enhance the dissemination of KPI’s management and accountability and change the way PM is perceived and integrated within an organization.
Therefore, the diagram on Figure 1 illustrates the three main points operation managers should focus when developing and managing an OPMS.
Firstly, there is an obvious need for a deep knowledge about the specific operational environment, with a systemic approach over the intertwined processes between internal and external boundaries, and the various interactions between people, machines, and people and machines. On the other hand, there is also the necessity for connecting the strategic and operational performance objectives, with a holistic and integrated view of the whole business. Finally, the operation managers can and should leverage the ICT developments to help reduce unfounded and biased decision-making and provide an automatic integration of the various performance systems to make up a systemic OPMS composed by function-specific operational performance management sub-systems.
2.2 Design thinking and data analytics for developing OPMS
To approach these OPMS issues, the authors explored two different theoretical backgrounds which could be combined to provide an adequate solution for the prior OPMS issues, at the same time they could be leveraged to overcome each theory’s intrinsic limitations. As such, the author initially summarizes each theory’s advantages and limitations on Figure 2.
This diagram depicts both DT and DA positive and negative points and it can be noticed there are two divergent ways of analyzing the real world according to these methodologies. To the left, Design Thinking takes a broad approach to problem-solving, where it explores different problem-solution frames, through intensive validation and by gathering a vast business knowledge allied with a user-centric approach.
As limitations, some argue over the designer’s intrinsic subjectivity on choosing the procedure’s iterations and artifacts used, which affects the traditional business’ requirement for milestones’ definition and compliance. Furthermore, there is a fundamental difference on design thinking and manager thinking methodologies for problem-solving, which can sometimes lead to misunderstandings and attrition on the organizational sphere.
On the other hand, Data Analytics is more focused on analyzing a specific perceived problem or system, with great power for creating valuable and meaningful information for supporting decision-making. It uncovers trends and patterns through immense computing power and algorithmic data processing which could not be accomplished with mere human capabilities.
As limitations, there is a fundamental need for exploring the problem-space to prevent tacking the wrong problem and to lead to the development of a faithful and robust model. Moreover, this type of analysis is very influenced by the intrinsic data quality and the reliable representation of the reality through the captured data. Finally, it needs a dedicated data pipeline to capture, clean and structure the collected data so that different analytic tools can be explored and tested on a flexible and effortless way.
Concluding, the authors propose a combination of these two methodologies to allow for a complete and reliable way of improving the operational performance management systems. The relations and leveraging points between both methodologies and OPMS needs are portrayed on Figure 3. Thus, it illustrates that DT is intended to provide the holistic and deep business knowledge, while DA is supposed to provide a data-driven and fact-based management. Additionally, their limitations are conversely minimized by the complementary methodology, making this a win-win methodology.
3. Methodology proposal
After exploring the perceived topics’ strengths and limitations and presenting the rationale for combining DT and DA to enhance the operational performance management systems, the methodological stages are laid down on Figure 4.
This diagram portrays six main stages. The first two stages (Stages 0 and 1) are only related to Design Thinking steps. These refer to the initial mapping of the organization’s activities and stakeholder’s relationships, while understanding their emotional and intuitive ways of reasoning. Altogether, this allows for multiple framing of the problem-space according to different perspectives.
The third stage (Stage 2) is intended to define the problem-space as it is perceived, and explore the solution-space, by conceiving different solutions for further narrowing and prototyping. This step is presented as being included both in DT and DA methodologies since the problem definition is based on DT practices, while for the solution space exploration the practitioner should already have a DA focus to try to come up with data-driven solutions.
The fourth stage (Stage 3) is focused on the models’ conception for the solutions idealized. To materialize these models, they require the development of proper datasets, along with and its inherent collection and storage processes. For further analysis and construction of the virtual model of the real system, this stage is primarily based on DA practices and tools, and it is a laborious and time-consuming step, since it needs diverse knowledge and infrastructure to capture, pre-process, structure and test different datasets for various analytic tools.
The fifth stage (Stage 4) requires a combination of DA and DT techniques and expertise to test and absorb feedback, so the developed prototypes and models can be improved or selected, grounded on a collaborative creation. This step’s outcome is intended to filter misconceptions and align stakeholder’s expectations, as well as narrowing down the solution space to concentrate the team’s efforts and walk towards the final solution.
The sixth and final stage (Stage 5) proposes the development of the desired solution into a real-life testable model. For this purpose, it is recommended the iterative testing and refining of the perceived solution until all parts are satisfied with the product/service.
This methodology entails an inherent iterative nature between all of the six stages, repeatedly exploring and refining the problem and solution space until the team is satisfied with the frames developed.
Beyond the methodological procedure, there must be a set of tools or practices which guide the practitioner, or at least are used as reference, for applying the conceptualized methodological steps. The proposed tools and practices presented on Table 1 are extracted from prior literature research, but these are not exclusively limited to its assigned stage, nor are they to use strictly as the literature proposes.
|Stage||Tools and practices||Reference literature|
|Stage 0—Constructing the organizational ‘picture’||Activity system mapping||[14, 15, 16]|
|Competitor activity system mapping||[14, 15];|
|Stakeholders mapping||[14, 17, 18]|
|Value Chain analysis|||
|PEST analysis||[14, 20];|
|Five Porter’s Forces|||
|Stage 1—Exploring users’ pains and needs||Shadowing||[14, 22, 23]|
|Semi-structured interviews||[14, 24]|
|Customer narratives/storytelling||[14, 24]|
|Personas||[14, 18, 25]|
|Customer journey||[14, 15];|
|Stage 2—Co-evolution of problem and solution space||Motivational mapping||[14, 15];|
|Concept development||[28, 29]|
|Stage 3—Solution exploration||Data types and data collection;||[30, 31];|
|Data pre-processing||[30, 32];|
|Data warehousing||[30, 33, 34]|
|Statistical tools and models||[30, 35];|
|Machine Learning models||[36, 37];|
|Business analytics and visualization techniques||[38, 39, 40]|
|Stage 4—Prototype and test||Rapid prototyping||[15, 25, 41]|
|Co-creation||[14, 28, 42]|
|Future customer journey||[14, 43];|
|Iterative prototyping||[14, 41];|
|Stage 5–Develop concrete product/service solution||Co-creation||[14, 28, 42, 44]|
|Iterative prototyping||[14, 41];|
|Value delivery||[2, 14, 15, 45, 46];|
|Integration with existing operations||[13, 14, 47, 48]|
4. Case study
The proposed methodology was explored on an empirical case study of an airport’s Baggage Handling System (BHS) of a European airport. This application followed the methodological procedure presented in Figure 4 and used some of the tools and practices presented on Table 1 to help on making sense out of the immense accounts of information gathered.
4.1 Stage 0—Constructing the organizational ‘picture’
This initial stage had the objective of giving an overview of the case study environment, and it was divided in two main topics: an overall industry contextualization; and a specific case study contextualization.
The first topic started with an overview of the air transport industry and its current trends, followed by a deep-down into the airport environment. Here, a brief overview over its control and supervision was made, and some generic characteristics were laid down concerning multi-airport systems, international differences, stakeholder’s mapping and economic chain, and the main regulatory drivers. Afterwards, the focus was directed to the airport’s Baggage Handling System (BHS), where a generic description of its objectives, work process and primary functions was made, along with the specific regulatory drivers and upcoming regulatory frames on BHS industry.
The second topic intended to be company-specific within the case study boundaries. It started by giving a technical description of the equipment and controlling systems used for its operation. Then, the specific BHS ecosystem was portrayed, along with its main stakeholders and their influence/impact on BHS operations. Following, the BHS’ operational performance and its main dimensions were addressed. Afterwards, a thorough description of the operational characteristics and the workflow along the various stakeholders involved on a baggage journey was made, depicting the intrinsic dependencies between the various BHS stakeholders. At last, an organizational overview was done showing the overall structure and the in-site hierarchy of operation and management, as depicted on Figure 5. Subsequently, a final description of the BHS operational management is addressed, in order to understand how the daily processes and concerns were managed and resolved.
4.2 Stage 1—Exploring users’ pains and needs
After constructing the initial industry picture and understanding the surroundings where the BHS is included and its basic characteristics, the next step was to empathize with the various stakeholders which interact or impact the system’s operations. A collection of tools was employed throughout this stage to harness on the gathered data and provide profound insights about the intuitive and relational aspects of human relations and people’s reasoning when performing their jobs.
Approximately 7 hours of shadowing were undertaken to carefully observe BHS control room operators while operating the BHS. Here, the researcher had the opportunity of first-hand contact with the real-time management, which allowed him to pose some specific questions about the interactions between the CR operators and the information systems utilized for the BHS’ operations management. Additionally, several semi-structured interviews with the BHS’ main stakeholders were conducted to obtain a rich and broad view from the various perspectives that come into play when taking BHS operation performance into account. A total of 12 interviews were performed by the researcher, segregated according to the Table 2.
Each semi-structured interview was used to construct a mind map that allow for a summary of the gathered information and themes discussed, which helps to mentally process and reflect over them to generate deeper insights. Alongside these interviews, the author also performed a total of five customer narratives with different passengers, with the objective of understanding some common pains and needs that the final customer of the whole baggage journey could have. In order to get a broader picture and minimize biases throughout these collected stories, the author diversified the customers’ profiles along the five customer narratives (e.g. old vs. new, frequent flyer vs. novice, family travel vs. business).
A total of seven personas were constructed in order to have a concise and coherent definition of the information gathered for further presentation and discussion with a focus group composed by experts. They comprised the main concerns and the global opinion between the interviewees belonging to each stakeholder’s persona. Some stakeholder’s categories were aggregated according to the relative importance to the BHS operational environment and management, resulting on the personas’ set depicted on Figure 6.
4.3 Stage 2—Defining and ideating on the problem-solution space
This stage focuses on developing an iterative and collaborative environment for the discussion and refinement of the perceived problem space, as well as for the exploration of the solution space. This was an iterative process of exploration and discussion, in order to clarify and improve the problem definition and consequently, the solution ideation.
The authors started by reviewing the current insights gathered during the exploration phase, to look deeper into any misconnection or discover profounder insights. Hence, it was reexamined the mind maps and personas created in the prior stage, in order to develop a motivational map. This was used to understand and connect the various stakeholders’ perspectives around a central concern identified throughout different interviews and, deconstruct it to find opportunities and clear the problem definition.
After trying to find connections to continue developing the problem space understanding, there were already many pains and needs identified throughout the data collection processes. Therefore, the authors had to filter them and find the most critical ones, in order to have an efficient and effective dialog with the experts’ group. With this purpose, every pain and need identified until this step were listed and a multi-criteria prioritization was made. As such, the authors created a set of four criteria consisting in:
Relative reference frequency
Relative occurrence frequency
Impact on BHS operations
From this exercise, five main problems were prioritized, which were then taken into discussion with a focus group of experts. This group gathered both managers and engineers with experience ranging from 5 to 20 years with this type of systems, and belonging to operational and back-office teams. The meeting started by presenting a summary of the overall organizational ‘picture’, followed by the presentation and discussion of the five main problems prioritized earlier. As a result, it was agreed on one most desirable opportunity to be pursued, which would be the one to be explored on the solution development stage:
“Prevent the sorters’ congestion”
4.4 Stage 3—Solution exploration
At this point, the pivotal opportunity had been identified and it was the one explored throughout this stage. This opportunity was related to an internal process of the BHS, thus having available automatically generated data to leverage on analytics tools, which was one of the essential research objectives. This stage had different progressive steps in order to delve deeper into this opportunity and approach different solution possibilities, according to their desirability and feasibility in terms of time and data resources available for analysis. Additionally, the BHS information systems’ functioning and their integration had to be fully understood, to comprehend the type of data being captured, and how it could be combined and leveraged with the power of analytics tools.
The first step undertaken by the researchers was to further explore the identified opportunity, using the Motivational Map tool one more time. This enabled the deconstruction of the problem into some more specific issues and concerns, as well as the identification of solution paths to take into consideration. Additionally, it also allowed the narrowing down of the solution space into more concrete and defined opportunities to be further investigated. Hence, the initial opportunity was transformed into a concrete solution to be explored:
“Sorter X congestions’ comprehension – causes and impacts”
Consecutively, the authors had to build a mental model which allowed him to know the dataset he would need to have in order to tackle this opportunity and develop some analytics’ prototypes. This was made with the use of a metaphor with a roundabout, through which the researcher could find an analogy between the real opportunity another common context to help him on reasoning and understanding the flow of baggage through a sorter and the possible causes for congestion.
In order to gather the necessary dataset for further exploration and prototyping on the solution space, the authors had to firstly understand the underlying information systems and the type of data collected by each, and how these could be integrated into a complete coherent dataset for analysis. To help on this task, the author resorted to the Tableau Software™, an interactive visualization software for business intelligence, which allowed to visually display various combinations of variables to understand their relations and the real meaning of the virtual captions. The initial understanding of these IS, along with the mental exercise provided by the roundabout metaphor, allowed for the author to understand what types of data he would need and could have access to, in order to comprehend the sorter’s congestions. Thus, these could be summarized into:
Data about the injection lines, i.e. the amount of baggage which enters the sorter through each injection point;
Data about the transfer chutes, i.e. the amount of baggage which leaves the sorter through each chute;
Data about the assigned baggage destinations, i.e. the amount of baggage inside the sorter which is destined for each transfer chutes;
Data about the sorter’s congestion, i.e. combination of operational factors which indicate the existence of the incapacity of the sorter to deliver baggage into their respective chute;
The construction of the necessary dataset resulted on a combination of different sub-sets retrieved from the various IS, as illustrated on Figure 7.
From system A, it wasn’t possible to get congestion variables since there wasn’t a specific event or a coherent combination of current events being captured that could effectively indicate the congestion. Therefore, the author had to assume a different variable which could indicate congestion: the sorter’s occupancy, i.e. the percentage of sorter trays being occupied relative to the total available trays. From system B, the objective was partially accomplished, since there was data characterizing the desired variables, but the time granularity with which some variables were captured wasn’t enough for the detail level required. As such, the only variable possible to consider was the occupation variable. Finally, the author stepped into system C in order to collect every missing variable to complete the desired dataset. Here, the process of getting this data was more laborious and involved more complex queries in order to align the captured data for the proceeding data pre-processing. From this IS, the author could arrange the variables of the injectors, chutes and assigned destinations of the baggage going through the specified sorter.
From prior data collection, which was gathered from different IS, the author had different data types (i.e. some were counts, others were events) which could not be integrated into a whole coherent dataset to be analyzed. Furthermore, this data came with inherent redundancy, lack of normalization, and even outliers which had to be processed. Hence, different specific steps had to be taken to cleanse and integrate these loose data sub-sets into a whole dataset. Different software tools were used for the various tasks implied on this pre-processing step:
SQL Developer™ to pre-filter and query the various databases;
Spyder™ interactive development environment (IDE) for Python programming language to manipulate and transform data and its structure;
Microsoft Excel™ to easily visualize and validate data and correct punctual errors;
Tableau Prep™ to provide a visual and user-friendly manipulation for joining the various subsets and checking for redundancies;
In the end, a total of 66 variables were taken into the next stage, comprising the following variable sets:
Injectors counts, with a total of 14 injector variables;
Transfer chutes counts, with a total of 27 chutes variables;
Baggage assigned destinations counts, with a total of 24 destination variables;
Sorter’s occupancy level, with one 1 variable;
4.5 Stage 4—Prototyping and testing
This stage was intended to undertake an initial analysis of the relationships within the constructed dataset, followed by the exploration of deeper analytic techniques. This procedure is necessary to build some concrete prototypes which will be tested and evaluated, thus becoming the basis for an iterative co-evolution of the final solution.
This stage unfolded in three main steps. On the first step, some descriptive statistics and the bi-variate correlations matrix were performed to better understand the variables within the dataset. This was performed using the Spearman correlation coefficient since all the variables’ distributions were skewed. This is explained by the large number of zeros when there is no baggage passing through the collecting points, and the existence of a tail towards the positive integer numbers, given the relatively few but dangerous levels of high-traffic. The matrix showed some strong relations (i.e. coefficient bigger than 0.6) between some explanatory variables and the target, which is a good indicator of the possibility of building a regression model.
On the second step, the author tested some linear regressions to create an explanatory model of the sorter’s functioning. From the initial dataset, encompassing every-minute counts from a one-and-a-half-month period along the 66 variables, four subsequent datasets were derived by sampling different time periods available on the initial dataset. This was aimed at mitigating the risk of overfitting, which can occur when a regression model rightfully explains the specific dataset, but it poorly performs when given a slightly different dataset. The resultant models created for each of the four datasets and their discussion gave a good overall confidence over an initial explanatory model for the sorter’s functioning. Thus, considering the regression model created for the whole dataset, since it has the biggest and most complete time period, it is represented on Eq. 1.
Basically, this expression provides a succinct and easy way of controlling the sorter’s occupancy level and consequently, its congestions. It means that the manager can concentrate his efforts on controlling these five variables (i.e. the number baggage that is inside the sorter with destination to the respective eight transfer chutes) and, consequently, he will be able to assure the sorter does not surpass critical levels which can result on congestions. By keeping a focus on these counts and by guaranteeing the respective chutes aren’t jammed, the manager can reduce the sorter’s congestion probability and, consequently, the number of delayed baggage.
On the third and final step, this initial prototype was presented and discussed on a group meeting, which served as a catapult to explore the possibilities of other types of analytic models for further development, as well as to diagnose some general considerations about the overall analytic journey in order to enable future easiness and agility when trying new and different analytic ventures.
This was the final stage conducted during this project’s case study since time motives impeded its further development and conclusion into a final and feasible solution for the operational performance management. However, Stage 4 is still to be continued, according to Figure 8. This is an iterative stage of creating and adapting prototypes according to the various feedbacks received, with the possible return to the problem-space exploration after realizing there is missing information for each new development of the solution-space.
Afterwards, stage 5 would be responsible for pursuing a concrete and defined final solution, also by iterative development and testing with the users. These users do not comprise only the end users, but also the influencers and enablers which help on conceiving and deploying the final product/service. Furthermore, the technical, operational and financial feasibility has to be assessed and leveraged in order to create a true value-adding solution possible to be integrated and used on the operational environment, and not a solution to be used once and dropped afterwards.
This chapter proposes a hybrid Design Thinking and Data Analytics methodology for the conception of Operational Performance Management Systems (OPMS). With this purpose, an extensive literature review was made about the three topics addressed by the methodology: Operational Performance Management, Design Thinking and Data Analytics. This proposition was tested resorting to a case study research methodology, where it was possible to analyze while being deployed on a practical application. It provided evidences of the power in combining DT and DA practices for a full comprehension of the operational context, both in subjective and objective terms. Furthermore, during the empirical application, it was developed an initial prototype which conveys the capacity for readily providing operational performance improvements, as well as to envision broader possibilities which could bring huge value to the operational performance management.
Immediate operational benefits were depicted with an estimate about the savings the regression model could bring to the operational performance management, when in operation. Moreover, this prototype allowed to envision much more outcomes to the operational performance by continuing the cycle of solution exploration and refinement, with the iterative evolution of the analytic solutions and, thus, their final added value to the company. In conclusion, the proposed methodology was found successful, not only as a promising method to better understand and enhance the operational performance management, but also with many collateral effects which point towards a possible broader application. In the end, this method is perceived as completely feasible for the overall conception of holistic operational performance management systems which can take into account the various operational perspectives and needs, along with dynamic and factual tools to help on daily decision-making.