Assessing the Capacity for Engineering Systems Thinking (CEST) and Other Competencies of Systems Engineers

This chapter introduces a tool for assessing engineers' interest in what is required from successful systems engineers, or in other words, assessing the extent of engineers' systems thinking. What is required from successful systems engineers (the characteristics of successful systems engineers) is commonly called 'competencies of successful systems engineers' and much activity to develop systems engineering competency models has been done in recent years. A summary of several systems engineering competency models is presented in the chapter. The competency model that has been used as the underpinning basis for the developing of the assessment tool presented in this chapter is the CEST (Capacity for Engineering Systems Thinking) model. The main reason for choosing this model is presented in the chapter and then the model itself and several principles for assessing engineers' systems thinking are presented. Finally, the assessment tool is presented as well as the main methods that have been used for validating the tool.


Introduction
This chapter introduces a tool for assessing engineers' interest in what is required from successful systems engineers, or in other words, assessing the extent of engineers' systems thinking.What is required from successful systems engineers (the characteristics of successful systems engineers) is commonly called 'competencies of successful systems engineers' and much activity to develop systems engineering competency models has been done in recent years.A summary of several systems engineering competency models is presented in the chapter.The competency model that has been used as the underpinning basis for the developing of the assessment tool presented in this chapter is the CEST (Capacity for Engineering Systems Thinking) model.The main reason for choosing this model is presented in the chapter and then the model itself and several principles for assessing engineers' systems thinking are presented.Finally, the assessment tool is presented as well as the main methods that have been used for validating the tool.

Systems thinking and CEST
Systems thinking is what makes systems engineering different from other kinds of engineering and is the underpinning skill required to do systems engineering" (Beasley & Partridge, 2011).Systems thinking, according to Senge (1994), is a discipline for seeing the whole.Engineering Systems Thinking is hypothesized as a major high-order thinking skill that enables individuals to successfully perform systems engineering tasks (Frank, 2000;2002).Systems engineers need a systems view or a high capacity for engineering systems thinking (CEST) to successfully perform systems engineering tasks.Research found that this ability is a consistent personality trait and that it can be used to distinguish between individual engineers (Frank, 2006).CEST may be developed through experience, education and training (Davidz & Nightingale, 2008;Kasser, 2011) and can be assessed (Frank, 2010).Moreover, well designed and taught systems engineering courses may accelerate systems thinking development.
The chapter introduces a tool for assessing engineers' CEST.Since there is no known way for directly 'measuring' thinking skills of individuals, an indirect way is needed, for example, IQ tests are pen-and-paper indirect tests for 'measuring' the intelligence of individuals.
One of the main assumptions made by Frank (2010) is that in order to be a successful systems engineer, one must have both a will and an interest to be a systems engineer.
In addition, as mentioned, successful systems engineers possess a high level of engineering systems thinking (CEST).Thus, the three components discussed here -success in a systems engineering position, an interest in systems engineering positions and CEST-are all interconnected and interrelated.The will and interest to be a systems engineer basically means the desire and interest to be involved with job positions that require CEST.In other words, we may hypothesize that there is a high positive correlation between the engineering systems thinking extent (CEST) of an individual and his/her interest in what is required from successful systems engineers.Figure 1 is a simple concept map that depicts the relationships between these three components: Fig. 1. the relationships between the desire, successful SE and CEST If this hypothesis is supported, then it enables developing a method for assessing the extent of CEST of individuals.This is because interests may be assessed by an interest inventory which is a very common and frequently used to help people choose a profession, and as a selection tool (to determine whether a certain individual is suitable for a certain role) in the recruiting process (Anastazi, 1988).This chapter introduces a tool for assessing engineers' interest in what is required from successful systems engineers, or in other words, assessing the extent of the engineering systems thinking.

Systems engineering competency models
What is required from successful systems engineers (the characteristics of successful systems engineers) is commonly called 'competencies of successful systems engineers' and much activity to develop systems engineering competency models has been done in recent years.A summary of the following models is presented below:

MITRE systems engineering competency model
The MITRE competency model (Metzger & Bender, 2007) consists of 36 competencies organized into five sections: enterprise perspectives, systems engineering life cycle, systems engineering planning and management, systems engineering technical specialties, collaboration and individual characteristics.For example, the section 'enterprise perspectives' consists of three competencies -comprehensive viewpoints, innovative approaches and foster stakeholder relationships and the section 'collaboration and individual characteristics' consists of nine competencies -building trust, building a successful team, communicating with impact, persuasiveness and influence, facilitating, managing and championing change, high quality standards, results orientation, adaptability and integrity.

Systems thinking enablers
According to Davidz and Nightingale (2008), the primary mechanisms that enable systems thinking development include: experiential learning, a supporting environment and certain individual characteristics, such as thinking broadly, curiosity, questioning, being openminded, communication, tolerance for uncertainty, strong interpersonal skills and 'thinking out of the box'.

Advancing the practice of systems engineering at JPL
The JPL (Jet Propulsion Laboratory) competency model presented by Jansma and Jones (2006) refers to personal behaviours and processes.The personal behaviours are presented in five groups:


Leadership Skills -has the ability to influence; has the ability to work with a team; has the ability to trust others; communicates vision and technical steps needed to reach implementation; mentors and coaches less experienced systems engineers.


Attitudes and Attributes -has intellectual self-confidence; has intellectual curiosity; has ability to manage change; remains objective and maintains a healthy scepticism.


Communication -advances ideas and fosters open two-way discussions; communicates through storytelling and analogies; listens and translates information.


Problem Solving and Systems Thinking -manages risk; thinks critically and penetrates a topic in a methodical manner. Technical Acumen -successfully expresses a technical grasp of system engineering at all levels; is a generalist in nature; with proven technical depth in one or two disciplines; has proven knowledge of systems engineering practices.

Characteristics of the ideal systems engineer
Burk ( 2008) found that the characteristics of the ideal systems engineer are: systems outlook, customer/user/consumer orientation, inquisitiveness, intuition, discipline, communication and cooperation (but not capitulation).

The maturity model framework
The maturity model for the competency of systems engineers is based on an assessment of an individual's skill against ability in each of three broad dimensions -knowledge (systems engineering and domain), cognitive characteristics (systems thinking and critical thinking) and individual traits.The maturity model is designed in such a manner so as to be a generic maturity model for assessing competency in many practitioner professions simply by changing the knowledge requirements (Kasser & Frank, 2010).
The maturity model is a two-dimensional model.The vertical dimension covers the following three broad areas:  Knowledge of systems engineering and the application domain in which the systems engineering is being applied. Cognitive characteristics, namely the ability to think, identify and tackle problems by solving, resolving, dissolving or absolving the problems in both the conceptual and physical domains. Individual traits, namely the ability to communicate with, work with, lead and influence other people.
The horizontal dimension is based on Kasser, Hitchins and Huynh (2009) who argue that anecdotal evidence exists for five types of systems engineers:  Type I.This type is an "apprentice" who can be told "how" to implement the solution and can then implement it.


Type IV.This type has the ability to examine the situation and define the problem. Type V.This type is rare and combines the abilities of the Types III and IV, namely has the ability to examine the situation, define the problem, conceptualize the solution and plan the implementation of the physical solution.
The two-dimensional maturity model framework shows the assessment of the competency in increasing levels of competency (Type I to V) as presented in the following Table .Declarative knowledge is knowledge that can be declared in some manner.It is "knowing that" something is the case.Describing a process is declarative knowledge.Procedural knowledge is about knowing how to do something.It must be demonstrated; performing the process demonstrates procedural knowledge.Conditional knowledge is about knowing when and why to apply the declarative and procedural knowledge (Woolfolk, 2011).This usually comes from experience.In the The maturity model may serve both as a competency model and a framework for assessing/comparing other competency models (Kasser et al., 2011).
Other systems engineering competencies models found in the literature include:

The CEST competency model
However, the competency model that has been used as the underpinning basis for the developing of the assessment tool presented in this chapter is the CEST model (Frank, 2002;2006).The main reason for choosing this model is that in order to assess systems thinking in engineers, it is necessary, first, to elaborate this thinking skill to elements that can be assessed.The CEST Competency Model presents a list of cognitive competencies that are all related to systems thinking and each one of them can be assessed separately.
Eighty-three competencies of successful systems engineers have been found in the studies and these findings were used to create the CEST Competency Model.These 83 competencies were then aggregated into 35 competencies -16 cognitive competencies, nine skills/abilities (all also related to cognitive competencies), seven behavioural competencies and three related to knowledge and experience.
The 16 cognitive competencies are as follows for successful systems engineers: 1. understand the whole system and see the big picture; 2. understand interconnections; closed-loop thinking; 3. understand system synergy (emergent properties); 4. understand the system from multiple perspectives; 5. think creatively; 6. understand systems without getting stuck on details; tolerance for ambiguity and uncertainty; 7. understand the implications of proposed change; 8. understand a new system/concept immediately upon presentation; 9. understand analogies and parallelism between systems; 10. understand limits to growth; 11. ask good (the right) questions; 12. (are) innovators, originators, promoters, initiators, curious; 13. are able to define boundaries; 14. are able to take into consideration non-engineering factors; 15. are able to "see" the future; 16. are able to optimize.
The nine skills/abilities that are all related to cognitive competencies of successful systems engineers are the ability to: 1. analyze and develop the needs and mission statement, and the goals and objectives of the system; 2. understand the operational environment and develop the concept of operation (CONOPS); www.intechopen.com 3. analyze the requirements (requirements analysis) including capturing requirements, defining requirements, formulating requirements, avoiding suboptimizing, generating System Requirements Documents (SRD), "translating" the concept of operations and the requirements into technical terms and preparing system specifications, validating the requirements, tracing therequirements, ensuring that all needs, goals and external interfaces (context diagram) are covered by the requirements, and allocating the system requirements into lower levels; 4. conceptualize the solution; 5. generate the logical solution -functional analysis; 6. generate the physical solution -architecture synthesis; 7. use simulations and SE tools; 8. manage systems processes including interface management, configuration management, risk management, knowledge/data management, resource management, integration, testing, verification and validation; 9. conduct trade studies, provide several options and rate them according to their costeffectiveness.
The seven behavioural competencies of successful systems engineers are as follows: 1. be a team leader; 2. be able to build, control and monitor the project (technical management); 3. possess additional management skills (negotiators, resolving conflicts.etc.); 4. be characterized by good communication and interpersonal skills; be able to collaborate; be strong team players; establish trusting relations with stakeholders; 5. be capable of autonomous and independent self-learning; 6. characterized by having a strong desire/will to deal with systems projects; 7. characterized by seeing failures not as "the end of the road" and by having tolerance for failure.
The three competencies related to knowledge and experience for successful systems engineers are as follows: 1. expert in at least one science or engineering discipline (core disciplines such as physics, electrical engineering, mechanical engineering, aeronautical engineering and industrial engineering); 2. possesses technical general knowledge in additional science/engineering disciplines (interdisciplinary and multidisciplinary knowledge); 3. experience of several years in working as a domain and as a junior systems engineer in several systems projects.
In organizations and projects there are many different kinds of job positions that may be included in the systems engineering category.Different positions require different competencies, for example, a systems engineer who works in marketing needs different knowledge, skills and behavioural competences from those of a systems engineer who deals with integration or a systems engineer who deals with verification and validation.In addition, it is unlikely that a successful systems engineer would possess all of these 35 competencies.It is more likely that a certain systems engineer possesses part of the listed competencies and is employed in a position that requires these specific characteristics.Thus, it is not enough to assess CEST by the final score of the assessment tool presented below.
Analyzing the answers to each question is important as well.
However, it appears that a set of core competencies do in fact exist, necessary to all systems engineers, independent of their specific position.It is a matter of hierarchy.Every job level requires competencies suitable for the said level.The higher the systems engineering position in the organization/project hierarchy, the higher the level of required cognitive competencies, skill/ability and behavioural competencies, and broader knowledge needed.

Assessing CEST
The battery for assessing CEST in its final stages will comprise:


Field tests.In the field test the examinee will be asked to develop and present a functional flow block diagram that describes the functional (logical) and physical architecture of a system that meets a given specification.


Lab test.In the future, the possibility of adding a lab test will be considered.In this lab test the capability for global processing by the right hemisphere will be tested (Evert & Kmen, 2003).The field test and the lab test are not in the purview of this chapter.

The interest inventory for assessing CEST
As said earlier, the will/desire and the interest to be a systems engineer (to be involved in systems projects) mainly means the will and interest to deal with situations that require systems thinking.In addition, one of the seven behavioural competencies of a successful systems engineer is a will/desire to be a systems engineer (to be involved in systems projects) -see competency number 6 in the list of the seven behavioural competencies aforementioned in the CEST competency model section.These two findings lead to the conclusion that the will/desire to be involved in positions that require engineering systems thinking predicts success in systems engineering positions.This will/desire can be assessed by an interest inventory.As mentioned above, an interest inventory is a very common tool which is frequently used to help people choose a profession and as a selection tool in the recruiting process (Anastazi, 1988).
Usually, the items in interest inventories deal with preferences, specifically likes and dislikes regarding a diverse group of activities, jobs, professions or personality types.Likewise, the items included in the tool discussed in this chapter refer to ranges of likes and dislikes regarding systems engineering activities, various disciplines and knowledge required from systems engineers, systems engineering activities and types of people involved in projects.
In its present version the tool consists of 40 pairs of statements.For each pair, the examinee has to choose between the two statements according to his/her preference.The examinee checks answer "A" if he/she prefers the first statement or answer "B" if he/she prefers the www.intechopen.com The following sample question is based on the finding that successful systems engineers possess interdisciplinary and multidisciplinary knowledge -see competency number 2 in the list of the competencies related to knowledge and experience aforementioned in the CEST competency model section.

Sample question
No. 17 The following sample question is based on the finding that successful systems engineers are able to analyze and develop the needs and mission statement, and the goals and objectives of the system -see competency number 1 in the list of the abilities and skills of successful systems engineers aforementioned in the CEST competency model section.

Sample question No. 22
-A.I like to discuss the needs with the customer.-B.I prefer to leave the contact with the customer to marketing experts.
The following sample question is based on the finding that successful systems engineers are innovators, originators, promoters, initiators and curious -see competency number 12 in the list of the cognitive competencies of successful systems engineers aforementioned in the CEST competency model section.

Validity of the interest inventory
Four types of validity have been checked in a series of studies (Frank, 2010) -content validity, contrasted groups validity, concurrent validity and construct validity.

Content Validity
The proposed tool was developed and the content validity was achieved by basing the items of the interest inventory discussed here on a literature review including the INCOSE SE Handbook Version 3 (INCOSE, 2006), laws of the fifth discipline and systems archetypes (Senge, 1994), systems thinking principles (Kim, 1994;Waring, 1996;O'Connor and www.intechopen.com, 1997;Sage, 1992), some principles of systems dynamics (Sweeney and Sterman, 2000;Ossimitz, 2002), the seven 'thinking skills' of systems thinking (Richmond, 2000) and on the findings of a Ph.D. study presented in Frank (2002).

Contrasted Groups Validity
This type of validity is determined by comparing the grades of two contrasted groups.In one study it was found that systems engineers achieved significantly higher scores, as compared to other engineers.In another study the contrasted group validity was checked by comparing the tool's CEST scores of four groups -senior Electrical Engineering students, senior Technology Management students, systems engineers and other engineers.Statistical analyses revealed that: (1) the systems engineers achieved significantly higher scores than the other engineers, (2) the systems engineers achieved significantly higher scores than the Technology Management students and the Electrical Engineering students, while (3) the senior Technology/Engineering Management students achieved significantly higher scores as compared to the senior Electrical Engineering students.This result is not surprising because Technology/Engineering Management students are trained to look at problems holistically.

Concurrent Validity
This type of validity is the correlation between the scores obtained by two assessment tools.
In one study, the concurrent validity was checked by calculating the correlation between the participants' scores using the proposed tool and the appraisal of their supervisor.It was found that the Pearson Correlation Coefficient was close to 0.4 (p<0.05).This result is very similar to the predictive validity of other selection tools.In another study the concurrent validity was checked by calculating the correlation between systems engineers' scores using the tool and the appraisal of their supervisor.The supervisor had been familiar with the participants' systems thinking capabilities for many years.The subjective assessments were all made by the same senior supervisor to decrease bias.It was found that the Pearson Correlation Coefficient between the participants' scores and the supervisor assessments was 0.496 (p<0.05).

Construct Validity
Construct validity indicates the extent to which the tool measures a theoretical construct or characteristic (Anastasi, 1988).The construct validity was checked by factor analysis.The analysis revealed five factors that may be labelled as follows: seeing the big picture, implementing managerial considerations, using interdisciplinary knowledge for conceptualizing the solution, analyzing the needs/requirements and being a systems thinker.These results are compatible with the factors found in an earlier study (Frank, 2006).

Some possible implementations of the assessment tool
Every enterprise strives to fill positions in the organization with employees who have the best chance to succeed.Employees are also interested in entering positions that fulfil their aspirations.Selection and screening processes can help match the interests of both parties, thus contributing both to the organization and the individual.
Many studies show that individuals do not behave and function in the same way in every organizational environment.The meeting point between the characteristics of an individual and the specific environment of his/her workplace often determines the quality of the functioning of the individual.Hence, the goal of the selection process is to help find the optimal meeting point and match the right employee to the right job within an organization.
The selection process for systems engineering positions should reliably predict those employees who can succeed and reject those who are likely to fail.Out of the employees who can succeed as systems engineers, it is necessary to choose those who have the highest chance of succeeding.Since no selection process is perfect, two types of errors are possiblechoosing candidates that fail after they have been placed and rejecting candidates who might have succeeded.These errors have an influence on both the organization and the individual.
From the organization's point of view, rejection of candidates who might have succeeded in systems engineering positions can be critical, especially under conditions of an everincreasing shortage of systems engineers.Likewise, placing engineers who later fail in systems engineering positions is also an expensive error, taking into consideration the necessary training which will be invested and the subsequent damage which might be caused to the projects in which they are involved.The tool presented in this chapter may be used for selection, filtering, screening of candidates for systems engineering job positions and for placing the 'right person to the right job'.
think that every employee should gain interdisciplinary knowledge and general knowledge in several fields.-B.I think that every employee should become an expert in his/her field.Learning more fields may lead to sciolism (to know a little about many subjects).
This type is the most common type of systems engineer.Type IIs have the ability to follow a systems engineering process to implement a physical solution once told what to do. Type III.Once given a statement of the problem, this type has the necessary know-how to conceptualize the solution and to plan the implementation of the solution, namely create the process to realize the solution. www.intechopen.com

Table 1 .
Table, where knowledge is required at the conditional level, it includes procedural and declarative.Similarly, where knowledge is required at the procedural level, it includes declarative knowledge.The two-dimensional maturity model www.intechopen.com Paper-and-penciltests.These tests will include three inventories:  An interest inventory -will be discussed in detail in Section 7 below. A knowledge and skills test.The present paper does not discuss the knowledge and skills test.Much work in this field has already been done by the International Council on Systems Engineering (INCOSE), the INCOSE Certification of Systems Engineers Exam working group.This exam is based on the INCOSE SE Handbook (INCOSE, 2006). An aptitude test.Please see several sample questions in Frank (2007).