Systems-of-Systems MS&A for Complex Systems, Gaming and Decision for Space Systems

Abstract

This chapter discusses advanced modeling, simulation and analysis (MS&A) approaches for supporting complex space system, gaming and decision support system (DSS) using systems-of-systems perspective. The systems-of-systems MS&A approaches presented here also address capability-based approach for supporting US defense acquisition life cycle with a laser focus on the pre-award acquisition phase and combined game theory and wargaming for acquiring complex defense space systems. The chapter also provides an overview of existing models and tools for the design, analysis and development of the government reference system architecture solution and corresponding acquisition strategy in a complex defense systems-of-systems environment. Although, the proposed MS&A approaches presented here are focused on defense space systems, but the approaches are flexible and robust that can be extended to any civilian and commercial applications.

1. Introduction

In the past twenty years, US DoD has been undergoing three major transformations concerning the way to (i) fight, (ii) conduct business, and (iii) collaborate with allied countries. These transformations have led to significant changes in US DoD acquisition process, moving away from requirement-based to capability-based acquisition, the adaptation of the Joint Capability Integration & Development System (JCIDS) and Systems-of-Systems perspective in the design and build of future space systems [1, 2, 3, 4, 5, 6]. Figure 1 illustrates the key differences between the requirement-based and capability-based approaches. The red dotted line shown in Figure 1 denotes the area of responsibility between the US Government (USG)¹ and its selected contractor. For requirement-based, the US DoD is responsible for (i) defining the reference system architecture, architecture performance attributes (APAs) and associated key performance parameters (KPP), and architecture trade-space, and (ii) developing Level-A specifications (spec) that can potentially achieving optimum KPP within the defined trade-space. A selected defense contractor is responsible for using the Level A spec to derive lower level requirements (subsystems and components), design and build the system.

Figure 1.

Description of requirement-based and capability-based approaches.

Unlike requirement-based approach, the capability-based approach requires USG to define user objectives and provide required capabilities for meeting warfighter needs. As shown in Figure 1, USG is also responsible for providing technical objectives and goals documents along with the Initial Capability Document (ICD)² that presents APAs, required capabilities, threshold, and objective criteria for meeting the required capabilities. On the other hand, a selected contractor is responsible for defining the architecture trade space and developing appropriate “TBD” documents for the derivation of Level A spec using the USG’s ICD and inputs (e.g., evolving adversary threats, existing US DoD systems’ capabilities, etc). The “TBD” documents shown in Figure 1 are dependent on the acquisition phase. The “TBD” document can be a Capability Development Document (CDD) or a Revised-CDD³. Like requirement-based approach, the selected contractor is also responsible for the (i) flow-down of Level A spec to lower level requirements, and (ii) design and build of the systems. For some defense system acquisition programs, USG also provides Technical Requirement Document (TRD) or System Requirement Document (SRD) along with the ICD to help the selected contractor concentrates on specific operational use cases, APAs and KPPs.

Designing a system for operation in complex Systems-of-Systems environment requires a good understanding of the types of systems-of-systems that the designed system would be operated in. There are three types of Systems-of-Systems, namely, Type 1: A family of System-of-Systems that provides similar core services, e.g., communication services - But each system provides different core service types, e.g., non-secure FDMA vs. secure TDMA communication services; Type 2: An integration of many families of System-of-Systems, when combined, this type of system provides unique Systems-of-Systems capabilities at the enterprise level (i.e., integrated level) - An example of this complex system is a combination of a family of communications Systems with a family of Global Position Satellite systems; and Type 3: An integration of many heterogenous, independent but interrelated types of systems with each system providing distinctive core services.

As pointed out in [7], most of current professional papers, technical reports and System-of-Systems standards considered integration of (i) many systems of the same type of systems together, which is identical to Type 1, and (ii) many different types of systems as a system consisted of many systems and referred to as System-of-Systems, which is identical to Type 3. In this chapter, we focus our discussion on Type 2, since existing System-of-Systems engineering standards and current MS&A approaches can be directly applied to Type 1 and Type 3 but not Type 2.

This chapter presents advanced concepts on systems-of-systems MS&A approaches to support capability-based acquisition of defense space systems. The MS&A frameworks and processes presented here are emphasized on the systems-of-systems architecture trade support phase of the US DoD defense acquisition life cycle. In addition, it addresses systems-of-systems MS&A frameworks, processes, models and tools using game theoretical modeling and DSS for developing optimum acquisition strategy to acquire complex systems. The complex systems discussed in this chapter are mainly focused on defense space systems, but they can be extended to any systems-of-systems for civilian and commercial applications.

The chapter is organized as follow: (i) Section 2 presents an advanced capability-based MS&A framework, including processes and required MS&A tools, to support US DoD acquisition life cycle from the system architecture design phase to sustainment phase; (ii) Section 3 describes a MS&A approach supporting architecture design and analysis of complex systems using systems-of-systems perspective; (iii) Section 4 provides a MS&A approach for acquisition strategy development and optimization supporting pre-award phase; (iv) Section 5 describes existing available systems-of-systems MS&A models and tools supporting the pre-award phase of the acquisition life cycle; and (v) Section 6 concludes the chapter with a conclusion and way-forward.

2. MS&A approach supporting defense acquisition life cycle

Existing US DoD acquiEsition life cycle employs capability-based acquisition and has three key milestones, namely, Milestone A, Milestone B and Milestone C, which correspond to (i) analysis of alternative (AoA) and technology development phase, (ii) system and prototype development and demonstration phase, and (iii) produce and deploy phase, respectively [4]. For US Air Force and Space Force, the MS&A domains⁴ for supporting the three key milestones and associated three phases can be defined as operation, training, test and evaluation, acquisition, analysis, education, experimentation and war-gaming exercise. Based on the identified MS&A domains, Figure 2 proposes an advanced MS&A framework and associated processes supporting US DoD defense acquisition life cycle. The proposed framework defines the (i) input in terms of systems-of-systems CONOPS meeting warfighter needs and stakeholders’ inputs and requirements, (ii) output in terms of data products, and (ii) related MS&A components to support the DoD acquisition life cycle. Following is a high-level description of the proposed framework, including four phases with required MS&A Models and tools and related processes supporting US DoD acquisition life cycle [4]: (i) Phase 1: systems-of-systems architecture trade study supporting pre and post-Milestone A – Addresses required MS&A tools for architecture refinement, evolution and spiral development planning, capability gap analysis, system solution development, convert capabilities to system requirements – This chapter focuses on systems-of-systems architecture design and analysis for the development of cost effective acquisition strategy and acquisition of optimum reference architecture solutions; (ii) Phase 2: systems-of-systems integration and operation support at pre-Milestone B – Addresses MS&A models and tools for assessing and evaluating incompatibility testing and operation; (iii) Phase 3: systems-of-systems testing and fielding support at post Milestone B – Addresses MS&A models and tools for evaluating hardware and software testing and fielding; and (iv) Phase 4: training and sustainment at pre and post Milestone C – Addresses models and tools for supporting on/off-line training and sustainment evaluation.

Figure 2.

MS&A framework supporting US DoD defense acquisition life cycle.

The proposed M&SA framework, including processes, models and tools, presented in Figure 2 can provide support the US DoD acquisition life cycle. The framework allows for USG stakeholders to incorporate their needs using systems-of-systems perspective taking into account (i) Warfighter’s needs (CONOPS and mission threads), (ii) M&SA domains (e.g., operation, training, test and evaluation, acquisition, analysis, education, experimentation and war-gaming), (iii) JCIDS analyses and US DoD Defense of Acquisition Guide (DAG) process [2, 3, 4], (iv) DoD Joint Tactical Architecture (JTA) MS&A standards, (v) USG Stakeholder M&S strategic plan (e.g., [8]), and (vi) USG Stakeholder’s goals, scopes, objectives, Statement of Objectives (SOO), and System Engineering Plan (SEP).

3. MS&A approach supporting architecture design and analysis

The proposed systems-of-systems MS&A approach presented in this section addresses the system architecture design and analysis for Phase 1 at pre and post Milestone A with an emphasis on achieving integrated capabilities at the enterprise level. As discussed in Section 1, US DoD has been using capability-based approach for developing government reference system architecture (GRA) solutions that would be used for generating optimum acquisition strategy to select appropriate contractor(s) for system acquisition. To avoid potential stovepipe GRA solutions, the capability-based approach allows the (i) USG team to develop desired GRA solution(s) in terms of high level required capabilities that are independent of technologies, and (ii) selected contractor to decide what technology enablers (TEs)⁵ should be used to meet the required capabilities dictated by the GRA. Based on the choices of TEs, the selected contractor defines the system trade space and derives the Level-A specification. From the USG perspective, at pre-Milestone A, the system architecture trade space is not well-defined⁶ since the choices of TEs are not available for the system architect to perform architecture design trade study making the search for an optimum GRA solution becomes very challenging. This section discusses how MS&A models and tools should be developed to support the system architecture trade study at (i) pre-Milestone A, where the USG team is responsible for the trade study to generate a reference architecture solution for the development of optimum acquisition strategy, and (ii) post-Milestone A, where a contractor (or multiple contractors) is (are) selected to work with USG team to refine the reference solution and develop associated CDD at Pre-Milestone B.

Practically, at pre-Milestone A, a contractor has not been selected⁷ and the USG team obtains required inputs from warfighter needs and associated stakeholders’ requirements in terms of desired systems-of-systems Enterprise (SOSE) CONOPS and associated mission threads for the desired system to be acquired. It is assumed at this stage that the Capability-Base Analysis (CBA) was completed and that the capability gaps were identified and associated potential capability solutions for the identified gaps were documented in the preliminary Initial Capability⁸ Document (ICD). From USG’s perspective, the USG team’s objective is two-fold, namely, (i) to develop an optimum reference architecture solution meeting warfighter needs along with affordable cost and deployment schedule, and (ii) to finalize the ICD for post-Milestone A. The goal of the MS&A models and tools for this phase is to support USG team achieving these objectives. The key challenge for Pre-Milestone A is the lack of a clearly defined system architecture design trade space due to the intent⁹ of capability-based approach. Figure 3 presents a MS&A approach to address this challenge and support key Milestone A activities, including pre- and post-Milestone A activities. As shown in Figure 3, Sections 3.1 and 3.2 discuss systems-of-systems MS&A approaches for pre-Milestone A and post-Milestone A, respectively.

Figure 3.

MS&A framework and processes supporting milestone A.

3.1 MS&A approach for pre-milestone A

This section emphasizes on systems-of-systems MS&A approach for the pre-award phase at pre-Milestone A. As mentioned earlier, at pre-Milestone A, a contractor has not been selected, and the USG team is responsible for developing reference system architecture with associated program and technical risks. Figure 3 shows that there are four key pre-Milestone A activities requiring MS&A support, including: (i) SOSE CONOPS assessment for identifying SOSE architecture solutions and generating corresponding alternative system architecture solutions, (ii) System architecture assessment and trade study for selecting optimum system architecture solutions, (iii) Acquisition strategy development and optimization, and (iv) Pre-award risk assessment. This section discusses MS&A models and tools for supporting these four key activities. Sections 3.1.1, 3.1.2, and 3.1.3 describe SOSE MS&A approaches for pre-Milestone A activities, including SOSE CONOPS assessment, system architecture assessment, and pre-award risk assessment, respectively. As shown in Figure 3, the SOSE MS&A approach for the acquisition strategy development and optimization will be addressed in Section 4.

3.1.1 Approach for SOSE CONOPS assessment in pre-milestone A

The MS&A approach for SOSE CONOPS assessment discussed in this subsection is derived from [9, 10] with an emphasis on the design and build of a new space system that can be deployed in a complex space SOSE. The complex space SOSE can be assumed to have three families of systems (FOSs), namely, FOS of communications satellites, FOS of sensing satellites and FOS of position-navigation-and-timing (PNT) satellites. This section describes a MS&A approach to design and build of a new space system in this complex space SOSE environment.

The proposed MS&A approach employs advanced orbital mathematical and complex space systems simulation models for the assessment of a pre-defined SOSE CONOPS to identify the alternative systems-of-systems architecture solutions meeting warfighter and stakeholders needs [9]. This approach allows the system architecture solution to be optimized within a selected set of alternative systems-of-systems architectures using appropriate APAs and KPPs. Recently, USG has been using the “Resilience” attribute for assessing and optimizing SOSE CONOPS performance [11, 12, 13, 14]. The Resilience attribute encompasses avoidance, robustness, reconstitution and recovery. Practically, Resilience Capacity (RC) metric is defined as the system resilience against an adversary threat, and RC is a value that represents a fraction of system capability that is retained after the recovery and reconstitution steps. Mathematically, RC is a function of:

• Avoidance - R_AV: is a measure of how likely it is that the threat can be fully avoided,

• Robustness - R_RO: is a measure of how much capability is preserved should avoidance failed,

• Recovery - R_RV: is a measure of the lost capability can be recovered, and perhaps how quickly it can be recovered for a specific mission, and

• Reconstitution - R_RC: is a measure of the total capability can be replaced, and perhaps how quickly it can be replaced.

Mathematically, RC can be expressed as follow [14]:

$$\begin{array}{l}RC={\mathrm{R}}_{AV}+\left(1-{\mathrm{R}}_{AV}\right){\mathrm{R}}_{RO}+\left(1-{\mathrm{R}}_{AV}\right)\left(1-{\mathrm{R}}_{RO}\right){\mathrm{R}}_{RV}\\ +\left(1-{\mathrm{R}}_{AV}\right)\left(1-{\mathrm{R}}_{RO}\right)\left(1-{\mathrm{R}}_{RV}\right){\mathrm{R}}_{RC}\end{array}$$E1

For defense space applications, the most pronounce threat is the radio frequency interference (RFI) threats from both friendly and unfriendly sources. Thus, R_AV, R_RO, R_RV, and R_RC can be defined in terms of the SOSE architecture¹⁰ performance as follow:

• R_AV = % of time SOSE is free from any RFI threats and the required SOSE network nodes relate to sufficient Link Margin (LM), i.e., no drops of communications links due to insufficient LM in the presence of RFI,

• R_RO = Mean SOSE Network Score when RFI present and/or no connectivity drops due to insufficient LM

• R_RV = Mean Network Score when band switching increases SOSE Network Score

• R_RC = Mean increase in SOSE Network Score due to optimal assisting satellite.

SOSE network score is used to assess and evaluate the SOSE network states. The SOSE network score is calculated by the number of communication pairings (e.g., Ground Terminal 1 connected to Satellite 1) possible in the current state divide by the number of pairings possible in an ideal State. It is the probability two arbitrary SOSE network nodes can communicate or connect to each other. Thus, the SOSE network score is defined as:

$$\mathit{SOSE}\mathit{Network}:\mathit{Score}=\frac{{\displaystyle {\sum }_{i=l}^N\left(\begin{array}{c}\hfill l\hfill \\ \hfill 2\hfill \end{array}\right)}}{\left(\begin{array}{c}\hfill N\hfill \\ \hfill 2\hfill \end{array}\right)}$$E2

where $l$ is the number of fragmented network i, N is the total number of fragmented networks, and $\left(\begin{array}{c}\hfill l\hfill \\ \hfill 2\hfill \end{array}\right)$ is the Binomial coefficient.

RC and SOSE network score models are used to evaluate and assess SOSE communications LM and SOSE network availability [9, 10]. In addition to RC model, [9, 10] also recommended two additional models that are very useful in the SOSE CONOPS assessment, namely, Resilience Assessment Index (RAI) and Spectrum Resiliency Assessment Index (SRAI). RAI Model is used to generate a “Heat-Map” for identifying areas impacted by RFI threats and associated reconstitution’s quality (R_RC). SRAI Model generates a “Heat-Map” to show the likelihood that a space system can access to the allocated frequency-band in the presence of RFI events. A description of RAI and SRAI models is provided in [10].

Figure 4 describes an advanced MS&A approach with desired simulation models for SOSE CONOPS assessment. Based on the warfighter needs and related stakeholders, SOSE CONOPS can be developed to address warfighter needs using required SOSE databases. The required SOSE databases include practical operational systems that can impact the pre-defined SOSE CONOPS. The pre-defined SOSE CONOPS focuses on defense space systems and defense space enterprise, and the operational space systems can impact the defense space enterprise operations, including civilian space and commercial space enterprises. in addition to RAI-RFI, SRAI and RC models, additional mathematical and simulations models are required to perform SOSE CONOPS assessment, including SOSE orbital analysis and simulation, dynamic LM calculation, satellite performance, and avoidance models.

Figure 4.

MS&A approach for SOSE CONOPS assessment.

SOSE orbital-analysis-and-simulation model is used to simulate the RFI threats and dynamics of space systems of interest providing accurate SOSE network nodes and associated positions and network nodes connectivity. The dynamic-LM-calculation model simulates and evaluates link margins of SOSE communications links among SOSE network nodes and calculates network score. The satellite-performance model simulates and evaluates satellite system performance, including signal-to-noise ratio (SNR) calculation and processing time estimation for assessing the recovery time from the threat. The avoidance model simulates space system threat avoidance techniques including antenna beam nulling and adaptive modulation-and-coding techniques to assess and evaluate if the RFI threats can be avoided Figure 4.

As described in Figure 4, the inputs to the SOSE CONOPS MS&A models and tools are required warfighter capabilities, RFI threats and sources of threats, operational use cases and operational constraints. The MS&A output is a set of optimum (or the best) alternative architectures based on a pre-defined SOSE CONOPS. Note that the USG team will adjudicate of what is the “best” or “optimum” set of alternative system architecture solutions based on warfighter and stakeholder needs and the SOSE network score for each operational use case associated with the pre-defined SOSE CONOPS.

3.1.2 MS&A approach using multi-criteria decision support system for system architecture assessment in pre-milestone A

To address the system requirements trade space challenges, the proposed system architecture assessment approach should be based on the required warfighter capabilities and market survey results to identify desired TEs for providing the required capabilities. The MS&A approach is derived from [15, 16], where system architecture assessment is based on the technical performance, market, cost, and schedule risks. Technical performance risk is referred to as technology risk and is quantified using Technology Readiness Level (TRL), while market risk is related to market uncertainty and is quantified by Manufacturing Readiness Level (MRL). Rough Order Magnitude (ROM) Cost, TRL and MRL data are collected from a market survey for cost and schedule risk assessment. Figure 5 illustrates a recent advanced MS&A approach for system architecture assessment and an example of an architecture solution output [15, 16]. The approach uses game theory combined with the war-gaming concept to assess and optimize the system architecture solutions using the market survey results. The approach requires input from warfighter and associated stakeholders along with a set of “optimum” alternative architectural solutions obtained from SOSE CONOPS assessment described in Section 3.1.1, and a pre-defined Payoff-and-Cost Function (PCF). The outputs are (i) optimum architecture solution, (ii) associated technology and market risks, and (iii) predicted related schedule and cost risks. The selected optimum architecture solution is captured in terms of selected TEs and DoD Architecture Framework (DoDAF) views, including Capability View-1 (CV-1) and CV-2.

Figure 5.

MS&A approach for SOSE system architecture assessment.

The MS&A approach requires systems-of-systems analysts to develop the USG game engine (a.k.a. DAA-PWGE) and Contractor game engine (a.k.a. KTR-PWGE) for assessing and optimizing the architecture solutions under USG perspective and contractor perspective, respectively [15, 16]. The objective of the USG game engine is to minimize cost and technical risk using an appropriate PCF for trading off the affordability and technical requirements. The objective of the contractor game engine is to maximize profit and minimize execution risk using an appropriate PCF for trading off the profit and execution risk. The game engines can play pure game or mixed game depending or survey results. Pure game is used when the contractors are surer of their risk assessments, and there are no “belief” and “weighting” functions are needed for assessing the TE risks. Mixed game is used when contractors are more uncertain of their risk assessments, and hence “belief” and “weighting” functions are needed to characterize the TE risks. For this case, TEs are weighted based on their priorities using either a uniform or triangular distribution. The games are static Bayesian games with the goal to reach Nash equilibrium, where the games have stable solutions to game theoretic problem involving multiple players in which no individual player can improve their payoff by a unilateral change in behavior. The objective of MS&A models and tools is to selects the best architectural solution and associated architecture solution type for risk assessment. Classification of architecture solution type depends on the system and associated systems-of-systems requirements and associated market and technology risks (i.e., uncertainty) [15, 16]. Figure 6 describes an acquisition strategy mapping framework and shows the mapping of requirement type to architecture solution type according to various market and technology risks. Section 4 describes a recommended MS&A approach for acquisition strategy development and optimization using this acquisition strategy mapping framework. Theoretically, for these games, the players can be the USG team and contractors to participate in the games playing action. In practice, during the pre-Milestone A, the USG team can also play the contractor role to determine the win-win acquisition strategy from both USG and contractor perspectives. Detailed description of the game engines can be found in [15, 16].

Figure 6.

Acquisition strategy mapping framework [15, 16].

In practice, when the architecture solution does not converge to a single optimum solution, a brute force approach can be used to force the solution to converge to a single system architecture solution for acquisition strategy development and optimization. Since the brute force approach might not converge or lead to an optimum solution, [17] proposed a multiple-criteria decision model based on the Marquis de Condorcet principle found in the ELECTRE models for addressing the situations when the game models do not yield optimal outcome.

3.1.3 MS&A approach for program risk assessment in pre-milestone A

Based on existing US defense acquisition life cycle, the MS&A approach for pre-award phase at pre-Milestone A, the USG team often assesses program risk associated with the following nine pre-award events, including (i) Program Go-Ahead, (ii) Early Strategy and Issues Session (ESIS) (see AFI 63–138, is a key event), (iii) Acquisition Strategy Review Board (ASRB), (iv) Acquisition Strategy Panel (ASP) (see AFI 63–101, is a key event), (v) Acquisition Strategy Document (ASD) (is considered as a key event), (vi) Strategy Review Board (SRB), (vii) Source Selection Plan (SSP) (see 2011 DOD Source Selection Procedures), (viii) Request for Proposal (RFP) (is also considered as a key event), and (ix) Source Selection and Proposals Evaluation. The MS&A objective for the pre-award risk assessment is to provide MS&A models and tools for evaluating and assessment of the program and technical baseline (PTB) risks at each of the key events. As pointed out in [18, 19], there are nine PTB components, including five Program Baseline (TB) and four Technical Baseline (PB) components, as shown in Figure 7(a) and (b), respectively. Detailed description of these PB and TB components can be found in [18, 19].

Figure 7.

Description of PTB elements [18, 19].

Figure 8 proposes a MS&A approach for the program risk assessment of four TB and five PB components. The approach recommends a set of three MS&A models and tools, namely, (i) MS&A model and tool #1, (ii) MS&A model and tool #2 and (iii) MS&A model and tool #3 for supporting three MS&A tasks, including (i) Program risk quantification task on the roll-up program cost, schedule and performance risks, (ii) PB and TB risks Quantification task assessing impact on (key) pre-award acquisition event, and (iii) Task on prediction of PTB risk impact at each (key) pre-award acquisition event, respectively.

Figure 8.

MS&A approach for pre-award risk assessment.

MS&A model and tool #1 is a set of mathematical models for evaluating the overall program risk based on individual TB and PB components’ risks. The overall PTB risk is quantified in terms expected values of the likelihood and consequence that will be placed on the pre-award PTB risk management matrix. A notional PTB risk management matrix is depicted in Figure 9. MS&A model and tool #2 is a set of mathematical models and software tools for (i) Assessing the PB and TB components’ risks, (ii) Quantifying PB and TB risks impact on a specific pre-award event, and (iii) Evaluating PTB risk rolled up and quantification from individual PB and TB components’ risks. The rolled up PTB expected likelihood and consequence values will also be placed on a program risk management matrix like Figure 9.

Figure 9.

A notional program risk management matrix.

Finally, the MS&A model and tool #3 is a set of mathematical models and software tools for predicting the PTB risk at a future acquisition event given the risk assessment results at the current acquisition event. The PTB risk results are quantified in terms of expected likelihood and consequence values.

3.2 MS&A approach for system architecture analysis in post-milestone A

This section describes a MS&A approach for supporting the post-award phase of the DoD acquisition life cycle. At post-Milestone A, a contractor is already selected, and the USG team is responsible for working with the selected contractor to refine the USG reference system architecture and minimize associated technical and execution risks. Figure 10 proposes a MS&A approach for post Milestone A.

Figure 10.

Proposed MS&A approach for supporting post milestone A.

The approach shown in Figure 10 shows the (i) required inputs including warfighter and stakeholder, Performance Assessment Documents (PADs) and Trade Analysis Assumption Documents (TAADs); (ii) desired MS&A activities and supporting MS&A models and tools; and (iii) essential outputs for supporting post Milestone A tasks. The figure is color coded to illustrate appropriate (i) warfighter input, USG Team input, activity and documents, (ii) contractor input, activity and documents, (iii) third party (i.e., related subcontractors) involvement, and (iv) joint USG and contractor teams’ activities. USG team provides ICD, SOSE CONOPS and associated threat scenarios, government reference architecture, and warfighter needs. Using USG’s inputs, including PADs, TAADs and SOSE perspective, the selected contractor team is responsible for developing desired trade space and performing the system architecture trade analysis and refine the government reference architecture (GRA) and providing the “best”(or optimum) system architecture solution and associated system requirements for the development of hardware prototype. The USG team serves as the final adjudicator of what is the “best,” and define which Technical Performance Measures (TPMs) are more important than the others for meeting the warfighter and stakeholder needs, and which residual risks are of the most concern. Typical PADs include (i) Program Technical Objectives and Goals (TOG), (ii) Program TPMs, (iii) Top-Level CONOPS, and (iv) Program Risk Assessment Plan (PRAP). Trade Analysis Assumption Documents (TAAD’s) with typical TAADs including (i) Adversary Capability Document (ACD), (ii) Scenarios Document (SD), (iii) Value Model Document (VMD), (iv) Master Test Plan (MTP), (v) Integrated Master Plan/Integrated Master Schedule (IMP/IMS), (vi) System Capability Baseline (SCB), and (vii) Technology Maturity Baseline (TMB).

For post-Milestone A, the selected contractor is responsible for (i) the architecture analysis (what-if analyses) on the selected alternative SOSE architecture solutions, and (ii) providing all MS&A models and tools for activities supporting SOSE architecture analysis associated with GRA refinement. The contractor MS&A models and tools should be developed for supporting the following SOSE architecture analyses, including, at the minimum, (i) Technology Insertion Assessment: What available technologies could be inserted to gain a significant increase performance without unacceptable increased in risk, (ii) System Capabilities Evaluation: increases/decreases in system capabilities vs. gains/losses in overall system performance, (iii) SOSE CONOPS Assessment: SOSE CONOPS Changes for increased performance vs. ease of integration, (iv) TPMs Evaluation: Benefits for not meeting threshold objective TPMs vs. not to exceed TPMs, (v) Threat/Scenarios Analysis: Benefits for not to address the full baseline operation under different threat/scenarios vs. Benefits to address scenarios beyond the baseline, (vi) Integrated Management Plan (IMP)/Integrated Master Schedule (IMS) Assessment: Where would the USG derive benefit from changing quality standards, cost management system, award fee structure, the schedule for implementation, and (vii) Master Test Plan Analysis: Address changes in planned test facilities, test resources, or test restrictions that would provide overall benefit to fully testing the capabilities of the system.

4. MS&A approach for acquisition strategy development and optimization supporting pre-award phase

The proposed MS&A approach for the acquisition strategy development was derived from [15, 16], where an acquisition strategy is developed based on the selected SOSE architecture solutions and associated technical and program risks and cost and schedule risks. Based on the technology (TRL) and market (MRL) uncertainty risk assessment results associated with the selected SOSE architecture solutions, the proposed MS&A approach uses advanced acquisition strategy mapping framework, as shown in Figure 6, to select an appropriate contract type with associated optimum architecture solution. Depending on the TRL and MRL assessment results, an optimum contract type is chosen and appropriate game theoretical type is selected to optimize the contract parameters, including target price, sharing ratios (SR) and contract fees allowing maximum USG savings with “increased competition” or “increased number of bidders.” Currently, References [15, 16, 17] only addressed the three contract types, including Firm Fixed Price (FFP), Fixed Price Incentive Firm (FPIF) and Cost Plus Incentive Firm (CPIF). Figure 6 also describes requirements classifications, risk assessment classification, acquisition strategy mapping, and architecture solution classification and risk assessment for architecture risk assessment.

Figure 11 describes a recent advanced MS&A approach for supporting acquisition strategy development and optimization along with an example of an acquisition solution output [15, 16]. As shown in Figure 11, the approach requires input from warfighter and associated stakeholders, a set of optimum alternative system architectural solutions obtained from SOSE CONOPS assessment, and a pre-defined PCF¹¹ to evaluate USG saving and contractor profit along with the contract’s parameters. Additionally, the required inputs to the proposed MS&A models and tools include USG architecture solution type, risk assessment results, cost distribution, corresponding contract type. The outputs include (i) Optimum acquisition strategy¹² and contract type and associated contract parameters, including incentives, target price, SRs and fees, (ii) USG saving (payoff), (iii) contractor profit (payoff), (iv) Number of potential bidders (i.e., increase competition), and (v) Risk results in terms of technology (technical and performance) and program (cost and schedule) risks. The acquisition strategy depends on the program and technical risks assessment of the selected optimum architecture solution obtained from the SOSE-CONOPS-assessment model in Section 3.1.1 and system-architecture-assessment model in Section 3.1.2, hence these two MS&A models will be tightly coupled with the acquisition strategy development and optimization MS&A models discussed in this section.

Figure 11.

MS&A approach for acquisition strategy development and optimization.

5. Existing SOSE MS&A models and tools

This section provides a summary of existing MS&A models and tools for supporting pre-award phase of US DoD acquisition life cycle. Section 5.1 focuses on the models and tools for SOSE CONOPS assessment, Section 5.2 for system architecture assessment, and Section 5.3 on space systems acquisition strategy development and optimization using game theoretic and multi-criteria decision support system.

5.1 Models and tool supporting SOSE CONOPS Assessment

The MS&A models and tool implemented in Matlab¹³ for SOSE CONOPS assessment presented in this section are derived from [10]. The models and Matlab tool focus on space SOSE [9, 20] in the presence of friendly RFI threats. The current Matlab models and tools include public open source databases for military, commercial and civilian satellites and ground systems. They can be used to evaluate key SOSE CONOPS performance metrics, including (i) Communication LM and communication link availability in terms of network score, and (ii) the three key resiliency metrics for measuring spectrum resiliency against RFI threats. The three key resiliency metrics are (a) Resilience Assessment Index against RFI (RAI-RFI), which is a measure of “Reconstitution” metric calculating the probability of a ground/satellite system being disrupted by RFI and its ability to reduce RFI by re-routing the desired signal to avoid RFI threats, (b) Spectrum Resiliency Assessment Index against RFI (SRAI-RFI), which is a measure of “Avoidance-Robustness-and-Reconstitution” metric for evaluation of the ability of a system that can access the spectrum and be able to response to a disruptive event - SRAI-RFI is a metric calculating the probability that a system can access to its allocated RF frequency band in the presence of RFI threats, and (c) Resilient Capacity against RFI (RC-RFI), which is a measure of “Avoidance, Robustness, Reconstitution, and Recovery.” The RAI-RFI Model generates a “Heat-Map” to show areas impacted by RFI threats, SRAI-RFI Model generates a “Heat-Map” to show the likelihood that a communication system can access to the allocated frequency-band in the presence of RFI events, and RC-RFI Model generates SOSE communication LM and link availability for the “areas identified by RAI-RFI and SRAI-RFI” models. Figure 12 shows the Matlab Graphic User Interface (GUI) of the Matlab tool describing 2-Dimension (2-D) and 3-D simulation of a notional SOSE CONOPS. The tool can generate a set of potential SOSE architecture solutions with minimum performance degradation in the presence of RFI threats.

Figure 12.

Matlab GUI for SOSE CONOPS assessment.

5.2 Models and tools supporting system architecture assessment and multi-criteria decision support system

The available MS&A models and tool implemented in Matlab¹⁴ for system architecture assessment presented in this section are collected from [18, 19, 21, 22, 23, 24]. The Matlab models implemented static Bayesian games with (i) complete information games and associated pure and mixed games, and (ii) incomplete information games and associated mixed games. The Matlab models also incorporated both brute force and multi-criteria decision approaches. The brute force approach iterates the PCFs adjusting USG and contractors gains and losses until the contactors’ architecture solutions converge to UGS solution. The brute-force’s criterion is set to a minimum of two contractors’ solutions are required to converge to USG solution [16, 24]. The multi-criteria decision approach implemented advanced ELECTRE II with five evaluation criteria, including market uncertainty, technological uncertainty, technical and performance risk, cost and schedule risks, and payoffs and costs [17]. Figure 13 depicts the Matlab GUI for evaluating system architectures under both USG and contractor perspectives [16, 24]. The figure shows a notional case assuming Pure game, four potential contractors bidding, with and without enterprise databases. Practically, enterprise database includes program of records (i.e., past programs data from open sources). Non-enterprise open source database includes only survey data collected from the potential bidding contractors. Detailed description of the models and tool can be found in [15, 16, 17, 24]. It should be noted that the MS&A models and tool presented in this section are not intended to predict what contractor has the winning system architecture solution. The intention is to gain insight into the winning architecture solution based on contractors’ supplied information for the development and selection of the USG reference architecture for Request for Proposal (RFP) preparation.

Figure 13.

Matlab GUI for system architecture assessment.

5.3 Models and tools supporting space systems acquisition strategy development and optimization using game theoretic

Like Section 5.2, the MS&A models and tools presented here were also implemented in Matlab [18, 19, 21, 22, 23, 24]. The Matlab simulation models also implemented static Bayesian with complete and incomplete information games and associated pure and mixed games for the acquisition strategy development and optimization. The current Matlab models implemented three common contract types, including FFP, FPIF and CPIF. Figure 14 illustrates the Matlab GUI of the tool for developing optimum acquisition strategy with associated contract parameters under both USG and contractor perspectives [16, 24]. The figure shows a notional case with Pure game, four potential contractors bidding, with and without enterprise databases. Note that enterprise database includes program of records with past programs data from public open sources. Non-enterprise database includes only notional survey data collected from potential bidding contractors.

Figure 14.

Matlab GUI for system acquisition strategy development and optimization.

Figure 14 also shows a notional use case for four contractors (a.k.a. Suppliers) bidding the contract assuming that (i) there are three contractors (Suppliers #1, #2 and #3) bidding using optimum Nash strategy and one contractor (Suppler #4) bidding using non-optimum strategy, (ii) the selected USG reference architecture is obtained from MS&A models and tool presented in Section 5.2, and (iii) CPIF is the selected contract type based on the risk assessment results for the selected system architecture solution. Bidding results show that Supplier #1 has the winning bidding strategy with 37% wins, followed by Supplier #3 with 37% wins and Supplier # 2 with 25% wins. Supplier #4 has the lowest winning bid with 5% due to non-optimum bidding strategy, i.e. not using Nash strategy. Acquisition results captured the key features of the winning bidding strategy, including USG saving for overrun and underrun cases, underrun and overrun formulas.

Again, the MS&A models and tool presented in this section are not intended to predict what contractor has the winning bidding strategy. The intention is to gain insight into the winning bidding strategy based on the selected optimum architecture solution for the development and selection of the USG reference architecture for RFP preparation.

6. Conclusion and way-forward

The systems-of-systems MS&A approaches presented in this chapter focused on recent advanced framework, processes and available models and tools for supporting pre- and post-Milestone A of the US defense acquisition life cycle with capability-based acquisition approach. Proposed MS&A approaches were derived from the USG point of view using systems-of-systems perspective to address optimum reference system architecture solu¬tion and associated acquisition strategy for acquiring the selected solution meeting desired cost, schedule and technical performance. The proposed MS&A approaches and associated Matlab models and tools were primarily focused on pre-Milestone A and developed based on SOSE CONOPS modeling and simulation of resilient space SOSE operations, Bayesian games combined with war-gaming concept and multi-criteria decision support system for optimizing system architecture solutions and associated acquisition strategy. Available Matlab tools were presented for assessing space SOSE CONOPS, evaluating alternative system architecture solutions and optimizing acquisition strategy of common contract types, such as FFP, FPIF and CPIF [15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. In general, the systems-of-systems MS&A approaches presented here can be extended to support other non-defense system and acquisition life cycle from non-government perspectives. Existing Matlab models and tools presented in Section 5 can also be extended to non-space SOSE CONOPS, Bayesian dynamic games with other contract types (such as FPEPA, FPAF, CPAF and CPFF).

The author hopes that this chapter provides MS&A concepts and source of ideas for the readers to develop commercial systems-of-systems frameworks, processes, models and tools for supporting of their own MS&A works.

Acknowledgments

The author would like to thank his esteemed colleagues, Professors Charles Lee and Sam Behseta at California State University in Fullerton, Ms. Navneet Mezcciani and Mr. Garick Lue-chung at The Aerospace Corporation for their continuous support. He also wants to express his deep appreciation to his wife, Thu-Hang Nguyen, for her constant moral support during the process of writing this chapter.

Conflict of interest

The author declares no conflict of interest. The MS&A approaches and system acquisition views presented in this chapter are those of the author and do not reflect endorsement of California State University in Fullerton or The Aerospace Corporation or the US DoD.