Evaluation Methods of Satellite Navigation System Performance Evaluation Methods of Satellite Navigation System Performance

With the development of global satellite navigation system, for example, global position- ing system (GPS) and so on, some regional navigation systems and augmentation systems are developing rapidly. The continuous development of satellite navigation system has attracted the users ’ attention to satellite navigation performance, which makes the navigation system performance become the key of satellite navigation system competition in the field of GNSS applications. The signal in space (SIS) continuity evaluation model based on the reliability is established, and the mean time between failures (MTBF) is used to characterize the probability that there is no continuity loss in unit time. Aiming at the incompleteness of the current availability model, a per-satellite availability evaluation models based on Markov process is established. Moreover, the constellation availability evaluation model is proposed by combining the satellite failure rate, repair rate and backup situation. By analyzing the measured data, the probability of the continuity and availability of GPS and BeiDou Navigation Satellite System (BDS) are calculated respec- tively. The results are instructive for the study of the availability performance monitoring and the evaluation of global BDS.


Introduction
With the development of global satellite navigation system, for example, GPS and so on, some regional navigation systems and augmentation systems are developing rapidly. The continuous development of satellite navigation system has attracted the users' attention to satellite navigation performance, which makes the navigation system performance become the key of satellite navigation system competition in the field of GNSS application [1][2][3][4]. Meanwhile, the Each stage of a satellite navigation system is inseparable from its performance assessment. The performance evaluation of a navigation system can not only verify whether the performance meets the original design requirements, but also monitor the operating status of the system in real time, provide a basis for performance enhancement of the system, and promote the modernization process of the navigation system [16]. Therefore, in the development of satellite navigation systems, the evaluation of the performance of navigation systems has become a very important part.
At present, China's Beidou satellite navigation system has been able to be compatible with other satellite navigation systems. The development of Beidou satellite navigation system follows the "three-step" strategy, namely: firstly to achieve navigation services in China, secondly to achieve regional navigation services in the Asia-Pacific region, and finally to build a global satellite navigation system with global navigation capabilities. Now it has completed the construction of the second-step regional navigation system and will achieve global coverage in 2020.
The construction of BDS has very important practical significance, but at present it has not yet established a complete performance evaluation system, and it faces severe performance test evaluation problems [17]. Therefore, considering the construction and subsequent development of BDS in China, it is very necessary to carry out research on the performance evaluation method of the satellite navigation system. In-depth analysis of the evaluation index system and method, and the use of measured data to verify it, in order to establish a sound BDS performance evaluation system to accumulate certain experience.

Evaluation algorithm of GNSS signal in space availability
There are lots of researches working on per-satellite availability and constellation availability. Ochieng et al. established a per-satellite availability evaluation model that calculates instantaneous availability based on failure rate and reliability without considering on-orbit restoration performance [18]. In fact, most satellite failures can be restored by ground control segments or resolved by spare satellites. Based on this, Section 2.1 mainly introduces and analyzes the definition and performance evaluation indicators of GNSS. Based on the "GPS Standard Positioning Service Performance Standards" and "Beidou Satellite Navigation System Open Service Performance Specification", the accuracy, integrity, continuity and availability definition as well as evaluation indicators of GPS and BDS are analyzed respectively, which is convenient for subsequent performance evaluation. The results provide the basis. In Section 2.1.2, the SIS availability of GPS and BDS are evaluated and analyzed. The model of satellite availability algorithm is constructed based on the Markov process. Based on the evaluation models, the performance of GPS SIS and BDS SIS is evaluated by using the measured data in Section 2.2. Finally, combined with the backup situation of satellites, the constellation availability evaluation models of SIS and service are proposed in Section 2.3.

Satellite navigation system performance evaluation criteria
Accuracy, integrity, continuity and availability are the four basic properties of satellite navigation systems. Performance indicators and assessment methods vary for different navigation systems. Compared with GLONASS and Galileo, GPS navigation system is more perfect for performance evaluation indicators and results, especially the release of GPS SPS PS, which is the leading direction of satellite navigation system performance standards and evaluation system.

Standard positioning service performance standard of GPS
GPS is the first satellite navigation system to perform performance evaluation, and its civilian standard location service is applicable to civilian users (L1 C/A code) worldwide. Compared with the precision positioning service and the enhanced system WAAS service, the civilian standard positioning service has the lowest positioning accuracy, but the user quantity is the largest.
Up to now, the US Department of Defense has released four versions of GPS SPS PS to show GPS service performance to users around the world [10][11][12][13]. Based on the comprehensive comparison, it is found that with the improvement of GPS performance level, the relevant performance evaluation theory is gradually improved. In the late 1990s, GPS service performance was greatly improved by the implementation of a series of performance enhancement programs. With the cancelation of the optional availability SA policy, the third edition of the GPS SPS PS released in 2001 was revised accordingly. Although the overall framework has not changed, the connotation of the accuracy index has changed a lot, the index parameters of the SIS precision are compressed, and the service reliability is also redefined. At present, the latest version of GPS SPS PS was released in 2008. Based on the summary of the first three editions, the original index system was reorganized and improved.

Satellite navigation system public service performance specification of Beidou
The space segment of the Beidou satellite navigation system consists of five geostationary orbit (GEO) satellites, 27 medium-Earth orbit (MEO) satellites and three inclined geosynchronous orbit (IGSO) satellites. Taking into account the characteristics of the BDS, the China Satellite Navigation System Management Office released the BDS-OS-PS-1.0, which is used to illustrate the performance characteristics and indicators of the public service B1I signal in the regional phase of the Beidou satellite navigation system [19,20].

SIS evaluation algorithm based on Markov process
GPS SPS PS pointed out that the availability of navigation system refers to the percentage of time that the navigation system can provide users with available services within its service area, which characterizes the service capability of satellite navigation systems [13]. As one of the four basic service performances of satellite navigation systems, usability has become a key performance indicator in civil aviation and other application fields, and is an important basis for judging whether the navigation system is reliable [21,22]. The usability study content of the four released versions of GPS SPS PS is compared, and the development trend is shown in Table 1. Table 1, as the research progresses, the availability of satellites is gradually divided into SIS layer availability and service layer availability, and SIS availability is receiving more and more attention. SIS availability can be described in terms of per-satellite availability and constellation availability [13]. From the definition of availability, the availability of SIS is essentially an assessment of satellite availability, i.e., the availability of per-satellite is analyzed, and then the study of per-satellite availability transitions to constellation availability, which in turn reflects the SIS availability of all satellites.

Evaluation model for per-satellite availability
The instantaneous reliability function of a satellite is given in the literature, which only considered the failure rate but ignored the restoration rate. In fact, part of satellite failures can be resolved by the repair of ground control section or the backup of spare satellites. And this is one of the functions of the Markov chain. The life distribution and the failure restoration time distribution are assumed to follow exponential distributions. As long as the satellite navigation system status is given properly, the system can always be described by the Markov process [23,24]. The exponential distribution model is used to describe the reliability of the satellites in many of the literature. When the failure rate of a satellite is assumed to be of exponential distribution, the Markov state transition process of per-slot is shown as Figure 1. GPS  In Figure 1, 0 indicates the system status is normal; 1 indicates the system status is failed; λ is the failure rate of satellite, it is the inverse of mean time between failure (MTBF); μ is the restoration rate of satellite, it is the inverse of mean time to restoration (MTTR); and Δt is the time interval of state transition.
According to reliability theory [25,26], we can get the instantaneous availability of per-slot by utilizing the Markov state transition process: In order to express the per-slot availability more reasonably, when t ! ∞, we get the steady state availability of per-slot through formula (1), and it would be as follows: According to the relationship between and MTBF, and the relationship between and MTTR, formula (1) is equivalent to: where A is the per-satellite availability; MTBF is the reciprocal of λ, and MTTR is the reciprocal of μ.
In Figure 2, λ i (i = 1, 2, 3, 4) is the satellite failure rate of different failure type. μ i (i = 1, 2, 3, 4) is the satellite restoration rate of different failure type. Considering the influence of different failure type, the satellite failure rate λ and the satellite restoration μ rate in formula (2) are equivalent to the following formula:

Evaluation algorithm for constellation availability
In the previous constellation availability models [27], a specific constellation with a defined number (generally 24) of space vehicles was considered, when more space vehicles than required for the constellation have actually been on orbit for most of the time since the declaration of Full  Operational Capability (FOC); Zheng and Ren pointed out that a study on constellation availability is based on per-satellite availability, and MTBF and MTTR of different per-satellite outages directly influence constellation availability in different states [28]; U.S. Department of Defense refers that constellation availability model is based on binomial probability distribution [15], and it can be described as the following formula: where, p k ð Þ is the constellation availability when there are faulted satellites. N represents the total number of satellites in the constellation. k is the number of faulted satellites, and A is the availability of per-satellite.
Formula (5) calculates the constellation availability based on the stationary state of per-satellite availability. The model calculates the availability of the constellation based on the availability of per satellite steady-state. Although it reflects the availability of the constellation under different fault conditions to some extent, it ignores the impact of satellite backup strategy on constellation availability and is a relatively static constellation availability. The availability of constellations is affected by many factors. Because the factors affecting the state of the constellation are more complicated, this section proposes a mathematical model of constellation availability considering the satellite's failure rate, repair rate and backup situation based on the traditional constellation availability model.
Let's explain the state of the constellation: suppose the constellation consists of a basic orbital satellite and a non-orbital satellite. When there is no satellite failure, the state of the constellation is N. if there is a failure of the basic orbital satellite, and the faulty satellite is replaced by the nonorbiting satellite in time, the state of the constellation is considered to have not changed. If the failed satellite is not replaced by the non-orbital satellite in time, the state of the constellation will change. When the base orbit satellite fails, the ground control station will repair the faulty satellite while replacing the non-orbital satellite, and only repair one satellite at a time.
Failure state of constellation system will change with the number of failed satellites. Therefore, on the condition of the above assumptions, the failure state spaces of baseline satellites and spare satellites respectively are 0; 1; 2; ⋯; N f gand 0; 1; 2; ⋯; M f g . The SIS constellation availability model based on Markov chain is set up. In addition, this model the baseline satellites constellation availability and the spare satellites constellation availability are considered respectively. Moreover, the backup situations of spare satellites are combined with the failure state of baseline satellites constellation. Finally, the availability of whole constellation system can be attained.
Þare used to indicate the failure rates of baseline satellites constellation; μ i i ¼ 1; 2; 3; ⋯; N ð Þ are used to indicate the restoration rates of baseline satellites constellation, and both failure rates and restoration rates are on different failure conditions. The corresponding Markov failure state transition process of baseline satellites is shown in Figure 3.
In Figure 3, If the state of failure state space is i at the time of t, then the probability of this situation can be indicated with G i t ð Þ i ¼ 0; 1; 2; ⋯; N ð Þ , and assuming that the time of initial state is t ¼ t 0 . When time increases to t ¼ t 0 þ Δt, the probability that the baseline constellation not having failed satellites is: The failure state space of baseline constellation also has other states, and the probability of these states is as follows:  Failure state space of baseline constellation has different states in different time. The formulas (6)-(10) can be shown by matrix form, then the probabilities of the above different states can be expressed as: Therefore, According to the formula (11) and the initial condition, the probabilities of different states of baseline constellation after the time of T ¼ kΔt can be calculated.
Similar to the baseline constellation, failure state space of spare satellites constellation also has different states in different time. And the probabilities of these states can be expressed as: According to the formula (12) and the initial condition of spare satellites constellation, the probabilities of different states of spare satellites constellation after the time of T ¼ kΔt can be also calculated.
Both baseline constellation and spare satellites constellation have important influence on SIS constellation availability. Considering these influences comprehensively, the model of SIS constellation availability is considered: In this formula, when the number of failed satellites is S, the probability of SIS constellation availability is P s t 0 þ kΔt ð Þ .

Per-satellite availability evaluation of GPS
According to the Markov state transition model, it can be seen that the key to calculate satellite availability is to get satellite failure rate λ and restoration rate μ. Therefore, by analyzing the GPS status report provided by the FAA, the MTBF and MTTR of each satellite with different types of failures can be respectively obtained, and then the availability of the satellite can be obtained.
In order to guarantee the reliability of the evaluated results, this paper makes the statistics based on all outages according to the GPS failure report of FAA. According to analysis, it can be seen that a total number of 51 satellites generates different types of outages from 1999 to 2015. By processing the MTBF and MTTR of satellites, the per-satellite average failure rate and restoration rate can be obtained, which are shown in Figure 4.
The spacecraft number (SVN) has a certain relationship with the order of launch of GPS satellites. Satellites with smaller numbers are launched earlier and satellite models are lower; while satellites with larger numbers are launched later, and satellite models are relatively high. From the analysis of Figure 4 in general, the average repair rate of the higher-model satellites is generally far greater than the average failure rate compared with the low-model satellites.
Combined with the analysis of formula (4), it can be inferred that the availability of higherprofile satellites is generally higher, which reflects the continuous development of GPS satellites to some extent.
By bringing the average failure rate and restoration rate of each of the above satellites into the per-satellite availability model, the availability of GPS per-satellite can be obtained, as shown in Figure 5.
The dotted line is the per-satellite availability minimum standard (0.957) of GPS SPS PS (2008) in Figure 5. If the availability of the satellite is greater than this value, the satellite is considered to meet the availability criteria. As can be seen from Figure 5, except for the availability of In addition, the section compares the availability of satellites according to the type of satellite, as shown in Figure 6.
As can be seen from Figure 6, the average availability of the BLOCK II series satellites is the lowest, and the average availability of the BLOCK IIF series satellites is the highest. With the continuous upgrading of satellite models, the availability of GPS satellites is significantly  enhanced. With the continuous advancement of GPS modernization, the availability of GPS satellites is constantly improving.

Per-satellite availability evaluation of BDS
BDS has not yet established a sound outage forecasting mechanism. Therefore, there is no FAA-like department in China to provide a measured fault report on BDS. Therefore, the paper mainly obtains the Beidou navigation message published by IGS, and extracts its "health word" data as the data source according to the Beidou satellite navigation system spatial signal interface control file (2.1) [29][30][31]. The average failure rate and average repair rate of the satellites obtained by collating the data sources are shown in Figure 7.
Since December 24, 2013, the C13 satellite is in the unhealthy state, and its restoration rate is zero. Comparing Figure 4 and Figure 7, it can be found that the average restoration rate of BDS satellites is generally higher than that of GPS, indicating that the BDS satellite system can repair the satellites more timely when the satellites fail. However, the average failure rate of BDS satellites is also significantly higher than that of GPS, indicating that the frequency of BDS failures is greater.
Bringing the average failure rate and average restoration rate of the BDS into the per-satellite availability model, you can get the single-star availability probability of the BDS, as shown in Figure 8. In Figure 8, since the restoration rate of the C13(MEO) satellite is zero, the satellite's persatellite availability is 0 (indicated by a circle). As can be seen from Figure 8, except for the C13 satellite, the availability of other MEO satellites is in line with the minimum availability standard of 0.91 for MEO satellites in BDS-OS-PS-1.0. In addition to the poor SIS availability of C03(GEO) and C08(IGSO) satellites due to factors such as switching satellites and system instability, the remaining GEO and IGSO satellites meet their respective minimum availability standards of 0.98.

Constellation availability evaluation of GPS SIS
As of 2015, the number of remaining in-orbit satellites was 27 except for satellites that were unable to count MTBF and MTTR values just after launch. The 27 satellites in orbit are distributed over six orbital planes, as shown in Table 2.
In order to keep the status that four baseline slot satellites located on the same orbital plane, the satellites of SVN23, SVN46, SVN67 were selected as non-orbital satellites according to the satellite distribution principle, and the remaining 24 satellites are based on the orbital position, that is, the constellation consists of 24 + 3 satellites.
Through the analysis of per-slot availability, the average failure rate of the 24 baseline constellation is 1.6551eÀ04, and the average restoration rate is 0.1796. Similarly, the average failure rate of the 3 spare satellites is 1.4881eÀ04, and the average restoration rate of the 3 spare The initial state G ! t 0 ð Þ and F ! t 0 ð Þ can be divided into two cases: 1. Regardless of the per-slot availability in initial state.
Regardless of the per-slot availability means that there is no failed baseline constellation and spare satellites at the beginning time. Therefore, it can be attained as follows.
Through the experiment, the constellation availability has been in stable state under the condition of T ¼ 300Δt. Then the constellation availability probability of SIS on different failure conditions is listed in Table 3.

2.
Considering the per-slot availability in initial state.
The initial state of baseline constellation and spare satellites constellation will be changed after considering the per-slot availability. In order to simplify the calculation process, the average per-slot availability probabilities of 24 baseline constellation and 3 spare satellites are calculated, respectively. And the calculation is in terms of the per-slot availability probability showed in Figure 7. Therefore, p 0 is the average per-slot availability probability of 24 baseline   Slot  A2  A4  A5  A6  B1  B2  B3  B4  C1   SVN  52  48  65  64  56  62  44  58  57   Slot  C2  C3  C4  D1  D2  D3  D4  D5  D6   SVN  66  59  53  61  63  45  34  46  67   Slot  E1  E2  E3  E4  E5  F1  F2  F3  F4   SVN  51  47  50  54  23  41  55  43  60   Table 2. Distribution of the satellites.  constellation, its value is 0.9988. And p 1 is the average per-slot availability probability of 3 spare satellites, its value is 0.9963. Then, it can be calculated as follows: ¼ 0:9889 0:0110 4:1056e-05 5:0911e-08 ½ T By the experiment, the constellation availability probability of SIS under the condition of considering the per-slot availability in initial state is listed in Table 4.
Analyzing the above two cases, it can be concluded that: 1. On the condition of considering the per-slot availability in initial state, since the on-orbit satellites all have a large per-slot availability probability, G 0 t 0 ð Þ and F 0 t 0 ð Þ are basically close to 1. In other words, whether considering per-slot availability in initial state have little influence on the element values of initial state matrix. When T ¼ kΔt is large enough, the constellation availability reaches stable state. Then, the tiny gaps between the initial state matrix of the above two cases have no effect on the system availability. So, when perslot availability is large enough, whether or not to consider per-slot availability in initial state has no effect on SIS constellation availability.

Tables 2-4
show that when there are no failed satellites in satellite constellation, the constellation availability probability can reach 0.999999963. It is basically close to 1. By contrast, the probability of other failure states can be neglected. Thus, it can be seen that the constellation, which is made up of the 27 satellites has a very high availability.

Constellation availability evaluation of BDS SIS
The spatial constellation of the BDS consists of 35 satellites. At present, BDS is in the construction stage of global satellite navigation system. As of the end of 2015, there are only 20 BDS satellites launched, which is not enough to form a complete constellation. Moreover, due to the small number of satellites, the BDS does not currently have a backup satellite dedicated to replacing faulty satellites. Therefore, the constellation availability model of GPS SIS is not suitable for the current BDS system.
In addition, due to the incomplete BDS constellation, only BDS SIS availability in BDS-OS-PS-1.0 and BDS-SIS-ICD-2.1 is indicated by per-satellite availability, and there is no definition of BDS SIS constellation availability and related indicators [22].  Table 4. SIS constellation availability (considering the per-slot availability in initial state).
Based on the above reasons, the constellation usability should be evaluated after the Beidou global satellite navigation system constellation is fully completed and the corresponding evaluation indicators are perfected. At present, the practical significance of analyzing the availability of BDS SIS constellation is not significant.

Conclusions
This chapter mainly studies the per-satellite availability and constellation availability evaluation methods of GPS and BDS SIS, including the establishment of evaluation models and performance verification based on measured data.