Entropic Complexity Measured in Context Switching

A complexity metric for concurrent software controlled systems is defined and derived. It is equivalent or comparable to the Shannon information metric, which essentially measures entropy of a system, but uses a novel and efficient technique based on a FFT to calculate its value. This can be extended to other temporal realizations of

New events can also be handled well, either by adding a new thread or task or interrupt to watch for their occurrence or by adding an event handler to a queue.Unfortunately these benefits of scalability and flexibility detract from being able to reason about the execution semantics of a program, thus leading to an element of non-determinism.One can thus consider the obvious potential for introducing stochastic ideas to qualitatively and quantitatively understand how complex a design has become.If we can attach a simple and fundamental measure to this complexity and thus generate a useful metric, it has the capacity for comparing software designs or of providing a continuous benchmark of growing potential complexity.That leads to the notion of supplying an entropy-based measure to characterize the behaviour of a concurrent software system.

Approach -The context-switching entropy measure
The premise is to evaluate the complexity of a concurrently executing program.A novel approach of creating such a complexity metric involves the analysis of the context-switching frequency spectrum.We take a Fourier transform (FT) of the temporally distributed context switching events, c(t), and treat that as a probability density function (Feller, 1957) in frequency space (i.e. a normalized power spectrum).
Then the Shannon information entropy (S) of p(f) (Leon-Garcia, 2008) will generate a simple complexity measure.

=− • ln
(2) The entropy of a spectrum has meaning because the power spectrum itself transforms the autocorrelation of the signal in the time domain.The autocorrelation is nothing more than the probability of a time-shift occurring between points of interest.Thus by transforming the time domain information to the frequency domain we do not lose the essential information pertaining to order and disorder in the signal, in particular when it comes to comparing one waveform against another.The information containing the essential order is largely retained across the transformation.
So the Fourier transform represents the same probabilities as that of the autocorrelation but computed in the frequency domain.For any captured sequence of events, the Fourier transform of the autocorrelation is practically obtained as the magnitude of the Fourier transform of the waveform.
For a given mean frequency <f>, the value of S will be the greatest when the probability spread of the context switching frequency components is maximized.In this case, complexity tracks entropy and the maximum entropy principle (Jaynes, 2003) can be applied to understand the trend toward greater complexity.By Parseval's theorem, we do not lose the integrated signal when we take the Fourier transform.This general approach is also known as Frequency Domain Entropy (Bao, 2004).In practice using a discrete Fourier transform such as the Fast Fourier Transform (FFT) (Brigham & Morrow, 1967) allows us to then use the summation variant of the entropy, where N is the order of the FFT.

Signal
The utility of the frequency domain approach is apparent when we consider that the frequency domain is partitioned into FFT bins and we have a very fast way of computing the metric, and in many cases more convenient than by calculating the autocorrelation, or by traditional periodogram (Schuster, 1898).If we did have the extra processing time, it may make sense to calculate the entropy off the autocorrelation as that would definitely retain more of the information measure, and thus not losing information due to the transformation process.In practice, however, many times we want to know only relative entropy measures, so the loss in precision is not as important as long as we maintain the relative ordering or rank.
As an intuitive interpretation of this measure consider that a value of S near 0 (if <f> normalized to 1) indicates that the program is very specialized and is only trying to a few things at once.As S approaches 1 it becomes a general purpose application that attempts to control many different behaviours or responds to events of a more random nature.If S goes much greater than 1, then the temporal complexity turns even more random, even approaching fractal.It becomes progressively harder to test such systems because of the diverging spread in time scales.This turns into a useful metric since any sufficiently capable controller program or eventdriven system will try to accommodate as much real-world functionality as possible, and since the real world tends to disorder and to maximize entropy, this will track an increasingly complex scenario.So in practice, we will have fast cycles interspersed with slower cycles corresponding perhaps to human events or sporadic signals.In further practical terms, to apply the metric we need to run a simulation or collect data from a real or prototype system to gather the statistics.It may be possible to evaluate a program statically but that would require an introspective analysis of a program's structure, a task much more involved than collecting the statistics during typical use or as a test scenario.The possibility of using this approach on non-concurrent programs such as a single-tasking event-driven program exists but these have less inherent complexity due to their sequential nature.When the concurrency becomes an integral aspect of the system and in particular when threads must interact with one another does the complexity start to increase quickly.That becomes the essence of the disorder or entropy that we try to capture.System designers will use tools such as state diagrams, activity diagrams, and Petri nets (Peterson 1981) to architect concurrency behaviors.A typical Petri net is shown below which illustrates the parallel activities and synchronous behaviors.These have various levels of contol, in that they can abstract some of the temporal scale differences by encapsulation of behaviours (Pukite & Ludwig, 2007).However, the actual synchronization will always reveal itself in the realized system.

Application
To effectively put the temporal complexity measure into practice, the program or simulation implementation will need a way to time stamp and data log context switches during execution.An implementation of a discrete event task scheduler such as the public source Degas (Ludwig & Pukite, 2006) provides such an instrumentation facility.
The simplest input stream is a list of events signified by time-stamps, which will serve to capture the c(t) signal.The values for c(t) can be as simple as delta functions with a value of unity indicating a context switch.

Computation
This algorithm needs a FFT routine.The following pseudo-code guides the calculation calling a generic FFT routine named Fourier8.This assumes that each line contains the number of events that occur in the time step and is read through standard input.(This is set to read in exactly 2N lines, where N is an order 15 FFT) If sparse input is provided with time stamps then the data must be transferred into a discrete set of lines with either 0 or N events on each line, due to the nature of the discrete time FFT.Note that for a running simulator, if the data is provided continuously through an output port or pipe, then a suitably fast FFT can report a context switching metric in real time.This is useful for diagnostics, for example correlating the detection of errors with high complexity regimes.

Example of use
We evaluated hypothetical systems with 15 concurrent tasks.One system featured asynchronous events.The other system had tasks cyclically scheduled at different periods.
The asynchronous system always had higher switching entropy.The context switching metric distinguishes complicated but synchronized architectures from those with complex temporal behaviour For this metric, by computing the entropy of the phase spectrum in addition to the amplitude may help in discriminating between complexity and noise.That would be straightforward to include because right now we discard the phase information.Any stationary signal that shows asymmetry must have some peculiar phase relationships going on.So it might be easier to discount randomness in favour of more complex phase relationships if we were to include both the amplitude spectrum and the phase spectrum in the final metric.
Another simple alternative that works on the timing alone is the gzip program.This looks at the distribution complexity in times and calculates the entropy metric.The usage of this application is trivial.Take the output file, if a sequence of discrete inputs (zeros and number of values), run it through gzip and record the size (Benedetto, et al, 2003).Comparisons of two different sequences of identical length will suggest that the less complex program is the one of the smaller zipped size.

Complementary approach -the multi-scale entropy measure
An alternate approach to measuring the temporal complexity involves the application of the multi-scale entropy metric suggested by (Costa, 2005).This differs from the just described context-witching metric in that it measures the complexity of a temporal behaviour or signal over a wide range of fundamental periods.Whereas the single-scale metric works best over a single-decade frequency spectrum scale, a multi-scale metric offers up the possibility of looking at complexities at a variety of time scales, ranging over potentially orders of magnitude.This is definitely useful but it can't be boiled down into a single complexity metric; instead we will need to depict this graphically over what amounts to a logarithmic frequency scale.
The basis for the multi-scale entropy metric is that many real-world behaviours often occur over time scales of varying dynamic ranges.Costa originally applied this to a biomedical application, trying to extract the temporal complexities of cardiac-driven circulatory systems.When such a pulsed cardiac signal is multi-scale, it is actually composed of a fundamental pulse and various arrhythmias, leading to a complicated spectrum of events.
The signal actually appears buried amongst competing behavioural periodicities at different time scales and so it becomes that much harder to extract the information.
At first glance, we would imagine that a Fourier transform would work well to extract the periods but in fact a typical FFT algorithm actually works best over a limited dynamic range.By expanding the scope to a multi-scale level, Costa showed that this complexity measure has use in a real-world application and we contend that it may also prove useful for cyber-physical applications such as a complex event-driven system.In this case, the time scales can range from fast interrupt-processing, to human-scale interactivity, to the even more sporadic environmental influences.The multi-scale aspect senses the different temporal frequencies in the underlying signal, comparable to what the contest-switching metric does, but instead pairs or groups the data points to measure a different coarse graining effect.This works out fairly straightforwardly in terms of a autocorrelation.The essential algorithm groups adjacent time samples together in a window of length Scale as the coarse graining or moving average measure.Then it counts the number of times, n, that the amplitude will change from one coarse-grained time step to the next.
If the amplitudes don't change for a given coarse-grain then it is predictable and the entropy will be low.To calculate the sample entropy they calculate over each of the scale factors Scale = 1 .. maxScaleFactor.

Usage domains
A graph of the multi-scale entropy will appear flat if it is measuring "1/f" (van der Ziel, 1950) or the so-called pink noise as the underlying behaviour.Pink noise shows a predictable constant change of amplitude density per scale factor; in other words it has a constant energy per frequency doubling while white noise shows constant per frequency interval (Montroll & Shlesinger, 2002).
In comparison to the structure-less noise, if structure does exist in the signal, you will see observable changes in the entropy from one scale factor to the next.For example a superimposed sine wave would show a spike downward in sample entropy when it crossed a harmonic in the scale factor.
A simple interpretation suggests that we scale the measured results relative to the 1/f noise part of the signal.The 1/f noise includes the greatest variety of frequencies of any behaviour known and therefore the highest entropy (Milotti, 2002).So by providing a good visualization or graph that plots the 1/f asymptotic value we can immediately gauge the complexity of a signal.Costa et al discuss the difficulty of distinguishing between randomness and increasing complexity, which has importance in the realm of event-driven systems.
This is a key insight and one echoed by researchers in complexity theory (Gell-Mann, 1994) in that the most interesting and challenging complexity measures occupy the middle range of the complexity scale.In other words, the most ordered signals can be described by a few harmonic periods and at the other extreme, the complexity reduces to simple stochastic measures akin to statistical mechanics (Reif, 1965).In between these extremes, we require a different level of sophistication.
"In fact, entropy-based metrics are maximized for random sequences, although it is generally accepted that both perfectly ordered and maximally disordered systems possess no complex structures.A meaningful physiologic complexity measure, therefore, should vanish for these two extreme states."

Comparison to single-scale metric
The same inputs as that for the Context-Switching Metric described earlier result in the following Figure 6.Note that in this case as well, the sample entropy is always higher for the disordered signal than for the ordered signal.The reference 1/f noise level is shown on the plot to indicate the asymptotic maximum entropy level achievable.
Fig. 6.For the same pair of inputs we used on the context-switching metric, the multiscale entropy appears as the following graph.It shows greater variety than the context-switching metric over the time scales because the metric compares at different levels of resolution.
In practice, the multi-scale algorithm requires only a basic periodogram method invoked over different time scales.The output is one value per temporal scale factor so the results are best displayed as a graph, via a spreadsheet or bar-charting software for example.The calculation is somewhat more brute force compared to the FFT, with complexity o(n 2 ) versus o(n*ln(n)).The context-switching metric operates over a narrower time scale so gets rolled into a single value, simplifying the presentation into a classical scalar metric.

Discussion
Interacting concurrent activities can produce behaviour that is difficult to anticipate.The combinatorial capacities of just the ordering of parallel thread execution will exhaust any brute force attempt at testing the possible permutations.Moreover, the possibilities of rare anomalies attributed to corner cases of executions may take non-determinant times to reveal, or worse, are not easily repeatable.The sporadic defect that occurs rarely in the lab usually has a real origin and should never be ignored (Kuhn, et al, 2004).
The figure below shows a representative timeline trace of the interacting subsystems on a typical automated ground vehicle.The task interactions are interleaved and pipelined.

Applications to design
The difficulty in designing and debugging of such systems necessitates practical simulation tools and simplified metrics such as the entropy measures described here.Both of these entropy metrics are potentially useful during system development as an analysis of alternatives or design space exploration tool.If several concurrent options are available for a design, either of these metric could provide criteria to establish the least complex design.In that sense, it serves as a simple utility function to measure unwanted or creeping complexity much like a duty-cycle utilization measures processor contention.

Applications to test and verification
A close connection exists between the entropy metrics and usage modelling (Whittaker, 1994) for program verification.Most non-exhaustive testing requires a mix of tests taken during nominal conditions along with tests sampled according to potentially rare conditions.This consideration takes into account the number of test vectors and the path coverage for testing.Any characterization at this stage will provide useful inputs to generating a stochastic measure of reliability.This could incorporate stochastic usage models and a log-likelihood metric is often used to compare between two state space probabilities.Between an entropic measure and a usage model, we can cover the temporal and path dimensions of a program's execution and its programmatic complexity.

Applications to diagnostics
As a diagnostic tool, the context switching metric can also detect potential complexities during execution.Since the FFT can easily compute in real-time for typical sample sizes N, parallel execution of the entropy algorithm with the context switching data can reveal deviations from expected operation.For example, if an execution profile shows a high regularity of frequent context switches during some interval and then transitions to a more irregular sequence of switches with the same overall density, the expected entropy measure will definitely increase.In that sense, the entropy metric measures an intrinsic property of the signal, and that strictly speaking, density fluctuations such as expected increases in the rate of context switches will not influence the measure.In other words, density alone does not affect the complexity.By the same token, the multi-scale metric has obvious benefit for detecting long term complexity changes or short-term bursts buried in a nominally sampled signal.The idea of using frequency domain entropy for diagnostics of complex machinery is further explored in (Shen, 2000).

Conclusion
We described a complexity metric for concurrent software controlled systems or concurrent realizations of behaviour.The novel approach of creating a complexity metric for context switching involves the analysis of the switching frequency spectrum.We take a Fourier transform of the temporally distributed context switching events (c(t)) and treat that as a probability density function in frequency space (i.e. a normalized power spectrum).Then the entropy (S) of p(f) will generate a simple complexity measure.The context switching metric can be used during system development as an analysis of alternative utility function.If several design options or algorithms are available, the context-switching metric can be used as selection criteria to minimize inherent algorithmic complexity.It is comparable or equivalent to the Shannon information metric, which essentially measures entropy of a system.As an alternative approach we compared this against a multi-scale entropy measure.
Although more involved in construction, the multi-scale entropy can be used as an orthogonal metric, perhaps more useful for measuring temporal behaviours of a wide dynamic range or as a more detailed diagnostic tool.This will reveal finer structures in complexity than the single-scale metric can.

Acknowledgment
We wish to thank the DARPA META program for providing encouragement to develop complexity metrics for cyber-physical system development.

Fig. 1 .
Fig. 1.Example of context-switching waveforms with low entropy (left) and high entropy (right).The low entropy signals are more regular.

Fig. 2 .
Fig. 2. Variation of power spectrum with increasing disorder.(a) Ordered signal with harmonics generates a low entropy signal.(b) Disordered signal generates a larger intropy.

Fig. 3 .
Fig. 3.A typical concurrent automata represented by a Petri net.Multiple threads are represented by filled tokens (places), and context switches occur when threads meet at synchronization points (bars).For large systems, the complexity of interactions canl grow well beyond this level.

Fig. 4 .
Fig. 4. Pseudo-code algorithm for calculating the context-switching entropy using a library package FFT routine

Fig. 7 .
Fig. 7. GANTT chart of a typical vehicle system execution trace showing interacting threads.Time proceeds left to right, and one thread exists per horizontal entry.The lines indicate thread synchronization points.This diagram is only meant to give a notional idea of complexity, and the text description along the left edge is irrelevant to the discussion.

Table 1 .
Mapping of signal entropies to their corresponding frequency domain entropies.The trend of increasing entropies with increasing disorder is maintained through the transformation process.