## 1. Introduction

The challenge of this chapter is to develop precise algorithms for modulation format recognition (MFR) with the highest identification accuracies in the presence of optical channel impairments. For that, we propose and demonstrate the use of ANN-based pattern recognition technique, trained by different feature-based approaches.

In the first method, we implement a new approach using ANN in conjunction with linear optical sampling (LOS) for direct and coherent systems at high data rates. Here, and in accordance to the IEEE 802.3 standards, we have considered the classification of 10 Gbps Non-Return-to-Zero On-Off Keying (NRZ OOK), 40 Gbps NRZ-Differential Quadrature Phase Shift Keying (DQPSK), 100 Gbps NRZ Dual-Polarized (DP)-QPSK, 160 Gbps DP-16 Quadrature Amplitude Modulation (16QAM) and 1 Tbps WDM-Nyquist NRZ-DP-QPSK digital modulation formats for high-speed communication systems. Numerical simulations demonstrate high identification precision in the presence of different impairments, such as chromatic dispersion (CD), differential group delay (DGD) and amplified spontaneous emission (ASE) noise.

In the second method, we propose a novel technique of MFR algorithms using the time-frequency analysis, which is wavelet transform. In conjunction with ANN pattern recognition algorithm, this method is efficient for features extraction when it approximate both the signal envelop and frequency content. Continuous wavelet transform (CWT) is used to extract the classification features of 40 Gbps NRZ-OOK, and used three multi-carriers modulation formats namely 160 Gbps OFDM DP-16QAM, 400 Gbps Dual-Carrier (DC)-Polarization Division Multiplexed (PDM)-QPSK and 1 Tbps WDM-Nyquist NRZ-DP-QPSK. Through simulations, the proposed technique is able to classify these modulation schemes under different transmission impairments with high accuracy.

## 2. MFR based on ANN trained by LOS

### 2.1. Principle of the proposed method

An implementation of automatic MFR method of the detected signals, at high data rates is proposed. We consider the recognition of 10 Gbps NRZ-OOK, 40 Gbps NRZ-DQPSK, 100 Gbps NRZ-DP-QPSK, 160 Gbps DP-16QAM and 1 Tbps WDM-Nyquist NRZ-DP-QPSK. The basis of this technique is the use of ANN-based pattern recognition trained by the features of linear optical sampling. The method is validated in the presence of various link impairments including CD, DGD and ASE noise. Thereby, the ANN concept with the principle of asynchronous sampling is described in the following sections.

#### 2.1.1. ANN architecture

The ANN is a computational tool trained by the use of input-output data to generate a desired mapping from an input stimulus to the targeted output. The architecture of an ANN consists of three layers: input, hidden and output layers, also called as multilayer perceptron 3 (MLP3), as shown in
**Figure 1**
. The role of the input layer is to pass the input vector to the network, without computational role. In addition, the ANN architecture has one or more hidden layers and finally an output layer [1]. Layer of processing elements gives independent computations of received data. Then, it passes the result to another layer. In turn, the next layer also passes on the result to another layer after making it independent computations. At the end, the output of the network is determined by a subgroup of one or more processing elements. Each processing element makes its computation based upon a weighted sum of its inputs.

As shown in
**Figure 1**
, it is used for MFR by assigning output nodes to represent each format type. In our case, to recognize these modulation formats, five output nodes such as 10 Gbps NRZ-OOK, 40 Gbps NRZ-DQPSK, 100 Gbps NRZ-DP-QPSK, 160 Gbps DP-16QAM and 1 Tbps WDM-Nyquist NRZ-DP-QPSK are required. In the training data, the target output vectors **t**
*
_{i}
*, (

*i*= 1, …,

*m*) can be considered as binary vectors with elements with values of “1” indicating the correct modulation formats and elements with values of “0” indicating the incorrect formats.

*m*is the number of modulation formats to be recognized (

*m*= 5). In this way, the target vectors of these five used modulation formats would be represented by [1, 0, 0, 0, 0], [0, 1, 0, 0, 0], [0, 0, 1, 0, 0], [0, 0, 0, 1, 0] and [0, 0, 0, 0, 1], respectively. The posterior probability is considered at the output of the multilayer perceptron. Hence, the final recognition goes to the node with the highest value

*ar*g

*max*(

**y**

*). Taken an example of output vector with elements [0.05, 0.01, 0.03, 0.9, 0.01], the most probable identification would be 160 Gbps DP-16QAM format.*

_{i}Amplitude histograms are represented at the input of the ANN with back propagation (BP) learning method. The basic processing elements of the ANN, called neurons, of neighboring layers are interconnected by varying the coefficients that represent the strengths or weights of the respective connections. Each neuron is computed as the weighted sum of the input signals **X**
*
_{i}
*, (

*i*= 1, …,

*m*) transformed by the transfer function, as shown in

**Figure 2**. The learning capability of an artificial neuron is achieved by adjusting the weights

**W**

*, (*

_{ki}*i*= 1, …,

*m*) in accordance with the chosen learning algorithm. Weights of the perceptron can amplify or reduce the original input signal. Adding the weighted signals before passing into the activation function is essential to convert the input into a more useful output (

**Y**

*). Different types of activation function exist but one of the simplest would be step function.*

_{k}The architecture of ANN for recognition problems requires some guidelines. More neurons need more computation and they have a tendency to overfit the data when the number is set too high, which justifies the choice of MLP3 architecture.

#### 2.1.2. Linear optical sampling for amplitude histograms

In the literature, optical sampling proved necessary for a variety of experiments. Optical signals are sampled by means of optical or even opto-electrical gates. This technique covers LOS systems that enable the measurement of amplitude histograms and eye diagrams.

Some publications in this field date back to the mid-1990s, when the first optically sampled eye diagrams of 10 Gbps optical data signals were published [2, 3]. The LOS technique allows us the processing of signals at high speeds, like in our case when using WDM-Nyquist system. Received signals are asynchronously oversampled to generate amplitude histograms. These histograms define the own signature or traces of the five used modulation formats.
**Figure 3**
shows an example of cleaned histogram on back-to-back (a) and an impaired one after a fiber transmission (b,c). Correspondent eye diagram is also showed in the figure (2^{nd} column).

It is clear from the histograms that the location of each peak correspond to a particular intensity level of the formats constellations. By definition, amplitude histograms are the empirical distribution of the received signal power. Therefore, they are sensitive to changes in optical signal-to-noise ratio (OSNR), CD and polarization mode dispersion (PMD) of the transmission link. As given in
**Figure 3(c)**
, histogram characteristics changes significantly regarding the optical link impairments and distances.

### 2.2. Simulation design

In
**Figure 4**
, the simulation design of the proposed method for automatic MFR is shown. Five different transmitters generates NRZ-OOK at 10 Gbps, NRZ-DQPSK at 40 Gbps, NRZ-DP-QPSK at 100 Gbps, DP-16QAM at 160 Gbps and WDM-Nyquist NRZ-DP-QPSK at 1 Tbps. The pseudorandom binary sequence (PRBS) is of length 2^{14} – 1. Then, using a variable optical attenuator (VOA), generated output signals are decreased, which affects the gain of the optical amplifier (OA). The latter is used to emulate the effect of optical signal-to-noise ratio (OSNR) by tuning the noise figure and add a variable amount of ASE noise into the signals. As a result, the OSNR values are adjusted in the range between 10 and 30 dB with steps of 2 dB and are then transmitted over a single mode fiber (SMF). By using the CD/PMD (polarization mode dispersion) emulator, the accumulated CD of the link varied in the range of 0–510 ps/nm (steps of 17 ps/nm) to reach 30 km fiber transmission, while the DGD is introduced in the range of 0–14 ps with steps of 2 ps. An optical filter optical band pass filter (OBPF) is used to select the carrier whose modulation format need to be recognized.

Using an optical switch (SW) in the detection stage, the NRZ-OOK signal is directly detected with single photodetector, while coherent receivers detect other formats. Using polarization beam splitter (PBS), the detected signal is split into two orthogonal polarization states. Each output is coupled with the signal derived from the local oscillator (LO). A single-polarization is detected in the case of NRZ-DQPSK signal. In the MFR block, signals are asynchronously oversampled with a rate of 16 samples/bit to have 262,144 amplitude samples. The amplitude histograms of the five used modulation formats are shown in
**Figure 5**
(2nd and 3rd columns). Besides, when considering the OSNR equal to 16 dB, DGD at 10 ps for 10 km fiber length (3rd column), these histograms stay again different from each other. Typical eye diagrams are also showed in the 1st column of the figure. Later, these histograms are used in the ANN module for formats classification, as described in the previous section. The number of neurons in hidden layer is optimized to be 30 neurons using the incremental-constructive approach [1].

With **n** input and **m** output neurons, the training set is given as {(**X**
_{1}; **t**
_{1}), …, (**X**
_{p}; **t**
_{p})}, where P = 27,280 is the size of the entire data corresponding to different OSNR (11 values), CD (31 values), DGD (8 values) and modulation types at two polarization states, **n** = 100 is the number of histograms bins. More precisely, we want to minimize the error function of the network, the MSE, defined as ||**y − t**||^{2}. When training multilayer networks, the overall data set is randomly divided, with 60% used for training, 15% for validation and 25% for testing. On the other side, the results of the simulations indicate that the Levenberg-Marquardt (LM) training algorithm provides high correct recognition ratios [5].

Using our previous settings, we reach the minimum of MSE on 3.67 × 10^{−7} for 27 epochs leading to abort the training process, as illustrated in
**Figure 6**
. This amelioration reduces the required time for ANN training process, which depends on ANN size, type of learning algorithm, size of the training data set and the desired degree of accuracy.

### 2.3. Modulation formats recognition

The previously described system of MFR has validated with test cases comprising five modulation formats. Training data sets are arranged into 16,368 input vectors **x**
*
_{i}
* (60%), and then numerous simulations using different combinations of OSNR, CD and DGD are performed. The results of the testing data sets (6820 input vectors x

*) are depicted in*

_{i}**Figure 7**and five elements of ANN output vectors

**y**

_{ i }are shown. Each output represents a particular modulation type among 10 Gbps NRZ-OOK, 40 Gbps NRZ-DQPSK, 100 Gbps NRZ-DP-QPSK, 160 Gbps DP-16QAM and 1 Tbps WDM-Nyquist NRZ-DP-QPSK.

Indeed, one output **y**
*
_{i}
* reaches a maximum value than the others, as clearly shown in the figure, which represents the modulation format. For that, there is no ambiguity in the classification process because of this large separation. The five-used modulation formats are well recognized with good accuracies, as given in the all cases represented in

**Figure 7**.

After ANN training, the recognition accuracy is validated with different set of testing data.
**Table 1**
shows the obtained results, and it is apparent that 100% accuracy is obtained for 10 Gbps NRZ-OOK, 40 Gbps NRZ-DQPSK and 160 Gbps DP-16QAM with small differences in recognition probabilities (i.e., 99.99% and 99.98%) for the other formats.

An unannounced precision ambiguity between 100 Gbps NRZ-DP-QPRK and 1 Tbps WDM-Nyquist NRZ-DP-QPSK modulation formats has observed. This is due to the use of the same coding scheme, which is the QPSK. As previously stated at the outset, our simulations are performed under various impairments of ASE noise, CD and PMD but the recognition accuracy remains high up to 99.99%. This result is an improvement compared to other works presented in the literature (99.6%, 99.95%) [6, 7].

## 3. MFR based on ANN trained by wavelet transform

### 3.1. Theoretical background

Different mathematical guidelines are used in the defined MFR block of this technique. Before moving to the ANN classifier, it allows the features extraction of all received signals. In our study, we chose the wavelet transform for the multiresolution analysis (MRA). In addition, to accomplish the classification mission, the singular value decomposition (SVD), as a factorization tool of matrix, is used. The different blocs of our MFR module are shown in
**Figure 8**
.

#### 3.1.1. Wavelet transform and features extraction

Wavelet transform analysis is one of the most popular non-stationary signals processing tool. In this method, continuous wavelet transform (CWT) is used for its ability to construct in time and frequency domain a good representation of treated signals. It is also used to extract the necessary features of each received modulation format [8].

In the following integral, the CWT of the signal **f**(**t**) ∈ **Z** is expressed at a scale **a** > 0 and translational value **b** ∈ R:

where CWT(a, b) define the wavelet transform coefficients, * denotes complex conjugate and Ѱ_{a, b}(t) is the baby wavelet comes from time-scaling and translation of mother wavelet Ѱ(t) as described in Eq. (3).

Recently, the selection of the mother wavelet function as well as the decomposition level of signal is the most indispensable challenge in wavelet analysis. It includes Haar, Meyer, Morlet, Symlet, Daubechies and coiflet wavelets [8]. In our case, the Haar wavelet is chosen due to its simple form and also its computation is still easy. It is given in the following equation as a continuous function in both; the time and frequency domain:

Give s(t), with 0 < t < T_{s}, the received optical waveform in a complex form described as:

where T_{s} is the symbol duration, *θ*
_{
c
} is the carrier initial phase, **f**
_{
c
} is the carrier frequency, **n(t)** is the complex ASE noise and

with *N* the number of observed symbols, *h _{Ts
}
*(

*t*) is the pulse shaping function of duration

*T*, S is the average signal power and

_{s}*φ*∈{(2

_{i}*π*/

*N*)(

*m*− 1) ,

*m*= 1 , 2 , … ,

*N*},

where (*A _{i}
*,

*B*) are the assigned QAM symbols.

_{i}with *d _{i}
* = {0, 1} symbols.

An example of CWT for the four used modulation formats (40 Gbps NRZ-OOK, 160 Gbps OFDM DP- 16QAM, 400 Gbps DC PDM-QPSK and 1 Tbps WDM-Nyquist NRZ-DP-QPSK) is shown in
**Figure 9**
, where we choose a scale a = 100. From the figure, it is clear that each modulation format has its own features in terms of CWT amplitude and number of peaks.

#### 3.1.2. Singular value decomposition

SVD is the most important applicable matrix factorization used for signal processing and statistics. This tool is used to solve the least squares problems, and provides the best way to approximate a matrix with one of lower rank.

Given a real or complex matrix **A** having **m** rows and **n** columns, the matrix product *U*Σ*V*
^{∗} is the singular value decomposition for the given matrix **A** if:

U and V, respectively, have orthonormal columns;

**Σ**has non-negative elements on its principal diagonal and zeros elsewhere and**A**= U**Σ**V*.

In addition, if σ is a non-negative scalar, and **
u
** and

**are nonzero**

*v*

*m**-*and

*n**-*vectors, respectively,

where σ is a singular value of A and **u** and **v** are corresponding left and right singular vectors, respectively.

In fact, the diagonal elements {σ_{i}} of Σ are the singular values of **A**. The columns
**A**, respectively.

#### 3.1.3. ANN classifier

Features extraction of received signals is accomplished using CWT and SVD. In the last step, the pattern recognition method based on ANN is used for our statistical learning model. Its architecture is described in
**Figure 10**
. It is structured as an interconnected group of artificial neurons, which use a computational model or mathematical model for information processing. As given in the figure, the ANN is an adaptive system that changes its structure based on external or internal information that flows through the network. Precisely, the architecture of the ANN varies, but generally, it consists of several layers of neurons.

In this technique, we also used the MLP3 for the modulation format identification by assigning output nodes to represent each format. Four output neurons in the output layer represent the 40 Gbps NRZ-OOK, 160 Gbps OFDM DP-16QAM, 400 Gbps DC PDM-QPSK and 1 Tbps WDM-Nyquist NRZ-DP-QPSK modulation formats. As designed previously, before training the network, the multilayer perceptron output can be considered as the highest probability, which represent one from the four modulation types. Eigenvalues after the SVD for each modulation format are represented at the input of the ANN. The multilayer perceptron used in this architecture requires four output neurons representing the number of format types.

### 3.2. Design of the proposed method

The setup of the proposed MFR technique is shown in
**Figure 11**
.

The four used modulation formats are generated with carrier frequency equal to 193.1 THz. Using a variable optical attenuator (VOA), the injected power is tuned and passed through an optical amplifier (OA) that undergo the ASE noise effect at higher gain amplifier. As a result, the OSNR of the signals is adjusted in the range of 10–35 dB (steps of 5 dB). Then, using a CD/PMD emulator, chromatic dispersion is varied from 85 to 510 ps/nm (steps of 85 ps/nm) to reach 30 km on SMF, and the DGD between 0 and 20 ps (steps of 5 ps). The appropriate carrier to be classified is selected by an OBPF. As described in
**Figure 8**
, in MFR block, using the CWT, each received signal is processed by scaling factor up to 128 without amplitude normalization. Moreover, with length equal to 3, the median filter was applied to extract the features set and remove the peaks. In addition, we make the SVD to the time-scale parameters of the wavelet coefficients matrix and obtain the eigenvectors as the final characteristic vectors for each received signal. Finally, we reach the ANN classifier as described in the previous section, where for each modulation format we generate 150 realizations with 65,536 symbols corresponding to different combinations of CD, DGD and OSNR.

The number of neurons in hidden layer is optimized to be 25 neurons. In addition, 128 input neurons representing the eigenvalues after SVD for each input modulated signal are used, in addition to 4 output neurons to design the modulation formats to be recognized.

In the training process, the input vectors are randomly divided, with 65% used for training, 20% for validation and 15% for testing. On the other side, using LM training algorithm and reducing the MSE (3.71 × 10^{−5} for 10 epochs) optimize the identification rates and minimize the computation time.

### 3.3. Modulation formats classification

The proposed classification method is demonstrated at high bit rates for the four used optical modulation formats including 40 Gbps NRZ-OOK, 160 Gbps OFDM DP-16QAM, 400 Gbps DC PDM-QPSK and 1 Tbps WDM-Nyquist NRZ-DP-QPSK. For each modulation type, to verify the robustness of our method, 150 realizations with 65,536 symbols have been used. All signals are transmitted through ideal (B-to-B) and impaired channels. The simulation results are given
**Figures 12**
–**14**.

The plot of the probabilities of recognition against OSNR is given in
**Figure 12**
. The OSNR is in the range of 10–35 dB. 100% correct identification was observed for 160 Gbps OFDM DP-16QAM and 1 Tbps WDM-Nyquist NRZ-DP-QPSK at higher OSNR values. In contrast, for lower OSNR values (lower than 15 dB), a challenging problem remains for NRZ-OOK identification, which reached values between 80 and 90%. This misclassification is because of its coding properties, where to extract its features, the wavelet transform located ambiguities.

The modulation format can also be recognized correctly when varying the CD, as described earlier in
**Figure 13**
. From the simulation, the residual dispersion varies between 85 and 510 ps/nm with OSNR = 20 dB. The 160 Gbps OFDM DP-16QAM and 1 Tbps WDM-Nyquist NRZ-DP-QPSK still having the highest recognition probabilities (~100%). Furthermore, 40 Gbps NRZ-OOK and 400 Gbps DC PDM-QPSK reached the minimum values of identification accuracies, that is, 90% for residual dispersion more than 350 ps/nm. In the main, when increasing the residual dispersion values, all probabilities of recognition decreases. This is interpreted by the fact that signal characteristic are widely modified in terms of phase and amplitude with the chromatic dispersion. For that, the wavelet transform will lose its principal features.

In optical transmissions, in the presence of several impairments that broadly modified the signals features and makes the formats recognition issue more difficult. We can cite in this case the impairment caused by the DGD. In
**Figure 14**
, we show the probabilities of identification versus this parameter.

The DGD is between 0 and 20 ps with steps of 5 ps. The misclassification percentage for 40 Gbps NRZ-OOK signal remains with a small difference for 160 Gbps OFDM DP-16QAM and rises when the DGD is greater than 15 ps. 400 Gbps DC PDM-QPSK and 1 Tbps WDM-Nyquist NRZ-DP-QPSK modulation formats still having the highest accuracies when the DGD is less than 10 ps and begin to decrease for DGD values more than 15 ps. It is suspected that this error is due to the effect of SOP induced by the PMD. It is based on the DGD and the frequency, and it varies differently for each modulation format with the optical carrier, after fiber transmission.

**Table 2**
shows the recognition results of the neural network when OSNR is equal to 30 dB. For 160 Gbps OFDM DP-16QAM and 1 Tbps WDM-Nyquist NRZ-DP-QPSK formats, we obtain 100% correct classification, while 40 Gbps NRZ-OOK and 400 Gbps DC PDM-QPSK were identified at 96.6% and 98%, respectively. An ambiguity of identification for these formats is increased to 3% at higher DGD and CD. This misclassification is due to the fact that the signal features (phase, amplitude, etc.) are infected in the presence of signal impairments. Thereby, CWT proves a difficulty to sign these formats and extract their eigenvalues.

The robustness of this technique is evaluated and the simulated results show that the correct modulation identification scheme is possible even at all channel parameters ranges: OSNR between 10 and 35 dB, CD from 85 to 510 ps/nm and DGD in the range of 0–20 ps for all modulation formats mentioned previously. We reached higher identification accuracies at high bit rates which facilitates the performance monitoring process (OPM) for the network management.

From the results obtained, it is found that the method employing the amplitude histograms is simple and more robust to impairments link, than the other one using CWT, for features extraction. Moreover, the asynchronous amplitude histograms (AAH) generation followed by ANN is rapid on time response compared to the method using CWT for features extraction. Besides the modulation formats, identification ambiguities is more apparent for the second method.

## 4. Summary

In this chapter, two cost-effective techniques for intensity and phase-modulated systems have been proposed and demonstrated. Both methods employ ANN for pattern recognition in conjunction with features extraction approaches and digital signal processing.

In the first method, asynchronous amplitude sampling is the features extraction method. For high-speed optical communications, new approach using ANN trained by the features of linear optical sampling is implemented. For the demonstration of the proposed method, 10 Gbps NRZ-OOK, 40 Gbps NRZ-DQPSK, 100 Gbps NRZ DP-QPSK, 160 Gbps DP-16QAM and 1 Tbps WDM-Nyquist NRZ-DP-QPSK modulation formats are considered. The efficiency of this technique is demonstrated in the presence of different transmission link parameters, such as CD, DGD and OSNR. Simulation results demonstrate successful recognition from a known bit rates with higher estimation accuracy, which exceeds 99.8%. For this method, asynchronous sampling with a rate greater than symbol rates is successfully utilized to have the maximum features for each received signal. Thereby, due to the simplicity of ANN implementation and the use of only amplitude samples, the proposed techniques enable the identification of various modulation formats at different bit rates with high accuracies.

The second technique presents a new achievement using the continuous wavelet transform (CWT) for features extraction. It offers the best time and frequency localization. In that case, Haar wavelet and SVD followed by ANN pattern-recognition are used to achieve the classification process. This method is advantageous because its cost effectiveness and its flexibility. To demonstrate the validity of this technique, we consider the classification of 40 Gbps NRZ-OOK, and three multi-carriers modulation schemes such as 160 Gbps OFDM DP-16QAM, 400 Gbps DC-PDM-QPSK and 1 Tbps WDM-Nyquist NRZ-DP-QPSK. The effect of each channel parameter to the probability of recognition has been also observed. In particular, it has been found that the correct identification was observed at higher OSNR values. While, an increase of CD and DGD affects the accuracy of recognition and limits the measurement ranges. Despite the presence of link impairments, and because the CWT is resistant to the noise in the signal, the classification of these formats remains possible with good precision.