Coded Modulation and Impairment Compensation Techniques in Optical Fiber Communication

Abstract

This chapter deals with coded modulation and impairment compensation techniques in optical fiber communication. Probabilistic shaping is a new coded modulation technology, which can reduce transmission power by precoding, reduce bit error rate and improve communication rate. We proposed a probabilistic shaping 16QAM modulation scheme based on trellis coded modulation. Experimental results show that this scheme can achieve better optical SNR gain and BER performance. On the other hand, in order to meet the demand of transmission rate of next generation high speed optical communication systems, multi-dimensional modulation and coherent detection are sufficiently applied. The imperfect characteristics of optoelectronic devices and fiber link bring serious impairments to the high baud-rate and high order modulation format signal, causes of performance impairment are analyzed, pre-compensation and receiver side’s DSP techniques designed for coherent systems are introduced.

1. Introduction

With the rapid development of Internet services, higher requirements are put forward for the transmission rate, system capacity and stability of communication. Optical fiber communication has become one of the main communication methods in the world because of its large transmission bandwidth, long-distance transmission and strong anti-interference ability.

The optical fiber communication systems are mainly divided into two types: intensity modulation-direct detection (IM-DD) and coherent optical communication systems. The IM-DD system is mainly used in access networks, passive optical networks (PON) and data centers, and its transmission distance is usually less than 80 km. Specified in the standard, the highest transmission rate with single-wavelength of IM-DD system is 100Gb/s, transmitting 56GBaud 4-level pulse amplitude modulation (PAM4) signal. The coherent optical transmission system, which uses multi-dimensional modulation to improve spectrum efficiency, local oscillator lasers to increase sensitivity, and digital signal processing (DSP) technology for impairment compensation, thereby can greatly improve transmission performance, increase transmission rate and distance, and is usually used in in large-capacity long-distance backbone networks and metropolitan area networks.

Quadrature amplitude modulation (QAM) is commonly used in coherent optical fiber transmission system. In this method, evenly distributed constellation points are arranged in two-dimensional space to form a constellation diagram. The performance loss of the modulation scheme tends to $\mathit{\pi e}/6$ (1.53 dB) from the Shannon limit. To further approach Shannon limit capacity and improve system performance, probabilistic shaping technology can be used [1]. Probabilistic shaping is a new coded modulation technology. It can reduce transmission power by precoding, reduce bit error rate and improve communication rate without changing the original system. And with the change of channel environment, the channel can be matched by changing the size of the shaping, thus improving the flexibility of the system.

The future research direction of high-speed optical fiber communication is digital coherent optical communication technology [2, 3, 4]. With the development of high-bandwidth optoelectronic devices, digital-to-analog converter (DAC), analog-to-digital converter (ADC) and application specific integrated circuit (ASIC) chips, beyond 800Gb/s transmission with single wavelength above becomes possible. In order to achieve such high-speed transmission, DSP technology plays an important role in dealing with chromatic dispersion (CD), polarization mode dispersion (PMD), frequency offset and phase noise, by compensating the signal at the transmitter and receiver in the electrical domain. With highly integrated and flexible digital coherent optical detection technology, high-speed, large-capacity and long-distance optical communication can be effectively realized.

The most important technologies in coherent optical communication system are introduced in this chapter, a novel coded modulation technology based on probabilistic shaping and DSP-based Impairment compensation techniques.

2. Novel coded modulation technology based on probabilistic shaping

2.1 Principle of probabilistic shaping

The main idea of probabilistic shaping is to reduce the probability of occurrence of constellation points in outer ring, increase ones in the inner ring, and change the constellation of uniform distribution into non-uniform distribution. The common distribution matcher is constant composition distribution matcher (CCDM), which makes the occurrence probability of each constellation point conform to Maxwell Boltzmann distribution:

$$P_X\left(x_i\right)=\frac{e^{-v{\left|x_i\right|}^2}}{{\sum }_{j=1}^m{e}^{-v{\left|x_j\right|}^2}}$$E1

Where, $X=\left\{x_1,x_2,\dots \dots ,x_m\right\}$ is the constellation symbol set, $v$ is the probability distribution factor, and the value range is 0 ∼ 1. The larger value of $v$, the higher the degree of constellation shaping [5].

Taking 16QAM modulation format as an example, the constellation probability distribution diagram is shown in Figure 1, and the constellation diagram after recovery at the receiving end is shown in Figure 2.

Figure 1.

Probability distribution of 16QAM signal. (a) Uniform 16QAM; (b) PS 16QAM.

Figure 2.

Constellation for receiver recovery using probabilistic shaping technique.

2.2 Experiment

To evaluate the performance of the proposed scheme, an experiment is carried out by employing the coherent optical communication system setup illustrated in Figure 3. The system parameters are shown in Table 1. At the transmitter, a 1550 nm lightwave with power of 10 dBm and line-width of 100 kHz is employed as the laser source, followed by a polarization beam splitter to divide the output light into orthogonal polarized pair. Then two I/Q modulator are applied to modulate two orthogonal light waves, respectively. Trellis-Coded Modulation (TCM)-PS-16QAM signals are generated offline by MATLAB program. And each modulator is driven by two 10Gbaud amplified electrical TCM-PS-16QAM signals with a frame length of 33,336 symbols, and 10 patterns (333,360 symbols) are collected for bit error ratio (BER) calculation. The peak-peak voltage of the two amplified electrical signals are both set as 2.0 V. And a polarized beam combiner is applied to combine two orthogonal modulated lightwaves, which is amplified by an erbium-doped fiber amplifier (EDFA) in the standard single mode fiber (SSMF) for transmission. The gain and the noise figure of the EDFA are 20 dB and 4 dB, respectively. The input power of the fiber is 0 dBm, and the fiber is a SSMF with attenuation of α = 0.2 dB/km, nonlinear coefficient of γ = 1.3 (W·km)⁻¹, and dispersion of D = 17 ps/nm/km. At the receiver, the ASE noise is added to the received optical signal to adjust the received optical signal noise ratio (OSNR) with the resolution of 0.1 nm. The optical signal added with noise is bandpass-filtered and converted into an electrical signal by a coherent receiver. The laser diode (LD) generates a local oscillating light with the power of 5 dBm and linewidth of 100 kHz. In the offline DSP module, timing recovery is implemented by Gardner algorithm and chromatic dispersion is compensated digitally. To process different shaped signals, a pre-convergence constant modulus algorithm with step of 2e-6 and taps of 9 followed by 1% pilot aided multi-modulus algorithm are used for polarization demultiplexing and compensation of polarization mode dispersion without training sequence. In addition, fast Fourier transform based frequency offset estimation (FFT-FOE) algorithm and blind phase search algorithm are applied to realize frequency offset estimation and compensate the laser phase noise, respectively. Finally, TCM-PS-16QAM decoding and statistics of BER are implemented.

Figure 3.

Experimental setup for PDM single carrier system.

Parameter	Specification
Center wavelength	1550 nm
Laser linewidth	100 kHz
Input power of fiber	0 dBm
Amplification	EDFA
EDFA gain	20 dB
EDFA noise figure	4 dB
Fiber	SSMF
Attenuation	0.2 dB/km
Nonlinear coefficient	1.3(W·km)^−1
Chromatic dispersion	17 ps/(nm·km)
Span length	25 km

Table 1.

System parameters.

To verify our proposed scheme, we compare the performance of 8QAM, TCM-16QAM-4state, TCM-16QAM-8state, TCM-16QAM-16state, and TCM-PS-16QAM-4state respectively. The convolutional encoders and information entropy parameters are displayed in Table 2. Here, some modulation formats have different entropy, and lower entropy means a higher baud-rate, which makes their bit-rates consistent.

Scheme	Entropy (bits/symbol)	Baud rate (GBaud)
8QAM	3	10
TCM-16QAM-4state	3	10
TCM-16QAM-8state	3	10
TCM-16QAM-16state	3	10
TCM- PS-16QAM-4state	2.9	10.4
	2.8	10.7
	2.7	11.1

Table 2.

Modulation format parameters.

In the experiment, the TCM-PS-16QAM (H = 2.8 bits/symbol) with OSNR = 15 dB is firstly transmitted. After the optical transmission system, the received signal of the subset S0, S1, S2, S3 and the overall constellation at the receiver is shown in Figure 4. The successful transmission of the TCM-PS-16QAM signal means that our proposed novel scheme is reasonable and realizable. Meantime, it can be seen that the probability distribution of the constellation points achieves the goal of low energy point density and high energy point sparseness.

Figure 4.

Probability distribution with the entropy of 2.8 bits/symbol of the proposed (a) subset S₀ constellation, (b) TCM-PS-16QAM constellation at the receiver.

Figure 5 depicts the BERs of 8QAM, TCM-16QAM-4state, TCM-16QAM-8state, TCM-16QAM-16state, and TCM-PS-16QAM-4state (H = 2.9 bits/symbol) with the same data rate of 60 Gb/s after 25-km SSMF transmission. It can be seen that when the OSNR is above 5 dB, the performance of TCM-16QAM-nstate (n = 4, 8, 16) is better than 8QAM. And with the increase of OSNR, the BERs of TCM-16QAM-nstate (n = 4, 8, 16) fall faster than 8QAM. At the BER of 1 × 10⁻³, TCM-16QAM-4state obtains the gain of 3.4 dB compared to 8QAM. However, TCM-16QAM-8state slightly outperforms TCM-16QAM-4state and the OSNR gain is just 0.4 dB. Meantime, compared to 8 state, the gain of TCM-16QAM-16state grows only 0.4 dB too. And the decoding complexity double as the number of state double. So, increasing the number of states is not a suitable way to obtain coding gain, especially when the coding gain is close to the limit. The required OSNR for BER of 1 × 10⁻³ is about 7.7 dB for TCM-16QAM-4state, and 6.8 dB for TCM-PS-16QAM-4state (H = 2.9 bits/symbol), The OSNR improvement is increased by 0.9 dB with a little more complexity. Compared to TCM-16QAM-8/16state, TCM-PS-16QAM-4state (H = 2.9 bits/symbol) has lower decoding complexity and better performance. And the gain of TCM-PS-16QAM-4state (H = 2.9 bits/symbol) grows 0.5 dB and 0.1 dB, respectively. This is mainly due to the shaping gain brought by PS. Under the condition that the minimum Euclidean distance in the constellation remains unchanged, the convergence of most constellation points greatly reduces the average power of the constellation, improving the performance of the entire modulation system.

Figure 5.

BER curves and constellations of different modulation formats (H = 2.9 bits/symbol) for 25-km transmission. (a)8QAM, (b)TCM-16QAM-4/8/16state, (c)TCM-PS-16QAM-4state.

The BER curves of TCM-PS-16QAM-4state with different information entropy after 25-km transmission are also measured, and the measured results are shown in Figure 6. In our experiments, according to their different information entropy, the baud rate is set differently, their information entropy is 2.9, 2.8, 2.7 bits/symbol, so the baud rate is 20.7, 21.4, 22.2Gbaud, namely, the bit rate is 60 Gb/s. It can be seen that the required OSNRs are 6.8 dB, 6.5 dB, and 5.9 dB at the BER of 1 × 10⁻³. In other words, TCM-PS-16QAM-4state (H = 2.7 bits/symbol) obtains OSNR gain of 0.6 and 0.9 dB compared with TCM-PS-16QAM-4state (H = 2.8 and 2.9bits/symbol), respectively. This advantage proves that as the information entropy decreases, more shaping gains can be obtained. Obviously, at the same OSNR, the lower the information entropy, the better the BER performance and the more flexible information entropy and gain compared to the traditional TCM-16QAM.

Figure 6.

BER curves and constellations of TCM-PS-16QAM-4state for 25-km transmission. (a) H=2.9 bits/symbol, (b) H = 2.8 bits/symbol, (c) H = 2.7 bits/symbol.

3. Impairment compensation techniques in coherent communication

Coherent optical communication systems are implemented in the form of optical modules in commercial communication equipment [6]. As shown in Figure 7, The transmitted information is encoded by forward error correction (FEC) through the electrical interface and then transmitted to transmitter-side (TX) DSP. The TX DSP includes constellation mapping, pulse shaping, pre-equalization, skew compensation, dispersion pre-compensation, etc. The main purpose of implementing DSP-based pre-compensation techniques at the transmitter is to increase the signal-to-noise ratio (SNR) of the transmitted signal while the transmission power is limited, reduce the bandwidth requirement for signal transmission, avoid inter-symbol interference (ISI), and reduce the impairment caused by the optoelectronic devices.

Figure 7.

Coherent optical module structure.

After single mode fiber channel, the coherent optical signal is transmitted into the coherent receiver and ADC to obtain four electrical signals, which consist of dual polarizations X/Y and in-phase/quadrature (IQ) components XI, XQ, YI, and YQ. The receiver-side (RX) DSP contains a series of impairment compensation and equalization algorithms for the optoelectronic devices and fiber link, including IQ signal orthogonalization, normalization, dispersion compensation, clock recovery, polarization demultiplexing, frequency offset estimation, and phase noise recovery. DSP compensation technology at the receiver side is the core of coherent optical communication. It can recover multi-dimensional modulated signals from distorted constellation, to realized large-capacity transmission.

This chapter will introduce the impairment compensation technology based on DSP in two parts, all common compensation techniques are described in detail.

3.1 Impairment compensation techniques at transmitter side

Pre-equalization and skew compensation are basic DSP techniques at transmitter side, pre-equalization technique is used to compensate the filtering effect caused by the bandwidth limitation of transmitting devices, and skew compensation is to compensate the delay of XI, XQ, YI and YQ signals while passing through the electrical and optical paths [7]. At the same time, in order to cope with more and more high-speed transmission, such as 400Gb/s, 800Gb/s and 1.2 Tb/s transmission, high-order modulation format and high baud rate signal is generated and transmitted, such as 96Gbaud 32QAM and 80Gbaud 64QAM, which is extremely sensitive to linear and nonlinear impairment of optoelectronic devices, therefore, other pre-compensation algorithms such as look-up table (LUT) and digital predistortion (DPD) are usually added to the next generation coherent optical module’s DSP algorithm. In this section, basic principles of the impairment compensation techniques at transmitter side are described for the development of next generation coherent optical transmission systems.

3.1.1 Bandwidth limitation and pre-equalization

Restricted by material and technical level, the frequency response of the optoelectronic device is not flat in the range of the bandwidth needed to transmit the signal, as shown in Figure 8, the transmitted signal will fade at high frequency, which leads to inter symbol interference, the smaller the bandwidth is, the more serious the ISI is and the worse the BER performance.

Figure 8.

Spectrum of transmitted signal with different bandwidth limitation.

To suppress the performance degradation of high-speed signal caused by bandwidth limited system, the pre-equalization technology based on DSP can alleviate the bandwidth shortage of the transmitter device, which is an effective bandwidth compensation method. To implement pre-equalization, it is necessary to obtain the frequency response of the transmitter, including DAC, electric driver, modulator and so on. First, we send specific training sequence $X$ without any compensation, then receive the signal $Y$ with a high bandwidth digital sampling oscilloscope, we can estimate the frequency response H of the transmitter by comparing the transmitted and received signals with least square (LS) algorithm

$$H_{\mathit{LS}}={\left(X^{H}X\right)}^{-1}{X}^{H}Y$$E2

Then we multiply the inverse of the estimated frequency response with the transmitted data in the frequency domain

$$X_{\mathit{TX}}=X·H_{\mathit{LS}}^{-1}$$E3

Thus, the high frequency component of the signal can be raised at the transmitter to resist the low-pass filtering effect of the device. The spectrum of the transmitted signal is flat to reduce the inter symbol crosstalk.

3.1.2 Look-up table algorithm

LUT is a pre-compensation method, which is used to compensate the memory effect of the amplifier. From the time domain point of view, the memory effect means that the current output symbol of the amplifier is not only dependent on the current input symbol, but also related to the past input symbol value. From the perspective of frequency domain, memory effect can be defined as the phenomenon that the amplitude and phase characteristics of intermodulation distortion term of amplifier change with the variation of envelope frequency of input signal. Considering the influence of $2N+1$symbols before and after the symbols at the intermediate time, all possible transmitted sequences are $X\left(k-N:k+N\right)$, and the corresponding received sequence is $Y\left(k-N:k+N\right)$. All data in lookup table are set to 0 in the initial state, and the sliding window selects $2N+1$ symbols in the transmission sequence each time, after looking up the index and finding the address of this pattern, the error $E\left(k\right)$ is obtained by subtracting the central symbol of the sending sequence and the receiving sequence. As the sliding window moves forward, error values of all transmission modes can be traversed. Suppose the lookup table index is $i$, the number of data stored in index is $M\left(i\right)$, and the lookup table is updated as follows

$$\mathit{LUT}\left(i\right)=\mathit{LUT}\left(i\right)+E\left(k\right)$$E4

$$M\left(i\right)=M\left(i\right)+1$$E5

$$\mathit{LUT}\left(i\right)=\frac{\mathit{LUT}\left(i\right)}{M\left(i\right)}$$E6

3.1.3 Summary

In terms of modulation and impairment suppression technology in transmitter of optical transmission system, researchers have carried out a lot of research, such as peak to average power ratio (PAPR) reduction technology, DAC resolution enhancement [8], joint pre-equalization in electrical and optical domain, etc. In recent years, artificial intelligence techniques have been recently proposed as a promising tool to address various challenges in optical communication, and machine learning technology based on indirect learning [9] and neural network [10] has also been used for impairment compensation of transmitter devices.

3.2 Impairment compensation techniques at receiver side

In the long haul and large capacity optical transmission system, the optical link will introduce chromatic dispersion [11], polarization mode dispersion (PMD) [12], fiber nonlinearity, etc., the laser linewidth and frequency jitter will bring frequency offset and phase noise [13, 14], and the ADC sampling frequency and phase cannot be synchronized with the DAC at the transmitter. All these problems can be solved by using mature DSP technology, so as to avoid the use of a series of complex devices such as phase locked loop. With the development of ASIC chip manufacturing technology, highly integrated and flexible digital signal processing technology can meet the needs of high-speed optical transmission system in the future.

As shown in the Figure 9, the basic DSP algorithm flow of a typical coherent optical communication receiver, including IQ imbalance compensation, CD compensation, timing recovery, polarization demultiplexing, frequency offset estimation, and carrier phase recovery. According to the algorithm design, different algorithms may have nested or parallel processing, these algorithms are closely linked and indispensable, which is the basis of coherent optical communication with high baud rate and high order modulation format.

Figure 9.

DSP flow of coherent optical transmission system.

3.2.1 I/Q imbalance compensation

Ideally, the I and Q components in the received signal are completely orthogonal, but in the actual system, for the extinction ratio of the two arms of the IQ modulator is not completely consistent, the division ratio of the 3 dB coupler in the receiver is asymmetric, and the response of the balance detector is inconsistent, the amplitude and phase of the two IQ components will be imbalanced. So, it is necessary to implement IQ imbalance compensation and normalization in the first step of digital signal processing. In general, Gram-Schmidt orthogonalization procedure (GSOP) algorithm is used to map one set of non-orthogonal vectors as reference variables and the other set as orthogonal variables. Suppose that $I_{\mathit{\text{in}}}\left(k\right)$ and $Q_{\mathit{\text{in}}}\left(k\right)$ are non-orthogonal vectors, and $I_{\mathit{\text{out}}}\left(k\right)$ and $Q_{\mathit{\text{out}}}\left(k\right)$ are vectors processed by GSOP, as

$$I_{\mathit{\text{out}}}\left(k\right)=\frac{I_{\mathit{\text{in}}}\left(k\right)}{\sqrt{P_I}}$$E7

$$Q^{\prime }\left(k\right)=Q_{\mathit{\text{in}}}\left(k\right)-\frac{\rho I_{\mathit{\text{in}}}\left(k\right)}{P_I}$$E8

$$Q_{\mathit{\text{out}}}\left(k\right)=\frac{Q^{\prime }\left(k\right)}{\sqrt{P_Q}}$$E9

where $\rho =E\left\{I_{\mathit{\text{in}}}\left(k\right)·Q_{\mathit{\text{in}}}\left(k\right)\right\},P_I=E\left\{{I_{\mathit{\text{in}}}}^2\left(k\right)\right\},P_Q=E\left\{{Q^{\prime }}^2\left(k\right)\right\}$, and $E\left\{·\right\}$ represents expectation.

3.2.2 Chromatic dispersion compensation

Chromatic dispersion is a static impairment for optical signal in fiber transmission. The main factor of CD is that the characteristics of optical fiber material lead to different propagation group velocity of different frequency components of optical signal, which is similar to the multipath effect in wireless communication, resulting in time-domain pulse broadening. For the early optical fiber communication system, chromatic dispersion is mainly compensated by negative dispersion coefficient media such as dispersion compensation fiber, fiber Bragg grating and other dispersion compensation modules. With the development of DSP technology, digital signal processing technology can completely replace the function of optical dispersion compensation module, it is easy to realize dispersion compensation based on DSP.

Generally, dispersion coefficient $D$ is used to quantify the pulse broadening caused by fiber dispersion, the unit is $\mathit{ps}/\mathit{nm}/\mathit{km}$. The partial differential equation of the influence of fiber dispersion on signal envelope is derived, and the frequency domain transmission equation can be obtained by Fourier transform

$$G\left(z,w\right)=\mathit{exp}\left(-j\frac{D{\lambda }^2}{4\mathit{\pi c}}{\omega }^2\right)$$E10

where $\lambda$ is the wavelength of light wave, $c$ represents light speed, and $\omega$ is arbitrary frequency component. The frequency-domain transfer function of the dispersion compensation filter is obtained by inversing the dispersion coefficient of the transfer function as

$$G\left(z,w\right)=\mathit{exp}\left(j\frac{D{\lambda }^2}{4\mathit{\pi c}}{\omega }^2\right)$$E11

In the long-distance optical communication system, the signal sub block must be large enough to compensate for the dispersion effect in the transmission. Therefore, an overlapped frequency domain equalization structure is proposed to improve the transmission and DSP efficiency by forming overlaps between sub blocks as shown in Figure 10.

Figure 10.

Frequency domain dispersion equalization with overlap method.

3.2.3 Clock recovery

After photodetection, the electrical signal is sampled and digitized by the ADC. However, in the actual system, because the local sampling clock is not synchronized with the transmitter signal clock, the sampling point of the ADC is not the best sampling point of the signal in most cases. On the other hand, due to the instability of the local clock source itself, it may also cause the sampling error of the system. This sampling error includes both the sampling phase error and the sampling frequency error. The clock error of the sampling signal, on the one hand, is due to the imperfection of the sampling point, causing interference between sampling symbols; on the other hand, the jitter of the sampling clock will also cause fluctuations in signal performance. Therefore, in order to achieve optimal digital signal recovery, a clock recovery module is needed in the actual system to eliminate the impact of clock sampling errors. Considering that the dispersion will cause the disappearance of the clock component, usually, the clock recovery module is placed after the dispersion compensation or works with the dispersion compensation module to form a unified balanced feedback module.

The feedback time-domain clock recovery algorithm is proposed by Gardner [15]. This algorithm uses a feedback clock synchronization structure to estimate the phase of the retiming digital clock source feedback by calculating the timing error. The estimation of the timing error can track the frequency jitter of the signal, and the application of this algorithm can achieve dynamic clock recovery. On the other hand, the Gardner clock recovery algorithm only needs two samples per symbol and the algorithm complexity is low. It is widely used in the digital signal processing module of the coherent optical communication system.

3.2.4 Adaptive equalization and polarization demultiplexing

As CD compensation technology has been well promoted in optical fiber communication systems, PMD becomes the primary impairment which limits the information capacity and transmission distance for over 40Gbit/s systems [16, 17, 18]. Besides, PMD is a stochastic impairment that converts with time, temperature, wavelength and fiber conditions which makes PMD hard to estimate and compensate. In recent years, the research on how to overcome the performance deterioration of optical communication system caused by PMD effect has been a hot topic, most of which focuses on PMD compensation [19, 20].

The adaptive equalizer is generally used to polarization demultiplex and PMD compensation of channels. A two-by-two multiple-input multiple-output (MIMO) structured finite impulse response (FIR) filters as shown in Figure 11 is used to estimate the inverse Jones matrix of the dynamic channel [17].

Figure 11.

Framework of 2 × 2 MIMO structured FIR filters.

The input sequence of the filter is symbol-spaced with index $n$, while the $N$ tap FIR filters, $h_{\mathit{xx}}$,$h_{\mathit{xy}}$, $h_{\mathit{yx}}$ and $h_{\mathit{yy}}$ are the column vector of length $N$. Tap weights are updated every two samples as the input sequence is two-fold sampled. Therefore, $x_i$ and $y_i$ represent a sliding block of $N$ samples such that

$$x_i\left(n\right)=\left[x_i\left(n\right),x_i\left(n-1\right)\dots x_i\left(n-N\right)\right]$$E12

$$y_i\left(n\right)=\left[y_i\left(n\right),y_i\left(n-1\right)\dots y_i\left(n-N\right)\right]$$E13

We consider that $u_i\left(\mathrm{n}\right)=\left[x_i\left(n\right),y_i\left(n\right)\right]$, $h_x\left(\mathrm{n}\right)=\left[h_{\mathit{xx}}\left(n\right),h_{\mathit{xy}}\left(n\right)\right]$, $h_y\left(\mathrm{n}\right)=\left[h_{\mathit{yx}}\left(n\right),h_{\mathit{yy}}\left(n\right)\right]$. And the filters outputs form as

$$x_o\left(n\right)=h_x^H\left(n\right)u_i\left(n\right)$$E14

$$y_o\left(n\right)=h_y^H\left(n\right)u_i\left(n\right)$$E15

where superscript${\left(.\right)}^H$ means the conjugate transpose.

For fast adaptive equalizer, we generally use stochastic gradient descent (SGD) optimizer to update the tap weights. Meanwhile, we need to choose a cost function to describe the error degree of the output samples so that the equalizer can get the response and update tap weights. For constant modulus algorithm (CMA), its cost functions are given as

$${\epsilon }_x\left(n\right)=R_2-{\left|x_o\left(n\right)\right|}^2$$E16

$${\epsilon }_y\left(n\right)=R_2-{\left|y_o\left(n\right)\right|}^2$$E17

where $R_2$ is the real-valued constant and given by $R_2=E{\left|X_{\mathit{sym}.}\right|}^4/E{\left|X_{\mathit{sym}.}\right|}^2$. The update equations of FIR filters are given as

$$h_x\left(n+1\right)=h_x\left(n\right)+\mu {\epsilon }_x\left(n\right)x_o^{\ast }\left(n\right)u_i\left(n\right)$$E18

$$h_y\left(n+1\right)=h_y\left(n\right)+\mu {\epsilon }_y\left(n\right)y_o^{\ast }\left(n\right)u_i\left(n\right)$$E19

where $\mu$ is the step size parameter and the superscript${\left(.\right)}^{\ast }$ means the complex conjugate operation.

The cost functions of CMA describe error degree between the amplitude of output symbols and the proposed convergence radius [17, 18]. By updating the tap weights of FIR filters with cost functions, the error degree can be minimized in a certain extent and the output symbols can converge on a circle with the proposed radius that equals to $\sqrt{R_2}$. The value of the constant $R_2$ and step size can significantly affect the convergence degree of CMA. Appropriate $R_2$ helps CMA to converge in less steps. Shorter step size can help CMA to get better convergence, but the computation of the algorithm is also increased. CMA needs enough steps to update and optimize its FIR filters. The output performance of CMA will be insufficient if the proposed convergence length is not long enough.

All tap weights are initialized to zero except the central tap of $h_{\mathit{xx}}$ and $h_{\mathit{yy}}$, which are initialized to unity. The filter taps of $h_{\mathit{xx}}$, $h_{\mathit{xy}}$, $h_{\mathit{yx}}$ and $h_{\mathit{yy}}$ estimate the components for data sequences. At the transmitter, data sequences for X and Y polarization are independent and only contain their own information. Thus, the central tap of $h_{\mathit{xx}}$ and $h_{\mathit{yy}}$ are set to unity. After fiber transmission, the received signal is contaminated by channel impairments and noise. With the contamination of noise and the channel impairments caused by linear effects such as CD, PMD etc. or fiber nonlinear effects, the received data sequences for X and Y polarization are no longer independent. One or more symbols in the received sequences can interfere other symbols, while the symbols for one polarization can interfere other symbols for another polarization. The interference between polarizations can be estimated by filter taps of $h_{\mathit{xy}}$ and $h_{\mathit{yx}}$. Though updating filter tap weights, the 2 × 2 MIMO structured FIR filters can gradually identify the components of each data sequence, and the dynamic impairments of channel can be compensated. This process can also achieve polarization demultiplexing.

It is worth noting that the CMA can be implemented in a full-blind mode, but it set no constrain with its outputs. Therefore, it is possible for the equalizer to converge on the same output, corresponding to the Jones matrix becoming singular. In practice, we need to check if $h_x$ and $h_y$ become singular after the algorithm running for certain steps, and if so, a mathematic process is necessary to make $h_x$ and $h_y$ nonsingular and the whole algorithm should be restarted.

CMA is especially suited to the modulation format with constant amplitude such as quadrature phase-shift keying (QPSK) and M-ary phase-shift keying (MPSK). For the formats with inconstant amplitudes such as quadrature amplitude modulation, the CMA error cannot converge to zero as extra noise is introduced during equalizing. To improve the SNR performance for the formats with inconstant amplitudes, several multi-modulus algorithms such as radius-directed algorithm (RDA) and cascaded multi-modulus algorithm (CMMA) are established [21, 22].

As CMA set at real-valued constant $R_2$, CMMA set several constants which depends on the ideal constellations. According to the distance between the input symbol and the constellation origin, the algorithm estimates which circle the symbol belongs to. Then CMMA calculates the error using the estimated radius. The rest of CMMA algorithm steps are same as CMA.

Compared with CMA, the CMMA significantly improves the SNR performance for high order QAM. But it reduces the robustness of the filter converging process. This is because multi-modulus algorithm depends on correct decision on symbol radius. For high order QAM format, the distance between different circles is less than the minimum symbol interval. Therefore, the algorithm may make massive mistake over the circle decision if the signal is severely contaminated by channel impairments and noise.

3.2.5 Frequency offset estimation

In the coherent optical communication system, the transmitting laser and the local oscillator laser work independently, so the central wavelength cannot be exactly the same, so there is a certain frequency deviation $∆f$. It will introduce a continuous phase variation along with time to the received signal, resulting in constellation rotation, so it is necessary to estimate and compensate the frequency offset by DSP. Through FFT operation, the spectrum of received symbol’s fourth power value is obtained and analyzed, it can be found that there is a peak component at the frequency of $4∆f$, therefore, $∆f$ can be obtained by searching for the maximum spectral component of the fourth-power value of received symbol. Usually, due to the lack of spectral resolution and other reasons, there will be residual frequency offset after estimation, which can be looked at as additional phase noise and recovered by carrier phase recovery.

3.2.6 Carrier phase recovery

Carrier phase recovery (CPR) is an essential DSP unit in coherent systems and has been extensively investigated for QPSK and QAM signals. Like frequency offset estimation algorithms, carrier recovery algorithms can be classified as either blind or data-aided estimation techniques. And the algorithms can be implemented in feedforward manner or in feedback structure [23, 24]. Compared to the constellation for QPSK, the constellation points for QAM vary in both phase and amplitude. Moreover, the modulated signal phase is with multiple values. Viterbi-Viterbi phase estimation (VVPE) algorithm using a fourth-power operation which is suitable for QPSK is hard to completely remove the signal phase and estimate the carrier phase. At present, the algorithms for QAM CPR mainly include Blind phase searching (BPS), improved BPS (BPS/maximum likelihood), decision aided maximum likelihood (DA-ML), etc. [25, 26, 27]

BPS is recognized as a favorable solution due to its high performance and suitability for parallel processing and can be used in feedforward manner.

Figure 12 shows the block diagram of the BPS algorithm in pure feedforward manner. The input signal $x_i$ is sampled at the symbol rate. The received signal $x_i$ is rotated by B test carrier angles ${\phi }_b$ with

Figure 12.

BPS diagram in feedforward manner.

$${\phi }_b=\frac{b}{B}\bullet \frac{\pi }{2},b\in \left\{0,1,\dots ,B-1\right\}$$E20

then all rotated symbols are fed into a decision circuit, which output the ideal constellation points with the minimum Euclidean distance to the input symbols. The squared distance ${\left|d_{k,b}\right|}^2$ to the closest constellation point is calculated in the complex plane

$${\left|d_{k,b}\right|}^2={\left|x_{k}exp\left\{j{\phi }_b\right\}-⌊x_{k}exp{\left\{j{\phi }_b\right\}}_D⌋\right|}^2$$E21

in order to remove the other initial noise distortions from the receiver, the distance of 2N + 1 consecutive test symbols rotated by the same carrier phase angle ${\phi }_b$ are summed up

$$s_{k,b}=\sum _{n=-N}^N{\left|d_{k-n,b}\right|}^2$$E22

the optimum value of the filter half width N depends on the laser linewidth times symbol rate product. N = 6, …,10 is generally a good choice.

After filtering the optimum phase angle $∆{\phi }_{\mathit{est}.}$ by searching the minimum sum of distance values, then the output symbols $y_i$, which is the input symbols $x_i$ rotated by $∆{\phi }_{\mathit{est}.}$, is outputted for the following DSP in coherent systems.

Due to the 4-fold ambiguity of the recovered phase in the square M-QAM, the blind algorithms may cause incorrect phase estimation by a multiple of $\mathrm{\pi }/2$ causing cycle slip. This problem can be solved by using framing information or by applying differential coding [23, 24, 25].

Though BPS shows a good tolerance to laser phase noise and can be flexibly applied to higher order QAM, with an increasing modulation order a larger number of test phases are required and the computation complexity increases. Therefore, an improved BPS algorithm with a two-stage diagram has been established [24]. The first stage of improved BPS just requires rough estimation and the required number of test phase ${\phi }_b$ can be reduced. A maximum likelihood phase estimator is introduced in the second phase to improve the accuracy. This two-stage BPS/ML algorithm effectively improves the performance with the computation complexity and availability of BPS algorithm remained.

3.2.7 Summary

This section introduces a series of impairment compensation algorithms for coherent optical communication system. Here, the constellation diagrams of 60GBaud 16QAM signal in DSP process is given in the Figure 13. Through the algorithms in this section, we can realize the signal orthogonalization and normalization, compensate the fiber dispersion, eliminate the clock sampling error, depolarize the multiplexing and equalize channel response at the same time, and finally recover the carrier phase to get the constellation of the original transmission signal. A series of impairment compensation algorithms based on DSP at the receiver end lay the foundation for high-speed, large-capacity and long-haul optical transmission.

Figure 13.

Constellations after CD compensation, polarization demultiplexing, and carrier phase recovery.

4. Conclusion

In this chapter, first we propose a probabilistic shaping 16QAM modulation scheme based on trellis coded modulation. Through non-uniform probability mapping of TCM-16QAM subset, an effective and good overall probability distribution of 16QAM constellation is obtained. The scheme is successfully demonstrated in a 25 km single-mode fiber transmission system, and better OSNR gain and BER performance are obtained. Then impairment compensation techniques in coherent optical communication are introduced from two aspects, transmitter side and receiver side, pre-emphasis and look-up table technology have been widely used in the impairment compensation of the transmitting devices, while the DSP process at the receiver side is more complicated, including GSOP, clock recovery, dispersion compensation, dynamic equalization, and carrier phase recovery. Using DSP technology can effectively mitigate the impairment of optoelectronic devices and optical fiber links, and it is an effective method to realize ultra-high speed, large-capacity and long-haul transmission.

Acknowledgments

This work is supported by National Natural Science Foundation of China No.62022016, No.61835002, No.61727817, National Key R&D Program of China from Ministry of Science and Technology, China No.2019YFA0706300, Open Fund of IPOC (BUPT) No.IPOC2020A007, No.IPOC2020A006, and China Postdoctoral Science Foundation No.2020 M680384, No.2020 M680385.