This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

Random‐noise‐induced biases are inherent issues to the accurate derivation of second‐order statistical parameters (e.g., variances, fluxes, energy densities, and power spectra) from lidar and radar measurements. We demonstrate here for the first time an altitude‐interleaved method for eliminating such biases, following the original proposals by Gardner and Chu (2020,

Second‐order statistics like atmospheric wave‐induced variances or fluxes of physical parameters calculated from lidar or radar data would have positive biases if detection‐noise‐induced variances or fluxes were not properly eliminated. If such biases remained, not only would atmospheric variances be overestimated, implying unrealistic wave activity and growth rates, but certain phenomena could be concealed, presenting misleading pictures of the atmospheric observations. While various methods have been developed to account for this, all have limitations to their use. This study summarizes the development, theory, and application procedures for three different methods, namely conventional variance subtraction, the spectral proportion method, and the interleaved method, which can be used to correct such biases. The newest of these methods is adapted from operating on the data time‐wise into an altitude‐wise approach, improving its applicability. The performances of the three methods are compared by calculating the variance‐based gravity wave potential energy density derived from lidar data taken at McMurdo, Antarctica. Their accuracies are assessed by using a forward model. This study aims to guide future research by providing information on how and when to apply each of these methods in order to enhance the outcomes of existing and future lidar/radar systems and datasets.

We demonstrate for the first time an altitude‐interleaved method for eliminating random‐noise‐induced biases from second‐order statistics

The interleaved method is compared to the spectral proportion and variance subtraction methods to reveal strengths and limitations of each

Lidar data from Antarctica & a forward model are used to assess accuracies and precisions of the three methods under different conditions

Lidar and radar systems provide unrivaled monitoring of the middle and upper atmosphere, allowing for high‐temporal/spatial resolution measurements of atmospheric parameters and constituents, which in turn enable the quantification of complex processes like atmospheric fluxes, constituent transport, turbulence, and gravity waves (e.g., Gardner & Liu, 2010; Hocking, 1996; Lu et al., 2015). These systems can observe a wide range of altitudes by taking advantage of various signal‐return mechanisms (e.g., Baumgarten, 2010; Chu et al., 2020; Chu, Yu, et al., 2011; Kaifler & Kaifler, 2021), which allows detailed studies into vertical coupling, analyzing how atmospheric processes develop as they travel over a wide range of altitudes. Such sophisticated systems have led to decades of impressive remote sensing campaigns (e.g., Li et al., 2020; She et al., 2019; Stober et al., 2021), yet to make full use of the data, advances must be made in data handling. Advances in science inevitably require the use of second‐order statistics such as variances and fluxes, which are inherently biased by random‐noise in the data generated during the detection processes (e.g., Chu et al., 2018; Gardner & Liu, 2014; Whiteway & Carswell, 1995). These biases grow increasingly problematic when analyzing the higher or wider reaches of lidar and radar data. To deal with these biases, various correction methods have been developed over the years (e.g., Chu et al., 2018; Gardner & Chu, 2020; Whiteway & Carswell, 1995), each with its own advantages and disadvantages. These methods have not yet been compared side‐by‐side to assess their effectiveness under various conditions, which are therefore the subject of this work.

This study focuses on lidar‐measured variance and covariance, but the same principles apply when calculating other second‐order statistics using radar and lidar data. Variance is a statistic dependent on the perturbation of a value from its mean, so it is important to understand the anatomy of the perturbation, which can be represented as

If calculating the variance of wave‐induced perturbation

On the righthand side of Equation 2, the second term is the noise‐variance

The most direct way to isolate *variance subtraction* method (e.g., Duck et al., 2001; Whiteway & Carswell, 1995). Estimating this term becomes problematic in low‐SNR conditions, as large uncertainties in the estimation often result in the estimated variance‐bias being greater than the geophysical variance itself, yielding a physically‐impossible “negative variance” when the noise term is subtracted.

To overcome this issue, Chu et al. (2018) developed a solution called the *spectral proportion* method, where Monte Carlo simulations based on parameter uncertainties are used to estimate the wave‐occupied proportion

Gardner and Chu (2020) developed a new approach named the *interleaved* method. In this method, the return photon counts are split into two separate but interleaved samples so that two statistically independent samples probe the same air parcel over the same time period. Consequently, the variance in Equation 2 is substituted with a covariance between these two samples (see Section 2.3 for details) which no longer contains the noise‐induced bias once a statistically‐sufficient sample size is used. This method improves the accuracy of the variance estimate by eliminating the noise‐induced bias yet decreases the precision both through increased uncertainty caused by photon count splitting and by any remaining terms containing the noise‐induced perturbations which have not approached zero under a limited sample size.

Since each approach has strengths and weaknesses, this paper compares these three methods in terms of their accuracy and precision using Antarctic lidar data as well as a forward model. The lidar data used here are the Rayleigh scattering signals collected with an Fe Boltzmann lidar from 2011 to 2020 at the Arrival Heights Observatory near McMurdo Station (Chu, Huang, et al., 2011; Chu et al., 2020; Chu, Yu, et al., 2011). Although these techniques are demonstrated on lidar measurements, they can be applied to radar data as both are similarly affected by noise‐induced biases in higher‐order parameters. Additionally, this paper demonstrates an alternative way to apply the interleaved method by interleaving in altitude as opposed to time‐interleaving as initially demonstrated in Gardner and Chu (2020).

This paper calculates gravity wave potential energy mass density (

In practice, we cannot calculate the

In Equation 5, the last two terms are not wave‐induced but introduced by noise, with the second introducing a positive bias and the third term introducing additional noise. Although

As seen in Equation 5, when the

The limitations of this method become obvious when attempting to use it on noisy data. Here, growing uncertainty in parameter error can cause

These negative values led to the development of the spectral proportion method in Chu et al. (2018) which eliminates the possibility of negative

After the noise floor is determined, the proportion of wave energy occupying the total energy

Examples of these averaged spectra and noise floors can be seen in Figure 1 for a variety of altitudes while the

This method can, however, introduce positive‐bias under high noise, which is discussed further in Sections 3 and 4.

The common idea of the previous two methods was based on the total variance calculation that includes both wave and noise‐induced variances, and then each method employed some algorithm to either remove the estimated noise‐variance or scale down the total variance to estimate the wave‐induced variance. The interleaved method instead eliminates the noise‐induced bias altogether by calculating the covariance of simultaneous, collocated samples taken in a way such that the noise‐terms are driven toward zero. Gardner and Chu (2020) point out that this bias elimination would optimally be done using two adjacent lidars, but such a setup would be complex, expensive, and uncommon. The interleaved method they propose instead introduces a practical way to achieve the same bias‐elimination using a single lidar (see diagrams in Figure 2). Gardner and Chu (2020) have demonstrated interleaving time bins (Figures 2a and 2c) for the covariance calculation but suggested that in many lidar systems it may make more sense to apply it to adjacent altitude bins. Here we describe the altitude‐interleaving method.

Implementation of this altitude‐interleaving process (Figure 2e) is best described by comparing it to a standard lidar data processing approach. In the standard data processing, the photon counts from *n*‐adjacent fine bins,

The difference between the wave‐induced variance and covariance depends on the level of correlation between

This flexibility in interleaving‐direction allows for the interleaved method to be used on data taken by many different lidar systems. For example, the Na Doppler lidar used as an example in Gardner and Chu (2020) saves its raw data in time intervals of 4.5 s, which means the data can be very finely time‐interleaved, while the Fe Boltzmann lidar used for this study saves its data in time intervals of 1 min, likely too coarse to use a time‐interleaved approach. However, the Fe Boltzmann lidar saves its counts at a high‐vertical resolution of 48 m, allowing the interleaved method to be utilized altitudinally.

Moreover, as

Results obtained from these three methods are compared in Figure 3, where four cases are shown—winter and summer observations representing high and low SNRs, respectively, with both large and small samples sizes. By comparing these cases, one can get a sense for how the accuracy and precision of each method respond to increased sample size, as well as how they behave in varying noise levels. Before discussing these results in Section 4, it is necessary to introduce the analyses of uncertainty in precision and bias in accuracy.

The measurement uncertainty

We tabulate the uncertainty equations in Table 1, among which Equations 18–20 consider the photon‐noise‐induced uncertainty only. That is, only the error propagation from

*Note*.

Variance uncertainties are propagated through Equation 12 to calculate

Equations 18–20 are suitable for estimating the random‐noise‐induced uncertainty for

While the precision of each method can be assessed from Figure 3 and Table 1, the analysis in Section 3.1 does not give much insight into the accuracies of these methods. To address this issue, a forward model was developed to test their performance. As the modeled wave energy is known, it is possible to assess any potential systematic bias introduced by each of the three methods, in addition to the assessment of their uncertainties in precision.

First, the atmospheric number density and wave‐induced perturbations in this density field are modeled as a background with wave‐induced perturbations:

The background density field

Next, we simulate the photon return from a lidar shooting vertically into this modeled density field. The photon counts are generated by

Equation 15 is essentially a modification of the general lidar equation for Rayleigh scattering as written in Chu and Papen (2005) with the addition of the density perturbations.

We now analyze how the three methods perform under different conditions of SNR and number of samples, in terms of their accuracy and precision. We define accuracy as how close the results are to the true atmospheric

We first compare the precision of these methods under various conditions. The precision of each method is largely determined by SNR and is increased with the use of a greater sample size (as the uncertainty is decreased). This is evidenced by Equations 18–23, as the

We then assess the accuracy of the methods using the forward modeled results in Figure 5. The performance of the variance subtraction method in all conditions clearly shows that this method has low accuracy under high‐noise due to its negative values in Figure 3 and its strong departure from the modeled

The negative‐bias of variance subtraction method is due to the uncertainty in the estimation of the noise‐variance. As the noise‐variance is computed using the temperature or density error as in Equation 6, large uncertainties in these error values inevitably occur near the top of the measurement (where the SNR has significantly declined) which cause the noise‐variance to increase dramatically. When subtracted from total variance via Equation 7 these often yield negative variances and thus, physically‐impossible negative

The positive bias of spectral proportion method seen in Figures 3g and 5d is caused by high‐noise in the initial sample contaminating the spectra at a given altitude. Looking at Figures 1a–1d, there is a regularly occurring peak near

The interleaved method does not suffer from either positive or negative bias and generally remains centered around the modeled

We now compare the performance of the interleaved method to that of the spectral proportion in Figure 7 by replicating results from a previously published study. In Chu et al. (2018), lidar data from 2011 to 2015 taken by the McMurdo Fe Boltzmann lidar was processed with the spectral proportion method to yield

The comparison in Figure 7 has demonstrated the repeatability of the seasonal asymmetry observations. The interleaved method generally agreed with the trends attained from the spectral proportion method. While some of these interleaved values are negative, it is now known from Section 4 that we must average over many samples (or apply a fit, as we do here) to reveal the true trend. Not only are the observational results replicated by the interleaved method, but since the interleaved method is more accurate in the high‐noise summer and does not overestimate

Random‐noise‐induced biases are inherent issues to the accurate derivation of second‐order statistical parameters (such as temperature, wind and species variances, momentum, heat and constituent fluxes, potential and kinetic energy densities of atmospheric waves, and power spectrum estimates) from lidar and radar measurements. As the boundaries of existing research expand, powerful techniques for removal of such biases must be developed to take full advantage of data collection campaigns. The variance subtraction, spectral proportion, and interleaved methods are all viable means to correct for the biases, yet the performance of each method varies depending on the conditions under which they are applied.

Based on the comparisons using the lidar observational datasets from Antarctica as well as the forward‐modeled cases, we draw the following conclusions. The variance subtraction method is best used with high‐SNR observations, as it is easily biased‐negatively by noise in the data. It provides a precise, yet not always accurate, measurement of atmospheric variance even with a relatively‐small sample size. The spectral proportion method is more robust, yielding precise and accurate measurements of variance in significantly noisier data than the variance subtraction method, and also does not rely on a large sample size (Chu et al., 2018). However, it begins to display a positive‐bias under high‐noise conditions. The interleaved method is the only method which will intrinsically not have a bias because it eliminates the random‐noise‐induced biases utilizing two statistically independent datasets that cover the same altitude range and time period (Gardner & Chu, 2020). However, such improved accuracy is attained at the price of reduced precision, necessitating a much‐larger sample size than the others even for high‐SNR measurements.

This work is the first demonstration of altitude/range‐interleaved method for deriving second‐order statistics, following the original proposal by Gardner and Chu (2020). Interleaving in altitude (or range) bins provides two statistically independent samples over the same time period and altitude range even if the original raw data were not saved in high temporal resolutions but sufficiently high spatial (range) resolutions. Therefore, the altitude/range‐interleaved method provides a suitable solution to many current and historic lidar and radar datasets for accurately deriving variances, fluxes, wave energy densities, and power spectrum estimates, etc.

Given the overall considerations we recommend applying the interleaved (either in time or in altitude/range bins) method and the spectral proportion method in real applications because they are superior to the noise subtraction method as demonstrated in this work. When the application goals are to derive statistically mean profiles with high accuracy and/or there are a large number of samples, the interleaved method would be the best choice because it inherently eliminates the noise‐induced biases to give the highest accuracy while the large sample size reduces uncertainties to ensure sufficient precision as well. However, if the application goals are to derive second‐order statistics within a small number of samples and then study the time evolution of such statistics over month, season, and/or year (i.e., non‐stationary signals in longer time periods), the spectral proportion may be a better choice for its higher precision and the ability to handle small sample sizes with a caveat of potentially positive‐biases in high‐noise conditions. Applying the proper bias‐removal method can unlock the full potential of a dataset, allowing retrieval of second or higher‐order parameters into lower‐SNR regions of the data. Additionally, it can reveal trends in the data that may otherwise be concealed by the bias, such as the seasonal asymmetry demonstrated prior, or altitudinal trends which have not yet been discovered.

Using Rayleigh lidar data, the gravity wave potential energy mass density

While the atmospheric density may seem superior overall in the

A primary downside of the interleaved method is the reduction of precision caused by splitting the samples into two groups. This is best countered by increasing the number of samples used for the measurements, which reduces the uncertainties roughly by a factor of

We gratefully acknowledge the graduate students and research scientists who made contributions to the McMurdo lidar campaign, including winter‐over lidar scientists Zhibin Yu (2011), Brendan Roberts (2012), Weichun Fong (2013), Cao Chen (2014), Jian Zhao (2015), Ian Barry (2016), Zhengyu Hua (2017), Dongming Chang (2018), Zimu Li and Ian Geraghty (2019), and Xianxin Li and Cissi Lin (2020), and summer scientists Wentao Huang, Zhangjun Wang, John A. Smith, Xian Lu, Muzhou Lu, and Clare Miller. We are grateful to Richard Dean, Nikolas Sinkola, and Adam Godfrey for their engineering help and support. We sincerely appreciate the staff of the United States Antarctic Program, McMurdo Station, Antarctica, New Zealand, and Scott Base for their superb support over the years. This work was partially supported by NSF grants OPP‐1246405, OPP‐1443726, OPP‐2110428, AGS‐2029162, and AGS‐1452351. The work of X. Chu was partially supported by NASA LWS grant 80NSSC20K0002. J. Jandreau is grateful to the generous support of Cooperative Institute for Research in Environmental Sciences (CIRES) Graduate Student Research Award, George C. and Joan A. Reid Memorial Scholarship, and Aerospace Engineering Sciences Smead Scholars Program.

The data shown in this work can be downloaded online from