Identifying and mitigating sources of measurement error is a critical task in geomatics research and the geospatial industry as a whole. In pursuit of such error, accuracy assessments of lidar data have revealed a range bias in low-cost scanners. This phenomenon is a temporally correlated instability in the lidar scanner where the measured distance between target and sensor changes over time while both are held stationary. This research presents an assessment of two low-cost lidar scanners, the Velodyne® HDL–32E and Livox® Mid–40, in which their temporal stability is analyzed and methods to mitigate systematic error are implemented. By immobilizing each scanner as it observes a stationary target surface over the course of multiple hours, trends in scanner precision are identified. Scanner accuracy is then determined using a terrestrial lidar scanner, the Riegl® VZ-400, to observe both subject scanner and target, and extracting the distances between scanner origin and observed surface. Patterns identified in each scanner’s distance measurements indicate temporal autocorrelation, and, by exploiting the high linear correlation between scanner internal temperature and measured distance in the HDL–32E, it is possible to mitigate the resulting error. Application of the proposed solution lowers the Velodyne® scanner’s measurement RMSE by over 60%, providing levels of measurement accuracy comparable to more expensive lidar systems.
While accuracy assessment requirements may differ between industries and data applications, the need for understanding a measurement’s accuracy is universal; without understanding an instrument’s capabilities and limitations, users are limited in their ability to use it to their own ends. A measuring instrument’s accuracy is optimized (or restricted) by a variety of factors: the experience and competence of users, the conditions of its use, and the process by which its data is processed all contribute to the overall uncertainty and must be considered when determining the accuracy of the end products.
An accuracy assessment is a deliberate process by which the closeness of a measured or computed value is compared to an accepted significantly higher accuracy (often termed “true”) value of a particular quantity across the entire area of interest being observed [
While they are useful evaluations of an instrument’s performance, there are deliberate distinctions between precision and accuracy. Precision is defined as the “closeness with which measurements agree with each other”, whether or not the measurements “contain a systematic bias” [
Parameters for assessing lidar system suitability for applications across different environments include the stability, precision (repeatability), and accuracy of their observations. This paper presents an accuracy assessment of two lidar scanners found in the topographic mapping community, and, using the assessment findings, proposes a methodology for deliberately mitigating error in their distance measurements.
The research presented in this paper was first inspired by the results from Lassiter et al. [
This finding is supported by existing literature, where Velodyne® lidar scanners have been found to consistently exhibit temporal instability within the first hour of operation. Glennie et al. [
Less established in existing literature is the stability and UAS-mapping utility of the ultra-low cost Risley-prism based laser scanner from Livox®, the Mid–40. First available on the market in 2019 for the cost of just $600, the Mid–40 was initially designed as an automotive lidar solution capable of detecting targets at up to 260 m within its circular scan area [
Given that there are a handful of published studies regarding the accuracy and precision of small form laser scanners, an important question remains: can the stability and absolute accuracy of distance measurements be directly observed in a robust, repeatable manner? Based on the demand for rigorous accuracy assessments of low-cost lidar scanners, the research presented here seeks to answer this question for the benefit of both the members of academia and the greater laser scanning community. As a whole, this study introduces methods to assess and increase the absolute accuracy of low-cost, sUAS-mountable laser scanners, informing their use in applications from rigorous scientific studies to practical utilizations. More specifically, this study presents: A methodology to directly observe and analyze the precision and absolute accuracy of low-cost lidar scanners; The identified range walk in low-cost lidar scanners and an analysis of its correlated scan parameters; Measures to correct, adjust out, or otherwise mitigate systematic errors in observations for high-accuracy sUAS surveys; A comparison between like-tier lidar scanners in terms of precision, accuracy, and temporal stability.
Through the execution of the outlined methodology, this study shows that low-cost laser scanners exhibit a temporal distance bias that is highly correlated with internal scanner temperature, and that survey accuracy can be improved through application of a per-laser linear regression in multi-laser scanners. Additionally, it is demonstrated that scan conditions, such as fluctuations in solar radiation on target surface and increased beam incident angle, have a measurable impact on scanner performance, which in turn can be effectively mitigated through deliberate selection of survey execution conditions and flight parameters.
The HDL–32E is a lightweight lidar scanner first introduced by Velodyne® in 2011 as an “ultra-compact and cost-effective optical sensor for many unmanned robotic vehicles [
The HDL–32E functions by rotating a vertical array of 32 individual laser emitter/detector pairs about a central axis, with each scanner deliberately aimed with a vertical offset angle creating a fan-like scanning pattern (
The Livox® Mid–40 is a recent addition to the lidar scanner industry, introduced by Livox® (a subsidiary of DJI®) in 2019 at a price point significantly below the automotive lidar standard price point ($599 at time of writing) [
The Mid–40 itself is capable of collecting up to 100,000 points per second within a circular field of view directly in front of the scanner’s glass window. Smaller than 10 cm3, it has been rated to collect points on objects at standoff distances of up to 260 m with precisions of up to 0.02 m (further specifications are listed in
In order to assess the performance (precision, accuracy, and stability) of the Velodyne® HDL–32E and the Livox® Mid–40, it was necessary to identify parameters that may influence the scanner performance over time. In multiple iterations of experiments, the two scanners were immobilized, oriented towards a vertical wall (painted drywall or exposed concrete) between 2 and 30 m away from the scanner, and run for a multiple-hour collection period. A precision assessment was executed by comparing each scanner’s measurements of distinct subsections of the stationary target surface over time. Then, using a highly accurate terrestrial lidar scanner (the Riegl® VZ-400), both scanner and target surface were mapped in a dense point cloud, enabling the standoff distance between scanner origin and target surface to be measured for comparison (Note that throughout the rest of this manuscript, these higher-accuracy distances are termed “true/truth” with the understanding that they are appropriate surrogates for actual true values used in accuracy assessment). The comparison of this “true” standoff distance to the measurements recorded by both the HDL–32E and Mid–40 provided an absolute accuracy assessment of each scanner, as well as a detailed understanding of their respective range walks, and how to mitigate any resulting error.
The subject scanners are positioned to maximize the target surface’s exposure to emitted lasers. Because the HDL–32E and Mid–40 have dissimilar scan patterns, the two scanners do not share the same specific orientation of each scan axis (X, Y, Z). The HDL–32E, being a line-type scanner, is laid on its side so its Z-axis is horizontal (rather than vertical), and its Y-axis is vertical (rather than horizontal) as shown in
Each experiment was executed in the same fashion: the scanner was immobilized on an adjustable, leveled tripod and oriented approximately perpendicular to the target surface. When conditions were set, the scanner was powered on and, while operating continuously for the entire experiment, observations were recorded for the first two out of every ten seconds. This two-out-of-ten collection method ensured that the lidar point clouds were manageable in size and enabled them to be easily isolated during analysis by timestamp, all while capturing any overall temporal trends in the data. The HDL–32E data were accompanied by a record of scanner diagnostic information, retrieved every 30 s of scanner operation, consisting of scanner internal temperature (°C) and power consumption (V). The Mid–40 lacks the capability to output this information, so no record of its internal temperature or power consumption was available for analysis.
While both the HDL–32E and Mid–40 record horizontal and vertical scan angles, the incident angle of the scanner’s laser beam on the target surface potentially varies between experiments and must be calculated after the conclusion of each study. The incident angle is the difference between the laser’s angle of emission and the normal of the target surface, and is calculated using Equation (1), where (X, Y, Z) is the vector between the scanner’s origin and a point in the point cloud, and (NX, NY, NZ) is the normal vector to the target surface. The normal of the target surface is determined by extracting the scanner’s observations of the target surface and fitting them to a plane, thereby ensuring both vectors are in the same coordinate system.
Indoor experiments, while not necessarily able to replicate the conditions of sUAS topographic surveys in both distances between scanner and target and environmental conditions, enabled a baseline of performance and stability of the laser scanners over time. Additionally, their repeatability allowed the identification of trends across datasets, reinforcing findings of scanner precision (or lack thereof) and associated stability. For these experiments, a temperature-controlled office with 2-m standoff distance from a blank, painted drywall wall was used. Maintaining a room temperature of approximately 22 °C and no sunlight exposure, the scanners were oriented towards the target wall, leveled, and immobilized (
In order to ensure the scanners only collected measurements from uniform sections of the target surface, the HDL–32E was restricted to scan azimuths (α) ±8.00° relative to the target surface normal for all 32 individual laser channels. The Mid–40 required no such restriction so long as it was carefully aimed to collect data from empty portions of the target surface. Because the target surface was a generally planar wall in a controlled environment, observations from each scanner at like scan angles should have been consistent in measured distance, intensity, and incident angle over time. Deviations in any of these measurements indicated a lack (or loss) of scanner stability.
After the completion of each experiment iteration, the two-second observations from the HDL–32E (
By repeating this process for all observation files, a list of average distances per channel at each two-second epoch was generated. This simplified and isolated systematic phenomena for each individual channel. These lists were, in turn, consolidated and used to generate basic summary statistics (mean, maximum, minimum, range, standard deviation) for measured distances for each laser and, when combined, for the scanner as a whole (
Diagnostic data collected for the HDL–32E was downloaded and processed in accordance with the guidance published in the Velodyne® HDL–32E User’s Manual, which contains step-by-step instructions for interpretation of the raw data provided by the scanner [
The raw observations from the Livox® Mid–40 were recorded in a .lvx file format, which is a proprietary file type used by Livox®. As with the HDL–32E, observations from every two-second epoch were saved and converted to a .csv file for analysis. Unlike the HDL–32E, this process could be done using just the Livox® software package Livox Viewer, “convert to .csv” function [
Because the Mid–40 only has one scanner/receiver pair (unlike the 32 in the HDL–32E) and due to its rosette-style scan pattern, its observation files cannot be easily divided by individual scan lines. Instead, the observations were simplified for analysis by dividing the scan area into an array using the range of observed Y and Z coordinates (
Outdoor experiments, used to more closely replicate the conditions of sUAS surveys, were conducted on the permanent parking infrastructure on the University of Florida campus. In order to maintain uninterrupted observation of a relatively flat target surface, the exposed concrete side of one of the university’s parking garages was used as the target surface, which was clearly visible from the top of the neighboring (shorter) garage (
By setting the scanners approximately 30 m away from the target wall, four distinct horizontal surfaces were visible in each scan (
As previously described, there is a known degree of error associated with edge effects while using scanners such as the Mid–40 and HDL–32E [
The design of the outdoor experiments also included the placement of alignment targets along the target surface (
During the HDL–32E data collection, alignment targets (
Following the conclusion of each scanner’s collection window, the VZ-400 was used to capture the scanner and target surface in the same reference frame in a highly dense and accurate point cloud (
The observations generated by the HDL–32E were subject to the same workflow as described in
Using the published dimensions of the HDL–32E [
Points representing the scan lines for each individual laser were then plotted along vectors originating at the laser’s origin out to the approximate distance of the target surface (
Like the HDL–32E, the workflow for processing observations made by the Mid–40 in this phase was identical to that from the precision assessments of phase two. However, additional steps were required to align and extract the truth data for the absolute accuracy assessment. First, as per the recommendations of [
To assess the performance of the HDL–32E as a whole, its measurements of the target surface (±8°) from all channels were averaged per epoch (2 out of every 10 s for a 3-h experiment duration) for a series of approximately 1080 average measurements. This series of means was plotted over time (
These overall average distance measurements can be broken down by individual scanner, and in doing so it is possible to plot the observations of each of the 32 channels within the HDL–32E over time and compare their stability at an individual level. While all 32 exhibited similar curves in their measurements to the overall plot shown in
The distribution of scanners with relatively high and low measurement range or standard deviation did not appear to be correlated with position along central axis (and resulting offset angle). Variations in measured range were only as high as approximately 0.02 m between the most- and least-precise channels, but this, along with the minimal range in standard deviations did not appear to be statistically significant. As these results were consistent across all three indoor experiments, no one individual channel appeared to negatively influence the HDL–32E’s overall precision to a significant degree.
Due to the controlled nature of the indoor experiments, one of the few parameters that changed over the course of the experiment was the internal temperature of the HDL–32E itself. This data, extracted from the self-diagnostic output of the instrument, was converted to °C and plotted, over time, with the overall average measured distances from
At a glance, there appears to be a correlation between the two curves (average measured distance and internal scanner temperature) that is generally consistent between both experiments. With the exception of a minor deviation in temperature curve in Control 2, the experiment results were almost identical: while operating in an indoor office kept at 22 °C, the HDL–32E took approximately 30 min to reach a steady state of 52.6 °C, at which time the distance measurements stabilized. Both internal temperature and measured distance remained consistent after reaching this steady state.
To confirm the suspected correlation between the two parameters, the HDL–32E’s internal temperature and overall average measured distance were plotted together and, using least squares to determine the slope and intercept coefficients, a linear trendline was fitted to the data (
The relationship between internal scanner temperature and measured distance was further explored by plotting the average measured distances (m) for each individual channel with the scanner’s internal temperature (°C). The resulting R2 values for the overall scanner and each individual channel are shown in
In the second phase, both the HDL–32E’s precision and absolute accuracy were assessed through use of measurements collected from approximately 30 m under both day and night conditions, which were then compared to the observations of a Riegl® VZ-400 terrestrial laser scanner. Both day and night iterations of the absolute accuracy assessment exhibit range walks similar to those that were observed in all previous iterations. Because of this, the scanner’s absolute accuracy varied over time; for approximately the first five minutes, the scanner’s average measurements were too “long” (greater than the “true” distance), and after that point they were (generally) shorter than the calculated “true” distance.
Between the night and day iterations (
While offset angle did not appear to have a significant impact on scanner precision, the accuracy assessment experiments demonstrated that offset angle appeared to have a potentially significant impact on scanner accuracy. While individual scanner error is generally well within 0.015 m for most channels with offset angles less than 15°, individual channels with greater offset angles tend to exhibit greater error across all scan surfaces. With the exception of channel 4 during the day iteration, there were few examples of individual channels with offset angles greater than 15° with residuals that were regularly and significantly below the overall scanner RMSE.
Finally, while the individual channels that were the most accurate could vary between night and day iterations and even target surface subsections, the least accurate individual channels were much more consistent across experiment iterations. Channels 0, 2, 6, 8, and 10 showed the highest RMSE for all garage walls combined, and for all but one individual garage wall (channel 14 has an error of −0.037 m for the day scan of garage wall 4). This implies that scanner accuracy could be improved by simply removing these channels from the data—a course of action that is possible with both Velodyne® and Phoenix Lidar Systems® software. While doing so may not necessarily improve the HDL–32E’s stability (channels 9, 19, 26, and 29 were the least precise during the night iteration and channels 11, 13, and 15 were least precise during the day iteration), the potential improvements to RMSE could justify the loss of collected data.
As expected, RMSE in the HDL–32E’s observations was not consistent over time. Across the two hours of data collected, the RMSE was shown to generally increase from the beginning to the end of the experiments. This is in line with the results shown in
With the absolute accuracy of the scanner determined, it was possible to compare experiment parameters to observations and identify any potential impacts made by the environment. Again, three environmental conditions were used for this analysis: air temperature, atmospheric pressure, and solar radiation, shown in
The higher air temperature of the day iteration versus the night iteration was expected (
This loss of stability correlated with an experiment factor other than internal scanner temperature potentially undermines the observed correlation between scanner temperature and measured distance. This theory is confirmed in
Following the same series of indoor test protocols as the Velodyne® HDL–32E, the Livox® Mid–40 was subjected to a series of indoor experiments to establish baseline of scanner behavior under controlled conditions. Because the Mid–40’s field of view is limited to ±19.2° in all directions from the normal of the scanner face, no restriction of scan angle was applied and the entire field of view was included in the analysis.
Taking the average distance measured across the field of view per epoch and plotting it over the duration of the experiment, it was possible to identify overarching trends in the scanner’s distance measurements.
As the Mid–40 does not contain the function of internal scanner temperature measurement, it was not possible to determine what (if any) degree of correlation existed between the observed range walk and the operating temperature of the laser.
The field of view raster in
To determine which trends observed in the indoor experiments carry over to survey-like conditions, a precision assessment was conducted at a scanner distance of approximately 30 m. As some obstructions blocked portions of the Mid–40’s field of view (
The final phase of the Mid–40 assessment added the observations of the Riegl® VZ-400 to the outdoor experiments, which were repeated in both night and day conditions. The only significant change to workflow was the addition of processing the differences between observed measurements (from the Mid–40) and truth measurements (from the Riegl® VZ-400).
As shown in
Of note in the outdoor experiments was the observed range walk of a lesser magnitude and in the opposite direction compared to the indoor experiments. While it is possible that this change in direction is dictated by the operating temperature, it is difficult to establish a correlation between the two. The study area temperature for indoor experiments was an average of 22.2 °C while the study area temperature for this outdoor experiment was an average of 14.2 °C. The calculated R2 between the precision (range of measurements) and incident angle was found to be minimal with a value of 0.16.
When the range of measurements across the field of view are shown in a colorized raster (
In order to collect “truth” measurements, the points on the target surface as observed by the Mid–40 must be identified in the VZ-400’s point cloud. This was accomplished using the alignment targets placed on the target surface observed by both scanners (
Subtracting the “true” distance between point and Mid–40 origin for each point yielded the residual arrays shown in
Calculating the RMSE for each experiment showed both night and day experiments had like levels of accuracy, with the RMSE of the night experiment at 0.004 m and the day at 0.009 m.
As there was no clear significant difference between the results from night and day experiments, and there was likely no impact from observed experiment conditions. This observation was consistent for atmospheric pressure, solar radiation, and air temperature (
The calculated R2 between error and incident angle was found to be minimal at less than 0.1 for both and night experiments.
Both the HDL–32E and Mid–40 exhibited a degree of temporal instability and autocorrelation in their indoor control experiments (
These findings confirm that is there a temporal bias in both scanners, though this bias is not uniform between the two scanners. The HDL–32E’s range walk being highly correlated with its internal temperature further suggests that this instability in measured distance is the result of the heat generated by its continued operation. As its operating temperature cannot be directly measured in the Mid–40’s current configuration, a like finding of correlation could not be confirmed in the Mid–40. However, given that both scanners have moving components, operate within similar voltage ranges, and demonstrate a range walk under like indoor conditions, it is possible that the Mid–40 is also subject to a similar internal temperature-correlated distance bias.
The loss of observed range walk at increased scanner standoff distance for the Mid–40 may be attributed to the minimal calibration of the scanner. The HDL–32E used in this study has been deliberately calibrated by the distributor, Phoenix Lidar Systems®, for high accuracy mobile mapping applications, whereas the Mid–40 only has the manufacturer’s factory calibration. While the effectiveness of calibration methods is outside the scope of this study, the overall increase of noise visible in the Mid–40’s point cloud at study distance versus the HDL–32E (and the HDL–32E’s general lack of significant outlier measurements) implies that Mid–40 lacks the same capabilities as the HDL–32E to compensate for potential variability in its observations. This lack of calibration may have removed any impact of range walk in the Mid–40’s outdoor experiments, but instrument bias being lost due to an increase in noise is hardly a viable solution to solving for scanner instability.
Regardless of why the range walk exists, the fact that it can be replicated under both controlled and survey conditions implies that it is an active source of error in the subject scanners, and must therefore be accounted for when executing surveys, topographic studies, or other high accuracy lidar applications.
The precision (repeatability) of the distance measurements made by the HDL–32E and Mid–40 were shown to have a moderate correlation with two separate scan parameters: the duration of scanner operation and the angle of beam emission relative to the scanner’s X-axis.
When range walk could be observed, the precision of both the HDL–32E and the Mid–40 had a direct relationship to the duration of their operation; the longer the scanner operates, the more precise its measurements became. During the first 30 min of use, the range walk resulted in a 66% and 75% reduction in standard deviations and 93% and 88% reduction in range of average measurements for the HDL–32E and Mid–40, respectively (
While range walk was shown to impact scan precision over extended observation periods, the scan angle was found to impact measurement precision regardless of duration of scanner operation; measurement precision generally decreased as scan angle increases from nadir (
Given the findings regarding emission angle impact on scanner precision, it can be concluded that the most precise surveys were executed when the scanner (HDL–32E or Mid–40) was observing targets near-nadir and after having been allowed to stabilize in its operating environment.
Scanner error, defined in this research by the residual between observed and “true” measurements, was shown to be influenced by the aforementioned factors that correlate to changes in temporal stability and precision. As any change in the scanner’s measurements while held stationary changes the amount of observed error, this is a practical conclusion. However, this research shows that the presence of relatively high or low temporal stability or precision do not necessarily indicate the collection of the most accurate data.
As a whole, the HDL–32E exhibited an overall RMSE at or within the manufacturer’s specifications (0.020 m) for both night and day experiments. In all accuracy assessments conducted with the HDL–32E, the initial overall measurements were regularly too far (positive error) before decreasing to and past the “true” distance and stabilizing at a consistent level measurement that was shorter than the “true” distance (negative error) (
The Mid–40 also demonstrated an overall RMSE that was within manufacturer specifications (0.020 m) in its accuracy assessments. While not consistent in its spatial distribution or magnitude, the measured error (after filtering potential outliers in the data) stayed on average within a centimeter of the “true” distance. The temporal distribution of error was shown to be relatively stable with no clear trends regarding level of error and scan conditions over time. However, the amount of error during the day experiment was more than double that of the night, implying that the addition of daytime conditions limits the average scanner accuracy (
This study identified two main sources of error in the HDL–32E: scanner temperature and inaccurate individual laser emitter/detector pairs. Using the two successful accuracy assessments and the derived relationships between parameters presented in this study, three potential solutions were implemented to maximize scanner accuracy: removal of channels with high error, application of linear regressions developed from the same accuracy assessment (adjusting night measurements with regressions developed from the night accuracy assessment iteration, vice versa), and application of linear regressions developed from a separate, independent accuracy assessment (adjusting night dataset with day accuracy assessment results, vice versa). The resulting levels of RMSE before and after each of these techniques were implemented are shown in
The effectiveness of the linear regressions was further highlighted by applying them to each subsection of the target surface, extracted by the HDL–32E’s scan azimuth. While accuracy and precision were shown to decrease as scan azimuth increases,
Removal of the least accurate channels from the dataset, while a relatively simple solution, did not demonstrate a large enough impact on the HDL–32E’s overall RMSE relative to other error mitigation techniques to justify loss of 20% or more of observed points. Instead, given the distribution of error over time, establishing and exploiting the relationship between error and scanner temperature demonstrated the most potential in maximizing measurement accuracy in a survey environment. Additionally, the net improvement in scanner RMSE using the regressions developed from alternate conditions (day regressions used for night data, night regressions for day data) implied that solutions developed under experiment conditions can be applied to observations from a variety of survey and mapping missions.
As the Mid–40 has no observed parameters with as high of a correlation with measured error as internal temperature in the HDL–32E, the findings regarding its precision and accuracy do not lend themselves to as straightforward an approach as a linear regression. Applying the recommendations of [
While only observed in the HDL–32E, the internal scanner temperature is the parameter that has the highest observed correlation with average distance measurement with, in indoor experiments, repeated examples of a R2 value in excess of 0.98. This correlation was observed in all experiments where the scanner internal temperature was recorded, with the lowest R2 observed still being greater than 0.73. Though not consistent between individual scanners (R2 ranging from 0.13 to 0.88), this study demonstrates that it is possible to build and implement linear regressions to reduce overall observed error. Given the range walk observed in the indoor tests with the Mid–40, it is implied that a similar relationship between internal scanner temperature and distance measurement may exist and could potentially be solved for in a like manner.
The disparity between night and day experiments for both the HDL–32E and Mid–40 in terms of precision and accuracy implies that the addition of sunlight on the target surface negatively impacts the overall performance of the scanners. Given that sunlight contains wavelengths of 905 nm (the wavelength used by both the HDL–32E and Mid–40), this is not unexpected. However, the fluctuations in observed solar radiation and the corresponding loss of precision observed in the HDL–32E implies that accounting for just the presence of sunlight during day experiments is insufficient for mitigating its effects. If collection at night is not possible due to safety concerns, local airspace restrictions (a lack of a waiver from the Federal Aviation Administration (FAA Part 107.29(b): operation at night) in the United States, for example), or survey requirements, it is recommended that consistent levels of solar radiation be prioritized during surveys to ensure consistent results.
The HDL–32E and Mid–40 exhibited a moderate degree of correlation between laser scan angle and both their respective levels of precision and accuracy. Generally, as emission angle (and incident angle) increased, precision and accuracy decreased, with the lowest error and range of measurements being found near the instrument’s nadir. However, no correlation between performance and scan angle was found to meet or exceed that of other parameters (e.g., scanner temperature), and therefore developing a solution to account for identified error has not proven to be effective. Rather than retroactively solving for increased uncertainty at higher scan angles, it is recommended that data collection be executed with the highest feasible swath overlap to maximize the percentage of the study area observed near nadir. The resulting increase in redundant observations will likely mitigate errors from other sources present in the study area or instrument.
By conducting a deliberate assessment of the Velodyne® HDL–32E’s and Livox® Mid–40’s distance measurements in controlled and survey-like conditions, a holistic understanding of each scanner’s spatial and temporal trends in accuracy and precision were developed. It was shown that, under the reported conditions and limitations, the HDL–32E and the Mid–40 each me and exceeded their published manufacturer’s accuracy specifications, demonstrating potential for use in high-accuracy topographic surveys for a fraction the cost of their high-tier counterparts.
Each scanner exhibited a clear range walk of just under 0.020 m during the first 30 min of indoor conditions, which in the HDL–32E was found to be highly correlated with scanner internal temperature. This range walk and like correlation with temperature was found again in outdoor experiments with the HDL–32E, but was lost in the noise of the observations in the Mid–40. Allowing the scanners to operate for 30 min and stabilize in collection conditions prior to use effectively mitigated this instability.
Beyond the range walk, scanner precision and accuracy were found to be directly impacted by scan angle; as the angle of emission increased from the scanner’s nadir, there was a moderately correlated loss of both precision and accuracy in both the HDL–32E and Mid–40. However, this angular offset was not equal in all axes, and the resulting loss of performance did not lend itself to a correction model. In order to mitigate this loss of precision and accuracy, it is recommended to condense flight lines for sUAS surveys and restrict scan angles to ±15° when observing generally flat or planar environments.
The presence of solar radiation on the target surface was found to decrease the precision and accuracy of scanner measurements to a minor degree, while fluctuations in solar radiation were found to be strongly correlated with potentially significant losses of precision. Limiting lidar collections to hours of night was found to reduce RMSE by up to 5% in the HDL–32E and 50% in the Mid–40.
By taking advantage of the relationship between scanner temperature and measured distance by each individual scanner of the HDL–32E, linear regressions were developed and used to adjust the reported distance measurements, resulting in a decrease in RMSE of over 88% when applied to like experiments. More importantly, a RMSE reduction of over 33% was found when regressions from one experiment were applied to the other, indicating that solutions developed in test conditions can be applied to datasets collected under a variety of conditions.
This research demonstrates a methodology to assess the temporal stability of lidar scanners that can be implemented by most lidar users. However, it is limited to just the two subject scanners. Future research should explore the performance of other low-cost lidar scanners from both Velodyne® and Livox®, as well as other manufacturers in the mobile mapping industry. Additionally, the impact of calibrations on the mapping of scanner stability (and its resulting mitigation) should be explored in future applications.
Conceptualization: C.K. and B.W.; experiment methodology, C.K., O.C., B.W. and H.A.L.; software: C.K. and B.W.; formal analysis: C.K.; writing—original draft preparation: C.K.; writing—review and editing: B.W., A.A.-E., H.A.L. and O.C.; project administration: C.K., B.W. and A.A.-E. All authors have read and agreed to the published version of the manuscript.
Data available on request from the primary author.
The authors would like to thank Ryan Brazeal for his subject matter expertise and advice regarding the use and study of the Livox® Mid–40. We also thank the University of Florida Police Department (UFPD) and the University of Florida’s Transportation and Parking Services (TAPS) for their coordinated efforts to ensure successful data collections, and to Phoenix Lidar Systems® for technical support.
The authors declare no conflict of interest.
While illustrated in
Calculated error (m) per individual channel, per subsections of target surface in Velodyne® HDL–32E during 30 m, night experiment. Subsections are labeled W1 through W4, with W1 being the lowest observed and W4 being the highest.
Channel | Offset | W1 | W2 | W3 | W4 | All |
---|---|---|---|---|---|---|
31 | 10.67° | 0.002 | 0.006 | 0.011 | 0.017 | 0.009 |
29 | 9.33° | −0.003 | 0.003 | 0.008 | 0.012 | 0.005 |
27 | 8.00° | 0.005 | 0.007 | 0.012 | 0.014 | 0.010 |
25 | 6.67° | 0.000 | 0.000 | 0.011 | 0.020 | 0.008 |
23 | 5.33° | 0.006 | 0.010 | 0.017 | 0.021 | 0.014 |
21 | 4.00° | 0.004 | 0.008 | 0.011 | 0.019 | 0.010 |
19 | 2.67° | 0.002 | 0.004 | 0.012 | 0.017 | 0.009 |
17 | 1.33° | 0.009 | 0.011 | 0.012 | 0.015 | 0.012 |
15 | 0.00° | 0.002 | 0.005 | 0.008 | 0.007 | 0.006 |
13 | −1.33° | 0.006 | 0.007 | 0.011 | 0.010 | 0.009 |
11 | −2.67° | 0.000 | 0.002 | 0.002 | 0.004 | 0.002 |
9 | −4.00° | 0.005 | 0.007 | 0.009 | 0.013 | 0.008 |
7 | −5.33° | 0.015 | 0.015 | 0.017 | 0.011 | 0.015 |
5 | −6.67° | 0.009 | 0.008 | 0.008 | 0.013 | 0.009 |
3 | −8.00° | 0.005 | 0.002 | 0.005 | 0.006 | 0.005 |
1 | −9.33° | −0.004 | −0.006 | −0.003 | −0.003 | −0.004 |
30 | −10.67° | 0.001 | −0.003 | −0.005 | −0.009 | −0.004 |
28 | −12.00° | 0.006 | 0.001 | −0.002 | −0.010 | −0.001 |
26 | −13.33° | −0.005 | −0.009 | −0.013 | −0.021 | −0.012 |
24 | −14.67° | −0.001 | −0.007 | −0.009 | −0.010 | −0.007 |
22 | −16.00° | −0.016 | −0.018 | −0.021 | −0.020 | −0.019 |
20 | −17.33° | −0.013 | −0.020 | −0.020 | −0.022 | −0.019 |
18 | −18.67° | −0.012 | −0.018 | −0.020 | −0.024 | −0.019 |
16 | −20.00° | −0.012 | −0.016 | −0.020 | −0.021 | −0.017 |
14 | −21.33° | −0.010 | −0.017 | −0.022 | −0.027 | −0.019 |
12 | −22.67° | −0.006 | −0.012 | −0.013 | −0.015 | −0.012 |
10 | −24.00° | −0.025 | −0.032 | −0.034 | −0.039 | −0.032 |
8 | −25.33° | −0.019 | −0.026 | −0.032 | −0.037 | −0.029 |
6 | −26.67° | −0.020 | −0.026 | −0.032 | −0.037 | −0.029 |
4 | −28.00° | −0.011 | −0.022 | −0.033 | −0.040 | −0.025 |
2 | −29.33° | −0.017 | −0.026 | −0.030 | −0.033 | −0.026 |
0 | −30.67° | −0.030 | −0.036 | −0.042 | −0.043 | −0.038 |
Overall Average Error | −0.004 | −0.006 | −0.006 | −0.007 | −0.005 | |
RMSE (m) | 0.011 | 0.015 | 0.019 | 0.022 | 0.018 |
Calculated error (m) per individual channel, per subsection of target surface in Velodyne® HDL–32E during 30 m, day experiment. Subsections are labeled W1 through W4, with W1 being the lowest observed and W4 being the highest.
Channel | Offset | W1 | W2 | W3 | W4 | All |
---|---|---|---|---|---|---|
31 | 10.67° | −0.012 | −0.007 | 0.007 | 0.008 | −0.001 |
29 | 9.33° | −0.013 | −0.009 | −0.001 | 0.002 | −0.005 |
27 | 8.00° | −0.009 | −0.006 | 0.006 | 0.008 | 0.000 |
25 | 6.67° | −0.004 | −0.006 | 0.007 | 0.012 | 0.002 |
23 | 5.33° | −0.004 | −0.001 | 0.011 | 0.006 | 0.003 |
21 | 4.00° | −0.008 | −0.004 | 0.001 | −0.001 | −0.003 |
19 | 2.67° | −0.008 | −0.005 | −0.003 | −0.013 | −0.007 |
17 | 1.33° | −0.004 | 0.002 | 0.001 | −0.005 | −0.002 |
15 | 0.00° | −0.014 | −0.008 | −0.014 | −0.027 | −0.016 |
13 | −1.33° | −0.001 | −0.004 | −0.020 | −0.033 | −0.014 |
11 | −2.67° | −0.004 | −0.009 | −0.026 | −0.033 | −0.018 |
9 | −4.00° | 0.006 | 0.001 | −0.008 | −0.017 | −0.004 |
7 | −5.33° | 0.003 | −0.003 | −0.016 | −0.022 | −0.009 |
5 | −6.67° | 0.006 | −0.001 | −0.010 | −0.018 | −0.006 |
3 | −8.00° | 0.006 | −0.002 | −0.010 | −0.023 | −0.007 |
1 | −9.33° | −0.007 | −0.005 | −0.016 | −0.027 | −0.014 |
30 | −10.67° | −0.004 | −0.007 | −0.015 | −0.021 | −0.012 |
28 | −12.00° | −0.004 | −0.005 | −0.019 | −0.026 | −0.014 |
26 | −13.33° | −0.007 | −0.010 | −0.022 | −0.029 | −0.017 |
24 | −14.67° | −0.003 | −0.006 | −0.019 | −0.030 | −0.015 |
22 | −16.00° | −0.010 | −0.013 | −0.023 | −0.033 | −0.020 |
20 | −17.33° | −0.004 | −0.009 | −0.014 | −0.029 | −0.014 |
18 | −18.67° | −0.004 | −0.011 | −0.015 | −0.027 | −0.014 |
16 | −20.00° | −0.010 | −0.015 | −0.018 | −0.030 | −0.018 |
14 | −21.33° | −0.008 | −0.017 | −0.025 | −0.037 | −0.022 |
12 | −22.67° | −0.003 | −0.009 | −0.013 | −0.027 | −0.013 |
10 | −24.00° | −0.021 | −0.028 | −0.028 | −0.039 | −0.029 |
8 | −25.33° | −0.013 | −0.020 | −0.024 | −0.032 | −0.022 |
6 | −26.67° | −0.015 | −0.024 | −0.030 | −0.037 | −0.026 |
4 | −28.00° | 0.001 | −0.009 | −0.014 | −0.005 | −0.021 |
2 | −29.33° | −0.015 | −0.022 | −0.023 | −0.033 | −0.023 |
0 | −30.67° | −0.021 | −0.029 | −0.027 | −0.042 | −0.030 |
Overall Average Error | −0.007 | −0.009 | −0.013 | −0.021 | −0.013 | |
RMSE (m) | 0.011 | 0.014 | 0.023 | 0.033 | 0.018 |
In order to explore the HDL–32E’s accuracy over time, the calculated RMSE can be broken down by garage wall, by collection window (every 2000 s or approximately 30 min, as was done for the precision analysis), shown in
Calculated RMSE (m) per 2000 s of iteration, per subsection of target surface in Velodyne® HDL–32E during 30 m, night experiment. Subsections are labeled W1 through W4, with W1 being the lowest observed and W4 being the highest.
Period | W1 | W2 | W3 | W4 | All |
---|---|---|---|---|---|
0–2000 s | 0.012 | 0.015 | 0.019 | 0.023 | 0.017 |
2000–4000 s | 0.012 | 0.016 | 0.019 | 0.023 | 0.017 |
4000–6000 s | 0.012 | 0.016 | 0.020 | 0.023 | 0.017 |
6000–8000 s | 0.012 | 0.016 | 0.019 | 0.024 | 0.017 |
All | 0.012 | 0.016 | 0.019 | 0.023 | 0.017 |
Calculated RMSE (m) per 2000 s of iteration, per subsection of target surface in Velodyne® HDL–32E during 30 m, day experiment. Subsections are labeled W1 through W4, with W1 being the lowest observed and W4 being the highest.
Period | W1 | W2 | W3 | W4 | All |
---|---|---|---|---|---|
0–2000 s | 0.008 | 0.011 | 0.016 | 0.025 | 0.012 |
2000–4000 s | 0.011 | 0.013 | 0.022 | 0.034 | 0.018 |
4000–6000 s | 0.012 | 0.016 | 0.027 | 0.039 | 0.022 |
6000–8000 s | 0.014 | 0.018 | 0.027 | 0.038 | 0.023 |
All | 0.011 | 0.014 | 0.023 | 0.033 | 0.018 |
(
Livox® Mid–40 as mounted on a camera tripod.
Rosette-style scan pattern of the Livox® Mid–40, over a course of three seconds of operation. From left to right, each circle depicts the addition of one-second’s worth of data from a stationary scanner observing a stationary target surface.
Orientation of (
Velodyne® HDL–32E raw observations as points representing scan lines from all 32 individual channels (even numbers 0 through 30 from left to right, followed by odd numbers 1 through 31 from left to right).
Velodyne® HDL – 32E measurement analysis workflow.
Breakdown of example Mid–40 data from (
Study site and target surface at the University of Florida. Top: The scanners were oriented at the side of an adjacent parking garage (yellow) at standoff distances to reflect a sUAS topographic data collection. Bottom: Scanner observations relative to each scanner are shown with extracted subsections highlighted in red. In order to facilitate reporting of results, the subsections of the target surface are designated W1 through W4, with W1 being the lowest and W4 being the highest observed by the scanners.
(
Point clouds generated by the Riegl® VZ-400, representing the (
Velodyne® HDL–32E with individual scanner and receiver pairs (32 total) plotted along central axis with focal point locations plotted for every 0.01° of rotation (
Projection of Velodyne® HDL–32E scan lines in (
Alignment of (
Velodyne® HDL–32E average distance measurements over time (average of all points from all 32 scanners within the −8° to +8° scan window) with variations in scanner precision highlighted.
Comparison of individual channels’ average measured distances for the Velodyne® HDL–32E. The two pairs shown are the ones with the highest (channel 20) and lowest (channel 7) precision in their averaged measurements over time.
Velodyne® HDL–32E indoor experiment overall average measured distances (m) plotted over time elapsed with scanner internal temperature (°C).
Linear relationship and correlation between Velodyne® HDL–32E internal temperature and overall average measured distance through the duration of indoor experiments. Equation for linear line of best fit and corresponding R2 values are shown on the figure.
R2 values for the linear trendlines and internal temperature/average measured distance for each of the Velodyne® HDL – 32E’s individual channels.
Linear relationship and correlation between Velodyne® HDL – 32E internal temperature and average measured distances for the individual channels with the highest (channel 14) and lowest (channel 29) R2 through the duration of indoor experiments. Equation for linear line of best fit and corresponding R2 values are shown on the figure.
Average measured distance of all 32 individual channels versus internal scanner temperature and “true” distance value for both (
Calculated error (m) per individual channel, per subsections of target surface in Velodyne® HDL–32E during a 30 m (
RMSE (m) per Velodyne® HDL–32E observation file for both (
Weather conditions during Velodyne® HDL–32E night and day absolute accuracy experiments: (
Linear relationship between Velodyne® HDL–32E average distance measurements and internal temperature for both night and day experiments, (
Average measured distance of the Livox® Mid–40’s field of view over time during indoor experiments.
Raster of Livox® Mid–40 field of view, colorized by range of distance measurements per cell over duration of three-hour indoor experiments.
Livox® Mid–40 indoor point cloud (subset of 10 observation files), colorized by surface point density (points per cm).
Raw observations of the Livox® Mid–40 during outdoor experiments, colored by measured distance from scanner origin.
(
Range of measurements for the Livox® Mid–40 during (
(
Livox® Mid–40 distribution of error in (
Livox® Mid–40 distance measurements (running average, per 60 s) and ongoing weather conditions (
Temporal Stability of Velodyne® HDL–32E and Livox® Mid–40 during indoor experiments.
Calculated (
Distribution of precision (measurement range) for the (
Accuracy of Velodyne® HDL–32E over time in night and day outdoor experiments. Shift in measured and “true” distance between night and day experiments is the result of minor adjustments made to scanner position after night iteration.
Amount of error observed in Livox® Mid–40 measurements in night and day scan conditions, by percentile.
Application of error mitigation techniques on Velodyne® HDL–32E measurements.
Applications of linear regressions on subsections of Velodyne® HDL–32E observations, split by scan azimuth.
Velodyne® HDL–32E Select Specifications [
Platform | |
---|---|
Overall Dimensions | 24.6 cm × 11.6 cm × 11.6 cm |
Operating Temperature | −10 °C to +40 °C |
Weight | 2.40 kg |
|
|
Laser Wavelength | 903 nm |
Laser Classification | Class 1 Eye Safe |
Channels | 32 |
Measurement Range (min, max) | 1 m, 100 m |
Measurement Range (resolution) | 0.002 m |
Manufacturer’s Reported Range Accuracy | ±0.02 m |
Measured Returns | 1 or 2 |
Field of View (Vertical) | +10.67° to −30.67° (41.33°) |
Field of View (Horizontal) | 360° |
Angular Resolution (Vertical) | 1.33° |
Angular Resolution (Horizontal/Azimuth) | 0.08°–0.33° |
Beam Divergence | 3 mrad |
Rotation Rate | 5 Hz to 20 Hz |
3D Lidar Data Points Generated: | |
Single/Dual Return Mode | 695,000/1,390,000 points per second |
Livox® Mid–40 Select Specifications [
Platform | |
---|---|
Overall Dimensions | 8.8 cm × 7.6 cm × 6.9 cm |
Operating Temperature | −20 °C to +65 °C |
Weight | 760 g |
|
|
Laser Wavelength | 905 nm |
Laser Classification | Class 1 (IEC 60825-1:2014) eye safe |
Channels | 1 |
Measurement Range (min) | 1 m |
Measurement Range (max) | 90 m at 10% reflectivity |
130 m at 20% reflectivity | |
260 m at 80% reflectivity | |
Manufacturer’s Reported Measurement Range (precision) (1 σ at 20 m) | 2 cm |
Beam Divergence | 0.28° (vertical) × 0.03° (horizontal) |
Measured Returns | 1 |
Field of View (Circular) | 38.4° |
Angular Accuracy | <0.1° |
Point Generation Rate | 100,000 points per second |
Experiment List and Conditions.
Lidar Scanner | Standoff Distance | Day/Night | Experiment Date |
---|---|---|---|
HDL–32E | 30 m | Day | 4 June 2021 |
HDL–32E | 30 m | Night | 9 July 2021 |
HDL–32E | 30 m | Day | 13 July 2021 |
HDL–32E | 60 m | Night | 15 July 2021 |
Mid–40 | 30 m | Night | 4 November 2021 |
HDL–32E | 30 m | Night | 21 November 2021 * |
Mid–40 | 30 m | Night | 21 November 2021 * |
HDL–32E | 30 m | Day | 21 November 2021 * |
Mid–40 | 30 m | Day | 21 November 2021 * |
* Experiment used in final accuracy assessment.
Summary of Velodyne® HDL–32E indoor experiment indoor measurements (average of all points from each scanner within the −8° to +8° scan window).
Scanner | Offset | Range (m) | St Dev (m) | Average (m) |
---|---|---|---|---|
31 | 10.67° | 0.011 | 0.001 | 2.226 |
29 | 9.33° | 0.013 | 0.001 | 2.217 |
27 | 8.00° | 0.011 | 0.002 | 2.196 |
25 | 6.67° | 0.016 | 0.001 | 2.180 |
23 | 5.33° | 0.012 | 0.002 | 2.172 |
21 | 4.00° | 0.018 | 0.002 | 2.162 |
19 | 2.67° | 0.016 | 0.002 | 2.143 |
17 | 1.33° | 0.024 | 0.003 | 2.158 |
15 | 0.00° | 0.021 | 0.002 | 2.124 |
13 | −1.33° | 0.015 | 0.001 | 2.119 |
11 | −2.67° | 0.015 | 0.001 | 2.111 |
9 | −4.00° | 0.018 | 0.002 | 2.107 |
7 | −5.33° | 0.026 | 0.004 | 2.105 |
5 | −6.67° | 0.021 | 0.002 | 2.101 |
3 | −8.00° | 0.027 | 0.003 | 2.099 |
1 | −9.33° | 0.014 | 0.001 | 2.093 |
30 | −10.67° | 0.017 | 0.002 | 2.103 |
28 | −12.00° | 0.019 | 0.002 | 2.105 |
26 | −13.33° | 0.013 | 0.002 | 2.105 |
24 | −14.67° | 0.019 | 0.002 | 2.114 |
22 | −16.00° | 0.019 | 0.002 | 2.120 |
20 | −17.33° | 0.011 | 0.001 | 2.131 |
18 | −18.67° | 0.017 | 0.002 | 2.128 |
16 | −20.00° | 0.013 | 0.002 | 2.144 |
14 | −21.33° | 0.022 | 0.003 | 2.150 |
12 | −22.67° | 0.021 | 0.003 | 2.160 |
10 | −24.00° | 0.018 | 0.002 | 2.168 |
8 | −25.33° | 0.015 | 0.002 | 2.192 |
6 | −26.67° | 0.017 | 0.002 | 2.207 |
4 | −28.00° | 0.010 | 0.001 | 2.225 |
2 | −29.33° | 0.015 | 0.002 | 2.250 |
0 | −30.67° | 0.012 | 0.001 | 2.253 |
Average | 0.017 | 0.002 | 2.151 |