Comparison of 2D and 3D semantic segmentation in urban areas using fused hyperspectral and lidar data

  • Agnieszka Kuras  
  • Anna Jenul
  • Maximilian Brell
  • Ingunn Burud
Faculty of Science and Technology, Norwegian University of Life Sciences, PB 5003, 1430 Aas, Norway

 https://orcid.org/0000-0002-6919-3483
 Search for papers by this author
Helmholtz Center Potsdam, GFZ German Research Centre for Geosciences, Telegrafenberg, 14473 Potsdam, Germany

 https://orcid.org/0000-0002-3759-7483
 Search for papers by this author
Faculty of Science and Technology, Norwegian University of Life Sciences, PB 5003, 1430 Aas, Norway

 https://orcid.org/0000-0003-0637-4073
 Search for papers by this author
 Corresponding Author
Faculty of Science and Technology, Norwegian University of Life Sciences, PB 5003, 1430 Aas, Norway
[email protected]
 https://orcid.org/0000-0002-0541-5614
 Search for papers by this author

Multisensor data fusion has become a hot topic in the remote sensing research community. This is thanks to significant technological advances and the ability to extract information that would have been challenging with a single sensor. However, sensory enhancement requires advanced analysis that enables deep learning. A framework is designed to effectively fuse hyperspectral and lidar data for semantic segmentation in the urban environment. Our work proposes a method of reducing dimensions by exploring the most representative features from hyperspectral and lidar data and using them for supervised semantic segmentation. In addition, we chose to compare segmentation models based on 2D and 3D convolutional operations with two different model architectures, such as U-Net and ResU-Net. All algorithms have been tested with three loss functions: standard Categorical Cross-Entropy, Focal Loss and a combination of Focal Loss and Jaccard Distance—Focal–Jaccard Loss. Experimental results demonstrated that the 3D segmentation of U-Net and ResU-Net with Focal and Focal–Jaccard Loss functions had significantly improved performance compared to the standard Categorical Cross-Entropy models. The results show a high accuracy score and reflect reality by preserving the complex geometry of the objects.

Keywords: semantic segmentation, 3D segmentation, urban environment, hyperspectral imaging, lidar, hyperspectral lidar fusion, data fusion, deep learning, multisensor fusion, remote sensing

Metrics

Downloads:

363

Abstract Views:

2,657