Robust Registration and Learning using Multi-radii Spherical Polar Fourier Transform.

Article Link in Signal Processing, Volume 217, April 2024, 109309: https://www.sciencedirect.com/science/article/abs/pii/S0165168423003833

Figure caption

• Top, surface data is transformed into voxels or volumetric representation using suitable resolution then holes are added, overlap between the original cat model and the distorted model is just about $36$ percentage i.e. only a small proportion of matching voxels exist even with a perfect alignment (for simplicity no rotation here).
• Middle row, SPFT based magnitude spherical images in increasing order of radii using original cat voxel model.
• Bottom row, SPFT data of the distorted model.

Abstract

This paper presents effective methods using spherical polar Fourier transform data for two different appli- cations, with active areas of research, one as a conventional volumetric registration algorithm and other as a machine learning classification network. For registration purposes the proposed method has the following advantageous features: (i) it is a unique and effective technique for estimating up to 7 degrees of freedom for 3D volumetric registration, that has a closed-form solution for 3D rotation estimation, and which does not resort to recurrence relations or search for point correspondences between two objects/volumes, (ii) it allows for robust rotation estimation determined simultaneously on multiple spectral spheres, therefore complete stack of such spherical layers can be processed concurrently to obtain accurate and optimal all three angles, and (iii) it has the ability to handle arbitrary large rotation angles, is shown to be robust against the presence of noise, holes/missing data, and partial overlaps. We demonstrate the effectiveness of our solution with extensive experimentation, including a set of scanned MRI images, a crashed car parking dataset, and the Princeton shape benchmark dataset with hundreds of 3D objects. For the classification solution we modify and adapt an existing network in the literature, a type of spherical convolutional network, that is suitable for processing multi-radii spectral spherical data, and showcase the resulting robustness achieved in classification of objects from the ModelNet40 dataset, especially in the presence of outliers, additive noise, and missing data.

General comment

Most of the folders in this repository should have readme files that describe the contents and purpose of it.

Overall organization

[1] SPFT-ArrayFire has C++ and ArrayFire library code, which is common for both volumetric registration as well as for machine learning, to simply compute the SPFT given volumes. However it is only for the machine learning work, involving tensorflow, python and jupyter that we had to extract this.

• Much of the experiments for volumetric registration were conducted in MATLAB where both SPFT and SOFT C/C++ projects had to be interfaced as MEX libraries.

[2] For Machine Learning:

• ModelNet40 preprocessing contains the scripts necessary to obtain just the magnitude spectrum from multi-radii SPFT.
• It also contains a helpful script to create tensorflow record that is necessary and used to train and test the classification network, the slightly modified spherical CNN. Much of the data examples, both in archive as well as the extracted images, had to be erased for the repository, otherwise it was bloating it over many GBs and causing git push unexpected hung-up failures.
• Common includes some useful and helpful MATLAB scripts extracted from the conventional volumetric registration algorithm for separately visualizing the images in this setting.
• Network used, after all the preprocessing and generation of tfrecords from the juypter notebook, is R-SCNN see external source link below as noted in the paper.

[3] For Volume Registration:

• SOFT2.0 contains the base C code for computing spherical correlations given two equiangular grids on a single sphere.
• SOFT_MEX is a project that allows for create of MEX file to be used in MATLAB with the registration scripts, this helps in visualizing the correlation results nicely in MATLAB.
• Rotation Estimation folder, take a look at this file TestMySOFTForComparisonPurposes.m
• 3D_SphericalPolarFT_Registration should have many helpful functions and useful scripts, much of it will try to add other folders from https://github.com/syedalamabbas/2D_3D_PolarFourierTransform or MEX_SOFT project that is included.
• Finally, take a look at this file/function ComputeSOFTRotation_SphericalPolarFT

Acknowledgements

Of some of the software/sources used, in no particular order, and please consider that each of them would have their own respective licenses:

[1] SOFT: SO(3) Fourier Transforms

• Kostelec, P.J., Rockmore, D.N., 2008. Ffts on the rotation group. Journal of Fourier analysis and applications 14, 145-179.
• Kostelec, P.J., 2015. The Soft Package. Retrieved from, http://www.cs.dartmouth.edu/ geelong/soft/.

[2] spherical-cnn

• Esteves, C., Allen-Blanchette, C., Makadia, A., Daniilidis, K., 2018. Learning so (3) equivariant representations with spherical cnns, in: Proceedings of the European Conference on Computer Vision (ECCV), pp. 52-68.
• https://github.com/daniilidis-group/spherical-cnn

[3] R-SCNN

• Mukhaimar, A., Tennakoon, R., Lai, C.Y., Hoseinnezhad, R., Bab-Hadiashar, A., 2022. Robust object classification approach using spherical harmonics. IEEE Access 10, 21541-21553.
• https://github.com/AymanMukh/R-SCNN

[4] 2D_3D_PolarFourierTransform

• Abbas, S.A., Sun, Q., Foroosh, H., 2016. An exact and fast computation of discrete fourier transform for polar and spherical grid. IEEE Transactions on Signal Processing 65, 2033-2048.
• https://github.com/syedalamabbas/2D_3D_PolarFourierTransform

[5] Rotation Estimation

• Althloothi, S., Mahoor, M.H., Voyles, R.M., 2013. A robust method for rotation estimation using spherical harmonics representation. IEEE Transactions on Image Processing 22, 2306-2316.

[6] ModelNet40 dataset

• Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J., 2015. 3d shapenets: A deep representation for volumetric shapes, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920.

[7] Princeton Shape Benchmark (PSB)

• Shilane, P., Min, P., Kazhdan, M., Funkhouser, T., 2004. The princeton shape benchmark, in: Proceedings Shape Modeling Applications, 2004., IEEE. pp. 167-178.