Proposing A Novel Framework Called LieBN: A Conclusion

Table of Links

Abstract and 1 Introduction

2 Preliminaries

3. Revisiting Normalization

3.1 Revisiting Euclidean Normalization

3.2 Revisiting Existing RBN

4 Riemannian Normalization on Lie Groups

5 LieBN on the Lie Groups of SPD Manifolds and 5.1 Deformed Lie Groups of SPD Manifolds

5.2 LieBN on SPD Manifolds

6 Experiments

6.1 Experimental Results

7 Conclusions, Acknowledgments, and References

\ APPENDIX CONTENTS

A Notations

B Basic layes in SPDnet and TSMNet

C Statistical Results of Scaling in the LieBN

D LieBN as a Natural Generalization of Euclidean BN

E Domain-specific Momentum LieBN for EEG Classification

F Backpropagation of Matrix Functions

G Additional Details and Experiments of LieBN on SPD manifolds

H Preliminary Experiments on Rotation Matrices

I Proofs of the Lemmas and Theories in the Main Paper

7 CONCLUSIONS

This paper proposes a novel framework called LieBN, enabling batch normalization over Lie groups. Our LieBN can effectively normalize both the sample and population statistics. Besides, we generalize the existing Lie groups on SPD manifolds and showcase our framework on the parameterized Lie groups of SPD manifolds. Extensive experiments demonstrate the advantage of our LieBN.

\ There are several other types of Lie groups in machine learning, such as special Euclidean groups. As a future avenue, we shall extend our LieBN to other Lie groups.

ACKNOWLEDGMENTS

This work was partly supported by the MUR PNRR project FAIR (PE00000013) funded by the NextGenerationEU, by the EU Horizon project ELIAS (No. 101120237), and by a gift donation from Cisco. The authors also gratefully acknowledge financial support from the China Scholarship Council (CSC).

REFERENCES

Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache. Fast and simple computations on tensors with log-Euclidean metrics. PhD thesis, INRIA, 2005. URL https://doi.org/10.1007/11566465_15.

\ Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. URL https://arxiv.org/abs/1607.06450.

\ Frederic Barbaresco. Gaussian distributions on the space of symmetric positive definite matrices from souriaus gibbs state for siegel domains by coadjoint orbit and moment map. In Geometric Science of Information: 5th International Conference, GSI 2021, Paris, France, July 21–23, 2021, Proceedings 5, pp. 245–255. Springer, 2021. URL https://doi.org/10.1007/978-3-030-80209-7_28.

\ Gary Becigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. arXiv preprint arXiv:1810.00760, 2018. URL https://arxiv.org/abs/1810.00760.

\ Marcel Berger. A panoramic view of Riemannian geometry. Springer, 2003. URL https://doi.org/10.1007/978-3-642-18245-7.

\ Rajendra Bhatia. Matrix analysis, volume 169. Springer Science & Business Media, 2013.

\ Victoria Bloom, Dimitrios Makris, and Vasileios Argyriou. G3D: A gaming action dataset and real time action recognition evaluation framework. In 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 7–12. IEEE, 2012. URL https://doi.org/10.1109/CVPRW.2012.6239175.

\ Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond Euclidean data. IEEE Signal Processing Magazine, 34(4):18–42, 2017. URL https://doi.org/10.1109/MSP.2017.2693418.

\ Daniel Brooks. Deep Learning and Information Geometry for Time-Series Classification. PhD thesis, Sorbonne universit´e, 2020. URL https://theses.hal.science/tel-03984879.

\ Daniel Brooks, Olivier Schwander, Fr´ed´eric Barbaresco, Jean-Yves Schneider, and Matthieu Cord. A hermitian positive definite neural network for micro-doppler complex covariance processing. In 2019 International Radar Conference (RADAR), pp. 1–6. IEEE, 2019a. URL https://doi.org/10.1109/RADAR41533.2019.171277.

\ Daniel Brooks, Olivier Schwander, Fr´ed´eric Barbaresco, Jean-Yves Schneider, and Matthieu Cord. Riemannian batch normalization for SPD neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019b. URL https://arxiv.org/abs/1909.02414.

\ Daniel Brooks, Olivier Schwander, Frederic Barbaresco, Jean-Yves Schneider, and Matthieu Cord. Second-order networks in pytorch. In Geometric Science of Information: 4th International Conference, GSI 2019, Toulouse, France, August 27–29, 2019, Proceedings 4, pp. 751–758. Springer, 2019c. URL https://doi.org/10.1007/978-3-030-26980-7_78.

\ Daniel Brooks, Olivier Schwander, Frederic Barbaresco, Jean-Yves Schneider, and Matthieu Cord. Deep learning and information geometry for drone micro-doppler radar classification. In 2020 IEEE Radar Conference (RadarConf20), pp. 1–6. IEEE, 2020. URL https://doi.org/10.1109/RadarConf2043947.2020.9266689.

\ Daniel A Brooks, Olivier Schwander, Frederic Barbaresco, Jean-Yves Schneider, and Matthieu Cord. Complex-valued neural networks for fully-temporal micro-doppler classification. In 2019 20th International Radar Symposium (IRS), pp. 1–10. IEEE, 2019d. URL https://doi.org/10.23919/IRS.2019.8768161.

\ Daniel A Brooks, Olivier Schwander, Frederic Barbaresco, Jean-Yves Schneider, and Matthieu Cord. Exploring complex time-series representations for riemannian machine learning of radar data. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3672–3676. IEEE, 2019e. URL https://doi.org/10.1109/ICASSP.2019.8683056.

\ Rudrasis Chakraborty. ManifoldNorm: Extending normalizations on Riemannian manifolds. arXiv preprint arXiv:2003.13869, 2020. URL https://arxiv.org/abs/2003.13869.

\ Rudrasis Chakraborty and Baba C Vemuri. Statistics on the Stiefel manifold: theory and applications. The Annals of Statistics, 47(1):415–438, 2019. URL https://doi.org/10.1214/18-AOS1692.

\ Rudrasis Chakraborty, Chun-Hao Yang, Xingjian Zhen, Monami Banerjee, Derek Archer, David Vaillancourt, Vikas Singh, and Baba Vemuri. A statistical recurrent model on the manifold of symmetric positive definite matrices. Advances in Neural Information Processing Systems, 31, 2018. URL https://arxiv.org/abs/1805.11204.

\ Rudrasis Chakraborty, Jose Bouza, Jonathan Manton, and Baba C Vemuri. Manifoldnet: A deep neural network for manifold-valued data with applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. URL https://doi.org/10.1109/TPAMI.2020.3003846.

\ Kai-Xuan Chen, Jie-Yi Ren, Xiao-Jun Wu, and Josef Kittler. Covariance descriptors on a Gaussian manifold and their application to image set classification. Pattern Recognition, 107:107463, 2020. URL https://doi.org/10.1016/j.patcog.2020.107463.

\ Kaixuan Chen, Shunyu Liu, Tongtian Zhu, Ji Qiao, Yun Su, Yingjie Tian, Tongya Zheng, Haofei Zhang, Zunlei Feng, Jingwen Ye, et al. Improving expressivity of GNNs with subgraph-specific factor embedded normalization. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 237–249, 2023a. URL https://doi.org/10.1145/3580305.3599388.

\ Kaixuan Chen, Jie Song, Shunyu Liu, Na Yu, Zunlei Feng, Gengshi Han, and Mingli Song. Distribution knowledge embedding for graph pooling. IEEE Transactions on Knowledge and Data Engineering, 35(8):7898–7908, 2023b. URL https://doi.org/10.1109/TKDE.2022.3208063.

\ Ziheng Chen, Tianyang Xu, Xiao-Jun Wu, Rui Wang, and Josef Kittler. Hybrid Riemannian graphembedding metric learning for image set classification. IEEE Transactions on Big Data, 2021. URL https://doi.org/10.1109/TBDATA.2021.3113084.

\ Ziheng Chen, Yue Song, Gaowen Liu, Ramana Rao Kompella, Xiaojun Wu, and Nicu Sebe. Riemannian multiclass logistics regression for SPD neural networks. arXiv preprint arXiv:2305.11288, 2023c. URL https://arxiv.org/abs/2305.11288.

\ Ziheng Chen, Tianyang Xu, Zhiwu Huang, Yue Song, Xiao-Jun Wu, and Nicu Sebe. Adaptive Riemannian metrics on SPD manifolds. arXiv preprint arXiv:2303.15477, 2023d. URL https://arxiv.org/abs/2303.15477.

\ Ziheng Chen, Tianyang Xu, Xiao-Jun Wu, Rui Wang, Zhiwu Huang, and Josef Kittler. Riemannian local mechanism for SPD neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7104–7112, 2023e. URL https://doi.org/10.1609/aaai.v37i6.25867.

\ Manfredo Perdigao Do Carmo and J Flaherty Francis. Riemannian Geometry, volume 6. Springer, 1992. URL https://link.springer.com/book/9780817634902.

\ Octavian Ganea, Gary B´ecigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in Neural Information Processing Systems, 31, 2018. URL https://arxiv.org/abs/1805.09112.

\ Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 409–419, 2018. URL https://arxiv.org/abs/1704.02463.

\ Alexandre Gramfort. MEG and EEG data analysis with MNE-Python. Frontiers in Neuroscience, 7, 2013. URL https://doi.org/10.3389/fnins.2013.00267.

\ Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016. URL https://arxiv.org/abs/1512.03385.

\ Marcel F. Hinss, Ludovic Darmet, Bertille Somon, Emilie Jahanpour, Fabien Lotte, Simon Ladouce, and Rapha¨elle N. Roy. An EEG dataset for cross-session mental workload estimation: Passive BCI competition of the Neuroergonomics Conference 2021, 2021. URL https://doi.org/10.5281/zenodo.5055046.

\ Sepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735–1780, 1997. URL https://doi.org/10.1162/neco.1997.9.8.1735.

\ Zhiwu Huang and Luc Van Gool. A Riemannian network for SPD matrix learning. In Thirty-first AAAI Conference on Artificial Intelligence, 2017. URL https://arxiv.org/abs/1608.04233.

\ Zhiwu Huang, Chengde Wan, Thomas Probst, and Luc Van Gool. Deep learning on Lie groups for skeleton-based action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6099–6108, 2017. URL https://arxiv.org/abs/1612.05877.

\ Zhiwu Huang, Jiqing Wu, and Luc Van Gool. Building deep networks on Grassmann manifolds. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. URL https://doi.org/10.1609/aaai.v32i1.11725.

\ Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448–456. PMLR, 2015. URL https://proceedings.mlr.press/v37/ioffe15.html.

\ Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Matrix backpropagation for deep networks with structured layers. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2965–2973, 2015. URL https://arxiv.org/abs/1509.07838.

\ Vinay Jayaram and Alexandre Barachant. MOABB: trustworthy algorithm benchmarking for BCIs. Journal of Neural Engineering, 15(6):066011, 2018. URL https://doi.org/10.1088/1741-2552/aadea0.

\ Hermann Karcher. Riemannian center of mass and mollifier smoothing. Communications on Pure and Applied Mathematics, 30(5):509–541, 1977. URL https://doi.org/10.1002/cpa.3160300502.

\ Reinmar Kobler, Jun-ichiro Hirayama, Qibin Zhao, and Motoaki Kawanabe. SPD domainspecific batch normalization to crack interpretable unsupervised domain adaptation in EEG. Advances in Neural Information Processing Systems, 35:6219–6235, 2022a. URL https://arxiv.org/abs/2206.01323.

\ Reinmar J Kobler, Jun-ichiro Hirayama, and Motoaki Kawanabe. Controlling the Fr´echet variance improves batch normalization on the symmetric positive definite manifold. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3863–3867. IEEE, 2022b. URL https://doi.org/10.1109/ICASSP43922.2022.9746629.

\ Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012. URL https://doi.org//10.1145/3065386.

\ Zhenhua Lin. Riemannian geometry of symmetric positive definite matrices via Cholesky decomposition. SIAM Journal on Matrix Analysis and Applications, 40(4):1353–1370, 2019. URL https://arxiv.org/abs/1908.09326.

\ Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the Fr´echet mean. In International Conference on Machine Learning, pp. 6393–6403. PMLR, 2020. URL https://proceedings.mlr.press/v119/lou20a.html.

\ Jonathan H Manton. A globally convergent numerical algorithm for computing the centre of mass on compact lie groups. In ICARCV 2004 8th Control, Automation, Robotics and Vision Conference, 2004., volume 3, pp. 2211–2216. IEEE, 2004. URL https://doi.org/10.1109/ICARCV.2004.1469774.

\ Meinard M ¨uller, Tido R¨oder, Michael Clausen, Bernhard Eberhardt, Bj ¨orn Kr ¨uger, and Andreas Weber. Documentation mocap database HDM05. Technical report, Universitat Bonn, 2007. URL https://resources.mpi-inf.mpg.de/HDM05/.

\ Iain Murray. Differentiation of the Cholesky decomposition. arXiv preprint arXiv:1602.07527, 2016. URL https://arxiv.org/abs/1602.07527.

\ Richard M Murray, Zexiang Li, and S Shankar Sastry. A mathematical introduction to robotic manipulation. CRC press, 2017. URL https://doi.org/10.1201/9781315136370.

\ Xuan Son Nguyen. The Gyro-structure of some matrix manifolds. In Advances in Neural Information Processing Systems, volume 35, pp. 26618–26630, 2022a. URL https://openreview.net/forum?id=eyE9Fb2AvOT.

\ Xuan Son Nguyen. A Gyrovector space approach for symmetric positive semi-definite matrix learning. In Proceedings of the European Conference on Computer Vision, pp. 52–68, 2022b. URL https://doi.org/10.1007/978-3-031-19812-0_4.

\ Xuan Son Nguyen and Shuo Yang. Building neural networks on matrix manifolds: A Gyrovector space approach. arXiv preprint arXiv:2305.04560, 2023. URL https://arxiv.org/abs/2305.04560.

\ Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019. URL https://arxiv.org/abs/1912.01703.

\ Xavier Pennec. Probabilities and statistics on Riemannian manifolds: A geometric approach. PhD thesis, INRIA, 2004. URL https://hal.science/inria-00071490.

\ Xavier Pennec and Nicholas Ayache. Uniform distribution, distance and expectation problems for geometric features processing. Journal of Mathematical Imaging and Vision, 9:49–67, 1998. URL https://doi.org/10.1023/A:1008270110193.

\ Xavier Pennec, Pierre Fillard, and Nicholas Ayache. A Riemannian framework for tensor computing. International Journal of Computer Vision, 66(1):41–66, 2006. URL https://doi.org/10.1007/s11263-005-3222-z.

\ Salem Said, Lionel Bombrun, Yannick Berthoumieu, and Jonathan H Manton. Riemannian gaussian distributions on the space of symmetric positive definite matrices. IEEE Transactions on Information Theory, 63(4):2153–2170, 2017. URL https://doi.org/10.1109/TIT.2017.2653803.

\ Yann Thanwerdas and Xavier Pennec. Is affine-invariance well defined on SPD matrices? a principled continuum of metrics. In Geometric Science of Information: 4th International Conference, GSI 2019, Toulouse, France, August 27–29, 2019, Proceedings 4, pp. 502–510. Springer, 2019a. URL https://arxiv.org/abs/1906.01349.

\ Yann Thanwerdas and Xavier Pennec. Exploration of balanced metrics on symmetric positive definite matrices. In Geometric Science of Information: 4th International Conference, GSI 2019, Toulouse, France, August 27–29, 2019, Proceedings 4, pp. 484–493. Springer, 2019b. URL https://arxiv.org/abs/1909.03852.

\ Yann Thanwerdas and Xavier Pennec. The geometry of mixed-Euclidean metrics on symmetric positive definite matrices. Differential Geometry and its Applications, 81:101867, 2022a. URL https://arxiv.org/abs/2111.02990.

\ Yann Thanwerdas and Xavier Pennec. Theoretically and computationally convenient geometries on full-rank correlation matrices. SIAM Journal on Matrix Analysis and Applications, 43(4): 1851–1872, 2022b. URL https://arxiv.org/abs/2201.06282.

\ Yann Thanwerdas and Xavier Pennec. O (n)-invariant Riemannian metrics on SPD matrices. Linear Algebra and its Applications, 661:163–201, 2023. URL https://arxiv.org/abs/2109.05768.

\ Loring W.. Tu. An introduction to manifolds. Springer, 2011. URL https://doi.org/10.1007/978-1-4419-7400-6_3.

\ Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016. URL https://arxiv.org/abs/1607.08022.

\ Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://arxiv.org/abs/1706.03762.

\ Raviteja Vemulapalli, Felipe Arrate, and Rama Chellappa. Human action recognition by representing 3D skeletons as points in a Lie group. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 588–595, 2014. URL https://doi.org/10.1109/CVPR.2014.82.

\ Rui Wang, Xiao-Jun Wu, and Josef Kittler. SymNet: A simple symmetric positive definite manifold deep learning method for image set classification. IEEE Transactions on Neural Networks and Learning Systems, 33(5):2208–2222, 2021. URL https://doi.org/10.1109/tnnls.2020.3044176.

\ Rui Wang, Xiao-Jun Wu, Ziheng Chen, Tianyang Xu, and Josef Kittler. DreamNet: A deep Riemannian manifold network for SPD matrix learning. In Proceedings of the Asian Conference on Computer Vision, pp. 3241–3257, 2022a. URL https://arxiv.org/abs/2206.07967.

\ Rui Wang, Xiao-Jun Wu, Ziheng Chen, Tianyang Xu, and Josef Kittler. Learning a discriminative SPD manifold neural network for image set classification. Neural Networks, 151:94–110, 2022b. URL https://doi.org/10.1016/j.neunet.2022.03.012.

\ Rui Wang, Xiao-Jun Wu, Ziheng Chen, Cong Hu, and Josef Kittler. SPD manifold deep metric learning for image set classification. IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2024. URL https://doi.org/10.1109/TNNLS.2022.3216811.

\ Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19, 2018. URL https://arxiv.org/abs/1803.08494v3.

\ Or Yair, Mirela Ben-Chen, and Ronen Talmon. Parallel transport on the cone manifold of SPD matrices for domain adaptation. IEEE Transactions on Signal Processing, 67(7):1797–1811, 2019. URL https://doi.org/10.1109/TSP.2019.2894801.

\ Hongwei Yong, Jianqiang Huang, Deyu Meng, Xiansheng Hua, and Lei Zhang. Momentum batch normalization for deep learning with small batch size. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pp. 224– 240. Springer, 2020. URL https://doi.org/10.1007/978-3-030-58610-2_14.

\

:::info This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license.

:::

:::info Authors:

(1) Ziheng Chen, University of Trento;

(2) Yue Song, University of Trento and a Corresponding author;

(3) Yunmei Liu, University of Louisville;

(4) Nicu Sebe, University of Trento.

:::

\