Research Highlights

Professor Seokju Kang's research team, has been accepted into the world's top AI conference with its AI model lightweight technology ‘NeurIPS 2026’ paper 2has been accepted

 

 

 

 

 

Professor Kangseokju's research team in the Department of Electronic Engineering at Sogang University Vision Transformerand Monocular Depth Estimation a paper on the topic of model lightweighting 2was accepted by the world's top artificial intelligence conference, ‘Neural Information Processing Systems, NeurIPS 2026’which was accepted. 1987starting from NeurIPSis the world's top international academic conference that has led the development of artificial intelligence and machine learning fields, until 12month 6from the 12month 13day to, will be held in Sydney, Australia, Atlanta, USA, and Paris, France.

 

This study involved Seokju Kang's research team, including Jincheol Yang, an integrated M.S.–Ph.D. program student and the first author, (the 1st author), Jaemin Choi, a master's program student and the first author, (the 1st author), Nahyun Lim, a master's program student and a co-first author, (a co-1st author), Yunsung Jung, a master's program student and a co-first author, (a co-1st author), Matti Zinke, a master's program student and a co-first author, (a co-1st author), and Hyunwoo Yu, an integrated M.S.–Ph.D. program student and a co-first author, (a co-1st author)participated in this study.

 

The title of the first paper is ‘Post-Training Quantization with Gradient-Projected Fisher Approximation for Vision Transformers’. It proposes GPFA-Q to reduce performance degradation that occurs when quantizing Vision Transformers to low bits.

 

This paper proposes two methods. The first method, Gradient-Projected Reconstruction (GPR), projects block-wise quantization errors in the direction of the loss function's gradient to reflect interactions between dimensions. This process does not involve directly constructing or inverting a curvature matrix. The second method is Soft Grid Rounding (SGR), which reduces the difference between continuous optimization and discrete quantization values used in actual inference. Combining these two methods, GPFA-Q improves low-bit quantization performance by reducing the dimension interaction and matrix inversion's numerical instability that were not fully addressed in previous studies, and the discrepancy between optimization and inference. This enables efficient operation of high-performance vision AI models on edge devices with limited memory and computational resources. This study contributes to the development of a lightweight technology that makes large AI models smaller and lighter without additional training, and is expected to contribute to the future proliferation of on-device AI.

 

Jincheol Yang, who participated in this research, said, “I am pleased to have my research achievements recognized, and I would like to thank my professor and colleagues who worked with me on this research. In the future, I will devote myself to lightweight research so that AI models can operate efficiently in various environments.” He expressed his feelings.

 

 ‣proposed GPFA-Q framework

 

The second paper title is ‘SPA-Q: Structure-Preserving Adaptive Post-Training Quantization for Monocular Depth Estimation’and, Monocular Depth Estimation model-specific quantization framework was proposed. The research team pointed out that existing quantization methods do not preserve the attention distribution created by the interaction between Query and Key during the quantization process, and the quantization error accumulates over multiple layers, changing the distribution of intermediate features.

 

This paper discusses Queryand Keyinteractions that result in a distribution closer to the original model's distribution attention by minimizing the difference to determine quantization parameters KL divergenceand proposes a method to minimize the difference to determine quantization parameters Attention-Preserving Calibration (APC)and proposes a method to minimize the difference. Additionally, it proposes applying a learnable channel-wise transformation to match the distribution of quantized intermediate features to the original model Channel-Wise Distribution Alignment (CWDA)and proposes a method to match the distribution of quantized intermediate features to the original model. The significance of this study lies in presenting a quantization framework that considers both the structure and feature distribution of the deep estimation model, going one step further from existing general quantization methods, of the deep estimation model's attention structure and feature distribution, and presents a quantization framework that considers both. Through this, it shows the need to consider not only minimizing simple numerical errors in model lightweighting, but also how important internal information for the model's prediction is maintained, and shows the need to consider how important internal information for the model's prediction is maintained. The proposed method SPA-Qis expected to contribute to efficiently applying the depth estimation model in fields such as autonomous driving and robotics, where understanding the surrounding environment's distance and structure is crucial, by increasing the accuracy of monocular depth estimation while reducing model size and computational complexity, in environments with limited memory or computational resources·such as robotics, where understanding the surrounding environment's distance and structure is important.

 

Jaemin Choi, who participated in this study, said, “I was able to achieve good results thanks to my lab colleagues who helped me think about the research direction and my professor who always trusted and supported me. Without being satisfied with this achievement, I will expand my research to larger and more complex AI models.”Jaemin Choi expressed his feelings.

 

  ‣Proposed SPA-Q framework

 

 

 ‣Qualitative comparison results of existing methodology and proposed methodology

 

‣Paper Information

1. Post-Training Quantization with Gradient-Projected Fisher Approximation for Vision Transformers

Jincheol Yang, Jaemin Choi, Nahyun Lim, Yunsung Jung, Matti Zinke, Hyunwoo Yu, Bongjun Hyun (Samsung Electronics), Gyomin Son (Samsung Electronics), Seokju Kang

2. SPA-Q: Structure-Preserving Adaptive Post-Training Quantization for Monocular Depth Estimation

Jaemin Choi, Jincheol Yang, Nahyun Lim, Yunsung Jung, Matti Zinke, Hyunwoo Yu, Seokju Kang

‣Other Information: This study is a result of the University ICT Research Center project by the Ministry of Science and ICT and the Institute for Information & Communications Technology Planning & Evaluation, and a result of the Next-Generation Intelligent Semiconductor Technology Development project.