Hongxu (Danny) Yin

I am a Senior.II (Staff) Research Scientist at NVIDIA Research, with the learning and perception research team led by Jan Kautz. I did my Ph.D. at Princeton University, where I was advised by Prof. Niraj Jha.

I am a recipient of Princeton Yan Huo 94* Graduate Fellowship, Princeton ECE Best Dissertation Finalist, Princeton Natural Sciences and Engineering Fellowship, Defense Science & Technology Agency gold medal, and Thomson Asia Pacific Holdings gold medal.

I have also been featured by Forbes as Top 60 Elite Chinese in North America and 36Kr with Global Outstanding Chinese Power 100 Award.

Google Scholar  /  Twitter /  CV  

I'm interested in a deeper understanding of neural nets.

This makes neural net efficient and secure. Topics cover resource efficient (data/computation) deep learning, overseeing CNNs and transformers, leveraging model inversion, knowledge distillation, dynamic inference, neural architecture search, pruning, quantization, model adaptation, etc.

I aim for fast and reliable networks for multi-modality tasks, autonomous driving, large language models, e-commerce, smart healthcare, among others.

We always welcome great research interns. Reach out to us if interested.

Dec. 2023 - I have research intern slots for VLM next spring/summer/fall 2024, and please drop a CV if interested.

[2024 Sep] Two papers accepted at NeurIPS 2024 on LLM and VLM, of which one as spotlight.

[2024 May] Three papers accepted at ICML 2024 on efficient and secure LLMs, of which two as oral (Top 1.5%).

[2024 Apr] We will host the Efficient Foundation Model workshop at ECCV 2024. See you in Milan.

[2024 Mar] FedBPT won the best paper award at AAAI symposium. Congrats to Jingwei!

[2024 Feb] Two papers accepted at CVPR 2024 on vision-language models.

[2024 Feb] We will host the Efficient Computer Vision workshop and GPU-based DL Acceleration tutorial at CVPR 2024. See you in Vancouver.

[2024 Jan] Two papers accepted at ICLR 2024 on efficient and robust large models.

Full list here. * equal contribution. ^ our great research interns at NVIDIA Research.

We introduce X-VILA, a foundation model for cross-modality understanding, reasoning, and generation in the domains of video, image, language, and audio. X-VILA demonstrates the ability to perceive (see, hear, and read) multi-modality inputs, and generate (draw, speak, and write) multi-modality responses.

We observe three rounds of free-lunch for VLM boosting, followed by a novel specialist augmentation mechanism.

DoRA consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding.

With an enhanced pre-training recipe we build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells and whistles.

We introduce RegionGPT that enables complex region-level captioning, reasoning, classification, and expression comprehension capabilities for the multimodal large language model.

Diffusion model for Plug-and-Play controllable generation.

Continual learning is now enabled for evolving model architecture upgrades. Your current model and new data are all you need to swap into a stronger architecture.

Transformers are vastly redundant, and one-click global structural pruning offers speed up right away. Stacking same block across depth is lazy, and a new distribution rule is proposed based on embedding only.

Deep equilibrium models smooth out flickering effect in video, enabled through recurrence without temporal-aware training, yielding efficient early stopping.

We show viability to train an invert network that maps intermediate tensors back to inputs. Works smoothingly for GANs and classifiers on high resolution tasks.

Sturcutral pruning is a quick knapsack problem to maximize accuracy through combining latency-guided parameter chunks.

We show that transformers can quickly drop redundant tokens and reserve computation on only informative ones, offering off-the-shelf cost saving.

We show Vision Transformer gradient encode sufficient information such that private original images can be easily reconstructed via inversion.

We push structural pruning into earlier training, cutting down on training costs.

We argue for a Train-Large-Swap-Faster model acceleration paradigm. Quickly adapting a large model to varying constraints in CPU-second search yields the quick finding of Pareto front.

We show under strong inversion, gradients are in essence original data via inversion, even for large datasets, large nets, for high resolution.

We aim at code-book oriented best scaled optimal quantization for deep nets.

We show that trained deep nets are in essence datasets. One can quickly invert from net outputs to synthesize a new dataset for off-the-shelf models. See ResNet-50 dreamed objects as left.

A genetic algorithm to quickly adjust the hyper architecture of a base model to target platforms (DSP, CPU, GPU) given constraints such as latency or memory.

Wearable medical sensors, backed by extremely efficient NNs, offers around-the-clock diabetes diagnosis.

We show that grow-and-prune yields faster yet more accurate H-LSTM family, surpassing LSTMs and GRUs.

We observe high degree of non-monoticity in latency surface given shrinking model dimensions, and propose a systematic structural Grow-and-Prune way to exlpoit this for faster inference.

Human brains grow before age 2 before pruning neurons for efficient synapsis afterwards. We propose grow-and-prune accordingly, and show consistent improvements over conventional pruning that always start from full models.

We aim to lay out foundations for pervasive healthcare from a wearables angle. Book now available at Amazon.

An FPGA solution augmented by a novel real-time tri-core SVD design for multi-coset signal reconstruction.

Partial list here.