Sangyun Lee

I’m Sangyun Lee (pronounced “Sang-Yoon”), a fourth-year Ph.D. student in Electrical and Computer Engineering at Carnegie Mellon University, advised by Giulia Fanti. Previously, I was a research intern at Microsoft Research, NVIDIA, NAVER AI Lab, Kakao Enterprise, and SI Analytics. I earned my Bachelor’s degree in Computer Science from Soongsil University in South Korea.

Contact: sangyunl@andrew.cmu.edu

Research Interest

I work on developing visual and digital intelligence. For visual intelligence, I work on understanding and improving generative models that synthesize realistic visual data [blur diffusion, curvature minimization, improved rectified flow, truncated consistency models]. For digital intelligence, I study better learning algorithms for training language models [BaNEL, LLMs need sleep].

News

Publications

Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference

Sangyun Lee, Sean McLeish, Tom Goldstein, Giulia Fanti

arxiv preprint

Abstract
Summary

Many have hypothesized that the remarkable learning ability of human brains has something to do with sleep. Can LLMs benefit from sleep, too? Our method is derived from three hypotheses: (1) the brain is just a gigantic recurrent network; (2) it updates its synapses during forward passes via local learning rules; and (3) sleep is simply a period during which forward passes and learning continue in the absence of input. Our method is the simplest possible instantiation satisfying these hypotheses: adding offline recurrent forward passes to state-space language models. The trained model uses this recurrence to learn good representations of the past by updating its fast weights, leading to improved performance after sleep.

BaNEL: Exploration Posteriors for Generative Modeling Using Only Negative Rewards

Sangyun Lee, Brandon Amos, Giulia Fanti

arxiv preprint

Abstract Blog
Summary

LLM RL works because the base model can already generate good outputs occasionally. However, when the problem is very hard and very far from the pretraining dataset, that is no longer the case (think about proving the Riemann hypothesis, for instance). What should we do when the base model obtains no positive reward? Our idea is that we can still learn from those failed attempts by learning a generative model of the negative samples and then using it to update the model's posterior distribution to avoid similar failures in the future.

Truncated Consistency Models

Sangyun Lee, Yilun Xu, Tomas Geffner, Giulia Fanti, Karsten Kreis, Arash Vahdat, Weili Nie

ICLR 2025

Project Page Abstract Code
Summary

This paper aims to improve the one-step generation quality of consistency models. We observe that, in consistency models, two conflicting objectives—denoising (t → 0 mapping) and generation (T → 0 mapping)—compete for model capacity. This is especially problematic because CM is a trajectory-based distillation method that already requires a much larger model to match the quality of other methods. To resolve this, we propose a method for specializing CM for generation while freeing its capacity from denoising. At the time of release and at the scale we considered, the resulting model was the state of the art among trajectory-based models.

Improving the Training of Rectified Flows

Sangyun Lee, Zinan Lin, Giulia Fanti

NeurIPS 2024

Abstract Code
Summary

Rectified flows can learn less curved generative trajectories than diffusion models by going through many "Reflow" training stages. This paper argues that only one Reflow stage should be enough to obtain near-straight trajectories and proposes several techniques for achieving good one-step generative performance with only one Reflow stage. Some of the techniques proposed here have been adopted in Step-Video-T2V's turbo model.

Sequential Data Generation with Groupwise Diffusion Process

Sangyun Lee, Gayoung Lee, Hyunsu Kim, Junho Kim, Youngjung Uh

arxiv preprint, also appeared at ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling

Abstract
Summary

Diffusion models vs. autoregressive models: Are they really different? This paper unifies the two by generalizing diffusion to be able to generate each part of data sequentially. This allows, for example, each patch or pixel of an image to be generated sequentially in any order, making autoregressive models a special case. We extend this to a frequency domain, where diffusion autoregressively generates spectral components from low to high frequencies, yielding a hierarchical, disentangled latent space.

Minimizing Trajectory Curvature of ODE-based Generative Models

Sangyun Lee, Beomsu Kim, Jong Chul Ye

ICML 2023

Abstract Code
Summary

Why should sampling from diffusion/flow models be iterative? This is because their generative trajectories are highly curved. This paper proposes a method for training low-curvature flow models by learning a neural coupling between data and noise that minimizes intersections.

Progressive Deblurring of Diffusion Models for Coarse-to-Fine Image Synthesis

Sangyun Lee, Hyungjin Chung, Jaehyeon Kim, Jong Chul Ye

NeurIPS 2022 Workshop on Score-Based Methods

Abstract Code
Summary

Diffusion models generate data through iterative denoising. But is that the only way, or can we generate data by inverting any signal-corruption process? This paper is one of the first to show that deblurring can be used for image generation. This is done by generalizing forward and reverse SDEs to different frequency domains.

High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions

Sangyun Lee*, Gyojung Gu*, Sunghyun Park, Seunghwan Choi, Jaegul Choo

ECCV 2022

Abstract Code

Learning Multiple Probabilistic Degradation Generators for Unsupervised Real World Image Super Resolution

Sangyun Lee, Sewoong Ahn, Kwangjin Yoon

ECCV 2022 Workshop on Learning from Limited and Imperfect Data

Abstract

(* denotes equal contributions.)

Scholarships

  • Bob Lee Gregory Fellowship for the 2024-2025 academic year.
  • ECE Department Recognition Award for Exemplary Qualifying Exam Performance, Spring 2025. Recognized by CMU ECE faculty for exemplary Ph.D. qualifying examination performance. This distinction was awarded by faculty vote to select students within the top 10% of Ph.D. student examinees during the Spring 2025 academic semester.

Talk

  • Jul 2026, Google DeepMind, “Do Language Models Need Sleep?” Slides
  • Jun 2026; FAIR, Meta Superintelligence Labs, Paris, “Do Language Models Need Sleep?” Slides
  • Mar 2025; Sewoong Oh’s group @ University of Washington, “Truncated Consistency Models”
  • Mar 2025; Stability AI, “Truncated Consistency Models”
  • Nov 2024; BioImaging, Signal Processing & Learning Lab @ KAIST, “Improving the Training of Rectified Flows”
  • Nov 2022 - Dec 2022; A three-week series of talks at Modulabs

Patent

Sangyun Lee and Kwangjin Yoon, “Super Resolution Imaging Method Using Collaborative Learning.” Korean Patent 1024062870000, filed Dec 31, 2021, and issued June 2, 2022.