Rishi Upadhyay

My research interests are in studying and improving the physical understanding of vision diffusion models. I also work on applying frontier vision techniques and physics-informed neural networks to scientific applications in medicine and battery management.

I am honored to be awarded the 2025 Amazon Fellowship and the 2026 UCLA Jonsson Comprehensive Cancer Center Fellowship.

I attended UC Berkeley for my undergrad where I was fortunate to work with Prof. Ren Ng and Prof. Avideh Zakhor.

Email  /  Google Scholar  /  CV  /  Github

I am on the job market for industry/postdoc research roles starting Spring 2027, focused on video-based world models, AI for Science, and world models for robotics.
Please reach out if you have any relevant opportunities.

Representative/Recent papers are highlighted

We evaluate how closely world models align with the physical world using a set of synthetic videos which test different physics concepts.

We train a machine learning model to estimate internal electrode potentials from standard battery operating data, preventing lithium plating and enabling safer fast-charging. Personally led the development of ML model from data to deployment.

Predicting pixel-aligned 3D bounding boxes from a single image without requiring camera intrinsics by using dense corner heatmaps and depth maps.

We develop VAE and Diffusion based generative models that model breathing motion in lung CT scans for better radiation targeting.

We integrate 3D Gaussian Splatting with depth regularization and diffusion priors to enable 360° novel view synthesis from just 12 views.

We introduce the first paired semantic segmentation dataset in adverse conditions and show that techniques that leverage this paired nature can significantly improve performance.

We propose a loss function for latent diffusion models that improves the perspective accuracy of generated images, allowing us to create synthetic data that helps improve SOTA monocular depth estimation models.

We characterize cone classification by ORG. Personally worked on the Retina Map Alignment Tool.

We create an automated tool to estimate building energy efficiency through drone and thermal imaging.

We use a smartphone, AR techniques, and object classifiers to create a system to automatically detect and map objects of interest in a 3D scene.

• Developed 3D native object-centric world model for multi-object dynamics prediction from a single image. Model disentangles simulation and rendering, allowing for fast and accurate simulation through a lightweight physics-conditioned transformer while maintaining high quality and diverse output videos through pre-trained diffusion models.
• Demonstrated that model architecture allows it to accurately model physical laws and can provide benefits for downstream robotics tasks through both synthetic data generation and in-the-loop planning on PushT and LanguageTable.
• Ported SoTA World Action Models (e.g. Cosmos 3, DreamZero, X-WAM) to AMD hardware to enable both inference and training.

• Developed end-to-end differentiable pipeline that combined deep learning and geometric eye tracking algorithms by creating a module to back-project an estimated 2D pupil ellipse into a 3D pupil.
• Demonstrated that the new pipeline was performant as a low-cost eye tracker and could be plugged into existing deep learning pipelines to improve performance across a variety of tasks.

Built a custom object detection pipeline designed to run in under 5 seconds on ~40 megapixel images.

Developed a tool to transfer vector fields such as force or pressure between 3D models with different shape, size, or resolution, including a fast 3D mesh parser.

Credit to Jon Barron for the website template.