Frank Yu

I am a second year M.Sc. student in Computer Science at the University of British Columbia (UBC), where I am supervised by Prof. Helge Rhodin. I am expecting to graduate in May 2023 and am currently looking for full-time positions!

Previously, I had the pleasure of interning with a wonderful group of people at Google (Project Starline). I also completed my B.Sc. in Electrical Engineering at the University of Manitoba, where I was also an undergraduate research assistant in Prof. Yang Wang's Lab.

Email / CV / Google Scholar / Github / LinkedIn / Twitter / Photography

News

New! June 2022: Started internship (Student Researcher) @ Google (Project Starline)
New! September 2022: Paper accepted to WACV 2023
New! September 2022: Demo accepted to SIGGRAPH Asia 2022 (XR)
New! June 2022: Started internship (Research Intern) @ Google (Project Starline)

September 2021: A-NeRF accepted to NeurIPS 2021 (Poster)
May 2021: TA'ing CPSC340: Machine Learning and Data Mining
March 2021: PCL Paper accepted to CVPR 2021 (Poster)
January 2021: A-NeRF paper available on arxiv
November 2020: I will be a student volunteer at NeurIPS 2020
September 2020: Starting my M.Sc in Computer Science at the University of British Columbia
August 2020: Paper accepted to ECCV 2020 (Spotlight)
June 2020: Graduated from the University of Manitoba with a B.Sc in Electrical Engineering

Research

My research interests focus on computer vision and neural rendering. My goal is to develop state-of-the-art technologies that seamlessly bend our world with the virtual world and vice versa.

New!Scaling Neural Face Synthesis to High FPS and Low Latency by Neural Caching
Frank Yu, Sid Fels, Helge Rhodin,
WACV 2023
Paper | Project Page

We present a novel method to reduce latency in neural rendering methods for telepresence by caching and implicitly warping neural network features between timesteps.

A-NeRF: Surface-free Human 3D Pose Refinement via Neural Rendering
Shih-Yang Su, Frank Yu, Michael Zollhoefer, Helge Rhodin,
NeurIPS 2021 (Poster)
Paper | Project Page

We present an analysis-by-synthesis approach for monocular motion capture that learns a volumetric body model and refines the 3D pose estimation of the user in a self-supervised manner.

PCLs: Geometry-aware Neural Reconstruction of 3D Pose with Perspective Crop Layers
Frank Yu, Mathieu Salzmann, Pascal Fua, Helge Rhodin,
CVPR 2021 (Poster)
Paper | Code

We propose PCL (perspective crop layer), a set of modular neural network layers that when inserted into MLPs or CNNs will deterministically remove location-dependent perspective effects leading to more precise 3D human pose estimation.

Few-shot Scene-adaptive Anomaly Detection
Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy, Yang Wang
ECCV 2020 (Spotlight)
Paper | Code

We propose a more realistic problem setting for anomaly detection in surveillance videos and solve it using a meta-learning based algorithm.

Demos

New!TeleViewDemo: Experience the Future of 3D Teleconferencing
Ziyi Xia, Frank Yu, Beibei Xiong, Emily Jia, Kaseya Zia, Seungyeon Baek, James Gregson, Xingzhe He, Helge Rhodin, Sid Fels,
SIGGRAPH Asia 2022 (XR)
Abstract (Coming Soon) | Project Page

We present an open-source telepresence platform capable of rendering on varioius view-dependent displays including, spherical, head-mounted VR, and flat screens.

Teaching

TA for CPSC340: Machine Learning and Data Mining [Summer 2021]

Led weekly tutorials and office hours. Assisted with assignment and final exam grading.

Credits to Jon Barron for the website design.