Title: Style transfer as a way of improving the realism of synthetic eye images while preserving data labels
Presenter: Dmytro Katrychuk
Advisor: Dr. Oleg Komogortsev
Date/Time: Thursday, December 2 @3:30 – 4:00 p.m.
Location: ZOOM: https://txstate.zoom.us/j/91243653692
Eye-tracking (ET) is a promising tool to greatly enhance the Virtual/Augmented Reality experience by enabling gaze-contingent interaction, eye-movement biometrics, life-like avatars for remote conferencing, etc. The existing ET solutions are not viable to provide signal quality for these use-cases within a low-power package. Instead of focusing on specific hardware implementation, we propose to develop a generic software framework for prototyping novel appearance-based gaze estimation methods. To be a reliable source of signal for various real-world applications, such a method usually requires a training set that reflects the expected variability in gaze and camera positions, subject appearance, ambient lighting, etc. To overcome the issue of the associated data collection being infeasible on a large scale, a synthetic source of eye images, such as a 3D rendering tool, can be utilized to expand on the real data with sparse availability. While preserving ground-truth labels in the simulated data by design, the lack of photorealism was proven to be a limiting factor in using resulting renders for data augmentation.
We pose the problem of improving the realism of synthetic eye images as a style transfer. In this study, we focus on enhancing the lack of variability in camera position in the data collected from real subjects via imposing their appearance on the Blender renders, where the camera position can be freely controlled. The prior work based on training with perceptual losses serves as a baseline for a subject-specific model. We propose few modifications to a Generative Adversarial Network with cycle-consistency (CycleGAN) to achieve a multi-subject style transfer within a single model. The assessment of photorealism and the accuracy of data labels in the generated data is presented.
Deadline: Dec. 3, 2021, midnight