Description of work
T1.1 – Generate rigged and skinned gray-scale avatars with body measurements (M5-M24)
The purpose of this task is to research and develop two methods that allow next generation pose detection on introduced scan data (images). Two approaches will be considered: either i) modifying the current matcher of QC to match the introduced poses, or ii) record captured poses and apply specific poses in post-processing to the generated model. Concerning the development track, QC’s current body measurement platform will be employed and be further improved with a wider variety of scanned poses, for the reconstruction of the 3D avatars. In generating realistic avatars, we will target automatic rigging; the current standard Skeleton that is used in animations consists of 65 parts. Also, soft tissue animation algorithms (Kim et al., 2017) will be examined to increase realism e.g. for fit and tightness of clothing in movement.
T1.2 – Increase the quality of the modelling of the head, hands, feet and the attractiveness of the avatars (M5-M24)
In this task, methods will be applied to smooth and improve the outcome of the current meshes and newly generated meshes from T1.1 by enhancing the quality of mesh topologies such as hands, feet and head. Such methods can include but are not limited to data labelling, mesh smoothing, hole filling and mesh replacements. To achieve the aforementioned task, automatic mesh modifiers from 3D sculpting software (e.g. Blender) will be used to optimize the visual fidelity of the model by smoothing edges and removing surfaces with noise.
T1.3 – Generate avatars from high resolution images (M5-M24)
Given an unposed avatar extracted in T1.1 and optimised in T1.2, the challenge in T1.3 is to predict the (3D) pose of the person in a given image, repose the avatar to the photographic pose and fine tune the muscle displacement and skinning so that the posed avatar exactly matches the image. Metail’s pose prediction framework is built on the Open Source DeeperCut deep learning package, with proprietary modules to implement multiview prediction and skeleton alignment. We will build on this and will create a new approach to map from the body landmarks predicted in those packages to the rigging model created in T1.1. To improve the fine matching between posed shape and photograph, we will use a combination of muscle deformation / displacement models, dual quaternion skinning and templated blend shapes. The output will be a posed 3D avata,r which can be used by either importing into VStitcher [VSTITCHER] or predicting garment fit.
T1.4 – Mobile application for self-3D-scanning (M5-M24)
In this task, we will build on QC’s scanning smartphone application and extend it by enabling self-scanning. Initially, we will evaluate the end-user needs for the UI and the interaction with the app, and how these needs translate into additional technical adaptations. Our overall plan is to collect trial data from end-users in a controlled environment to benchmark the following process steps: scanning process experience, segmentation quality, body creation quality (compared to manual measurement). Depending on the outcome, one or more of the following actions will be taken: i) UI changes will be made to provide better guidance and the built-in pose detector will be improved to accommodate the change in poses due to the self-scanning scenario; ii) a UI wizard will be developed to provide guidance through the scanning process; and iii) the utilization of the front camera will be adapted to the mobile application, including the implementation of different camera angles to the backend as well as an automatic countdown capture system.