Structure

Description of work

T1.1 – Generate rigged and skinned gray-scale avatars with body measurements (M5-M24)

The purpose of this task is to research and develop two methods that allow next generation pose detection on introduced scan data (images). Two approaches will be considered: either i) modifying the current matcher of QC to match the introduced poses, or ii) record captured poses and apply specific poses in post-processing to the generated model. Concerning the development track, QC’s current body measurement platform will be employed and be further improved with a wider variety of scanned poses, for the reconstruction of the 3D avatars. In generating realistic avatars, we will target automatic rigging; the current standard Skeleton that is used in animations consists of 65 parts. Also, soft tissue animation algorithms (Kim et al., 2017) will be examined to increase realism e.g. for fit and tightness of clothing in movement.

T1.2 – Increase the quality of the modelling of the head, hands, feet and the attractiveness of the avatars (M5-M24)

In this task, methods will be applied to smooth and improve the outcome of the current meshes and newly generated meshes from T1.1 by enhancing the quality of mesh topologies such as hands, feet and head. Such methods can include but are not limited to data labelling, mesh smoothing, hole filling and mesh replacements. To achieve the aforementioned task, automatic mesh modifiers from 3D sculpting software (e.g. Blender) will be used to optimize the visual fidelity of the model by smoothing edges and removing surfaces with noise.

T1.3 – Generate avatars from high resolution images (M5-M24)

Given an unposed avatar extracted in T1.1 and optimised in T1.2, the challenge in T1.3 is to predict the (3D) pose of the person in a given image, repose the avatar to the photographic pose and fine tune the muscle displacement and skinning so that the posed avatar exactly matches the image. Metail’s pose prediction framework is built on the Open Source DeeperCut deep learning package, with proprietary modules to implement multiview prediction and skeleton alignment. We will build on this and will create a new approach to map from the body landmarks predicted in those packages to the rigging model created in T1.1. To improve the fine matching between posed shape and photograph, we will use a combination of muscle deformation / displacement models, dual quaternion skinning and templated blend shapes. The output will be a posed 3D avata,r which can be used by either importing into VStitcher [VSTITCHER] or predicting garment fit.

T1.4 – Mobile application for self-3D-scanning (M5-M24)

In this task, we will build on QC’s scanning smartphone application and extend it by enabling self-scanning. Initially, we will evaluate the end-user needs for the UI and the interaction with the app, and how these needs translate into additional technical adaptations. Our overall plan is to collect trial data from end-users in a controlled environment to benchmark the following process steps: scanning process experience, segmentation quality, body creation quality (compared to manual measurement). Depending on the outcome, one or more of the following actions will be taken: i) UI changes will be made to provide better guidance and the built-in pose detector will be improved to accommodate the change in poses due to the self-scanning scenario; ii) a UI wizard will be developed to provide guidance through the scanning process; and iii) the utilization of the front camera will be adapted to the mobile application, including the implementation of different camera angles to the backend as well as an automatic countdown capture system.

Description of work

T2.1 – Generate the 3D garment collection (M5-M18)

In this task, the selection of the appropriate 3D garments of ODLO’s collection will be made. Overall, the selection will be based on the user requirements defined in T6.1 while simultaneously aiming at wide variety in terms of fabric properties. Moreover, in case this is necessary, modifications to the existing garments will be made in order to meet the user requirements and also clasp well with the virtual Models.

T2.2 – Simulate the avatar-garment interactions in professional software (M5-M18)

The work in this task will be two-fold. First of all, ODLO will manually simulate the garments on a smaller number of poses. There are some manual skills in using the Browzwear software [BROWZ] to pinch and drape the garment, which simulates how the garment would look in a real photoshoot. This output would be used in use cases 1 (fashion designer app) and 2 (social app). The second part of this task will involve Metail creating a way to feed different shapes and poses into Browzwear for automatic simulation to produce ground truth data for T2.4.

T2.3 – Automatically simulate garment-avatar interactions for videos (M5-M24)

In this task, Metail will research video segmentation approaches which will separate a short video into several frames. These frames will be used as input to generate an avatar with a series of poses matching each of these frames. This avatar will be then imported to Browzwear for garment simulation. Once the frames have been dressed and styled in Browzwear, the frames will be recombined to produce the same video with the person wearing digital garments.

T2.4 – Create a machine learning model to predict a game engine simulation approximation of the physics model (M5-M24)

The research in this area is quite evolved and there are a few possible approaches which will have to be investigated. In principle, Metail will start with frames of video of digital garments exported as OBJs and will output a model which for a given pose will be able to generate how the garment will drape. The approach will either by based on a 4D-based ML approach similar to Skinned Multi-Person Linear Model (SMPL) (Loper et al., 2015) but based on Metail proprietary technology, or by rigging and posing the garments and using this input combined with the ground truth data to model the output garment shape.

Description of work

T3.1 – Fashion emergent trend detection algorithms (M5-M24)

We will carry out object detection work training models on MLZ’s own set of fashion products imagery, with a goal of high precision in detection quality. After this sub-task is completed, we will apply time-series forecasting methods on MLZ’s own dataset of user preferences, alongside with the visual content detected. This will allow for the forecasting of patterns. Moreover, on the new digital imagery provided by WP2 (after the simulation of new digital garments on 3D avatars), we will be able to predict the success of the garments and test them with real customers to collect their opinions utilizing MLZ’s application.

T3.2 – User profiling and segmentation: (M5-M24)

We will work on the machine learning task of clustering consumers into groups by their fashion preferences. This will subsequently allow the insights generated on new products, such as those provided by Metail, to be segmented at the fashion group level.

T3.3 – Garment recommendations for the consumers: (M5-M24)

We will improve on MLZ’s recommendation algorithm by including the results of task T3.1 (the visual clues extracted from fashion imagery). This will then enable new features in the existing algorithm, which recommend garments to users based on the preferences they expressed on patterns, and subsequently recommend outfit matches which can determine what items go well together.

Description of work

T4.1 – System architecture and integration protocol design (M1-M8)

The purpose of this activity is to provide direction and specifications for consortium partners who will develop software components through the R&D and pilot phases of the project. It consists of defining the components, including their functional and non-functional requirements and the interfaces between them. These will be described as a series of related diagrams in the C4 model, and interfaces using the Swagger specification for APIs. The development of these specifications will involve a high level of collaboration between consortium partners to assess point-to-point validity. Integration validity will be assessed by describing how the components will collaborate for each of the pilot use cases.

T4.2 – eTryOn middleware and component integration (M9-M18)

This covers the development and deployment of server-targeted components to fulfil the specifications described in D4.1. These components will be developed to run in the AWS platform as serverless components (using AWS Lambda) for preference, or containerised services if this is not possible. Software development will use an agile methodology that includes regular, timeboxed releases, giving us a regular measure of progress, and we will use vertical slice development to uncover complexities early, and spike investigations to further reduce risk.

T4.3 – APIs for back-end and front-end applications (M9-M18)

This covers both the software development of service components that enable access to the business logic components developed in T4.2 and also the hosting and support of the assembled services for the rest of the consortium in R&D and the pilots. The software development parts will run in the same way as described in T4.2, with the addition of configurations for regional distribution of the services, load balancing, autoscaling and caching. We will monitor the performance of the services and components, and target and resolve server and network latency issues as they arise.

Description of work

T5.1 – Enhancing the creative experience of the fashion designers through a VR application for immersive garment fitting (M5-M24): The purpose of this task is the development of a VR application that will aid fashion designers to better design clothes through a virtual reality application. More specifically, a user-friendly VR interface will be developed using state of the art game engines (Unity, Unreal engine) that will enable designers to get a better idea of their designs, by providing them with realistic simulations on how their garments fit on models/mannequins, based on the garment fitting algorithms developed in T2.2 and realistic 3D models from T2.1, T1.3. Moreover, with the advancements in the field of human-like animations that modern game engines provide, designers will be able to edit the movement of the model animations and capture their own unique virtual fashion runaways. To improve the visual fidelity and user immersion, state of the art virtual reality sdks (e.g. Oculus rift, HTC Vive) will be utilized for achieving the creation of the aforementioned application.

T5.2 – Enhancing the social experience of the fashion lovers through a mobile application for use in social media (M5-M24): With this task, we aim to develop an application that will allow users (fashion consumers, models, influencers and lovers in general) to upload a short video including some simple movement and the application will return to them the video with the digital garment simulated on it in order to upload to social media (e.g.  instagram). For the development track, the mobile interface and backend architecture system will be designed with current state of the art game engines (Unity, Unreal) handling a variety of tools provided for mobile application development and physics simulations. Moreover, the utilization of the developments from WP2 and WP3 namely, garment simulation for videos and garment recommendations for consumers will serve as leading components in the backend architecture of the application.

T5.3 – Enhancing the shopping experience of the fashion consumers through an AR application for virtually trying on clothes (M5-M24): This activity aims to adopt the latest advancements of augmented reality for improving the overall experience of virtual garment fitting. The users of the AR application will be able to scan themselves or friends using their phone’s camera and observe in real-time through the screen of their mobile device how a particular garment fits their body. To achieve this result, AR APIs such as AR foundation offered from Unity technologies will be the leading tool for the development both of the interface and backend architecture of the application. Moreover, to further improve the virtual try on experience, the application will use as input the selected garments that the user inspected as well as his/her body measurements and recommend a variety of garments that fit the consumer both in terms of comfort as well as aesthetics based on the recommendation system developed in WP3.

Description of work

T6.1 – Define the user-based system requirements (M1-M4)

T6.1 is concerned with the collection of end-user requirements that will define the functionalities of the eTryOn technologies as well as with the definition of the experience scenario for each use case. More specifically, using a number of different instruments such as surveys/questionnaires, interviews and focus groups, we will capture the current situation with regard to challenges and difficulties in the way people experience fashion. The collected input will then be analysed to define a list of requirements, including both general requirements concerning the eTryOn technical functionalities (e.g. 3D scanning for clothes and human bodies, communication with eTryOn system for adding or removing assets, etc.) but also requirements focusing on the specific use case (e.g. VR experiences, AR experiences).

T6.2 – Pilot design, recruitment and evaluation plan (M5-M18)

This task will focus on developing a plan for testing the eTryOn platform as a whole, based on the user based requirements identified in T6.1, during the pilot phase that will be implemented as part of T6.3. The target audience will be identified as fashion industry professionals (e.g. designers, retailers) and end-users (e.g. online customers, fashion influences, models, lovers, etc.). We will identify a set of effective strategies/tools for reaching out to them, mainly through use case partners ODLO and MLZ. Subsequently, a schedule for the pilot operation as well as equipment requirements will be defined for each pilot test. Finally, the testing and evaluation procedures, both attended (i.e. supervised by project personnel) and unattended (mobile apps used by users at home) will be specified, including the methodology/protocol, the evaluation criteria/parameters, the tools/instruments (questionnaires, interviews, online automatic collection mechanisms) for user feedback collection and the training material. Pilot plans will comply with ethics principles from T8.4 and will be used to guide testing and evaluation activities in T6.3.

T6.3 – Pilot execution and evaluation of the system by end-users (M19-M24)

This task will involve the deployment and evaluation of the VR/AR and mobile applications generated by WP6 for the selected pilot cases (cf. Section 1.3.2). Based on the pilot plan of T6.2, this task will execute all the necessary activities regarding ethical approvals, user recruitment, recruitment events organization (Sec. 2.2.1.2), etc. Based on the evaluation plan set in T6.2, the effectiveness of the eTryOn technologies will be tested and evaluated.

Description of work

T7.1 – Project communication and dissemination activities (M1-M24)

T8.1 includes two main action pillars in order to maximize awareness for eTryOn and promote its outcomes. First, the project communication kit will be established consisting of the project website, social media accounts and a set of printed material (leaflets, factsheet, poster) for distribution at key academic and public events. The project’s website will be hosted beyond project duration, to promote exploitation of project’s results and serve as a hub and reference point for subsequent projects. The second pillar includes the dissemination plan and the reporting of its activities. A template for collecting inputs from dissemination activities will be also set up, documenting the important characteristics of the intended dissemination activity (cf. Section 2.2.1). The filled-in templates will be subsequently summarized into tables, composing a list of concrete dissemination actions that will allow us to assess, follow up and eventually correct the planned actions in terms of the reached measured impacts. Moreover, this task will deal with eTryOn’s participation, engagement and contribution to the exchange platforms developed in the frame of the CSA on Interactive Technologies funded under H2020 ICT-25-2018 eXtended Reality for All (XR4All).

T7.2 – Consumer engagement strategy and marketing activities (M1-M24)

T6.2 includes marketing activities targeting the consumers with the scope of maximizing the potential adoption of the eTryOn technologies as well run a marketing campaign to recruit people for the eTryOn pilots. In this direction, MLZ will design these activities and along with ODLO will run the marketing campaigns through MLZ’s application as well as their social media and online stores targeting their customers. For example, digital online experiences will be designed based on MLZ’s experience on digital marketing and user acquisition.

T7.3 – Exploitation and business plan creation (M1-M24)

The objective of T7.3 is to establish a detailed and efficient business plan that will ensure the commercialization and sustainability of the eTryOn technologies and interactive systems during and beyond the project lifetime. The first step is to assemble a detailed competitive landscape analysis and market analysis, which will ensure and maximize the innovation capacity of the eTryOn technologies. The second step is to establish a detailed business plan, including the exploitable assets, market segmentation, detailed marketing strategies, pricing estimations and sales projections for the exploitation of the project results. At this stage, the business plan will also include econometric models based on the refined competitiveness landscape analysis. This will enable us to establish precise market penetration rates and define the pricing plan, sales forecasts and sustainability analysis for the eTryOn systems that are closer to reach the market. Finally, in order to ensure fast and successful adoption of all the eTryOn exploitable assets regardless of time-to-market, this task will include guidelines in the form of a roadmap to commercialization.

Description of work

T8.1 – Administrative and financial management (M1-M24)

With respect to administrative management, the responsibility of T8.1 is to establish a stable management basis and ensure a firm foundation to the project. This task includes building upon the terms and conditions specified in the Consortium Agreement to define and set up the practical mechanisms implementing the management functions. These include communication between the Steering Board and eTryOn partners, EC officials and external organizations and individuals. The administrative management entails also the organization of the consortium and technical meetings, and meetings with the advisory board. The financial management entails the establishment of a plan ensuring that the expected income, expenditure, and disbursements are clear and verifiable. It also includes reviewing the financial status, report on progress to the Steering Board at quarterly meetings, verifying that audits are carried out for all partners and that audit certificates are received.

T8.2 – Technical management and quality assurance (M1-M24)

The technical management aims at monitoring and establishing the specific form and content of each deliverable, resolving what activities are necessary for their creation and delivery. The technical management includes also the monitoring and control of the activities of the Project Management Board (PMB). The goal of T8.2 is to ensure that all WPs stay on course and react to unexpected events. This process forms the core effort of the Project’s Technical Manager (PTM) since it is the process that handles day-to-day project issues. Throughout the project there will be a cycle consisting of authorizing work to be done, gathering progress information about that work, watching for changes, reviewing the situation, reporting and taking any necessary corrective action. The role of T8.2 is to cover these activities, together with the on-going work of risk analysis and contingency planning, and reporting to the PMB. The quality management of the project will be implemented through a Quality Assurance Plan (QAP) that will specify all quality control procedures, including responsibilities of project participants (PTM, WP Leaders, Task Leaders and other partners), contract management, documentation control, documentation formats and exchange rules, deliverable assessment, etc.

T8.3 – Management of knowledge and intellectual properties (M1-M24)

This task deals with the detection and assessment of the knowledge generated by the consortium. This knowledge will be initially identified at the beginning of the project and will be constantly monitored throughout the project’s lifetime based on an IPR plan. This plan, as described in more detail in Section 2.2.3, shows how all WPs will be evaluated for their expected patent applications and software copyrights. This task will also make sure that the necessary steps are taken for the protection of this intellectual property.

T8.4 – Ethical, legal and privacy requirements and guidelines for implementation (M1-M24)

This task deals with all the ethical, legal and privacy requirements of eTryOn. The Ethics & Privacy Coordinator (EPC) will create an internal ethical board including both technical members from the consortium as well as legal advisors from the legal departments of the eTryOn pilot partners. This board will supervise all data-related activities for the whole duration of the project to ensure that the project activities are in-line with the EU regulations in terms of ethical, legal and privacy practices. This board will set the guidelines and supervise their implementation for the data collection and ownership processes during the project pilots and the data collection during the research activities. Finally, this task will be responsible for getting ethical approvals for all data collection processes (e.g. from designated ethical committees).