Digital mask

The digital transformation of traditional art

Digital Mask utilizes AR filters to transform traditional theatrical mask art into digital art. Transform these on-stage masks into digital masks that everyone can wear. I'm thrilled to use multiple techniques to convey traditional mask art more broadly.

Time

2022

Project Type

AR filter
generated art

Skills&Tools

expression recognition
Lens Studio, Adobe PS, AI

28_edited.jpg

Traditional opera art is fading from the public's view.

Beijing Opera is a synthesis of stylized action, singing, dialogue, mime, acrobatic fighting and dancing to represent a story or depict different characters and their feelings of gladness, anger, sorrow, happiness, surprise, fear and sadness. The characters may be loyal or treacherous, beautiful or ugly, good or bad. Their images are always vividly manifested in bright costumes that show the styles of ancient China.

The masks of Beijing opera incorporates the personality, role and background of the characters. Different mask styles and colors represent different historical figures.

How can design bring traditional art into digital era?

Customized digital AR masks

In the Peking Opera makeup, artists use exaggerated and abstract lines to express the characters’ face, use different colors to express the personality of the characters.

Digital masks digitize styles and colors of Peking Opera mask and use expression recognition to customize each mask.

资源 30.png

Scan in snapchat

29.png

Design process

31.png

Research

Understand the essential of opera mask

Ideate

Generate ideas
weigh main elements

Technical demonstration

Ensure technical path
choose the right tools

Design

Iterate

Prototype design

Test the prototype
improve the performance

The relationship between color and personality in Beijing opera masks.

In Peking Opera masks, people use exaggerated and abstract lines to express the characters’face,  use different colors to express the personality of the characters. Different color schemes help viewers understand thepersonality of the roles. For example, red often represents loyalty, blue stands for straight and unruly.

Styles + Color shifting

45.png

Link emotions with mask colors.

expression recognition supported by deep learning network

Color can affect mood, and emotion can be expressed in different colors. 
In order to preserve the relationship of color and personality in the digital mask. I use emotions to replace personality. Using facial expression recognition technology, emotions can be quickly identified. This can serve as the basis for the real time changing of digitized masks.

34.png

A simple neural network can distinguish A from B.

33.png

A more complex neural network can further distinguish same color shape into circle and square.

Facial expression recognition.

The principle of deep learning networks is similar to neurons in the human brain. It can greatly improve the intelligence of the computer and make it possible to deal with problems that was previously only humans could do, for example, art creation.

35.png

The neural network I use, called convolutional neural networks, can classify and identify different emotions.

36.png

Successfully recognized the expression in the picture

Design digital opera masks as a snapchat fliter

I want people to experience traditional mask art in a fun and easy way. Snapchat has powerful filters in it. I use Lens Studio to create a filter and publish it in snapcht. 
Through this snap fliter, people can get their own Peking Opera masks and interact with AR masks. They are also able to post these in social networks.

37.png

Technical validation of customized AR masks

Two parts of technical issues need to be tested and validated.

-The first was a custom mask. I decided to use self-generated art instead of designing a lot of mock-ups and letting users choose their own. Each user's mask should be unique. Just like the masks for characters in Peking Opera are exclusive.
-The next step is to make filters in Lens Studio. I want it to be able to accomplish the effect of changing faces. By turning back once or using gesture to change the AR mask, just like the face changing magic in the traditional theater stage.

Develop customized masks

Step1   Build database

 

Collect Peking Opera masks and extract the elements to create a database for deep learning network learning and training.

38.png

Step2   Recognition + classification

 

Using the recognition program to detect and classify the face, and then the tester's facial expression is recognized. Then pick the appropriate parts in the image library and recombine them.

40.png

Step3   Refactoring

 

According to the appearance of a person, the program selects the appropriate materials in the content library to combine, and then selects different color combinations according to the facial expressions.

41.png

​recombine

​Work flow of self-generated digital mask

42.png

Test the basic function — generate 2D masks

43.png

Fliter development in Lens Studio

44.png

Face Mask Function

Hand Gesture Function

Convert 2D mask images into 3D mesh that fits the face

Lens Studio development interface

Hand Gesture makes it easy for users to change their masks (They can have several generated masks based on their six expressions )

Final test before publish

Digital Masks

资源 30.png