There’s no fool like a ‘Painting Fool’ – AI award for Imperial engineers

Department of Computing team win prestigious Artificial Intelligence award – News

By Colin Smith
21 January 2008

Could Artificial Intelligence ever be taken seriously as a creative force in the world of high art?

If a team from Imperial College London’s Department of Computing have their way, bespoke, individualised and beautiful pieces of artwork generated by Artificial Intelligence could become a reality.

Last month, Dr Simon Colton, PhD student Michel Valstar, and Dr Maja Pantic from the Department of Computing, won the prestigious award for the best Machine Intelligence demonstration at the British Computer Society’s Specialist Group on Artificial Intelligence (SGAI) conference, held at Cambridge University.

The SGAI award, voted by some of the world’s foremost authorities on Artificial Intelligence (AI), recognised the team’s innovative and creative automated software technology – The Painting Fool – which creates portraits by reading the facial expressions of a human subject and interpreting the emotion conveyed by expressions.

Dr Colton, who designed The Painting Fool, said he was excited by the challenge of creating an automated artist, which produces beautiful pieces of artwork. He also said his work raised some interesting questions about how people perceive creativity.

“I’m a computational creativity researcher, which is a sub-area of AI research, so the Painting Fool, can, and has been used to test out some theories about how people perceive software as creative - or not. For instance, because people have to watch The Painting Fool produce its artwork with painstaking detail they seem to project more value onto it than they would if Photoshop produced something in milliseconds,” he said.

In order to create a painting, The Painting Fool programme first takes an image of its subject, via a video camera, and segments it into a number of regions of colour. It then takes each segment and simulates how a human artist might paint that region. For instance, The Painting Fool may choose to imitate an artist’s brushstrokes by painting a circular region with ever decreasing circles, or straight lines with fine strokes.

In addition to simulating the artist’s style of work, it also replicates what art materials, like acrylics, pastels, chalk or watercolours would look like.

“In recent developments, I’ve enabled the choice of colour palette and materials to be controlled by other software. For the competition we used machine vision software to control the painting styles, so the portraits were appropriate to the emotion being displayed by the sitter,” Dr Colton explained.

He said this technology allows the The Painting Fool to use a number of techniques to depict emotions - for instance, if a person displays anger it may paint the subject in bold angry reds and black and use harsh brushstrokes to represent rage.

This type of intelligent interpretation of human feelings was possible because of the work of fellow contributor Michel Valstar, who said he felt “absolutely fantastic” about winning the award.

Michel, who is a PhD student of Dr Maja Pantic, contributed software which helped The Painting Fool interpret human emotions. As part of his PhD, Michel developed the Facial Action Analysis system used by The Painting Fool to recognise complicated facial features.

“I developed a computer system that is capable of detecting 23 out of a total 27 facial muscle actions which are known as Action Units - or AUs. These AUs help to describe typical expressions of the six basic emotions which include anger, disgust, fear, happiness, sadness or surprise,” explained Michel.

Apart from detecting which AUs were present, Michel says this technology was the first in the world to actually pinpoint when an AU starts, reaches its peak and when it starts to return to its normal expression.

“We use this knowledge to select from a video frame where the emotion shown was most intense. We call that the apex frame,” he said.

The apex frame is sent to The Painting Fool programme, together with information about which emotion was detected and where the facial points were at that time. Michel said he was pleased to see his work actually demonstrated in the real world.

“It is great to see my thesis work function in the real world in a useful way and I think the combination with The Painting Fool is brilliant. It all feels like a justification of all my PhD efforts,” said Michel proudly.

Dr Colton and Michel Valstar conducted their demonstration in front of delegates at the conference. Michel was the first ‘guinea pig’ to have his portrait painted, followed by Paulo - a delegate. Unfortunately, the exhibition did not go without a hitch.

“About ten people in the audience decided to use flash photography to capture the moment. Unfortunately, this meant the machine vision software was put off. It actually thought Paulo was angry when he was smiling. So, instead of painting him in happy colours, The Painting Fool chose angry greens and reds. Michel expertly joked his way out of that,” laughed Dr Colton.

And what prize did the team receive for all their efforts?

“A trophy which we keep for a year and a cheque of £1,000 from the sponsors Electrolux,” said Dr Colton.

The team plan to spend their winnings on an evening for the people who helped them to fine tune the system.

“We asked four or five people to be our pet subjects in the days leading up to the competition. They had to patiently sit and pull faces in front of a camera. Without their help we wouldn’t have done so well,” said Dr Colton.

Dr Colton said he would also like to purchase a robotic arm to do the painting in the real world - instead of a computer screen. He is also working on enabling The Painting Fool to design and paint its own 3D scenes.

Commenting about other future projects he said: “I also hope to announce a new project with a linguist from Oxford University to get The Painting Fool to read the front page of the New York Times and paint a picture which represents the events and emotional content of the story.”

Dr Maja Pantic said she was pleased with the win, because it gave the team the chance to demonstrate their facial recognition technology in real life.

“I have been working on automatic facial expression and emotion recognition for almost 10 years and I am glad that we finally managed to develop the technology to a stage where we could use in applications like The Painting Fool,” said Dr Pantic.

And what of the original question - could Dr Colton envisage a day when artwork generated by Artificial Intelligence could ever be taken seriously by the community and the art world?

“People want bespoke, individualised, unique and beautiful paintings. They can get all of this for £500 pounds or more from a human artist. But people without a budget have to settle for artwork which is not unique, or not quite right, or not individual. If you begin to appreciate computer art, then it can supply all of these aspects. I want to do for bespoke art what Ikea did for well designed furniture R 11; bring it to the masses.”

The SGAI is one of the leading Artificial Intelligence societies in Europe. It has organised an international AI conference annually since 1981. Imperial’s entry beat stiff competition from Heriot-Watt University, Nottingham Trent University, Leeds University and Grays Knowledge Engineering – based in Sydney, Australia.