The Fundamental AI Research (FAIR) team at Meta AI is excited to release Animated Drawings, an open source project aimed at helping creators and developers easily create their own drawing-to-animation experiences or products. It started in 2021 when researchers from FAIR decided to apply recent advances in computer vision towards a fast and intuitive pipeline to animate the human-like figures found in character drawings.
Drawing is a familiar and fun mode of expression, and the characters we draw as beginners are often wonderfully odd, abstract, and imaginative. Who hasn’t wished that those delightful characters might come alive, moving around on the page? There are even books and TV episodes that speak to this universal fantasy. Unfortunately, actually achieving that effect is quite hard. Creating many drawings to give the illusion of motion (such as a flip book) is tedious, and existing animation tools can be challenging for newer users. As a result, many incredible characters remain static on the page.
Our researchers envisioned using object detection models, pose estimation models, and image processing-based segmentation methods to quickly create a digital version of the drawing. This could then be deformed and animated using traditional computer graphics techniques. At Meta, we see this vision as an opportunity to connect people and communities through the power of creativity.
But there was a problem: character drawings, at the scale needed to train a computer vision model, are quite hard to come by. They also need annotations, such as bounding boxes, segmentation masks, and joint locations. No such dataset exists. To solve this problem, the researchers created the Animated Drawings Demo, a publicly-accessible web-based tool where creators could upload their drawings, view/correct a few model predictions, and receive an animation starring a character in the drawings - all in less than a minute. As part of the user flow, parents could opt to allow Meta to retain the image and annotations for future research purposes; this was completely optional, and the answer didn’t affect the functionality of the tool in any way. The researchers released the demo with the hopes of eventually collecting 10,000 drawings with which to improve their model’s performance.
The reaction to the Animated Drawings Demo was overwhelmingly positive. Within the first few months, users had uploaded and provided consent to use over 1.6 million images. Interestingly, many of the demo’s users tried to use it for purposes other than the original intent. Many of the images uploaded weren’t amateur drawings at all. They were pictures of company logos, stuffed animals, anime characters, pets, action figures, and a variety of other things that people wanted to animate. While the demo specified the necessity for human figures in the instructions, users uploaded quadrupeds, birds, fish, and many other forms. Users also indicated their desire for a more comprehensive tool via social media posts and in-demo bug reporting. They wanted transparent backgrounds, support for different skeleton types, multiple interacting characters, sound effects, background scenery, and text overlays.
Clearly, there was interest in more drawing-to-animation-based experiences. Rather than extend the demo to support all the new feature requests, the researchers decided instead to release an open-sourced version of the models and code used by the Animated Drawings Demo. Meta hopes that by implementing Animated Drawings in our own projects, we can inspire other developers to create demos and experiences that utilize animation assets from user drawings.
The Animated Drawings project is a great example of how open source technology can be used to lower the barrier to entry for people who want to experiment with animation and create their own drawing-to-animation experiences. By releasing the models and code as open source, the project provides a starting point for developers to build on and extend the project, fostering a culture of innovation and collaboration within the open source community.
If you're interested in exploring the FAIR Animated Drawings project, check out the installation and quick start instructions on the project's README. For a more visual exploration of the open source Animated Drawings project, we've also created a fun YouTube video with examples that you can check out here. We can't wait to see what you create!