ELI5: Captum - Easily Interpret AI Models

Artificial Intelligence is often described as a “black box”; we plug inputs in and the AI model gives us an output back, but we don’t have an easy way to understand how the model came to produce that output. The process to train AI generally goes like this - we submit inputs into a computer, train the computer to obtain an AI model, and use this model to predict our outputs. But how is that model actually working to make its conclusions?

Captum, from the Latin word for “comprehension,” aims to help us peer behind the curtain so that we can better interpret how the AI we program is “thinking,” giving us critical info on how to improve it.

In this blog post, we explain Captum in a way that is simple to understand (or as it’s commonly known online, ELI5. If you're interested in learning by watching or listening, check out the linked video about Captum on our Facebook Open Source Youtube channel.

What is Captum?

Captum is an open-source tool built on PyTorch that helps researchers and developers easily understand which features are contributing to an AI model’s prediction outputs. What Captum does is called model interpretability because it interprets the features the AI models used to make predictions.

For example, say we want to train an AI model to identify pictures of dogs. We would input lots of pictures of dogs into the computer so that it can start to understand what are defining features of a dog. It then creates a model that is able to recognize a dog the next time it sees one; however, we can’t know exactly what the computer is looking at in the images to come to the conclusion of whether or not there is a dog. Captum can help us see exactly which features the AI model is relying on to recognize if there is a dog in our picture.

In this example, Captum shows us that these green pixels are the ones most critical to helping us identify the dog in the image.