Since its release just a few months ago, PyTorch 1.0 has been rapidly adopted as a powerful, flexible deep learning platform that enables engineers and researchers to move quickly from research to production. As part of Facebook’s F8 Conference this year, we are highlighting some of the ways the AI engineering and research community is using PyTorch 1.0. We’re also sharing new details about the latest release, PyTorch 1.1, and showcasing some of the new development tools created by the community.
Building on the initial launch of PyTorch in 2017, Facebook partnered with the AI community to ship the stable release of PyTorch 1.0 last December. Along with enhanced production-oriented capabilities and deep integration with leading cloud platforms, PyTorch 1.0 expands on the open source library’s core features, with the addition of PyTorch JIT (Just in time compilation) that seamlessly transitions between eager mode and graph mode to provide both flexibility and speed.
Leading businesses across industries are beginning to use PyTorch to both facilitate their research and then also deploy at large scale for applications such as translation, computer vision, conversational interfaces, pharmaceutical research, factory optimization, and autonomous driving. Community adoption of PyTorch has also continued to expand. Stanford, UC Berkeley, Caltech, and other universities are using PyTorch as a fundamental tool for their machine learning (ML) courses; new ecosystem projects have launched to support development on PyTorch; and major cloud platforms have expanded their integration with PyTorch.
Facebook is now using PyTorch 1.0 end-to-end workflows for building and deploying translation and natural language processing (NLP) services at scale. These systems provide nearly 6 billion translations a day for applications such as realtime translation in Messenger and, as the foundation of PyText, power complex models that rely on multitask learning in NLP.
PyTorch’s unified framework has allowed us to iterate our ML systems more quickly. We leverage the eager frontend API (define by run) to build our neural machine translation (NMT) models allowing for full flexibility to explore the design space, train on distributed systems and iterate, and finally produce a graph with various optimizations (such as fusion, etc) for inference in production. This overall workflow demonstrates the full power of PyTorch 1.0 for large-scale production.
Moving to PyTorch has delivered other benefits as well. It has allowed us to reduce complexity by maintaining a single code base for both research and production. This has helped bring our researchers and product teams closer together and overall allowed teams at Facebook to significantly accelerate their research-to-production cycle, enabling better experiences for the people using our products. Teams across Facebook are actively developing with end to end PyTorch for a variety of domains and we are quickly moving forward with PyTorch projects in computer vision, speech recognition and speech synthesis.
Beyond Facebook, many leading businesses are moving to PyTorch 1.0 to accelerate development and deployment of new AI systems. Here are some examples:
Following the release of PyTorch 1.0 in December 2018, we’re now announcing the availability of v1.1, which improves performance, adds new model understanding and visualization tools to improve usability, and provides new APIs.
Key features of PyTorch v1.1 include:
We’ve also continued to partner with the community to foster projects and tools aimed at supporting ML engineers for needs ranging from improved model understanding to auto-tuning using AutoML methods. AutoML approaches have been valuable for applications at Facebook including computer vision and personalization. With the release of Ax and BoTorch (below), we will be sharing some of our core algorithms, including meta-learning for efficiently optimizing hyperparameters from based on historical tasks. We are exciting to see this work open-sourced for the community to build upon.
This ecosystem of projects and tools includes open source resources from Facebook that have been deployed at production scale, as well as products and services from our partnership with industry leaders such as Google, who share our vision of an open and collaborative AI community. Here are a few of the latest tools:
We’re also excited to see many interesting new projects from the broader PyTorch community. Highlights include:
PyTorch is ideal for teaching ML development because it enables rapid experimentation through its flexible, dynamic programming environment and user-friendly Pythonic interface. In addition, Google Colab now offers an interactive Jupyter Notebook environment that natively supports PyTorch, allowing developers to run any PyTorch tutorial immediately with free CPU and GPU resources.
University-level classes — including Stanford NLP, UC Berkeley Computer Vision, and Caltech Robotics courses — are now being taught on PyTorch. In addition, massive open online courses (MOOCs) are training thousands of new PyTorch developers.
Today, we’re announcing our sponsorship of a new Udacity course, building upon the Intro to Deep Learning course launched last year. This new course, led by Andrew Trask of Oxford University and OpenMined, covers important concepts around privacy in AI, including methods such as differential privacy and federated learning. Facebook will also be providing scholarships to support students as they continue their ML education in Udacity’s full Nanodegree programs.
The fast.ai community is also continuing to invest energy and resources in PyTorch. In June, fast.ai will launch a new course called Deep Learning from the Foundations, which will show developers how to go all the way from writing matrix multiplication from scratch to how to train and implement a state-of-the-art ImageNet model. The course will include deep dives into the underlying implementation of methods in the PyTorch and fast.ai libraries, and will use the code to explain and illustrate the academic papers that underlie these methods.
As part of the course, fast.ai will also release new software modules, including fastai.audio, which brings the power of fast.ai’s deep abstractions and curated algorithms to the new PyTorch.audio module, and show how fastai.vision can be used to create stunning high-resolution videos from material such as old classic movies, and from cutting-edge microscopy sequences through a collaboration with the Salk Institute. In addition, fast.ai is contributing its new X-ResNet module, including a suite of models pretrained on ImageNet, to the newly launched PyTorch Hub.
Everyone in the AI community — including those new to ML development as well as researchers and engineers looking for ways to accelerate their end-to-end workflows — can experiment with PyTorch instantly by visiting pytorch.org and launching a tutorial in Colab. There are also many easy ways to get started both locally and on popular cloud platforms.
As Facebook and others in the AI community continue to expand these resources, PyTorch will become more capable and versatile, as well as more accessible. With a hybrid front end that enables tracing and scripting models from eager mode into graph mode, along with a growing set of tools and resources such as PyTorch-BigGraph, BoTorch and Ax, and Tensorboard support, PyTorch is a powerful framework for taking breakthrough research in artificial intelligence to production deployment. The ongoing evolution of PyTorch serves as an example of the power of open, community-led development in AI, and we look forward to the many new collaborations still to come.