Swift for TensorFlow (In Archive Mode)

Swift for TensorFlow was an experiment in the next-generation platform for machine learning, incorporating the latest research across machine learning, compilers, differentiable programming, systems design, and beyond. It was archived in February 2021. Some significant achievements from this project include:

This site will not receive further updates. The API documentation and binary downloads will continue to be accessible as well as the Open Design Review meeting recordings.

Swift

Swift is an open source general-purpose programming language, which has a large and growing user base. We chose Swift because it has an open language design process and for specific technical reasons detailed in the "Why Swift for TensorFlow" document. We assume that most readers are unfamiliar with it, so we’ll briefly touch on some additional important things about it here.

The development of Swift started in 2010, and aimed to bring the best practices in programming language design together into one system rather than trying for academic novelty or to religiously propagate programming methodologies. As a result, it supports multi-paradigm development (e.g. functional, OOP, generic, procedural, etc) all in one system, and brings many well-known concepts from academic languages (e.g. pattern matching, algebraic data types, and type classes) into the forefront. Instead of strongly encouraging developers to rewrite all their code in Swift, it pragmatically focuses on interoperability with other languages, e.g., allowing you to directly import C header files and use them without an FFI and (now) the ability to use Python APIs without wrappers.

Swift has the audacious goal of spanning all the way from low-level systems programming to high-level scripting, with a focus on being easy to learn and use. Because Swift needs to be easy to learn and use but also powerful, it relies on the principle of progressive disclosure of complexity, which aggressively factors the cost of complexity onto the people who benefit from that complexity. The "scripting language feel" combined with high performance is very useful for machine learning.

A final pertinent aspect of the design of Swift is that much of the Swift language is actually implemented in its standard library. "Builtin" types like Int and Bool are actually just structs defined in the standard library that wrap magic types and operations. As such, sometimes we joke that Swift is just "syntactic sugar for LLVM".

There is a lot more that is cool about Swift and a ton of content available online. If you are interested in learning more about general Swift programming concepts, here are a few links to get started:

One warning: Swift evolved rapidly in its early years, so you should be careful with anything before Swift 3 (released in 2016).

Why Swift for TensorFlow?

Swift for TensorFlow is a new way to develop machine learning models. It gives you the power of TensorFlow directly integrated into the Swift programming language. We believe that machine learning paradigms are so important that they deserve first-class language and compiler support.

A fundamental primitive in machine learning is gradient-based optimization: computing function derivatives to optimize parameters. With Swift for TensorFlow, you can easily differentiate functions using differential operators like gradient(of:), or differentiate with respect to an entire model by calling method gradient(in:). These differentiation APIs are not just available for Tensor-related concepts—they are generalized for all types that conform to the Differentiable protocol, including Float, Double, SIMD vectors, and your own data structures.

// Custom differentiable type.
struct Model: Differentiable {
    var w: Float
    var b: Float
    func applied(to input: Float) -> Float {
        return w * input + b
    }
}

// Differentiate using `gradient(at:_:in:)`.
let model = Model(w: 4, b: 3)
let input: Float = 2
let (𝛁model, 𝛁input) = gradient(at: model, input) { model, input in
    model.applied(to: input)
}

print(𝛁model) // Model.TangentVector(w: 2.0, b: 1.0)
print(𝛁input) // 4.0

Beyond derivatives, the Swift for TensorFlow project comes with a sophisticated toolchain to make users more productive. You can run Swift interactively in a Jupyter notebook, and get helpful autocomplete suggestions to help you explore the massive API surface of a modern deep learning library. You can get started right in your browser in seconds!

Migrating to Swift for TensorFlow is really easy thanks to Swift's powerful Python integration. You can incrementally migrate your Python code over (or continue to use your favorite Python libraries), because you can easily call your favorite Python library with a familiar syntax:

import TensorFlow
import Python

let np = Python.import("numpy")

let array = np.arange(100).reshape(10, 10)  // Create a 10x10 numpy array.
let tensor = Tensor<Float>(numpy: array)  // Seamless integration!