Tutorial: Algorithmic Differentiation

Algorithmic differentiation is also known as automatic differentiation. It is a powerful tool in many fields, especially useful for fast prototyping in machine learning research. A couple of weeks ago, I started building a simple algorithmic differentiation module atop of Owl.

Currently I have implemented the prototype of both forward and backward mode. E.g., the following code first defines a function f0, then calculates from the first to the fourth derivative by calling Algodiff.AD.diff function.

open Algodiff.AD;;

let map f x = Vec.map (fun a -> a |> pack_flt |> f |> unpack_flt) x;;

(* calculate derivatives of f0 *)
let f0 x = Maths.(tanh x);;
let f1 = diff f0;;
let f2 = diff f1;;
let f3 = diff f2;;
let f4 = diff f3;;
let x = Vec.linspace (-4.) 4. 200;;
let y0 = map f0 x;;
let y1 = map f1 x;;
let y2 = map f2 x;;
let y3 = map f3 x;;
let y4 = map f4 x;;

(* plot the values of all functions *)
let h = Plot.create "plot_021.png" in
Plot.set_foreground_color h 0 0 0;
Plot.set_background_color h 255 255 255;
Plot.plot ~h x y0;
Plot.plot ~h x y1;
Plot.plot ~h x y2;
Plot.plot ~h x y3;
Plot.plot ~h x y4;
Plot.output h;;

Start your utop, then load and open Owl library. Copy and past the code above, the generated figure will look like this.

In the following weeks, I will keep polishing the module and try to overload more useful mathematical operators.

If you replace f0 in the previous example with the following definition, then you will have another good-looking figure :)

let f0 x = Maths.(
  let y = exp (neg x) in
  (F 1. - y) / (F 1. + y)
);;

Caveat: AD is a quite experimental feature at the moment. I will keep working on this in the following weeks/months.