Automatic Sparse Differentiation: How ML Researchers Can Leverage Sparsity for Faster, Memory-Efficient Jacobian and Hessian Computation

An Illustrated Guide to Automatic Sparse Differentiation

Contents The chain rule AD is matrix-free Forward-mode AD Reverse-mode AD From Jacobian operators back to Jacobian matrices Sparse matrices Leveraging sparsity Pattern detection and coloring Index sets Efficient propagation Abstract interpretation Local and global patterns Graph formulation Greedy algorithm Bicoloring Hessian-vector products Second order pattern detection Symmetric coloring Necessary packages Test function Backend switch Jacobian computation Preparation Colorin...