“Hot” Topics using Tensorflow
Writing programs for distributed systems involves thinking about
- Scalability: runtime proportionally decreasing with the number of cores
- Big Data: inputs and intermediates are too large to be in one machine
Supercomputers differ from typical cloud infrastructure by consisting of one large machine with thousands of cores and individual memories connected with a high-speed interconnect.
Writing programs for it usually meant spending quite some time with MPI (Message Passing Interface), and it operated at a rather painful level of abstraction, where communication between machines or even cores (depending on the version) had to be explicitly specified.
Modern distributed programming tools are starting to think in terms of distributed matrices, tensors, and graphs, and the whole idea of working with the message as the entity of distribution seems outdated.
Apache Spark operates with a resilient distributed dataset (RDD) as the basic unit and with it came the benefit of “fault tolerance” — calculations remaining robust even under partial hardware failure.
Doing this with MPI would involve writing a protocol (as if the application itself wasn’t tiring enough) that repeated messages appropriately based on the chance of core failure!
So it seems that the newer frameworks are providing benefits such as scalability and fault tolerance along with increased support in terms of numerous libraries. But MPI is still very common in scientific computing and demonstrating benefits would involve showing it on a scientific problem.
An opinion piece titled “HPC is dying and MPI is killing it” is a must-read in this direction where the 1D Heat Diffusion equation was illustrated using modern frameworks such as Chapel and Spark.
My efforts along that line are primarily towards including Tensorflow as well, and what better time when it is also heading towards fault tolerance while being familiar to us as the standard choice for machine learning applications.
The heat diffusion equation (∂u/∂t = α Δu) also nicely ports to modern frameworks as the second-derivative (or even the Laplacian in higher dimensions) can be written as a convolution.
Here, for 1D, the kernel is [1, -2, 1], corresponding to the numerical coefficients on discretizing the derivative.
With that in place, we can solve it in Tensorflow!
And this is how it looks like on solving
Woohoo! And that’s how we solve differential equations using Tensorflow.
The link to the repository is available here. Make sure to star it :)
