Here’s a quick roundup of distributed deep learning efforts running on Apache Spark. This will only list active(-ish) projects rather than academic experiments (of which there are too many to list) There’s roughly two approaches:
Linking Spark with an existing framework
- SparkNet from UC berkeley connects Apache Spark with Caffe. You can read the paper
- CaffeOnSpark from Yahoo takes the same approach, see the blog post.
- Arimo distributed TensorFlow on Spark for hyper-parameter tuninig, but that was before the release of the distributed version. Here’s a video from Spark Summit East.
- Elephas connects Keras with Apache Spark
Implementing a full-fledged frameworrk
- DeepDist (repo) is a framework for DBNs implementing downpour gradient descent. The approach is reminiscent of Splash
- DeepLearning4J is reimplementing a wide range of NNs, from a fast Java array lib. They run distributed on Spark, with GPU acceleration.
This is just a quick preview, and the criteria for notability are somewhat arbitrary : e.g. I chose not to include OpenDL, because it’s a seemingly unmaintained experiment based on Jeff Dean’s “Large Scale Distributed Deep Networks” paper. Feel free to mention anything I would have forgotten in comments !