TensorFlow


#1

While I can Spark SQL Stream data into an RDD on a MPP cluster to run XGBOOST, or Load Images to perform Deep Convolutional Neural Networks, so; the question is there such a thing as a Tensor Database, where operations end up in a Matrix/Vector/Scalar? Think about running TensorFlow on a Tensor Database.
I dont think I can run Spark/Tensorflow in AWS using SciDB? Imagine Spark SQL Stream data into a Tensor Database In Memory/RDD with System on a Chip (Memory + TPU). SCIDB admins, Please email me directly if you would like to discuss further offline, in addition to the responses we will see here. Peace,
Eric


#2

Hi @ewittry

We have commercial installations running TensorFlow today.

One way to do it is through Python Streaming - similar to here https://rvernica.github.io/2017/10/streaming-machine-learning but calling out to TensorFlow instead of SciKit-Learn

There is some interesting work to do in terms of SciDB-GPU balancing. If you have a node with 8 SciDB instances and 2 GPUs, they may not use the HW optimally, so some sort of scheduling layer would be useful.

There was also some very good work at integrating GPUs at the C++ level here: https://github.com/simonmarcin/SciDB_GPUs

More work in this area would be very exciting indeed!