Sending R matrices to scidb takes a very long time


#1

Hello,

I use scidb from R. The cluster configuration is 12 instances on 2 servers.
I build R sparse matrices that I send to scidb using the as.scidb function. I am surprised by the time it takes to complete.

Sending an array of 50000 entry takes 3s

The following piece of code takes 80 seconds to complete even though the amount of data to be transferred should be similar to the above query.

Do you know why I observe this behavior?

Kind regards,

Sébastien


#2

It seems the sparseMatrix is 3 times larger (with indexes), must inspect for duplicate entries in row and column indexes, and must update the entries. Creating each array of indexes and then the values might well take on the order of 3-fold the time to create the individual array.

Ray