Garbage collection/ temporary object in scidb


#1

Hello,

Should we take care of cleaning up objects that are created in scidb by R?
For example I have an iterative algortihm where a matrix is constantly updated according to M(t) = f(M(t-1))
M is pretty huge and for each time step a new matrix will be created. I am afraid these computations will eat up all of scidb’s available memory pretty quickly.

Are the corresponding scidb object destroyed when the R object is destroyed by the garbage collection. Or should it be handled manually?

Thanks & Regards,

Cristiano


#2

SciDB-R’s garbage collector cleans up the intermediate arrays once they get out scope.

To make sure that everything works OK, I would suggest

  • turn on debug mode using options(scidb.debug=TRUE), then viewing the R log to check that calls like remove(R_array...) are being called after each iteration
  • Temp arrays created by SciDBR have the signature R_array.... You can check for the existence and timely removal of such arrays by running
# on the terminal
iquery -aq "project(list(), name)" | grep R_array
  • If the result of each iteration is of the same shape, you might consider writing the result of each iteration onto the same array. Then the results of each iteration are stored as versions of the same array. You can clean up versions separately using the remove_versions operator
remove_versions(array, N)
  • Finally, if you do see memory leaks, please post a minimal repro of your problem with some small dummy data and a minimal function f(). We can then run such a script to suggest improvements / issue software fixes as needed.