Using flash or faster device for temp array cache


#1

This topic comes up with some regularity and I thought I’d offer an example here.

As you may know, the create temp array command can be used to create arrays that reside in memory, to the extent possible; spilling any data in excess of cache size to disk. The cache size is controlled by the mem-array-threshold setting, specified in Megabytes per instance. After cache size is exceeded, additional chunks are written to disk in the tmp subdirectory that is located under each respective instance data directory.

We’ll illustrate with an example, using 16.9. I have intentionally set mem-array-threshold low to cause spillover. This is how much room we have in the cache:

$ iquery -aq "_getopt('mem-array-threshold')"
{Inst} old
{0} '32'
{1} '32'
{2} '32'
{3} '32'
{4} '32'
{5} '32'
{6} '32'
{7} '32'

$ iquery -aq "aggregate(apply(_getopt('mem-array-threshold'), size_mb, int64(old)), sum(size_mb))"
{i} size_mb_sum
{0} 256

On my single-node installation, I have the following data directory in config.ini:

base-path=/data/scidb_data

And so all the temp arrays shall be written here:

$ du -hs /data/scidb_data/0/*/datastores/1_tmp
4.0K	/data/scidb_data/0/0/datastores/1_tmp
4.0K	/data/scidb_data/0/1/datastores/1_tmp
4.0K	/data/scidb_data/0/2/datastores/1_tmp
4.0K	/data/scidb_data/0/3/datastores/1_tmp
4.0K	/data/scidb_data/0/4/datastores/1_tmp
4.0K	/data/scidb_data/0/5/datastores/1_tmp
4.0K	/data/scidb_data/0/6/datastores/1_tmp
4.0K	/data/scidb_data/0/7/datastores/1_tmp

So let’s try to put a few gigs’ worth of random numbers into a temp array:

$ iquery -aq "create temp array foo <val:double>[x=1:500000000:0:1000000]"
Query was executed successfully

$ iquery -anq "store(build(foo, random()), foo)"
Query was executed successfully

$ du -hs /data/scidb_data/0/*/datastores/1_tmp
481M	/data/scidb_data/0/0/datastores/1_tmp
481M	/data/scidb_data/0/1/datastores/1_tmp
481M	/data/scidb_data/0/2/datastores/1_tmp
481M	/data/scidb_data/0/3/datastores/1_tmp
474M	/data/scidb_data/0/4/datastores/1_tmp
474M	/data/scidb_data/0/5/datastores/1_tmp
474M	/data/scidb_data/0/6/datastores/1_tmp
474M	/data/scidb_data/0/7/datastores/1_tmp

Now suppose I have a flash or faster IO device I want to use mounted at /local_flash. I can write a simple script that sets some symlinks and restarts scidb:

$ cat restart_with_flash.sh 
scidb.py stopall mydb

rm -rf  /local_flash/scidb_temp/tmp0
rm -rf  /local_flash/scidb_temp/tmp1
rm -rf  /local_flash/scidb_temp/tmp2
rm -rf  /local_flash/scidb_temp/tmp3
rm -rf  /local_flash/scidb_temp/tmp4
rm -rf  /local_flash/scidb_temp/tmp5
rm -rf  /local_flash/scidb_temp/tmp6
rm -rf  /local_flash/scidb_temp/tmp7

mkdir -p /local_flash/scidb_temp/tmp0
mkdir -p /local_flash/scidb_temp/tmp1
mkdir -p /local_flash/scidb_temp/tmp2
mkdir -p /local_flash/scidb_temp/tmp3
mkdir -p /local_flash/scidb_temp/tmp4
mkdir -p /local_flash/scidb_temp/tmp5
mkdir -p /local_flash/scidb_temp/tmp6
mkdir -p /local_flash/scidb_temp/tmp7

rm -rf /data/scidb_data/0/0/datastores/1_tmp
rm -rf /data/scidb_data/0/1/datastores/1_tmp
rm -rf /data/scidb_data/0/2/datastores/1_tmp
rm -rf /data/scidb_data/0/3/datastores/1_tmp
rm -rf /data/scidb_data/0/4/datastores/1_tmp
rm -rf /data/scidb_data/0/5/datastores/1_tmp
rm -rf /data/scidb_data/0/6/datastores/1_tmp
rm -rf /data/scidb_data/0/7/datastores/1_tmp

ln -s /local_flash/scidb_temp/tmp0 /data/scidb_data/0/0/datastores/1_tmp
ln -s /local_flash/scidb_temp/tmp1 /data/scidb_data/0/1/datastores/1_tmp
ln -s /local_flash/scidb_temp/tmp2 /data/scidb_data/0/2/datastores/1_tmp
ln -s /local_flash/scidb_temp/tmp3 /data/scidb_data/0/3/datastores/1_tmp
ln -s /local_flash/scidb_temp/tmp4 /data/scidb_data/0/4/datastores/1_tmp
ln -s /local_flash/scidb_temp/tmp5 /data/scidb_data/0/5/datastores/1_tmp
ln -s /local_flash/scidb_temp/tmp6 /data/scidb_data/0/6/datastores/1_tmp
ln -s /local_flash/scidb_temp/tmp7 /data/scidb_data/0/7/datastores/1_tmp

scidb.py startall mydb

$ ./restart_with_flash.sh 

$ iquery -aq "remove(foo)"
Query was executed successfully

$ iquery -aq "create temp array foo <val:double>[x=1:500000000:0:1000000]"
Query was executed successfully

$ iquery -anq "store(build(foo, random()), foo)"
Query was executed successfully

And now the chunks of foo are stored in the new location:

$ du -hs /local_flash/scidb_temp/*
481M	/local_flash/scidb_temp/tmp0
481M	/local_flash/scidb_temp/tmp1
481M	/local_flash/scidb_temp/tmp2
481M	/local_flash/scidb_temp/tmp3
474M	/local_flash/scidb_temp/tmp4
474M	/local_flash/scidb_temp/tmp5
474M	/local_flash/scidb_temp/tmp6
474M	/local_flash/scidb_temp/tmp7

In essence, this technique lets us separate the “temporary” array storage from persistent and make SciDB write temporary chunks into a separate location. Temporary arrays become “invalidated” upon SciDB restart (note we removed foo after restart in the example above) - so it is generally safe to remove and switch these directories with SciDB stopped. You should however take care not to disturb any of the adjacent directories.

Note also - the example uses 16.9. Some prior releases used a differently-named tmp location under [data]/X/Y/tmp.

… and I was recently informed that in the upcoming 17.x the temp directory names may change once again - do take note.