Change the directory of memarray


#1

Hi, there
My memarray is located in “/home/scidb/mydb-DB/000/0/tmp/memarray” and my data is also in “/home/scidb/mydb-DB/000/0/”. And the memarray occupys a lot of disk. So I want to change it to another one. I want to store the data and mamarray separately. I change "src\network\entry.cpp"
memArrayBasePath="/mnt/disk"+memArrayBasePath+"memarray";
But it doesn’t work. So is there any suggestions? I will very appreciate.


#2

We often accomplish the same effect with symlinks.

  1. stop scidb
  2. mkdir other_location/tmp1 other_location/tmp2 …
  3. ln -s other_location/tmp1 /home/scidb/mydb-DB/000/0/tmp

make sense?


#3

Hi, Mr apoliakov,
Thanks a lot. I got it. So the memarray is stored the intermediate result of query and it is like tempdb in SQL Server? When there is a query, there are memarray?
Q2: I modify the entry.cpp.
LOG4CXX_INFO(logger, "#########yq##########Start SciDB instance (pid="<<getpid()<<"). " << SCIDB_BUILD_INFO_STRING(". ")); LOG4CXX_INFO(logger, "#########yq##########Configuration:\n" << cfg->toString());
and recompile the source code.
vim /opt/scidb/15.7/share/scidb/log4cxx.properties, change
log4j.rootLogger=DEBUG, file into
log4j.rootLogger=TRACE, file.

./run.py setup -f  # to configure build directories and cmake infrastructure
./deployment/deploy.sh build_fast /tmp/packages
./deployment/deploy.sh scidb_install /tmp/packages 192.168.100.90 192.168.100.91 192.168.100.92
./deployment/deploy.sh scidb_prepare scidb "" mydb mydb mydb /home/scidb/mydb-DB 2 default 1 192.168.100.90 192.168.100.91 192.168.100.92

But “#########yq##########Start SciDB instance” cannot print in the scidb.log.
I even./run.py cleanup. And it didn,t work too.

Thank you.


#4

Correct about mem arrays - not all queries generate them; depends on which operator your query uses.
Also when you “create temp array…” it also creates a mem array that is invalidated after restart.

The logging should work - forget to re-install or something?


#5

I ln -s /mnt/disk1 /home/scidb/mydb-DB/000/0/tmp
I want to write memarray to /mnt/disk1 instead of /home/scidb/mydb-DB/000/0/tmp/memarray.
[scidb@coordinate memarray]$ du -sh
8.0K .
After I create b.txt(4K) in the directory of memarray. Now memarray occupy 12K. But the size of /mnt/disk1 don’t change. It doesn’t increase 4K.

[scidb@coordinate tmp]$ ll
total 16
lrwxrwxrwx. 1 scidb scidb   10 May 17 10:38 disk1 -> /mnt/disk1
drwx------. 2 scidb scidb 4096 May 17 18:45 memarray

I want to write to memarray, the data of memarray is redirect to /mnt/disk1. So is there any suggestions? Thanks.


#6

You also need to take into account the mem-array-threshold configuration variable. If the size of the array (as reported by summarize for example) is less than mem-array-threshold per instance, then the array is in memory. Otherwise, a portion of the array may still be held in memory. The mem-array-threshold is a shared piece of memory (per instance) that all active MemArrays can use. This is why they’re called “MemArrays” and hopefully the benefit is obvious.

If you create a larger array and / or decrease mem-array-threshold (requires scidb restart) I am confident you will see more space used by the directory.

Also - to find how much space is occupied by a directory, use a different command like “du -hs [dirname]” . The 4096 reported by ll is only the filesystem overhead.