Scidb consuming all available memory


#1

It seems immediately after startup scidb consumes all the available memory(12 GB). This might be due to the fact that scidb loads all metadata into memory at startup as mentioned here viewtopic.php?f=11&t=1437&p=3236&hilit=memory+usage#p3236.

But our use case randomly access data. So is there a way to stop all the metadata from loading into memory hence preventing scidb from taking up all the memory?

Here is the config I am using.

[cluster]
server-0=server,1
install_root=/opt/scidb/14.12
pluginsdir=/opt/scidb/14.12/lib/scidb/plugins
logconf=/opt/scidb/14.12/share/scidb/log4cxx.properties
db_user=scidb_pg_user
db_passwd=scidb_pg_user_pasw
base-port=1239
base-path=/home/scidb/scidb_data
redundancy=0

Threading: max_concurrent_queries=2, threads_per_query=1

max_concurrent_queries + 2:

execution-threads=4

max_concurrent_queries * threads_per_query:

result-prefetch-threads=2

threads_per_query:

result-prefetch-queue-size=1
operator-threads=1

Memory: 16000MB per instance, 12000MB reserved

network: 4800MB per instance assuming 5MB average chunks

in units of chunks per query:

sg-send-queue-size=240
sg-receive-queue-size=240

caches: 4800MB per instance

smgr-cache-size=1024
mem-array-threshold=1024

sort: 2400MB per instance (specified per thread)

merge-sort-buffer=1024

NOTE: Uncomment the following line to set a hard memory limit;

NOTE: queries exceeding this cap will fail:

max-memory-limit=10000

####################

WARNING: Make sure all of the below directories are created

before you execute ‘initall’

Instance storage directories on server0.domain.com

data-dir-prefix-0-0=/data/cluster.1
data-dir-prefix-0-1=/data/disk2/cluster.1


#2

Hello,

Sorry about the trouble.

More specifically, at startup SciDB loads the chunk map into memory, and the chunk map should be much smaller (in total size) unless you have 10s of terabytes data all loaded into this single node. If the chunk map is taking up too much memory, it could mean that some arrays’ chunk sizes are too small leading to too many total chunks.

Here’s a query that can give you a per-array summary:
paradigm4.com/forum/viewtopic.ph … 1950#p1950

You can post that output here and we can advise further.


#3

Now scidb crash and restart when I run any scidb command. I believe it might be something to do with 100 percent memory usage.


#4

Remove the “max-memory-limit” from the configuration file and try again?


#5

still same

SystemException in file: src/network/BaseConnection.h function: receive line: 321
Error id: scidb::SCIDB_SE_NETWORK::SCIDB_LE_CANT_SEND_RECEIVE
Error description: Network error. Cannot send or receive network messages.


#6

Hi — We need to see the error messages from the log files to know exactly why scidb is failing to start. Take a look at the files in:

/home/scidb/scidb_data/000/<instance #>/

You should see a “scidb.log” file and a “scidb-stderr.log” file. If you scan those files for lines beginning with “ERROR” you should be able to see the actual errors that are preventing scidb from starting…

If you don’t see any error messages at all, then its possible that the system is killing the process for using too much memory. In that case, you might find evidence of that in the “/var/log/messages” file.

Post the error message that you can find here and we’ll take a look.
–Steve F


#7

It seems the issue with using pandas Dataframe. It seems to work fine when I use flat arrays but as soon I move to dataframe scidb takes up all the memory and keeps crshing.