Load file OOM error


Following this method viewtopic.php?f=11&t=1452&p=3283&hilit=load+data#p3283
to load 2TB file into 128 and 256 instances.

It works at 128 instances but fails at 256 instances.

Only error information I got is

"[NID 01855] 2014-12-05 08:10:55 Apid 8867134: initiated application termination
[NID 01855] 2014-12-05 08:10:58 Apid 8867134: OOM killer terminated this process.
iquery -anq “load(VPIC, ‘data2.bin’, -1, ‘(float, float, float, float, float,float, float)’)”

data2.bin is binary file on each instance.

Does anyone know How to resolve this problem ?


So … first things first. You loaded 2T of data at a single swallow? That’s pretty impressive.

Quite why that worked at 128 instances, but not at 256, has us slightly mystified. . .

We’re going to need more details.

  1. SciDB Version, OS distro, etc.
  2. Would you be so kind as to share the config.ini for the 128 and 256 instance clusters? (This will give us a handle on your physical configuration, also: things like “number of physical instances”).
  3. Some details about those physical nodes? Memory, CPU, cores, disk#, etc?
  4. Would you be so kind as to post the CREATE ARRAY statement for VPIC? (Want to check out chunk sizing, etc).