Error Starting SciDB Server


#1

Hello,

Scidb was killed after a window-aggregation call maxed system memory. I had to do a hard reboot of the server.

I am now getting the following error:

/opt/scidb/14.3/bin/scidb.py: ERROR: Failed to start SciDB!

Here is the only error I could find in the log-files (scidb-stderr.log):

2014-10-15 16:7:10 (ppid=23633): Started.
log4cxx: Could not read configuration file [/kyle2/nick/scidb_data/share/scidb/log4cxx.properties].
log4cxx: No appender could be found for logger (scidb.entry).
log4cxx: Please initialize the log4cxx system properly.
2014-10-15 16:23:30 (ppid=23633): Terminated.

Also noticed that the status command shows instances online but there are no running scidb processes, not sure if this is related.

instance_id | host | port | online_since -------------+-----------+------+---------------------------- 0 | localhost | 1239 | infinity 1 | localhost | 1240 | 2014-10-13 13:22:06.96962 2 | localhost | 1241 | infinity 3 | localhost | 1242 | 2014-10-13 13:22:06.301049 4 | localhost | 1243 | 2014-10-13 21:20:07.016421 5 | localhost | 1244 | infinity 6 | localhost | 1245 | infinity 7 | localhost | 1246 | infinity 8 | localhost | 1247 | infinity 9 | localhost | 1248 | infinity 10 | localhost | 1249 | 2014-10-13 13:23:09.077303 11 | localhost | 1250 | infinity 12 | localhost | 1251 | infinity 13 | localhost | 1252 | infinity 14 | localhost | 1253 | infinity 15 | localhost | 1254 | 2014-10-13 13:22:09.840754 16 | localhost | 1255 | infinity 17 | localhost | 1256 | infinity 18 | localhost | 1257 | infinity 19 | localhost | 1258 | infinity 20 | localhost | 1259 | 2014-10-13 13:22:25.741177 21 | localhost | 1260 | infinity 22 | localhost | 1261 | infinity 23 | localhost | 1262 | infinity 24 | localhost | 1263 | infinity 25 | localhost | 1264 | infinity 26 | localhost | 1265 | 2014-10-13 13:22:17.723002 27 | localhost | 1266 | infinity 28 | localhost | 1267 | 2014-10-13 13:22:18.269076 29 | localhost | 1268 | infinity 30 | localhost | 1269 | infinity 31 | localhost | 1270 | infinity

Any advice would be greatly appreciated.

Thanks in advance for your help on this.

Nick


#2

Please run the command to stop all instances first:
scidb.py stopall

Then restart all instances with a verbose switch:
scidb.py -v startall

Check to see if all instances start successfully. If not, post the output of the start command.


#3

OK here is what I am getting:

Starting SciDB server.
/opt/scidb/14.3/bin/scidb.py: DEBUG: Remote command=exec /bin/bash -c $'(cd /kyle2/nick/scidb_data/000/0 && rm -f /kyle2/nick/scidb_data/000/0/SciDB-000-0-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/0/SciDB-000-0-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/0/SciDB-000-0-kyle2 -i localhost -p 1239 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/0/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/0/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/0/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/1 && rm -f /kyle2/nick/scidb_data/000/1/SciDB-000-1-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/1/SciDB-000-1-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/1/SciDB-000-1-kyle2 -i localhost -p 1240 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/1/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/1/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/1/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/2 && rm -f /kyle2/nick/scidb_data/000/2/SciDB-000-2-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/2/SciDB-000-2-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/2/SciDB-000-2-kyle2 -i localhost -p 1241 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/2/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/2/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/2/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/3 && rm -f /kyle2/nick/scidb_data/000/3/SciDB-000-3-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/3/SciDB-000-3-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/3/SciDB-000-3-kyle2 -i localhost -p 1242 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/3/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/3/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/3/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/4 && rm -f /kyle2/nick/scidb_data/000/4/SciDB-000-4-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/4/SciDB-000-4-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/4/SciDB-000-4-kyle2 -i localhost -p 1243 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/4/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/4/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/4/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/5 && rm -f /kyle2/nick/scidb_data/000/5/SciDB-000-5-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/5/SciDB-000-5-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/5/SciDB-000-5-kyle2 -i localhost -p 1244 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/5/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/5/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/5/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/6 && rm -f /kyle2/nick/scidb_data/000/6/SciDB-000-6-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/6/SciDB-000-6-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/6/SciDB-000-6-kyle2 -i localhost -p 1245 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/6/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/6/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/6/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/7 && rm -f /kyle2/nick/scidb_data/000/7/SciDB-000-7-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/7/SciDB-000-7-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/7/SciDB-000-7-kyle2 -i localhost -p 1246 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/7/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/7/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/7/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/8 && rm -f /kyle2/nick/scidb_data/000/8/SciDB-000-8-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/8/SciDB-000-8-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/8/SciDB-000-8-kyle2 -i localhost -p 1247 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/8/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/8/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/8/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/9 && rm -f /kyle2/nick/scidb_data/000/9/SciDB-000-9-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/9/SciDB-000-9-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/9/SciDB-000-9-kyle2 -i localhost -p 1248 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/9/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/9/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/9/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/10 && rm -f /kyle2/nick/scidb_data/000/10/SciDB-000-10-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/10/SciDB-000-10-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/10/SciDB-000-10-kyle2 -i localhost -p 1249 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/10/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/10/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/10/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/11 && rm -f /kyle2/nick/scidb_data/000/11/SciDB-000-11-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/11/SciDB-000-11-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/11/SciDB-000-11-kyle2 -i localhost -p 1250 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/11/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/11/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/11/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/12 && rm -f /kyle2/nick/scidb_data/000/12/SciDB-000-12-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/12/SciDB-000-12-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/12/SciDB-000-12-kyle2 -i localhost -p 1251 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/12/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/12/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/12/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/13 && rm -f /kyle2/nick/scidb_data/000/13/SciDB-000-13-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/13/SciDB-000-13-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/13/SciDB-000-13-kyle2 -i localhost -p 1252 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/13/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/13/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/13/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/14 && rm -f /kyle2/nick/scidb_data/000/14/SciDB-000-14-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/14/SciDB-000-14-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/14/SciDB-000-14-kyle2 -i localhost -p 1253 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/14/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/14/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/14/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/15 && rm -f /kyle2/nick/scidb_data/000/15/SciDB-000-15-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/15/SciDB-000-15-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/15/SciDB-000-15-kyle2 -i localhost -p 1254 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/15/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/15/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/15/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/16 && rm -f /kyle2/nick/scidb_data/000/16/SciDB-000-16-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/16/SciDB-000-16-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/16/SciDB-000-16-kyle2 -i localhost -p 1255 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/16/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/16/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/16/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/17 && rm -f /kyle2/nick/scidb_data/000/17/SciDB-000-17-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/17/SciDB-000-17-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/17/SciDB-000-17-kyle2 -i localhost -p 1256 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/17/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/17/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/17/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/18 && rm -f /kyle2/nick/scidb_data/000/18/SciDB-000-18-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/18/SciDB-000-18-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/18/SciDB-000-18-kyle2 -i localhost -p 1257 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/18/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/18/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/18/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/19 && rm -f /kyle2/nick/scidb_data/000/19/SciDB-000-19-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/19/SciDB-000-19-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/19/SciDB-000-19-kyle2 -i localhost -p 1258 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/19/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/19/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/19/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/20 && rm -f /kyle2/nick/scidb_data/000/20/SciDB-000-20-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/20/SciDB-000-20-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/20/SciDB-000-20-kyle2 -i localhost -p 1259 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/20/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/20/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/20/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/21 && rm -f /kyle2/nick/scidb_data/000/21/SciDB-000-21-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/21/SciDB-000-21-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/21/SciDB-000-21-kyle2 -i localhost -p 1260 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/21/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/21/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/21/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/22 && rm -f /kyle2/nick/scidb_data/000/22/SciDB-000-22-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/22/SciDB-000-22-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/22/SciDB-000-22-kyle2 -i localhost -p 1261 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/22/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/22/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/22/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/23 && rm -f /kyle2/nick/scidb_data/000/23/SciDB-000-23-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/23/SciDB-000-23-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/23/SciDB-000-23-kyle2 -i localhost -p 1262 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/23/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/23/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/23/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/24 && rm -f /kyle2/nick/scidb_data/000/24/SciDB-000-24-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/24/SciDB-000-24-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/24/SciDB-000-24-kyle2 -i localhost -p 1263 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/24/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/24/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/24/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/25 && rm -f /kyle2/nick/scidb_data/000/25/SciDB-000-25-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/25/SciDB-000-25-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/25/SciDB-000-25-kyle2 -i localhost -p 1264 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/25/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/25/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/25/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/26 && rm -f /kyle2/nick/scidb_data/000/26/SciDB-000-26-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/26/SciDB-000-26-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/26/SciDB-000-26-kyle2 -i localhost -p 1265 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/26/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/26/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/26/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/27 && rm -f /kyle2/nick/scidb_data/000/27/SciDB-000-27-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/27/SciDB-000-27-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/27/SciDB-000-27-kyle2 -i localhost -p 1266 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/27/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/27/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/27/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/28 && rm -f /kyle2/nick/scidb_data/000/28/SciDB-000-28-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/28/SciDB-000-28-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/28/SciDB-000-28-kyle2 -i localhost -p 1267 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/28/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/28/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/28/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/29 && rm -f /kyle2/nick/scidb_data/000/29/SciDB-000-29-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/29/SciDB-000-29-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/29/SciDB-000-29-kyle2 -i localhost -p 1268 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/29/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/29/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/29/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/30 && rm -f /kyle2/nick/scidb_data/000/30/SciDB-000-30-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/30/SciDB-000-30-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/30/SciDB-000-30-kyle2 -i localhost -p 1269 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/30/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/30/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/30/scidb-stderr.log ) & (cd /kyle2/nick/scidb_data/000/31 && rm -f /kyle2/nick/scidb_data/000/31/SciDB-000-31-kyle2 && ln -fs /opt/scidb/14.3/bin/scidb /kyle2/nick/scidb_data/000/31/SciDB-000-31-kyle2 && LD_LIBRARY_PATH=/opt/scidb/14.3/lib:/opt/scidb/14.3/lib/scidb/plugins:/opt/scidb/14.3/lib:  exec /kyle2/nick/scidb_data/000/31/SciDB-000-31-kyle2 -i localhost -p 1270 --cache 256 -k -l /kyle2/nick/scidb_data/share/scidb/log4cxx.properties --plugins /kyle2/nick/scidb_data/lib/scidb/plugins --parallel-sort=True --rle-chunk-format=True --enable-catalog-upgrade -s /kyle2/nick/scidb_data/000/31/storage.cfg -c \'host=localhost port=5432 dbname=kyle2 user=kyle2 password=kyle2\' 1> /kyle2/nick/scidb_data/000/31/scidb-stdout.log 2> /kyle2/nick/scidb_data/000/31/scidb-stderr.log ) &'

/opt/scidb/14.3/bin/scidb.py: ERROR: Failed to start SciDB!

There are processes, 56 of them for some reason.

$ scidb.py check_pids kyle2

checking (server 0 (localhost)) 147231 147232 147233 147234 147235 147236 147237 147238 147243 147245 147246 147248 147251 147252 147255 147257 147259 147263 147266 147268 147271 147272 147274 147275 147276 147278 147279 147282 147283 147287 147290 147294 147331 147332 147333 147349 147350 147363 147365 147368 147377 147378 147379 147380 147381 147382 147384 147385 147386 147387 147388 147389 147515 147516 147517 147518...
Found 56 scidb processes

I am getting a network error when trying to run a query.

Nick


#4

It looks like what you are trying to do it start a 32-instance scidb cluster on a single host machine. Can you post your config.ini file? That will clarify some things about what your cluster should look like. Given that, it looks like not all of the instances are able to start up. My guess is that you should end up with at least 64 processes on your host machine.

Also, please attach scidb-*.log files from the coordinator instance. They are in scidb data directory. There may be some error indications there that could be explored a bit further.


#5

I am running scidb on a 32-core machine with 132 GB memory.

My config.ini:

[kyle2]
server-0=localhost,31
install_root=/opt/scidb/14.3
metadata=/kyle2/nick/scidb_data/share/scidb/meta.sql
pluginsdir=/kyle2/nick/scidb_data/lib/scidb/plugins
logconf=/kyle2/nick/scidb_data/share/scidb/log4cxx.properties
db_user=kyle2
db_passwd=kyle2
base-port=1239
base-path=/kyle2/nick/scidb_data
enable-catalog-upgrade=true

From ./000/0/scidb-stderr.log:

2014-10-20 10:48:2 (ppid=147231): Started.
log4cxx: Could not read configuration file [/kyle2/nick/scidb_data/share/scidb/log4cxx.properties].
log4cxx: No appender could be found for logger (scidb.entry).
log4cxx: Please initialize the log4cxx system properly.

Nothing in ./000/0/scidb-stdout.log.

Thank you very much for your help on this.


#6

Also from $ scidb.py status kyle2, I see that 8 of the processes are from last week. Could be related to only 56 processes starting instead of 64.

instance_id | host | port | online_since -------------+-----------+------+---------------------------- 0 | localhost | 1239 | 2014-10-20 10:48:15.32697 1 | localhost | 1240 | 2014-10-13 13:22:06.96962 <--- * FROM LAST WEEK * 2 | localhost | 1241 | 2014-10-20 10:48:15.593795 3 | localhost | 1242 | 2014-10-13 13:22:06.301049<--- * FROM LAST WEEK * 4 | localhost | 1243 | 2014-10-13 21:20:07.016421<--- * FROM LAST WEEK * 5 | localhost | 1244 | 2014-10-20 10:51:04.246696 6 | localhost | 1245 | 2014-10-20 10:51:08.025521 7 | localhost | 1246 | 2014-10-20 10:51:41.004638 8 | localhost | 1247 | 2014-10-20 10:51:21.916644 9 | localhost | 1248 | 2014-10-20 10:51:07.435246 10 | localhost | 1249 | 2014-10-13 13:23:09.077303<--- * FROM LAST WEEK * 11 | localhost | 1250 | 2014-10-20 10:48:13.868524 12 | localhost | 1251 | 2014-10-20 10:48:15.319521 13 | localhost | 1252 | 2014-10-20 10:48:15.188483 14 | localhost | 1253 | 2014-10-20 10:48:15.20609 15 | localhost | 1254 | 2014-10-13 13:22:09.840754<--- * FROM LAST WEEK * 16 | localhost | 1255 | 2014-10-20 10:48:15.052025 17 | localhost | 1256 | 2014-10-20 10:48:14.569861 18 | localhost | 1257 | 2014-10-20 10:48:14.852652 19 | localhost | 1258 | 2014-10-20 10:48:15.700578 20 | localhost | 1259 | 2014-10-13 13:22:25.741177<--- * FROM LAST WEEK * 21 | localhost | 1260 | 2014-10-20 10:48:13.702643 22 | localhost | 1261 | 2014-10-20 10:51:28.059345 23 | localhost | 1262 | 2014-10-20 10:51:21.934968 24 | localhost | 1263 | 2014-10-20 10:52:01.11852 25 | localhost | 1264 | 2014-10-20 10:51:36.404086 26 | localhost | 1265 | 2014-10-13 13:22:17.723002<--- * FROM LAST WEEK * 27 | localhost | 1266 | 2014-10-20 10:51:41.469166 28 | localhost | 1267 | 2014-10-13 13:22:18.269076<--- * FROM LAST WEEK * 29 | localhost | 1268 | 2014-10-20 10:51:56.220547 30 | localhost | 1269 | 2014-10-20 10:51:46.966112 31 | localhost | 1270 | 2014-10-20 10:51:09.679535


#7

The following entries in config.ini seem suspicious:

install_root=/opt/scidb/14.3 # ok
metadata=/kyle2/nick/scidb_data/share/scidb/meta.sql # not sure why this is not pointing to some location under install_root
pluginsdir=/kyle2/nick/scidb_data/lib/scidb/plugins # same as above
logconf=/kyle2/nick/scidb_data/share/scidb/log4cxx.properties # same as above

The error you posted indicated that the log4cxx.properties was not found. Please check the config.ini settings above and make sure they are pointing to the appropriate files/directories.

Also, please try restarting scidb one more time. This time check if any of scidb processes are still running after you issue a stop command. If you still find scidb processes running, you may have to terminate them manually before attempting to start scidb.


#8

OK, I see that the links are wrong. These are the defaults from the configurator on github and I never checked the paths.

I tried startall again linking the /opt/scidb/1413/share folder to where it is supposed to be (/kyle2/nick/scidb_data). Again 56 processes start. From grepping the running processes, I see that 8 of 32 instances only have 1 process associated with them. These are shown by the “scidb.py status” command to be “online” since last week.

Now that the logging properties are in the right place, I get these errors only for the instances that fail to start:


$cat ./000/0/scidb-*.log
2014-10-20 12:51:51 (ppid=157962): Started.

$cat ./000/3/scidb-*.log
2014-10-20 12:51:51 (ppid=157965): Started.
2014-10-20 12:51:52 (ppid=157965): SciDB child (pid=158075) exited with status = 1
2014-10-20 12:51:54 (ppid=157965): SciDB child (pid=158288) exited with status = 1
2014-10-20 12:52:0 (ppid=157965): SciDB child (pid=158365) exited with status = 1
2014-10-20 12:52:12 (ppid=157965): SciDB child (pid=158449) exited with status = 1
2014-10-20 12:52:36 (ppid=157965): SciDB child (pid=158519) exited with status = 1
2014-10-20 12:53:24 (ppid=157965): SciDB child (pid=158599) exited with status = 1
2014-10-20 12:55:0 (ppid=157965): SciDB child (pid=158680) exited with status = 1
2014-10-20 12:56:36 (ppid=157965): SciDB child (pid=158767) exited with status = 1
2014-10-20 12:58:12 (ppid=157965): SciDB child (pid=158869) exited with status = 1

#9

Also, when I run scidb.py stopall, all of the processes are stopped.


#10

Sorry, also found this in the scidb.log on the failed instances

2014-10-20 13:18:29,279 [0x7f4ea9d777c0] [DEBUG]: NetworkManager::run()
2014-10-20 13:18:29,280 [0x7f4ea9d777c0] [ERROR]: Error during SciDB execution: UserException in file: src/network/NetworkManager.cpp function: run line: 142
Error id: scidb::SCIDB_SE_STORAGE::SCIDB_LE_STORAGE_NOT_REGISTERED
Error description: Storage error. Storage is not registered in system catalog.
2014-10-20 13:18:29,280 [0x7f4ea9d777c0] [INFO ]: SciDB instance. SciDB Version: 14.3.7383. Build Type: RelWithDebInfo. Copyright (C) 2008-2013 SciDB, Inc. is exiting.

#11

Ok - how about the pluginsdir setting:

pluginsdir=/kyle2/nick/scidb_data/lib/scidb/plugins
?

Relinking /kyle2/nick/scidb_data/share would not do anything for this setting.

Also, please take a look in /kyle2/nick/scidb_data/000/0/scidb.log. There may be additional error info bits there.

What OS are you running on this host machine?


#12

The plugins directory was in the right place. I am running this on Ubuntu12.04.

I am attaching the scidb.log from the coordinator instance. See post above to see the error I am getting on the instances that fail to start

Thank you very much for your help here
scidb.log (2.15 MB)


#13

Ok. It seems that there is a problem with the scidb storage folder (base path). The errors from the log indicate that scidb instances cannot get to the original storage files they used before the crash.

You can do a couple of things now:

  1. Ensure that the base path setting is correct and is pointing to a valid folder; there may be data corruption in the files of that folder (not sure how to check for this); restart scidb after checking
  2. Re-initialize scidb by running scidb.py initall -v command (warning: this will clear out all previous arrays that were in the database)

Re-initializing the database should be done after stopping all scidb instances. If you decided to re-initialize the database, please post any errors produced by the verbose switch. If re-initialization succeeds, proceed to starting scidb with the startall command.

Let us know how you get on.


#14

I think I have id’d some of the corrupt files in the datastore because 1)They are smaller than the rest of the files in the folder by an order of magnitude and 2) the integer prefix on the *.data or *.data.fl only exist on those instances that fail to start.

Here are some examples:
./000/1/datastore/52863.data.fl
./000/1/datastore/52863.data

If I were to delete these files is there a chance to avoid re-initializing the arrays?


#15

Smaller file sizes may not necessarily indicate data corruption. However, deleting files in the datastores directory will probably result in bad data in arrays (if scidb starts correctly, of course).


#16

Up and running again with a more sane config.ini. Unfortunately had to re-init all array data. I didn’t realize how import it is to set the max-memory-limit. You can easily corrupt all of your data. You may want to emphasize that more in your configurator script.

Thanks again for your help.

Nick


#17

max-memory-limit should NOT cause data corruption even if it results in shutting down some instances.
So, the possibilities include: we have a bug, you are using volatile memory for your data files (RAM disk, tmpfs, etc.), the configuration somehow causes different instances to share the same data directories (several soft links point to the same loaction, etc.). If you think it is our bug (i.e. the issue comes back), please let us know and please execute 'scidb.py dbginfo-lt ’ and provide the tar file generated by that command from /000/0.
Thanks!