13.9 Woes: Failed to start SciDB!


#1

I’ve been searching around this forum and it looks like I wasn’t the only person having this problem… I just upgraded from 13.6 to 13.9 and now I’m getting an error while trying to start SciDB:

# scistart checking (server 0 (127.0.0.1) local instance 0) ... checking (server 0 (127.0.0.1) local instance 1) ... checking (server 0 (127.0.0.1) local instance 2) ... checking (server 0 (127.0.0.1) local instance 3) ... Found 0 scidb processes start(server 0 (127.0.0.1) local instance 0) Starting SciDB server. start(server 0 (127.0.0.1) local instance 1) Starting SciDB server. start(server 0 (127.0.0.1) local instance 2) Starting SciDB server. start(server 0 (127.0.0.1) local instance 3) Starting SciDB server. Failed to start SciDB!

Like other users have mentioned, when I check the status of the servers it says they’re up, but when I try to open iquery I get this:

# iq /opt/scidb/13.9/bin/iquery: error while loading shared libraries: libscidbclient.so: cannot open shared object file: No such file or directory

I tried renaming localhost in my config to 127.0.0.1, but that didn’t work. I checked all my environment variables and they’re correct, but that didn’t help. What should I try next?


#2

Try

# export LD_LIBRARY_PATH=/opt/scidb/13.9/lib:$LD_LIBRARY_PATH
# iq

#3

[quote=“apoliakov”]Try

# export LD_LIBRARY_PATH=/opt/scidb/13.9/lib:$LD_LIBRARY_PATH
# iq
[/code][/quote]
Didn't do anything; the environment variables are definitely correct.

Also, I double-checked, and libscidbclient.so is definitely in /opt/scidb/13.9/lib, so I don't know why it's not finding it.

EDIT: So I tried running scidb.py startall db as the scidb user and it hung on trying to create the final server for four minutes before I stopped it. When I did, it spat out this, which hopefully will help:

[code]start(server 0 (localhost) local instance 3)
Starting SciDB server.
^CTraceback (most recent call last):
  File "/opt/scidb/13.9/bin/scidb.py", line 1286, in <module>
    while not check_scidb_ready(masterSrv,0):
  File "/opt/scidb/13.9/bin/scidb.py", line 727, in check_scidb_ready
    useShell=True)
  File "/opt/scidb/13.9/bin/scidb.py", line 446, in executeIt
    p.wait()
  File "/usr/lib64/python2.6/subprocess.py", line 1302, in wait
    pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
  File "/usr/lib64/python2.6/subprocess.py", line 462, in _eintr_retry_call
    return func(*args)
KeyboardInterrupt

#4

Are you sure that you have set up password-less ssh correctly? Just to be sure, try running these commands:

ssh-copy-id -i ~/.ssh/id_rsa.pub scidb@localhost ssh-copy-id -i ~/.ssh/id_rsa.pub scidb@0.0.0.0 ssh-copy-id -i ~/.ssh/id_rsa.pub scidb@127.0.0.1

You could also try to update your config.ini, replacing “localhost” or “127.0.0.1” with the actual IP address of your SciDB server. Another thing to check is to make sure that you have write access to your SciDB storage folder.


#5

[quote=“smarcus”]Are you sure that you have set up password-less ssh correctly? Just to be sure, try running these commands:

ssh-copy-id -i ~/.ssh/id_rsa.pub scidb@localhost ssh-copy-id -i ~/.ssh/id_rsa.pub scidb@0.0.0.0 ssh-copy-id -i ~/.ssh/id_rsa.pub scidb@127.0.0.1

You could also try to update your config.ini, replacing “localhost” or “127.0.0.1” with the actual IP address of your SciDB server. Another thing to check is to make sure that you have write access to your SciDB storage folder.[/quote]

Thank you for the reply, but unfortunately I’m still stuck. I just double checked with the password-less ssh and those all were still correct (I should reiterate that 13.6 did work for me perfectly; I only ran into trouble upon upgrading to 13.9), and changing the IP unfortunately didn’t help. The permissions of the folder are correct for all the relevant users.


I did some poking around with scidb.py, and I isolated the issue in the script, but I am unsure why it is occuring. After scidb.py starts the instances, it then tries to run check_scidb_ready(). During that, it attempts to call the following command:

/opt/scidb/13.9/bin/iquery -c localhost -p 1239 -naq “list(‘queries’)” 1>/dev/null 2>/dev/null

On my current SciDB setup, the return value for that call is always 1, not 0 (both when it is called inside of scidb.py and if I just call that function by itself outside of the script). Because of that, the loop continues until it hits 30, at which point it states that SciDB failed to start. I do not know WHY this is returning 1, but I was hoping this might shed some light on the issue.


#6

I grabbed the log from the coordinator in case that could shed light on this any better:

[code]2013-11-19 15:12:25,581 [0x7f4eb6da97e0] [INFO ]: Network manager is started on localhost:1239 instance #0
2013-11-19 15:12:25,581 [0x7f4eb6da97e0] [DEBUG]: Start connection accepting and async message exchanging
2013-11-19 15:14:08,559 [0x7f4eb6da97e0] [DEBUG]: Waiting for the first message
2013-11-19 15:14:08,560 [0x7f4eb6da97e0] [DEBUG]: Connection started from CLIENT (127.0.0.1)
2013-11-19 15:14:08,560 [0x7f4eb6878700] [DEBUG]: Generated queryID: instanceID=0, time=1384895648, clock=0, nextID=0, queryID=1100896523424
2013-11-19 15:14:08,560 [0x7f4eb6878700] [DEBUG]: Query 1100896523424 is not found
2013-11-19 15:14:08,560 [0x7f4eb6878700] [DEBUG]: Allocating query (1100896523424)
2013-11-19 15:14:08,560 [0x7f4eb6878700] [DEBUG]: Number of allocated queries = 1
2013-11-19 15:14:08,562 [0x7f4eb6878700] [DEBUG]: Initialized query (1100896523424)
2013-11-19 15:14:08,562 [0x7f4eb6878700] [DEBUG]: Parsing query(1100896523424): list(‘queries’)
2013-11-19 15:14:08,562 [0x7f4eb6878700] [DEBUG]: Inferred schema for operator list: not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
2013-11-19 15:14:08,562 [0x7f4eb6878700] [DEBUG]: Acquired 0 array locks for query 1100896523424
2013-11-19 15:14:08,562 [0x7f4eb6878700] [DEBUG]: Inferred schema for operator list: not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
2013-11-19 15:14:08,562 [0x7f4eb6878700] [DEBUG]: Inferred schema for operator list: not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
2013-11-19 15:14:08,562 [0x7f4eb6878700] [DEBUG]: The query is prepared
2013-11-19 15:14:08,562 [0x7f4eb6878700] [DEBUG]: Prepared query(1100896523424): list(‘queries’)
2013-11-19 15:14:08,562 [0x7f4eb6878700] [DEBUG]: The result preparation of query is sent to the client
2013-11-19 15:14:08,563 [0x7f4eb6979700] [DEBUG]: Creating Habilis optimizer instance
2013-11-19 15:14:08,563 [0x7f4eb6979700] [DEBUG]: Query is optimized
2013-11-19 15:14:08,563 [0x7f4eb6979700] [DEBUG]: The physical plan is detected as DML
2013-11-19 15:14:08,563 [0x7f4eb6979700] [DEBUG]:
[pPlan]:

[pNode] physicalList agg 0 ddl 0 tile 0 children 0
schema not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
output full chunks: yes
changes dioutibution: no
props sgm 1 sgo 1
diout loca instance 0
bound start {0} end {0} density 1 cells 1 chunks 1 est_bytes 161

2013-11-19 15:14:08,563 [0x7f4eb6979700] [DEBUG]: (Pre)Single executing queryID: 1100896523424
2013-11-19 15:14:08,563 [0x7f4eb6979700] [DEBUG]: Query is serialized: [pPlan]:

[pNode] physicalList agg 0 ddl 0 tile 0 children 0
schema not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
output full chunks: yes
changes dioutibution: no
props sgm 1 sgo 1
diout loca instance 0
bound start {0} end {0} density 1 cells 1 chunks 1 est_bytes 161

2013-11-19 15:14:08,563 [0x7f4eb6979700] [DEBUG]: Prepare physical plan was sent out
2013-11-19 15:14:08,563 [0x7f4eb6979700] [DEBUG]: Waiting confirmation about preparing physical plan in queryID from 3 instances
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896523424
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896523424
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896523424
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:08,564 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:10,582 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:10,582 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:10,582 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:10,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:11,565 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:11,565 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:11,565 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:11,565 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:11,565 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:11,565 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:11,565 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:11,565 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:14,565 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:14,565 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:15,582 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:15,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:15,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:15,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:15,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:15,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:17,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:17,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:17,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:17,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:17,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:17,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:18,563 [0x7f4eb6979700] [DEBUG]: Query::done: queryID=1100896523424, _commitState=0, errorCode=120
2013-11-19 15:14:18,564 [0x7f4eb6979700] [ERROR]: executeClientQuery failed to complete: SystemException in file: src/network/NetworkManager.cpp function: handleConnectionError line: 854
Error id: scidb::SCIDB_SE_NETWORK::SCIDB_LE_CONNECTION_ERROR2
Error description: Network error. Connection error while sending.
Failed query id: 1100896523424
2013-11-19 15:14:18,564 [0x7f4eb6979700] [DEBUG]: Query (1100896523424) is being aborted
2013-11-19 15:14:18,564 [0x7f4eb6979700] [ERROR]: Query (1100896523424) error handlers (1) are being executed
2013-11-19 15:14:18,564 [0x7f4eb6979700] [DEBUG]: Broadcast ABORT message to all instances for query 1100896523424
2013-11-19 15:14:18,564 [0x7f4eb6979700] [DEBUG]: Deallocating query (1100896523424)
2013-11-19 15:14:18,564 [0x7f4eb6979700] [DEBUG]: Releasing locks for query 1100896523424
2013-11-19 15:14:18,564 [0x7f4eb6979700] [DEBUG]: SystemCatalog::deleteArrayLocks instanceId = 0 queryId = 1100896523424
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896523424
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [DEBUG]: Query 1100896523424 is not found
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896523424
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [DEBUG]: Query 1100896523424 is not found
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896523424
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [DEBUG]: Query 1100896523424 is not found
2013-11-19 15:14:18,564 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:18,566 [0x7f4eb6da97e0] [DEBUG]: Disconnected
2013-11-19 15:14:20,564 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:20,564 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:20,565 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:20,565 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:20,565 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:20,565 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:20,582 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:20,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:20,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:20,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:20,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:20,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:23,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:23,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:23,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:23,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:23,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:23,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:25,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:25,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:25,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:25,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:25,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:25,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:26,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:26,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:26,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:26,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:26,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:26,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:30,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:30,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:30,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:30,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:30,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:30,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:30,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:30,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:30,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:30,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:30,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:30,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:35,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:40,583 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:45,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:45,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:45,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:45,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:45,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:45,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:45,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:45,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:45,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:45,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:45,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:45,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:50,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:50,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:50,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:50,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:50,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:50,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:50,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:50,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:50,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:50,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:50,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:50,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:14:55,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:14:55,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:14:55,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:14:55,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:14:55,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:14:55,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:14:55,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:55,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:14:55,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:55,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:14:55,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:14:55,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:00,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:00,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:00,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:00,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:00,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:00,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:00,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:00,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:00,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:00,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:00,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:00,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:05,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:05,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:05,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:05,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:05,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:05,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:05,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:05,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:05,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:05,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:05,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:05,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:09,021 [0x7f4eb6da97e0] [DEBUG]: Waiting for the first message
2013-11-19 15:15:09,021 [0x7f4eb6da97e0] [DEBUG]: Connection started from CLIENT (127.0.0.1)
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Generated queryID: instanceID=0, time=1384895709, clock=30000, nextID=1, queryID=1100896553486
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Query 1100896553486 is not found
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Allocating query (1100896553486)
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Number of allocated queries = 1
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Initialized query (1100896553486)
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Parsing query(1100896553486): list(‘queries’)
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Inferred schema for operator list: not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Acquired 0 array locks for query 1100896553486
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Inferred schema for operator list: not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Inferred schema for operator list: not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: The query is prepared
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: Prepared query(1100896553486): list(‘queries’)
2013-11-19 15:15:09,021 [0x7f4eb6b7b700] [DEBUG]: The result preparation of query is sent to the client
2013-11-19 15:15:09,022 [0x7f4eb6777700] [DEBUG]: Creating Habilis optimizer instance
2013-11-19 15:15:09,022 [0x7f4eb6777700] [DEBUG]: Query is optimized
2013-11-19 15:15:09,022 [0x7f4eb6777700] [DEBUG]: The physical plan is detected as DML
2013-11-19 15:15:09,022 [0x7f4eb6777700] [DEBUG]:
[pPlan]:

[pNode] physicalList agg 0 ddl 0 tile 0 children 0
schema not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
output full chunks: yes
changes dioutibution: no
props sgm 1 sgo 1
diout loca instance 0
bound start {0} end {0} density 1 cells 1 chunks 1 est_bytes 161

2013-11-19 15:15:09,022 [0x7f4eb6777700] [DEBUG]: (Pre)Single executing queryID: 1100896553486
2013-11-19 15:15:09,022 [0x7f4eb6777700] [DEBUG]: Query is serialized: [pPlan]:

[pNode] physicalList agg 0 ddl 0 tile 0 children 0
schema not empty queriesname:string,query_id:int64,creation_time:datetime,error_code:int32,error:string,idle:bool [No=0:0,1,0]
output full chunks: yes
changes dioutibution: no
props sgm 1 sgo 1
diout loca instance 0
bound start {0} end {0} density 1 cells 1 chunks 1 est_bytes 161

2013-11-19 15:15:09,022 [0x7f4eb6777700] [DEBUG]: Prepare physical plan was sent out
2013-11-19 15:15:09,022 [0x7f4eb6777700] [DEBUG]: Waiting confirmation about preparing physical plan in queryID from 3 instances
2013-11-19 15:15:09,022 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:09,022 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896553486
2013-11-19 15:15:09,022 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:09,022 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:09,022 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896553486
2013-11-19 15:15:09,022 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:09,022 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:09,022 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896553486
2013-11-19 15:15:09,022 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:09,023 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:09,023 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:09,023 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:09,023 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:09,023 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:09,023 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:10,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:10,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:10,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:10,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:10,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:10,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:12,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:12,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:12,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:12,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:12,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:12,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:15,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:15,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:15,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:15,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:15,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:15,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:15,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:15,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:15,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:15,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:15,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:15,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:19,022 [0x7f4eb6777700] [DEBUG]: Query::done: queryID=1100896553486, _commitState=0, errorCode=120
2013-11-19 15:15:19,022 [0x7f4eb6777700] [ERROR]: executeClientQuery failed to complete: SystemException in file: src/network/NetworkManager.cpp function: handleConnectionError line: 854
Error id: scidb::SCIDB_SE_NETWORK::SCIDB_LE_CONNECTION_ERROR2
Error description: Network error. Connection error while sending.
Failed query id: 1100896553486
2013-11-19 15:15:19,022 [0x7f4eb6777700] [DEBUG]: Query (1100896553486) is being aborted
2013-11-19 15:15:19,022 [0x7f4eb6777700] [ERROR]: Query (1100896553486) error handlers (1) are being executed
2013-11-19 15:15:19,022 [0x7f4eb6777700] [DEBUG]: Broadcast ABORT message to all instances for query 1100896553486
2013-11-19 15:15:19,022 [0x7f4eb6777700] [DEBUG]: Deallocating query (1100896553486)
2013-11-19 15:15:19,023 [0x7f4eb6777700] [DEBUG]: Releasing locks for query 1100896553486
2013-11-19 15:15:19,023 [0x7f4eb6777700] [DEBUG]: SystemCatalog::deleteArrayLocks instanceId = 0 queryId = 1100896553486
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896553486
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [DEBUG]: Query 1100896553486 is not found
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896553486
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [DEBUG]: Query 1100896553486 is not found
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [ERROR]: NetworkManager::handleConnectionError: Conection error in query 1100896553486
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [DEBUG]: Query 1100896553486 is not found
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:19,023 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:19,025 [0x7f4eb6da97e0] [DEBUG]: Disconnected
2013-11-19 15:15:20,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:20,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:20,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:20,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:20,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:20,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:22,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:22,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:22,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:22,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:22,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:22,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:25,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:25,583 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:25,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:30,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:30,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:30,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:30,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:30,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:30,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:30,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:30,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:30,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:30,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:30,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:30,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:35,583 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:35,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:35,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:35,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:35,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:35,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:35,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:35,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:35,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:35,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:35,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:35,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:40,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:40,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:40,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:45,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:50,584 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:15:55,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:15:55,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:15:55,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:15:55,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:15:55,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:15:55,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:15:55,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:55,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:15:55,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:55,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:15:55,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:15:55,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:16:00,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:16:00,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:16:00,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:16:00,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:16:00,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:16:00,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:16:00,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:00,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:16:00,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:00,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:16:00,586 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:00,586 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:16:05,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:05,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:16:10,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:16:10,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:16:10,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:16:10,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:16:10,585 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:16:10,585 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:16:10,587 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:10,587 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:16:10,587 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:10,587 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:16:10,587 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:10,587 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:16:15,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:16:15,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:16:15,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:16:15,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:16:15,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:16:15,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:16:15,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:15,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:16:15,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:15,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:16:15,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:15,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:16:20,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:16:20,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:16:20,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:16:20,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:16:20,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:16:20,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:16:20,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:20,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:16:20,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:20,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:16:20,585 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:20,585 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242
2013-11-19 15:16:25,584 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 1
2013-11-19 15:16:25,584 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 1
2013-11-19 15:16:25,586 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 2
2013-11-19 15:16:25,586 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 2
2013-11-19 15:16:25,586 [0x7f4eb6da97e0] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 3
2013-11-19 15:16:25,586 [0x7f4eb6da97e0] [DEBUG]: Recovering connection to instance 3
2013-11-19 15:16:25,587 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 1. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:25,587 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 1, localhost:1240
2013-11-19 15:16:25,587 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 2. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:25,587 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 2, localhost:1241
2013-11-19 15:16:25,587 [0x7f4eb6da97e0] [ERROR]: Could not get the remote IP from connected socket to/frominstance 3. Error:107(‘Transport endpoint is not connected’)
2013-11-19 15:16:25,587 [0x7f4eb6da97e0] [DEBUG]: Connected to instance 3, localhost:1242[/code]


#7

Hmmm…

Try using 0.0.0.0 instead of 127.0.0.1 in the config file? What else does it say by the way?

It looks like there are multiple problems:

  1. your iquery cannot find the .so file.
  2. your scidb instances keep restarting

The startup script just tried to do a simple stupid list(‘instances’) query to check to see if the system came up. It exits with 1 because that query failed, just like it fails when you try to do list(‘instances’) manually.

Also - check for path collisions from 13.6 to 13.9. Are you, by chance, using an old copy of iquery that belongs to 13.6 for example?


#8

[quote=“apoliakov”]Hmmm…

Try using 0.0.0.0 instead of 127.0.0.1 in the config file? What else does it say by the way?

It looks like there are multiple problems:

  1. your iquery cannot find the .so file.
  2. your scidb instances keep restarting

The startup script just tried to do a simple stupid list(‘instances’) query to check to see if the system came up. It exits with 1 because that query failed, just like it fails when you try to do list(‘instances’) manually.

Also - check for path collisions from 13.6 to 13.9. Are you, by chance, using an old copy of iquery that belongs to 13.6 for example?[/quote]
0.0.0.0 didn’t change it, unfortunately. What else were you wanting from the log, exactly?

I removed the 13.6 folder entirely prior to installing the update, so it can’t be using an old version of iquery.


#9

Your config.ini file?


#10

I figured it out; the environment variables on the SciDB user weren’t sticking despite being in ~/.bash_profile, such that “sudo -u scidb” no longer works (I have literally no idea what changed). I had to change all my scripts to su - scidb -c “command” and set the environment variables in ~/.bashrc, and now it works again. Hopefully I’ll figure out a way to set the environment variables more permanantly on the scidb user so I can go back to the much more useful “sudo -u scidb” format, but until then at least it works now. Sorry for sending us on such a wild goose chase; hopefully if this happens to someone else this will help.


#11

Roger that! Glad things are working.


#12

Can you tell me what environment variables need to be set for the scidb user? I am also getting the “SciDB Failed to Start!” message when I try to run scidb.py startall db


#13

Can you tell me which environment variables need to be set for the scidb user? I am also getting a “SciDB failed to start” message when I try to run scidb.py startall db as the scidb user.