Scidb cluster wont start


#1

i have shutdown a normally working scidb cluster (community edition 14.12, 6 nodes)
now when i am trying to start it i up i am getting this015-11-04 00:07:00,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 19
2015-11-04 00:07:00,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 20
2015-11-04 00:07:00,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 21
2015-11-04 00:07:00,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 22
2015-11-04 00:07:00,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 23
2015-11-04 00:07:00,604 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 16. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:00,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 17. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:00,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 18. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:00,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 19. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:00,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 20. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:00,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 21. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:00,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 22. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:00,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 23. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:05,602 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 16
2015-11-04 00:07:05,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 17
2015-11-04 00:07:05,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 18
2015-11-04 00:07:05,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 19
2015-11-04 00:07:05,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 20
2015-11-04 00:07:05,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 21
2015-11-04 00:07:05,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 22
2015-11-04 00:07:05,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 23
2015-11-04 00:07:05,604 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 16. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:05,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 17. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:05,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 18. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:05,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 19. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:05,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 20. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:05,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 21. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:05,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 22. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:05,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 23. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:10,602 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 16
2015-11-04 00:07:10,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 17
2015-11-04 00:07:10,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 18
2015-11-04 00:07:10,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 19
2015-11-04 00:07:10,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 20
2015-11-04 00:07:10,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 21
2015-11-04 00:07:10,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 22
2015-11-04 00:07:10,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 23
2015-11-04 00:07:10,604 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 16. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:10,604 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 17. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:10,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 18. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:10,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 19. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:10,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 20. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:10,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 21. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:10,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 22. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:10,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 23. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:15,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 16
2015-11-04 00:07:15,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 17
2015-11-04 00:07:15,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 18
2015-11-04 00:07:15,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 19
2015-11-04 00:07:15,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 20
2015-11-04 00:07:15,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 21
2015-11-04 00:07:15,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 22
2015-11-04 00:07:15,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 23
2015-11-04 00:07:15,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 16. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:15,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 17. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:15,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 18. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:15,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 19. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:15,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 20. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:15,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 21. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:15,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 22. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:15,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 23. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:20,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 16
2015-11-04 00:07:20,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 17
2015-11-04 00:07:20,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 18
2015-11-04 00:07:20,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 19
2015-11-04 00:07:20,604 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 20
2015-11-04 00:07:20,604 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 21
2015-11-04 00:07:20,604 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 22
2015-11-04 00:07:20,604 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 23
2015-11-04 00:07:20,605 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 16. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:20,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 17. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:20,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 18. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:20,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 19. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:20,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 20. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:20,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 21. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:20,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 22. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:20,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 23. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:25,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 16
2015-11-04 00:07:25,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 17
2015-11-04 00:07:25,603 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 18
2015-11-04 00:07:25,604 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 19
2015-11-04 00:07:25,604 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 20
2015-11-04 00:07:25,604 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 21
2015-11-04 00:07:25,604 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 22
2015-11-04 00:07:25,604 [0x7f6830c0d800] [ERROR]: Network error in handleSendMessage #32(‘Broken pipe’), instance 23
2015-11-04 00:07:25,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 16. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:25,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 17. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:25,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 18. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:25,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 19. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:25,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 20. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:25,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 21. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:25,606 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 22. Error:107(‘Transport endpoint is not connected’)
2015-11-04 00:07:25,607 [0x7f6830c0d800] [ERROR]: Could not get the remote IP from connected socket to/frominstance 23. Error:107(‘Transport endpoint is not connected’)


#2

i have checked scidb data directories on each node and noticed one node is missing them. so maybe a case of data corruption. i wish a scidb.log file helped diagnose that instead of cryptic low-level messages


#3

Have you solved this problem? It happens to me too!