Interface problem with large array redimension


#1

Hello.

I’m trying to redimension large array using Python and ScidbPY using following syntax

db_obj.input(’<i_2:int64, x:float,y:float,z:float>[j]’,
** upload_data=db_obj.arrays[array_name2].fetch(as_dataframe=False)**
** ).apply(‘t’, ‘1’, ‘x1’, ‘int64(x - 0.5)’, ‘y1’, ‘int64(y - 0.5)’**
** ).redimension(db_obj.arrays[merged_array_name]**
** ).insert(merged_array_name)**

Input 1D array size is 431.921.262
Redimensioned array should be 3D.
SciDB version is 18.1 and I’m using Python3.6.
OS: CentOS7 with enough RAM and disk space.

New array has the following schema:

<z:float>[t=0:*:0:1;x1=0:*:0:1000;y1=0:*:0:1000"]

Redimension part is going fine until insert() is called. Python program crashes with the following message:

Traceback (most recent call last):
** File “/usr/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 600, in urlopen**
** chunked=chunked)**
** File “/usr/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 354, in _make_request**
** conn.request(method, url, httplib_request_kw)
** File “/usr/lib64/python3.6/http/client.py”, line 1239, in request**
** self._send_request(method, url, body, headers, encode_chunked)**
** File “/usr/lib64/python3.6/http/client.py”, line 1285, in _send_request**
** self.endheaders(body, encode_chunked=encode_chunked)**
** File “/usr/lib64/python3.6/http/client.py”, line 1234, in endheaders**
** self._send_output(message_body, encode_chunked=encode_chunked)**
** File “/usr/lib64/python3.6/http/client.py”, line 1065, in _send_output**
** self.send(chunk)**
** File “/usr/lib64/python3.6/http/client.py”, line 986, in send**
** self.sock.sendall(data)**
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
** File “/usr/lib/python3.6/site-packages/requests/adapters.py”, line 445, in send**
** timeout=timeout**
** File “/usr/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 638, in urlopen**
** _stacktrace=sys.exc_info()[2])**
** File “/usr/lib/python3.6/site-packages/urllib3/util/retry.py”, line 367, in increment**
** raise six.reraise(type(error), error, _stacktrace)**
** File “/usr/lib/python3.6/site-packages/urllib3/packages/six.py”, line 685, in reraise**
** raise value.with_traceback(tb)**
** File “/usr/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 600, in urlopen**
** chunked=chunked)**
** File “/usr/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 354, in _make_request**
** conn.request(method, url, httplib_request_kw)
** File “/usr/lib64/python3.6/http/client.py”, line 1239, in request**
** self._send_request(method, url, body, headers, encode_chunked)**
** File “/usr/lib64/python3.6/http/client.py”, line 1285, in _send_request**
** self.endheaders(body, encode_chunked=encode_chunked)**
** File “/usr/lib64/python3.6/http/client.py”, line 1234, in endheaders**
** self._send_output(message_body, encode_chunked=encode_chunked)**
** File “/usr/lib64/python3.6/http/client.py”, line 1065, in _send_output**
** self.send(chunk)**
** File “/usr/lib64/python3.6/http/client.py”, line 986, in send**
** self.sock.sendall(data)**
urllib3.exceptions.ProtocolError: (‘Connection aborted.’, ConnectionResetError(104, ‘Connection reset by peer’))

After this, connection by Python cannot be established until shim is restarted. I’ve tried it many times, and the result is always the crash of shim interface. On small arrays the same code works fine.

Please let me know what I can do, may be some code modifications or shim settings?
Than you in advance.