Problem with fetch using arrow in SciDB 19.11

Hello.
I’m having a problem with fetch() function when use_arrow = True in SciDB 19.11. The same coide worked correctly with 18.1.

Here are the packages versions:

SciDB Version: 19.11.4
Build Type: RelWithDebInfo
Commit: 9086943
Copyright © 2008-2019 SciDB, Inc.

scidb-py (19.3.3)
pyarrow (0.14.0)

PostgreSQL 9.4.26 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit

Scidb-py was installed using pip and it sais that it is the last version.

Here is the array schema:

AFL% list(‘arrays’);
{No} name,uaid,aid,schema,availability,temporary,namespace,distribution,etcomp
{0} ‘KOMPSAT_10_10’,1,17,‘KOMPSAT_10_10elevation:uint16 [date=0::0:1; utm_zone=-60:60:0:1; latitude=-2147483648::0:1000; longitude=-2147483648:*:0:1000]’,true,false,‘public’,‘hashed’,‘none’

And I’m trying o select data using the following commands:

import scidbpy
db = scidbpy.connect()
db.load_library(‘accelerated_io_tools’)
array = db.arrays[‘KOMPSAT_10_10’]
aaa = db.between(array, 20190715, 15, 1107488, 5545535, 20190715, 15, 1107498, 5545545)

If I just fetch the data it works correctly

aaa.fetch()
/usr/lib/python3.6/site-packages/scidbpy/schema.py:747: UserWarning: 1 type(s) promoted for null support. Precision loss may occur
self._promo_warning()
date utm_zone latitude longitude elevation
0 20190715 15 1107488 5545535 2109.0
1 20190715 15 1107488 5545536 1895.0
2 20190715 15 1107488 5545537 2049.0
3 20190715 15 1107488 5545538 2330.0
4 20190715 15 1107488 5545539 2285.0
… … … … … …
116 20190715 15 1107498 5545541 2444.0
117 20190715 15 1107498 5545542 2263.0
118 20190715 15 1107498 5545543 2270.0
119 20190715 15 1107498 5545544 3204.0
120 20190715 15 1107498 5545545 3262.0

[121 rows x 5 columns]

But if I set use_arrow to True it fails

aaa.fetch(use_arrow = True)
Traceback (most recent call last):
File “”, line 1, in
File “/usr/lib/python3.6/site-packages/scidbpy/db.py”, line 887, in fetch
**kwargs)
File “/usr/lib/python3.6/site-packages/scidbpy/db.py”, line 434, in iquery
pyarrow.BufferReader(buf)).read_pandas()
File “/usr/lib64/python3.6/site-packages/pyarrow/ipc.py”, line 46, in read_pandas
table = self.read_all()
File “pyarrow/ipc.pxi”, line 295, in pyarrow.lib._CRecordBatchReader.read_all
File “pyarrow/error.pxi”, line 87, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Array length did not match record batch length

Could you please tell what could cause this problem. In shim config file aio is set to 1.

Thank you.

I think the problem is the pyarrow version, it has to be before 0.14, so 0.13 should work. See here

Thank you very much. It helped right away.