Invalid chunking position error


#1

now i am trying to load 1D array, its not very large 100MB of data
getting invalid chunk position
Error in file ‘load_data.afl’ near line 1
UserException in file: src/query/ops/input/InputArray.cpp function: end line: 196
Error id: scidb::SCIDB_SE_IMPORT_ERROR::SCIDB_LE_FILE_IMPORT_FAILED
Error description: Import error. Import from file ‘/home/scidb/current_speed_19790602_00_wnp.scidb’ (instance 0) to array ‘current_speed_flat’ failed at line 1000003, column 8, offset 49117532, value=‘1000000’: Invalid chunk position.

the array defintion
scidb@ubuntu:~/my_scidb_scripts/acaf_load$ more ddl/current_speed_flat.afl
CREATE ARRAY current_speed_flat <lat:int64, lon:int64, level:int64, time:string, speed:double> [i=0 :*,100000000,0];

What does invalid chunk position means?
if i reduce the file size from 2.5M rows to 1M rows it loads successfully.
if i split the file into 3 chunks 1M rows each each chunk loads successfully.
which gives?


#2

Hi,
Sorry about the problem.
It’s hard to tell without the query. Can you give us the exact query that is used to load the file?
If this is text file, there is a newer load prototype at github.com/paradigm4/load_tools. It usually gives faster performance and comes with some useful tools for string processing. May work better for this case.


#3

this was the command
LOAD current_speed_flat
FROM
’/home/scidb/my_scidb_scripts/curent_speed.scidb’

the scidb is 14.8 running on ubuntu with 4GB ram, its a vm

this is my config.ini
cluster]
server-0=172.16.85.189,1
install_root=/opt/scidb/14.8
metadata=/opt/scidb/14.8/share/scidb/meta.sql
pluginsdir=/opt/scidb/14.8/lib/scidb/plugins
logconf=/opt/scidb/14.8/share/scidb/log4cxx.properties
db_user=pguser
db_passwd=xxx
base-port=1239
base-path=/home/scidb/scidb_data
redundancy=0
mem-array-threshold=169
smgr-cache-size=169
merge-sort-buffer=85
network-buffer=85
execution-threads=1
result-prefetch-threads=1
result-prefetch-queue-size=1
operator-threads=1
ata-dir-prefix-0-0=/datadisk0/cluster.0.0
data-dir-prefix-0-1=/datadisk1/cluster.0.1


#4

in release notes to 14.12 there are notes indicating there was an issue with load operator, but without elaboration.
scidb.org/forum/viewtopic.php?f=14&t=1526

Could it be fixed in 14.12?


#5

Hi All,

Was the cause of problem found ? Kindly inform me of any possible issues.
I am also getting import error “invalid chunk position”. i am trying to load a csv file to scidb using loadcsv.py script.
loadcsv.py -i './file1.csv' -a array1 -n 1 -t NNsNsN

even if i exclusively define the schema in loadcsv argument i am getting the error. i dont know if i am missing something in script arguments. there are lot of them.

    loadcsv.py -i '/home/user/file1.csv' -n 1 -t NNsNsN -a array1
    Retrieving load array schema from SciDB.
    Computing type-pattern from load schema.
    Getting SciDB configuration information.
    This SciDB installation has 17 instance(s).
    Creating CSV fragment FIFOs.
    Creating DLF fragment FIFOs.
    Starting CSV splitting process.
    Starting CSV distribution and conversion processes.
    Converter is tsv2scidb
    Loading data into "array1" array (may take a while for large input files). 1-D load only since no target       array name was provided.

  ##### ERROR ##################
  Load failed.
  UserException in file: src/query/ops/input/InputArray.cpp function: moveNext line: 371
  Error id: scidb::SCIDB_SE_IMPORT_ERROR::SCIDB_LE_FILE_IMPORT_FAILED
  Error description: Import error. Import from file 'file1.csv.IVOCbT.dlf' (instance 0) to array 'array1' failed at line 0, column 2, offset 2, value='': Invalid chunk position.

 ##############################

  Removing CSV fragmemt FIFOs.
  Removing DLF fragment FIFOs.
  Failure: Error Encountered.

#6

please reach out me directly to discuss - my email senya72@gmail.com