Load Error: Number of errors exceeds threshold


#1

Hi, there! I am Cherry. Recently I load data with the function called load(). There are some mistakes.

iquery -aq "load(sppLines,'/home/scidb/sppLines.scidb');" iquery -aq "load(sppParams,'/home/scidb/sppParams.scidb');" iquery -aq "load(zooNoSpec,'/home/scidb/ZooNoSpec.scidb');"

UserException in file: src/query/ops/input/InputArray.cpp function: moveNext line: 371 Error id: scidb::SCIDB_SE_IMPORT_ERROR::SCIDB_LE_FILE_IMPORT_FAILED Error description: Import error. Import from file '/home/scidb/sppLines.scidb' (instance 0) to array 'sppLines' failed at line 1, column 347, offset 352, value='1.4728967': Number of errors exceeds threshold. UserException in file: src/query/ops/input/InputArray.cpp function: moveNext line: 371 Error id: scidb::SCIDB_SE_IMPORT_ERROR::SCIDB_LE_FILE_IMPORT_FAILED Error description: Import error. Import from file '/home/scidb/sppParams.scidb' (instance 0) to array 'sppParams' failed at line 1, column 689, offset 694, value='NA': Number of errors exceeds threshold. UserException in file: src/query/ops/input/InputArray.cpp function: moveNext line: 371 Error id: scidb::SCIDB_SE_IMPORT_ERROR::SCIDB_LE_FILE_IMPORT_FAILED Error description: Import error. Import from file '/home/scidb/ZooNoSpec.scidb' (instance 0) to array 'zooNoSpec' failed at line 1, column 166, offset 171, value='0.88899999856948853': Number of errors exceeds threshold. : command not found:

iquery -aq "load(sppLines,'/home/scidb/sppLines.scidb',0,99);" iquery -aq "load(sppParams,'/home/scidb/sppParams.scidb',0,99);" iquery -aq "load(zooNoSpec,'/home/scidb/ZooNoSpec.scidb',0,99);"
So I set max_errors=99. Here is the errors.

UserException in file: src/query/FunctionLibrary.cpp function: _findConverter line: 861 Error id: scidb::SCIDB_SE_TYPE::SCIDB_LE_CANT_FIND_IMPLICIT_CONVERTER Error description: Type error. Cannot find implicit converter from type 'int64' to 'string'. UserException in file: src/query/FunctionLibrary.cpp function: _findConverter line: 861 Error id: scidb::SCIDB_SE_TYPE::SCIDB_LE_CANT_FIND_IMPLICIT_CONVERTER Error description: Type error. Cannot find implicit converter from type 'int64' to 'string'. UserException in file: src/query/FunctionLibrary.cpp function: _findConverter line: 861 Error id: scidb::SCIDB_SE_TYPE::SCIDB_LE_CANT_FIND_IMPLICIT_CONVERTER Error description: Type error. Cannot find implicit converter from type 'int64' to 'string'. : command not found:
My arrays are below.

CREATE ARRAY sppParams <SPECOBJID:int64,BESTOBJID:int64,FLUXOBJID:int64,TARGETOBJID:int64,PLATEID:int64,sciencePrimary:int16,legacyPrimary:int16,seguePrimary:int16,FIRSTRELEASE:string,SURVEY:string,PROGRAMNAME:string,CHUNK:string,PLATERUN:string,MJD:int64,PLATE:int64,FIBERID:int64,RUN2D:string,RUN1D:string,RUNSSPP:string,TARGETSTRING:string,PRIMTARGET:int64,SECTARGET:int64,LEGACY_TARGET1:int64,LEGACY_TARGET2:int64,SPECIAL_TARGET1:int64,SPECIAL_TARGET2:int64,SEGUE1_TARGET1:int64,SEGUE1_TARGET2:int64,SEGUE2_TARGET1:int64,SEGUE2_TARGET2:int64,SPECTYPEHAMMER:string,SPECTYPESUBCLASS:string,FLAG:string,TEFFADOP:float,TEFFADOPN:int8,TEFFADOPUNC:float,TEFFHA24:float,TEFFHD24:float,TEFFNGS1:float,TEFFANNSR:float,TEFFANNRR:float,TEFFWBG:float,TEFFK24:float,TEFFKI13:float,TEFFHA24IND:int8,TEFFHD24IND:int8,TEFFNGS1IND:int8,TEFFANNSRIND:int8,TEFFANNRRIND:int8,TEFFWBGIND:int8,TEFFK24IND:int8,TEFFKI13IND:int8,TEFFHA24UNC:float,TEFFHD24UNC:float,TEFFNGS1UNC:float,TEFFANNSRUNC:float,TEFFANNRRUNC:float,TEFFWBGUNC:float,TEFFK24UNC:float,TEFFKI13UNC:float,LOGGADOP:float,LOGGADOPN:int8,LOGGADOPUNC:float,LOGGNGS2:float,LOGGNGS1:float,LOGGANNSR:float,LOGGANNRR:float,LOGGCAI1:float,LOGGWBG:float,LOGGK24:float,LOGGKI13:float,LOGGNGS2IND:int8,LOGGNGS1IND:int8,LOGGANNSRIND:int8,LOGGANNRRIND:int8,LOGGCAI1IND:int8,LOGGWBGIND:int8,LOGGK24IND:int8,LOGGKI13IND:int8,LOGGNGS2UNC:float,LOGGNGS1UNC:float,LOGGANNSRUNC:float,LOGGANNRRUNC:float,LOGGCAI1UNC:float,LOGGWBGUNC:float,LOGGK24UNC:float,LOGGKI13UNC:float,FEHADOP:float,FEHADOPN:int8,FEHADOPUNC:float,FEHNGS2:float,FEHNGS1:float,FEHANNSR:float,FEHANNRR:float,FEHCAIIK1:float,FEHCAIIK2:float,FEHCAIIK3:float,FEHWBG:float,FEHK24:float,FEHKI13:float,FEHNGS2IND:int8,FEHNGS1IND:int8,FEHANNSRIND:int8,FEHANNRRIND:int8,FEHCAIIK1IND:int8,FEHCAIIK2IND:int8,FEHCAIIK3IND:int8,FEHWBGIND:int8,FEHK24IND:int8,FEHKI13IND:int8,FEHNGS2UNC:float,FEHNGS1UNC:float,FEHANNSRUNC:float,FEHANNRRUNC:float,FEHCAIIK1UNC:float,FEHCAIIK2UNC:float,FEHCAIIK3UNC:float,FEHWBGUNC:float,FEHK24UNC:float,FEHKI13UNC:float,SNR:float,QA:int8,CCCAHK:float,CCMGH:float,TEFFSPEC:float,TEFFSPECN:int8,TEFFSPECUNC:float,LOGGSPEC:float,LOGGSPECN:int8,LOGGSPECUNC:float,FEHSPEC:float,FEHSPECN:int8,FEHSPECUNC:float,ACF1:float,ACF1SNR:float,ACF2:float,ACF2SNR:float,INSPECT:string,ELODIERVFINAL:float,ELODIERVFINALERR:float,ZWARNING:int64,TEFFIRFM:float,TEFFIRFMIND:int8,TEFFIRFMUNC:float,LOGGNGS1IRFM:float,LOGGNGS1IRFMIND:int8,LOGGNGS1IRFMUNC:float,FEHNGS1IRFM:float,FEHNGS1IRFMIND:int8,FEHNGS1IRFMUNC:float,LOGGCAI1IRFM:float,LOGGCAI1IRFMIND:int8,LOGGCAI1IRFMUNC:float,FEHCAIIK1IRFM:float,FEHCAIIK1IRFMIND:int8,FEHCAIIK1IRFMUNC:float>[j=0:*,1000000,0];

CREATE ARRAY zooNoSpec <specobjid:int64,objid:int64,dr7objid:int64,ra:float,dec:float,rastring:string,nvote:int32,p_el:float,p_cw:float,p_acw:float,p_edge:float,p_dk:float,p_mg:float,p_cs:float>[j=0:*,1000000,0];

CREATE ARRAY sppLines <SPECOBJID:int64,bestObjID:int64,TARGETOBJID:int64,sciencePrimary:int16,legacyPrimary:int16,seguePrimary:int16,PLATE:int64,MJD:int64,FIBER:int64,RUN2D:string,RUN1D:string,RUNSSPP:string,H83side:float,H83cont:float,H83err:float,H83mask:int8,H812side:float,H812cont:float,H812err:float,H812mask:int8,H824side:float,H824cont:float,H824err:float,H824mask:int8,H848side:float,H848cont:float,H848err:float,H848mask:int8,KP12side:float,KP12cont:float,KP12err:float,KP12mask:int8,KP18side:float,KP18cont:float,KP18err:float,KP18mask:int8,KP6side:float,KP6cont:float,KP6err:float,KP6mask:int8,CaIIKside:float,CaIIKcont:float,CaIIKerr:float,CaIIKmask:int8,CaIIHKside:float,CaIIHKcont:float,CaIIHKerr:float,CaIIHKmask:int8,Hepsside:float,Hepscont:float,Hepserr:float,Hepsmask:int8,KP16side:float,KP16cont:float,KP16err:float,KP16mask:int8,SrII1side:float,SrII1cont:float,SrII1err:float,SrII1mask:int8,HeI121side:float,HeI121cont:float,HeI121err:float,HeI121mask:int8,Hdelta12side:float,Hdelta12cont:float,Hdelta12err:float,Hdelta12mask:int8,Hdelta24side:float,Hdelta24cont:float,Hdelta24err:float,Hdelta24mask:int8,Hdelta48side:float,Hdelta48cont:float,Hdelta48err:float,Hdelta48mask:int8,Hdeltaside:float,Hdeltacont:float,Hdeltaerr:float,Hdeltamask:int8,CaI4side:float,CaI4cont:float,CaI4err:float,CaI4mask:int8,CaI12side:float,CaI12cont:float,CaI12err:float,CaI12mask:int8,CaI24side:float,CaI24cont:float,CaI24err:float,CaI24mask:int8,CaI6side:float,CaI6cont:float,CaI6err:float,CaI6mask:int8,Gside:float,Gcont:float,Gerr:float,Gmask:int8,Hgamma12side:float,Hgamma12cont:float,Hgamma12err:float,Hgamma12mask:int8,Hgamma24side:float,Hgamma24cont:float,Hgamma24err:float,Hgamma24mask:int8,Hgamma48side:float,Hgamma48cont:float,Hgamma48err:float,Hgamma48mask:int8,Hgammaside:float,Hgammacont:float,Hgammaerr:float,Hgammamask:int8,HeI122side:float,HeI122cont:float,HeI122err:float,HeI122mask:int8,Gblueside:float,Gbluecont:float,Gblueerr:float,Gbluemask:int8,Gwholeside:float,Gwholecont:float,Gwholeerr:float,Gwholemask:int8,Baside:float,Bacont:float,Baerr:float,Bamask:int8,C12C13side:float,C12C13cont:float,C12C13err:float,C12C13mask:int8,CC12side:float,CC12cont:float,CC12err:float,CC12mask:int8,metal1side:float,metal1cont:float,metal1err:float,metal1mask:int8,Hbeta12side:float,Hbeta12cont:float,Hbeta12err:float,Hbeta12mask:int8,Hbeta24side:float,Hbeta24cont:float,Hbeta24err:float,Hbeta24mask:int8,Hbeta48side:float,Hbeta48cont:float,Hbeta48err:float,Hbeta48mask:int8,Hbetaside:float,Hbetacont:float,Hbetaerr:float,Hbetamask:int8,C2side:float,C2cont:float,C2err:float,C2mask:int8,C2MgIside:float,C2MgIcont:float,C2MgIerr:float,C2MgImask:int8,MgHMgIC2side:float,MgHMgIC2cont:float,MgHMgIC2err:float,MgHMgIC2mask:int8,MgHMgIside:float,MgHMgIcont:float,MgHMgIerr:float,MgHMgImask:int8,MgHside:float,MgHcont:float,MgHerr:float,MgHmask:int8,CrIside:float,CrIcont:float,CrIerr:float,CrImask:int8,MgIFeIIside:float,MgIFeIIcont:float,MgIFeIIerr:float,MgIFeIImask:int8,MgI2side:float,MgI2cont:float,MgI2err:float,MgI2mask:int8,MgI121side:float,MgI121cont:float,MgI121err:float,MgI121mask:int8,MgI24side:float,MgI24cont:float,MgI24err:float,MgI24mask:int8,MgI122side:float,MgI122cont:float,MgI122err:float,MgI122mask:int8,NaI20side:float,NaI20cont:float,NaI20err:float,NaI20mask:int8,Na12side:float,Na12cont:float,Na12err:float,Na12mask:int8,Na24side:float,Na24cont:float,Na24err:float,Na24mask:int8,Halpha12side:float,Halpha12cont:float,Halpha12err:float,Halpha12mask:int8,Halpha24side:float,Halpha24cont:float,Halpha24err:float,Halpha24mask:int8,Halpha48side:float,Halpha48cont:float,Halpha48err:float,Halpha48mask:int8,Halpha70side:float,Halpha70cont:float,Halpha70err:float,Halpha70mask:int8,CaHside:float,CaHcont:float,CaHerr:float,CaHmask:int8,TiOside:float,TiOcont:float,TiOerr:float,TiOmask:int8,CNside:float,CNcont:float,CNerr:float,CNmask:int8,OItripside:float,OItripcont:float,OItriperr:float,OItripmask:int8,KI34side:float,KI34cont:float,KI34err:float,KI34mask:int8,KI95side:float,KI95cont:float,KI95err:float,KI95mask:int8,NaI15side:float,NaI15cont:float,NaI15err:float,NaI15mask:int8,NaIredside:float,NaIredcont:float,NaIrederr:float,NaIredmask:int8,CaII26side:float,CaII26cont:float,CaII26err:float,CaII26mask:int8,Paschen13side:float,Paschen13cont:float,Paschen13err:float,Paschen13mask:int8,CaII29side:float,CaII29cont:float,CaII29err:float,CaII29mask:int8,CaII401side:float,CaII401cont:float,CaII401err:float,CaII401mask:int8,CaII161side:float,CaII161cont:float,CaII161err:float,CaII161mask:int8,Paschen421side:float,Paschen421cont:float,Paschen421err:float,Paschen421mask:int8,CaII162side:float,CaII162cont:float,CaII162err:float,CaII162mask:int8,CaII402side:float,CaII402cont:float,CaII402err:float,CaII402mask:int8,Paschen422side:float,Paschen422cont:float,Paschen422err:float,Paschen422mask:int8,TiO5side:float,TiO5cont:float,TiO5err:float,TiO5mask:int8,TiO8side:float,TiO8cont:float,TiO8err:float,TiO8mask:int8,CaH1side:float,CaH1cont:float,CaH1err:float,CaH1mask:int8,CaH2side:float,CaH2cont:float,CaH2err:float,CaH2mask:int8,CaH3side:float,CaH3cont:float,CaH3err:float,CaH3mask:int8,UVCNside:float,UVCNcont:float,UVCNerr:float,UVCNmask:int8,BLCNside:float,BLCNcont:float,BLCNerr:float,BLCNmask:int8,FEI1side:float,FEI1cont:float,FEI1err:float,FEI1mask:int8,FEI2side:float,FEI2cont:float,FEI2err:float,FEI2mask:int8,FEI3side:float,FEI3cont:float,FEI3err:float,FEI3mask:int8,SRII2side:float,SRII2cont:float,SRII2err:float,SRII2mask:int8,FEI4side:float,FEI4cont:float,FEI4err:float,FEI4mask:int8,FEI5side:float,FEI5cont:float,FEI5err:float,FEI5mask:int8>[j=0:*,1000000,0];

Attachments are sppParams.scidb and ZooNoSpec.scidb.

Please help me. Thanks a lot. :smile:
sppParams.rar (476 KB)


#2

Hi Cherry,

These errors

Error id: scidb::SCIDB_SE_TYPE::SCIDB_LE_CANT_FIND_IMPLICIT_CONVERTER Error description: Type error. Cannot find implicit converter from type 'int64' to 'string'.
are because the error limit of 99 you wanted to use is in the wrong parameter position for load. Try that with

instead (specifying the format… I assume ‘text’ because the files have .scidb extension… and name for the “shadow array”, which collects the error information.

In the meantime I’ll have a look at your attached sample files and see if I can figure out the original problem.

Best,
Mike L.


#3

OK, I looked at your data files. There are format problems.

Looking at your data, I think that it might be easier for you to use the TSV (tab-separated value) format to load these one-dimensional arrays. We like TSV a lot; of the various non-binary formats supported by SciDB, it is the fastest to parse and the most well-defined. The dialect of TSV that SciDB supports is well described at http://dataprotocols.org/linear-tsv/ (we add a few additional ways to specify database nulls, but this page will get you started). The format string for the load() operator is ‘tsv’, as you would expect.

If you would rather stick with the SciDB text format, here are the problems I see with your data (looking specifically at ZooNoSpec.scidb):

  1. You have a lot of floating point numbers for int32 attributes. For example 0.88899999856948853 will not parse into the nvote:int32 attribute.

  2. Fields need to separated by commas.

  3. Strings need to be enclosed in single quotes.

In general, check that the data fields in your file (in left-to-right order) match the types of the attributes in your schema (also left-to-right).

I hope this helps!


#4

P.S. Since you have timestamps in your data, the SciDB built-in types datetime and datetimetz might be useful:

$ iquery -aq "build(<t:datetimetz>[i=0:0,1,0], '2014-05-20 21:00:00 -05:00')"
{i} t
{0} '2014-05-20 21:00:00 -05:00'
$ 
$ iquery -aq "build(<t:datetime>[i=0:0,1,0], now())"
{i} t
{0} '2015-05-20 20:14:58'

#5

[quote=“mjl”]OK, I looked at your data files. There are format problems.

Looking at your data, I think that it might be easier for you to use the TSV (tab-separated value) format to load these one-dimensional arrays. We like TSV a lot; of the various non-binary formats supported by SciDB, it is the fastest to parse and the most well-defined. The dialect of TSV that SciDB supports is well described at http://dataprotocols.org/linear-tsv/ (we add a few additional ways to specify database nulls, but this page will get you started). The format string for the load() operator is ‘tsv’, as you would expect.

If you would rather stick with the SciDB text format, here are the problems I see with your data (looking specifically at ZooNoSpec.scidb):

  1. You have a lot of floating point numbers for int32 attributes. For example 0.88899999856948853 will not parse into the nvote:int32 attribute.

  2. Fields need to separated by commas.

  3. Strings need to be enclosed in single quotes.

In general, check that the data fields in your file (in left-to-right order) match the types of the attributes in your schema (also left-to-right).

I hope this helps![/quote]

Thanks a lot! That’s right. I use

Problem is solved.