csv2scidb


#1

when I decide use csv2scidb tool on SCIDB CE 15.7 ,but I cannt find it .what should I do ?


#2

a number of utils has been removed from scidb 15.7 CE.
i am working on release 15.7 release to add them back


#3

CSV and TSV formats can now be loaded directly with the input() and load() operators, so csv2scidb is no longer needed. For example:

load(A, '/tmp/mydata.csv', -2, 'csv')

See the 15.7 doc for these operators. The format string (‘csv’ above) can include options specifiers that modify the behavior in (hopefully) useful ways.

The Linear TSV tab-separated-values dialect is also supported. Especially if you have string attributes with tricky quoting and embedded newlines, this is a nicer format. See http://dataprotocols.org/linear-tsv .


#4

P.S. If you are doing very large loads of mundane CSV or TSV data, you should look at https://github.com/paradigm4/accelerated_io_tools


#5

yes! I see. but my csv file come from sql-server database . when I run progress like you ! it will happen a problem ,such as
value=’%FFFFFFEF%FFFFFFBB%FFFFFFBF1237645942366274027’: Failed to parse string ‘1237645942366274027’ as int64 (non-digits).
could you help me


#6

yes! I see. but my csv file come from sql-server database . when I run progress like you ! it will happen a problem ,such as
value=’%FFFFFFEF%FFFFFFBB%FFFFFFBF1237645942366274027’: Failed to parse string ‘1237645942366274027’ as int64 (non-digits).
could you help me


#7

What we’re confused about is the format of the file. I don’t think those ‘%FFFFFFEF%FFFFFFBB%FFFFFFBF’ things are data. They’re some kind of artifact generated by the SQL Server data dumper.

Would you post a “head - 5” of the .csv file, please?


#8

I think you should
$head -2 file1.csv > file2.csv
$cat -A file1.csv
M-oM-;M-?1237645942366274027,2,109,301,3,114,491
M-oM-;M-? it’s the problem here! I think I should remove it from file .


#9

I think you should
$head -2 file1.csv > file2.csv
$cat -A file1.csv
M-oM-;M-?1237645942366274027,2,109,301,3,114,491
M-oM-;M-? it’s the problem here! I think I should remove it from file .


#10

Indeed!

The SQL Server data dumper seems to have put some rubbish at the start of each line.