Importing datetimes from csv


#1

Hi,

I just upgraded to SciDB 14.8 and it seems that the csv import doesn’t treat datetimes as before: I’ve got a csv file with (among other things) a column of dates, given in format “2013-10-01 16:56:26”. With 14.3, I could use loadcsv.py with the -t option to declare this csv column as a string and then it was automagically converted to a datetime during import. Now, loadcsv.py insists that the date column in the csv must be number because the schema has a “number” (=datetime). It tells me:
Warning: type pattern NNSSsNNNNNNssss conflicts with load schema, using NNSSSNNNNNNSSSN instead
Not surprisingly, 2013-10-01 16:56:26 is not a valid number and so I get an error.

Is there something (other than converting my csv file date column to seconds since 1970) I can do to load my csv file in parallel (with loadcsv.py)?


#2

Hi,

Thanks for telling us about this. We will take a look.

For now, can you load the data as a string first and then cast it to datetime in the system?
Also note - the superfunpack prototype has a function that could help: github.com/Paradigm4/superfunpack#strpftime


#3

Thanks a lot for looking at this :smile:

I realised that my call to

loadcsv.py -n 1 -t NNSSsNNNNNNssss

can be replaced by this

csv2scidb -s 1 -p NNSSsNNNNNNssss ...
iquery -aq "load(...)"
iquery -aq "store(...)"

and then it works again…

As long as that csv file stays small, this fixes my issue.