Date Time format in csv


#1

Hi . Im building csv files to import into scidb. What is the best format for a date and time within a csv field to make life easy for SciDB? Cheers Kevin


#2

dimensions or attributes? in case of dimensions i would keep both as integers - 20170901 for date and 161000 for time


#3

At the moment they look like 2017-08-13:11:02:11 as one field. I expect Date and time will be important dimensions. Grateful for any reccommendations


#4

yea so scidb only supports integer dimensions so you need to convert them to integers
like 20170813 and 13110211


#5

Alex Polyakov, is there plan for scidb to support non-integer dimensions? its kind of pain in the rear to make dimensions integer all the time. i understand its non-trivial for system to maintain the mapping from dimensions values to integer dimensions but it seems it was there in r13 and it was dropped.
i think the attributes and dimension should be completely symmetric as far as types.


#6

@senya72 yes it was dropped from r13. The idea is definitely sound but the implementation in r13 was sub par. We realized it was doing more harm than good.

One thing I find very often is that “dimensions” want to be shared between arrays and are sometimes multi-attribute. For example:

SAMPLE <name, date, type...>        [sample_id]
GENE   <hugo_symbol, gene_name,...> [gene_id]

RNA_EXPRESSION  <value> [sample_id, gene_id]
DNA_METHYLATION <value> [sample_id, gene_id]
...

…And perhaps you want the removal of a sample to cascade down to everywhere it’s used… So - yes - we are considering these examples and discussing what kind of syntax and features would work best.


#7

if you look at OLAP/data cubes - dimensions are always orthogonal, multi-hierarchical and shared. They also change over time and they represent actual entity whereas arrays represent facts about the entity. The traditional mathematical definition of arrays indicates the dimensions are integer. Since humans manipulate symbols rather than integers, its natural to provide the mapping between symbols and integers and it also does not violate notion that purely mathematical array definition provides.