Load Netcdf directly?


#1

I found that SciDB can only load txt file and the ASCII format is so strange for me who familiar with Netcdf. Do you have and plan to support these classic formats: Hdf5 and Netcdf ?
I hope there is a simple command “LOAD x.nc4” can create the array and load data from Netcdf file together.


#2

Hello,
We currently do not support these formats, but a few volunteers are actively working on them.
We do support csv data, using csv2scidb as a preprocessor.


#3

Hi,

since this post is a little bit old. Has there been some process in this section?
What is the best (read fastest) way to import netCDF in SciDB?

(I even read in the “Overview of SciDB” paper from the SciDB Development Team in “Proceedings of the 2010 international conference on Management of data - SIGMOD '10” that they are trying to implement the feature to address the data in netCDF or HDF files without importing them)


#4

[quote=“cayo”]Hi,

since this post is a little bit old. Has there been some process in this section?
What is the best (read fastest) way to import netCDF in SciDB?

(I even read in the “Overview of SciDB” paper from the SciDB Development Team in “Proceedings of the 2010 international conference on Management of data - SIGMOD '10” that they are trying to implement the feature to address the data in netCDF or HDF files without importing them)[/quote]
i feel very interesting ,too


#5

Hello all,

There is a community-written tool for loading HDF5 and FITS formats available in our github repo: github.com/Paradigm4/SciDB-HDF5 . It would be great if someone could check out this software against the new 14.7 release (just announced today!).

Unfortunately we don’t as yet have any tools to help with loading NetCDF. We would certainly welcome any community effort in this area.

Cheers,
Mike Leibensperger
Paradigm4


#6

[quote=“mjl”]Hello all,

There is a community-written tool for loading HDF5 and FITS formats available in our github repo: github.com/Paradigm4/SciDB-HDF5 . It would be great if someone could check out this software against the new 14.7 release (just announced today!).

Unfortunately we don’t as yet have any tools to help with loading NetCDF. We would certainly welcome any community effort in this area.

Cheers,
Mike Leibensperger
Paradigm4[/quote]
i find a fileparse.cpp file in the scidb source code,but find nothing about it in the userguide,errr :astonished:


#7

Hi,

I looked in SciDB release tarballs as far back as 12.2 and cannot find the fileparse.cpp file you mention. Nobody around here seems to have heard of it, so I’m not sure what more I can add. Someone did tell me that NetCDF and HDF5 are very similar formats, so perhaps something in the link posted above may be helpful.

Cheers,
Mike Leibensperger
Paradigm4


#8

[quote=“mjl”]Hi,

I looked in SciDB release tarballs as far back as 12.2 and cannot find the fileparse.cpp file you mention. Nobody around here seems to have heard of it, so I’m not sure what more I can add. Someone did tell me that NetCDF and HDF5 are very similar formats, so perhaps something in the link posted above may be helpful.

Cheers,
Mike Leibensperger
Paradigm4[/quote]
Sorry,i didn’t write it correctly. it’s FITSParser.h


#9

Hi, apologies for the late response. Fires to fight!

The code in the examples subdirectory does not typically get any mention in the User Guide. I gather that example/fits is community-contributed code for parsing that format, but I do not know how close it is to the NetCDF family of formats. If it is close, by all means use it as a starting point.

Right now we do not have any roadmap items for support of these formats. (A big pile of money could change that, wink wink.) Instead in the next several releases we are working to make the load of existing supported formats go more smoothly. We will also be moving more and more toward TSV as the “lowest common denominator” format.

Cheers,
Mike Leibensperger
Paradigm4


#10

If SciDB-HDF5 can import HDF5 into SciDB, then it should also import netCDF-4 without modification (perhaps renaming the file extension .nc to .h5?) because netCDF-4 files are valid HDF5 files.

From unidata (creators of netcdf): unidata.ucar.edu/software/ne … .html#fv16


#11

i am working on scidb adapter for netCDF files.
my current naive implementation is to convert netcdf to csv and then load csv files. i can share the code with you - email me at senya72@gmail.com