Most efficient way to export an array


#1

It seems that all formats supported by the iquery option -o are text-based (even the opaque one). E.g. exporting a 40MB 2D array with opaque takes ~15s and results in a 125MB file.

Is there any faster way to export arrays?


#2

Right - don’t use iquery for fast exports, it is only meant to be for quick interactive use.

For opaque format, the fastest is to use save operator as in

save( ARRAY, 'relative/path', -1, 'opaque')

That makes each instance generate an opaque file at "relative/path’ inside their respective data directories. That will generate one file per instance. These files can be re-loaded all at once or one or more at a time. More info here: https://paradigm4.atlassian.net/wiki/spaces/ESD169/pages/50856208/save

For TSV or binary format, the fastest way is to use aio_save from this plugin: