New data into array


#1

I’ ve being studying scidb with so much enthusiasm, however I have a question that has not been remedied. The user guide states that each store or update operation generates a new version.

Could I do “insert” of new data into an array without creating a new version?

I ask that because there is new data coming every day and i’d like to keep it inside scidb.


#2

Hi,

Yes this is a current issue. In the next version (out in October) we will have an insert() operator. Until then…

The real problem is not creating a new version, but that versions have too much overhead.

One way to go around the problem is to use an immutable array and design it such that NEW data always gets loded into NEW chunks. For example you could have a “day” dimension with chunk size 1 along that dimension and perform one load per day.

If that can’t be done, then to get the semantics of insert, you have to use store(merge( new_data, array ), array). Then, to get rid of overhead you periodically have to “collapse” the versions like so:

store(array, temp); --at this point, array has K versions
remove(array);
rename(temp, array); --at this point array has 1 version

This may not be feasible, depending on your hard drive usage. If it’s doable you could schedule it with some frequency, like once a week…

Will try to deliver insert as fast as we can…


#3

Thanks Apoliakov, I’ll work with daily loads until next version.