Store(): does it ensure disk and OS file cache flushes



does SciDB store operator guarantees that the array data were saved to disk and the data does not remain in OS or disk caches?


There are two separate questions here, so I’ll answer both.

When the store() query has completed successfully, the data are guaranteed to have been persisted to disk.

However, if you are concerned that the data remains in main memory or in various disk caches, the answer is possibly. SciDB maintains an LRU cache of data chunks so the data, which has been persisted to disk, may still remain in main memory.


Thank you!

Do I understand correctly that if
(1) an array is guaranteed to have been persisted to disk
(2) some chunks are cached in memory
then the two copies of the same chunk may reside both on disk and in memory (in SciDB cache)?

Can I configure the size of that cache?
If yes, what Is the parameter name? –


The chunk which has been persisted to disk for a given attribute for a given version of a given array may remain in the cache. This is for performance purposes. Rather than re-read the chunk from disk immediately after writing it, the cache allows subsequent queries to use the chunk that is still in memory. There are not really “two copies”. There is the persisted form on disk, and the value which is in memory. It is like viewing a text file (like less mycoolfile). The file is on disk, but the contents of the file reside in memory while viewing it. It’s not really two distinct copies.

The size of the cache is controlled by the following setting:

In version 16.9, two parameters control the cache size. One for TEMP array and one for Persistent arrays:

| mem-array-threshold (optional) | Cluster | SciDB Instance | Maximum size in MB of temporary data to cache in                      |
|                                |         |                | memory before writing to temporary disk files. Default: 1024 MB.      |
| smgr-cache-size (optional)     | Cluster | SciDB Instance | Size of memory in MB allocated to the shared cache of array chunks.   |
|                                |         |                | The cache is used only for the chunks belonging to persistent arrays. |
|                                |         |                | Default: 256 MB.                                                      |

In 18.1, the mem-array-threshold is used for all the buffers (TEMP and Persistent).

Please note that this value is per-instance.


Thank you again!

I feel I have got the idea but stil it is a bit confusing –

What do you mean when you say file?

Let’s better say data for chunk C_i
In this case, there is the
(1) data for chunk C_i on disk, in file
(2) data for chunk C_i in memory, in cache
but the data have different formats, one for storing on-disk data for chunk C_i
the other is for storing in-memory data for chunk C_i?

I am asking because you say There are not really "two copies".
However, to have data both on disk and in memory you should really keep two copies of data of the same chunk unless you just create a file and delay writing there by keeping data in cache.


“It is like viewing a text file”. This is an analogy.

If I type less mycoolfile (completely separate from any aspect of SciDB). The file in this case is the text file “mycoolfile”. When I type less mycoolfile, the text file, mycoolfile, remains on the disk. The contents of the text file, mycoolfile, is read into main memory and then is displayed to the terminal. The bits on the disk that make up mycoolfile are not changed. They are merely copied into main memory so they can be displayed.

There are not two different binary patterns (versions). This is fairly standard computer architecture: disk -> main memory -> cpu cache --> register.

Why do you say this? That is simply not true. The different forms are “one maintained by CMOS transistors in main memory” and the the other as “North-south magnetic polarization sections on a magnetic media”(assuming standard HDD).

(Minor caveat… with one possible exception of using compression. In that case, the data in memory is compressed before writing to disk and the uncompressed form would remain in main memory.).

Say you are using an Office Product like “Word” or “LibreOffice”. And you start writing a document. When you “save the file” the “bits in main memory are written to disk” but they remain in main memory so that you can continue to edit the file. In fact in DocX and ODT formats, the XML which is in main memory is compressed into a “blob” that is stored on disk. When you open a “docx” file, Microsoft Word will “decompress the ‘blob’ in the file” and store the uncompressed bits in main memory.


According to your description this is really true.

Probably we do not want to discuss the physical organisation of memory (CMOS transistors or magnetic media), but the data formats. I come from the computer science domain and the simplifications like “Word” or “less” make the discussion even more obscure (unless we are from the Word team of Microsoft and know how Word internally treats its files :slight_smile:

You said
It's not really two distinct copies.
and later
There are not two different binary patterns (versions).
How does this go together?

Now my understanding is that SciDB treats chunks something like this
CHUNK_DATA chunk write(chunk) function write(CHUNK_DATA chunk){ chunk_compressed <- compress(chunk) write_to_disk(chunk_compressed) // ***DISK COPY*** (1) data for chunk C_i on disk, in file write_to_cache(chunk) // ***CACHE COPY*** (2) data for chunk C_i in memory, in cache }

so, we have a chunk copy in RAM and on DISK, those copies may have different logical formats: one is possibly compressed and the other is possibly no. In this case we have two copies of the same data, one is in RAM and the other is on DISK.
Probably, the on-disk format has some other differences to the in-memory format.


I still stand by this. If the bits on disk are 10101010 then they are 10101010 in memory.

If it is a compressed chunk, then it is read into memory as 10101010 and then decompressed so that the system can use it.

If you are having trouble with the concept of cacheing, there are numerous resources on the web and in Computer Science text books.


Sorry, I do not have troubles with the concept of caching.
If the chunk data are compressed, their compressed FORMAT really differs from that of the uncompressed FORMAT.

The definition which creates confusion is not “cache”, but the “format”. The cache has nothing to do here.
Please, kindly think of it.


Please start a new topic, thread.

This asked about store() guarantee of persisting data upon completion(yes), and whether any data remained in OS caches (depends upon the OS caching policy), disk cache (again depends upon the FS caching policy), or in main memory (yes. There is an LRU cache).