One thing to try would be to write the array into a fifo device (instead of a file – see bash command “mkfifo”) and then have scidb read from it. It would be faster as the data would never hit the disk. Still, there would be an expensive conversion to and from string. Expensive and unnecessary…
What would it take to speed it up? You could take a look at the input() operator – LogicalInput.cpp and PhysicalInput.cpp. That’s the piece that reads data from a file (or filesystem device like a fifo) and emits it into a scidb array. All of our load operations go through input(). You could write an operator similar to input() - that takes data from a different source (socket?) and creates an array out of that. We’ve had some discussions about this idea but no one’s currently actively pursuing it. It would definitely be very helpful for a variety of scenarios!
Come to think of it - the first step would be to write a binary-file input(). Then you could do:
-output binary data into a fifo (from your python script or what have you)
-have scidb load() read binary data from the fifo
This would also be useful for general “save an array as a binary file for fast loading” use case. And it would be pretty much as fast as it can go. Only minimal OS overhead. The binary format would have to be chunk-centric - data would have to be organized a chunk at a time.