Can't load sparse arrays


#1

Loading a sparse array from a file gives the following error:

Error message: literal expected: tkn == TKN_LITERAL

I can create a sparse array as such:

AFL% store(build_sparse(<a1: double> [x=0:5,3,0, y=0:5,3,0], 1.0*x+100.0*y, x=y), junk);
[[{1,1}(101),{2,2}(202)]];
[[{3,3}(303),{4,4}(404),{5,5}(505)]]

but if I then save the array to a file, and try to reload it into a new array, it does not work:

AFL% save(junk,'/tmp/junk.txt');
[[{1,1}(101),{2,2}(202)]];
[[{3,3}(303),{4,4}(404),{5,5}(505)]]
AFL% CREATE EMPTY ARRAY junk2 <a1:int32> [x=0:5,3,0, y=0:5,3,0];
Query was executed successfully
AFL% show(junk2);
[("junk2<a1:int32 NOT NULL,empty_indicator:indicator NOT NULL> [x=0:5,3,0,y=0:5,3,0]")]
AFL% load(junk2,'/tmp/junk.txt');
Error query id: 1309036159
Error message: literal expected: tkn == TKN_LITERAL
Error code:27 ( Error on load operation )

Also, the directions for creating sparse arrays in the wiki are confusing (http://trac.scidb.org/wiki/Docs/Release_11.06/ArrayFunctionalLanguage_11.06#Sparsedataformat). First, they have you create a 10x10 array with a single chunk, and without the EMPTY keyword. Then, the example data is divided into multiple chunks, and none of the coordinates actually fit within the 10x10 array.


#2

Hello, and thank you for reporting the issue.

This is somewhat of a confusing area.
“CREATE EMPTY ARRAY” creates an array where some cells may be missing.
In 11.06 all such arrays have an empty tag attribute that is set to true for not missing cells. You can see this attribute in your declaration:

[(“junk2<a1:int32 NOT NULL,empty_indicator:indicator NOT NULL> [x=0:5,3,0,y=0:5,3,0]”)]

Your load fails because the array a1 does not have such an attribute. The load op tries to find the “true” symbol doesnt find it and throws an error. Here’s a sequence that works:

apoliakov@scalpel:~$ iquery -aq "create empty array a1 <val:int64> [x=0:10,10,0]"
Query was executed successfully
apoliakov@scalpel:~$ iquery -aq "build_sparse(a1, x, x%3=0)"
[{0}(0,true),{1}(),{2}(),{3}(3,true),{4}(),{5}(),{6}(6,true),{7}(),{8}(),{9}(9,true)]
apoliakov@scalpel:~$ iquery -aq "store(build_sparse(a1, x, x%3=0),a1)"
[{0}(0,true),{1}(),{2}(),{3}(3,true),{4}(),{5}(),{6}(6,true),{7}(),{8}(),{9}(9,true)]
apoliakov@scalpel:~$ iquery -aq "save(a1, '/tmp/a1')"
[{0}(0,true),{1}(),{2}(),{3}(3,true),{4}(),{5}(),{6}(6,true),{7}(),{8}(),{9}(9,true)]
apoliakov@scalpel:~$ iquery -aq "create empty array a2 <val:int64> [y=0:10,10,0]"
Query was executed successfully
apoliakov@scalpel:~$ iquery -aq "load(a2, '/tmp/a1')"
[{0}(0,true),{1}(),{2}(),{3}(3,true),{4}(),{5}(),{6}(6,true),{7}(),{8}(),{9}(9,true)]

Notes:

  1. You might wonder why this empty tag attribute is there. Seems ugly. Feels like it should be hidden from the user. That’s under discussion for next release.
  2. You also might wonder what’s the difference between your “a1” above (that is not EMPTY) and my “a1”.
    When an array is not “EMPTY” and there is no “empty tag” attribute, unspecified cells are assumed to be all 0. When the array is EMPTY, then such cells simply do not exist. To see the difference, you can run something like
    apply ( myArray, newAttribute, attribute+1 )

We tried to clarify these points in the AFL doc. Evidently not well enough.