During loading a big dataset, you can see things going by by looking at the scidb.log file with "tail -f"
I noticed, that when the chunks are small, then this listing flies by really fast. If the chunks are (too) big, then it crawls to an apparent halt.
This gave me an idea for monitoring a long ingest by taking the rate of change of the size of the log file, and connecting it to a visual dashboard widget with a gauge on it. I know it sounds like I am going crazy from looking at this too much, but I do see patterns that tell me whether or not the ingest is going ok, or something weird is happening. And this is from the speed and shape of the lines in the listing. I could even connect the gauge to a sound-generating thingie that purrs nicely during a well-formed well-behaved load, and makes different sounds when the pattern of the logfile entries changes.
It is also nice, that I can write up this widget non-invasively, because it just watches the logfile and hums…
I am waiting for the fsync error I had before. I switched to 12.7 and have a clean array store. Got rid of the store I made with 12.3 as I expect that each version change will require a rebuild anyway, right?
BTW: loading one month went from 22 minutes to 15 minutes just by switching from 12.3 to 12.7 and leaving everything else the same.
That is good news.
I am on month 2, have an hour before the anticipated fsync error. Last time it happened after 5 months of ingested data…
If I sound nuts, that’s because I spent too much time looking at climate data in this context. Last night I dreamt in AFL… but woke up with a parsing error because I neglected to set the language to AFL. By the way did you know that AFL% and AQL% look very similar? I’d like a more pronounced visual difference in the prompt inside iquery. What is the point of ‘AFL’ and ‘AQL’ in the prompt, when there is no information in the first and last letter?
Getting a bit punchy here… I wanted to have all the MERRA data loaded by Monday, but it doesn’t seem like that’s going to happen that soon. I’ll see.
Thanks for reading this.