Cool and Useful Links from the Community


This post takes a quick survey of the recent developments in our community. We aim to share our findings among SciDB users to promote collaboration.

I. Tutorial Blogs

Excited and grateful to see folks starting to write instructional blogs on SciDB.

  • Thinking in SciDB
    Bringing people’s attention once again to this very informative website by Dr. Rares Vernica. A superb tutorial on data loading and hopefully many more topics to come. Complete with amazing pictures!

  • Scalable Earth Observation Analytics with R and SciDB
    By Marius Appel and Dr. Edzer Pebesma. The authors have also significantly extended SciDB with the scidb4gdal and scidb4geo packages. So they have developed an entire approach for scaling up earth observation analytics using SciDB and R.

II. Docker Containers

This looks like a very popular thing to do and perhaps folks can start re-using each-other’s work. Here’s a sampling of repositories in no particular order:

III. UD* Extensions

Various User-Defined Functions, Aggregates, Data Types and Operators for SciDB. Quite useful - both directly and as a starting point for someone writing their own. Note that a lot of work exists targeting older versions of SciDB. Here’s a list of a few repositories that have been updated recently:

IV. Plugins from P4

You can find many other plugins at the P4 Github Page. Many of these are exploratory prototypes that are candidates for future productization. So far, the plugins we use most often are as follows:

  • dev_tools: simply, a plugin to easily install other plugins
  • accelerated_io_tools: fast and error-tolerant text loading
  • streaming: run various programs, such as R scripts, invoked on SciDB data in parallel
  • equi_join: easily join large arrays by attributes and/or dimensions
  • grouped_aggregate: easy aggregation grouped by attributes and/or dimensions
  • limit: return the first K cells of an array, just like the SQL LIMIT clause
  • summarize: very quick chunk density and size statistics
  • superfunpack: a few miscellaneous UDFs: Fisher’s exact test, regular expressions,…

V. Additional work

Some folks have also built extra R packages, interfaces, connectors and so on:

VI. Whom did we miss?

Apologies to anyone we didn’t notice. Always excited to hear about folks’ work as it relates to SciDB. Please feel free to share your results here. Cheers!


Iteration in AQL/AFL