General Mac OS X compatiblity


#1

Hello,

First of all thank you for developing SciDB. It looks very interesting and also looks like a lot of effort. Nice to see someone is trying to address the problems of big data analysis.

What I wanted to ask is whether Mac OS X is in broad sense a supported platform for building/running SciDB? I’ve encountered couple of issues which are described below and hence the question.

Stock SciDB 12.10 on Mac OS X 10.8 Mountain Lion

I’ve tried to build it on a fairly fresh copy of Mountain Lion (OS X 10.8) but I got the following error:

scidb-12.10.0.5777-Cheshire-final [trunk:5777] % make
[  1%] Built target json_lib
[  2%] Built target bsdiff
[  2%] Built target bs_tests
[  2%] Built target MurmurHash_lib
[  2%] Building CXX object src/util/CMakeFiles/util_lib.dir/Job.cpp.o
clang: warning: argument unused during compilation: '-ggdb3'
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/src/util/Job.cpp:32:
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/query/Query.h:44:
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/array/MemArray.h:32:
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/array/Array.h:38:
/Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/query/Aggregate.h:186:12: error: use of undeclared identifier 'isnan'; did you mean 'std::isnan'?
    return isnan(value);
           ^~~~~
           std::isnan
/usr/include/c++/4.2.1/cmath:550:5: note: 'std::isnan' declared here
    isnan(_Tp __f) { return ::__gnu_cxx::__capture_isnan(__f); }
    ^
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/src/util/Job.cpp:32:
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/query/Query.h:44:
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/array/MemArray.h:32:
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/array/Array.h:38:
/Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/query/Aggregate.h:192:12: error: use of undeclared identifier 'isnan'; did you mean 'std::isnan'?
    return isnan(value);
           ^~~~~
           std::isnan
/usr/include/c++/4.2.1/cmath:550:5: note: 'std::isnan' declared here
    isnan(_Tp __f) { return ::__gnu_cxx::__capture_isnan(__f); }
    ^
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/src/util/Job.cpp:32:
In file included from /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/query/Query.h:48:
/Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/include/SciDBAPI.h:169:32: warning: 'getSciDB' has C-linkage specified, but returns user-defined type 'const scidb::SciDB &' which is
      incompatible with C [-Wreturn-type-c-linkage]
EXPORTED_FUNCTION const SciDB& getSciDB();
                               ^
1 warning and 2 errors generated.
make[2]: *** [src/util/CMakeFiles/util_lib.dir/Job.cpp.o] Error 1
make[1]: *** [src/util/CMakeFiles/util_lib.dir/all] Error 2
make: *** [all] Error 2

Now, I’m aware that Apple has tinkered with the compiler lately, it’s not a stock GCC anymore but the GCC-like interface to LLVM. And this issue looks like it might be caused by that change. The compiler change to LLVM hasn’t been an issue so far though. I’ve seen some warnings when building dependencies for SciDB and other software as well (we’re developing in Python and I have full Python stack running; including C-extensions which compiled just fine), but other than that everything worked just fine. So I’d be very grateful if someone could confirm that it’s indeed the compiler causing this and GCC might solve the problem before I go ahead and replace LLVM with GCC which is potentially dangerous operation.

Modified version by Timothy Shead (SciDB 12.10 for OS X 10.8)

I’ve also tried to build the 12.10 version modified by Timothy Shead (thank you Timothy) - viewtopic.php?f=13&t=649

ScaLAPACK detection issue (OS X 10.8) - 1.x vs 2.x incompatibility?

One more issue is the ScaLAPACK detection. Despite installing ScaLAPACK (version 2.0.2) the cmake script didn’t pick it up. Surprisingly that was the case also on Ubuntu 12.04 - that’s why I figured something strange might be going on. It turned out that on Ubuntu it wasn’t enough to install ScaLAPACK (from source - version 2.0.2) but it was also required to install libscalapack-mpi1 (via apt-get as suggested in the cmake output). This is a 1.x package so it would seem that SciDB is compatible with version 1.x of ScaLAPACK but not with version 2.x? Perhaps would be good to note somewhere or make the cmake message more informative.

Anyway, FYI here’s the output from cmake (on a mac):

scidb-12.10.0.5777-Cheshire-final [trunk:5777] % cmake .
-- CMAKE_INSTALL_PREFIX not Initialized to default keeping: /opt/scidb/12.10
-- Reading revision from svn: 5777
-- Build type (use -DCMAKE_BUILD_TYPE=[RelWithDebInfo]/Debug/Release/Profile/CC/Valgrind): 
DISTRO_NAME_VER is Darwin
-- Boost version: 1.52.0
-- Found the following Boost libraries:
--   system
--   program_options
--   serialization
--   regex
--   filesystem
--   thread
-- GPB - /usr/local/lib/libprotobuf.dylib
-- LOG4CXX - /usr/local/lib/liblog4cxx.dylib
-- Found LibPQ: /usr/local/bin/pg_config  
-- Found LibPQ: /usr/local/Cellar/postgresql/9.2.2/lib  
CMake Warning at CMakeLists.txt:185 (message):
  Python binding broken on Mac! Forcing WITHOUT_PYTHON_BINDING=1


-- Could NOT find docbook.xsl (missing:  DOCBOOK_XSL_FILE) 
-- Configuring /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/src/system/Constants.cpp.in as /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/src/system/Constants.cpp
-- Configuring /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/src/system/System.h.in as /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/src/system/System.h
-- Configuring /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/src/util/scidb-sample.conf.in as /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/bin/packaging_only/scidb-sample.conf
-- ****************** BEGIN mpi/CMakeLists.txt ******************
-- MPI_Fortran_COMPILER=/usr/local/bin/mpif77
-- Using platform BLAS
-- A library with BLAS API found.
-- Using platform LAPACK
-- A library with BLAS API found.
-- A library with LAPACK API found.
-- Using platform ScaLAPACK stack
CMake Warning at src/mpi/CMakeLists.txt:216 (message):
  ScaLAPACK not found.  ScaLAPACK-based functionality will NOT be built.


CMake Warning at src/mpi/CMakeLists.txt:217 (message):
  Ubuntu Hint: $ sudo apt-get install libscalapack-mpi1


-- Debug: FINAL PACKAGE SETTINGS ================================>
-- Debug: SCALAPACK_LIBRARIES is 
-- Debug: LAPACK_LIBRARIES is /System/Library/Frameworks/Accelerate.framework;/System/Library/Frameworks/Accelerate.framework
-- Debug: BLAS_LIBRARIES is /System/Library/Frameworks/Accelerate.framework
-- Debug: MPI_LIBRARIES is /usr/local/Cellar/open-mpi/1.6.2/lib/libmpi_cxx.dylib;/usr/local/Cellar/open-mpi/1.6.2/lib/libmpi.dylib;/usr/lib/libm.dylib
-- Debug: MPI_INCLUDE is 
-- Debug: MPI_Fortran_COMPILER is /usr/local/bin/mpif77
-- Debug: CMAKE_C_FLAGS is  -pedantic -W -Wextra -Wall -Wno-long-long -Wno-unused-parameter -fPIC -D__STDC_FORMAT_MACROS
-- Debug: CMAKE_CXX_FLAGS is  -pedantic -W -Wextra -Wall -Wno-strict-aliasing -Wno-long-long -Wno-unused-parameter -fPIC -D__STDC_FORMAT_MACROS -Wno-system-headers -isystem /opt/local/include/ -isystem /usr/local/include/ -D__STDC_LIMIT_MACROS -DPROJECT_ROOT=\"/Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/\" -DOMPI_IGNORE_CXX_SEEK
-- Debug: CMAKE_Fortran_FLAGS is  -fno-f2c -I/usr/include/mpi 
-- Debug: CMAKE_Fortran_COMPLIER is /usr/local/bin/gfortran
-- Debug: CMAKE_CXX_COMPILER is /usr/bin/c++
-- Debug: FINAL PACKAGE SETTINGS <================================
CMake Warning at src/mpi/CMakeLists.txt:286 (message):
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@


CMake Warning at src/mpi/CMakeLists.txt:287 (message):
  Can't build ScaLAPACK extensions


CMake Warning at src/mpi/CMakeLists.txt:288 (message):
  without ScaLAPACK, Open MPI, LAPACK, or a BLAS


CMake Warning at src/mpi/CMakeLists.txt:289 (message):
  or without MPI_CXX


CMake Warning at src/mpi/CMakeLists.txt:290 (message):
  or without MPI_Fortran interfaces.


CMake Warning at src/mpi/CMakeLists.txt:291 (message):
  This is only a warning and is not a problem unless


CMake Warning at src/mpi/CMakeLists.txt:292 (message):
  you want the libdense_linear_algebra.so plugin to be built.


CMake Warning at src/mpi/CMakeLists.txt:293 (message):


CMake Warning at src/mpi/CMakeLists.txt:294 (message):
  If you intended to built it, the following will help you debug what is
  missing.


CMake Warning at src/mpi/CMakeLists.txt:295 (message):
  -------------------------------------------------


CMake Warning at src/mpi/CMakeLists.txt:296 (message):
  Debug: MPI_Fortran_COMPLIER is /usr/local/bin/mpif77


CMake Warning at src/mpi/CMakeLists.txt:297 (message):
  Debug: CMAKE_Fortran_COMPLIER is /usr/local/bin/gfortran


CMake Warning at src/mpi/CMakeLists.txt:298 (message):
  Debug: CMAKE_CXX_COMPILER is /usr/bin/c++


CMake Warning at src/mpi/CMakeLists.txt:299 (message):
  Debug: SCALAPACK_FOUND is


CMake Warning at src/mpi/CMakeLists.txt:300 (message):
  Debug: LAPACK_FOUND is TRUE


CMake Warning at src/mpi/CMakeLists.txt:301 (message):
  Debug: BLAS_FOUND is TRUE


CMake Warning at src/mpi/CMakeLists.txt:302 (message):
  Debug: MPI_CXX_FOUND is TRUE


CMake Warning at src/mpi/CMakeLists.txt:303 (message):
  Debug: MPI_Fortran_FOUND is TRUE


CMake Warning at src/mpi/CMakeLists.txt:304 (message):
  ----------------------------------------------------


CMake Warning at src/mpi/CMakeLists.txt:305 (message):
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@


-- *****BEGIN linear_algebra/CMakeLists.txt ***********************
-- MPI_Fortran_COMPILER=/usr/local/bin/mpif77
-- A library with BLAS API found.
-- A library with BLAS API found.
-- A library with LAPACK API found.
-- Using platform ScaLAPACK stack
CMake Warning at src/linear_algebra/CMakeLists.txt:166 (message):
  ScaLAPACK not found.  ScaLAPACK-based functionality will NOT be built.


-- Please look to doc/build.txt for list of dependencies
-- Debug: BLAS_LIBRARIES is /System/Library/Frameworks/Accelerate.framework
-- Debug: LAPACK_LIBRARIES is /System/Library/Frameworks/Accelerate.framework;/System/Library/Frameworks/Accelerate.framework
-- Debug: SCALAPACK_LIBRARIES is 
-- Debug: MPI_LIBRARIES is /usr/local/Cellar/open-mpi/1.6.2/lib/libmpi_cxx.dylib;/usr/local/Cellar/open-mpi/1.6.2/lib/libmpi.dylib;/usr/lib/libm.dylib
-- Debug: CMAKE_C_FLAGS is  -pedantic -W -Wextra -Wall -Wno-long-long -Wno-unused-parameter -fPIC -D__STDC_FORMAT_MACROS 
-- Debug: CMAKE_CXX_FLAGS is  -pedantic -W -Wextra -Wall -Wno-strict-aliasing -Wno-long-long -Wno-unused-parameter -fPIC -D__STDC_FORMAT_MACROS -Wno-system-headers -isystem /opt/local/include/ -isystem /usr/local/include/ -D__STDC_LIMIT_MACROS -DPROJECT_ROOT=\"/Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/\" -DOMPI_IGNORE_CXX_SEEK
-- Debug: CMAKE_Fortran_FLAGS is  -fno-f2c -I/usr/include/mpi 
CMake Warning at src/linear_algebra/CMakeLists.txt:281 (message):
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@


CMake Warning at src/linear_algebra/CMakeLists.txt:282 (message):
  Can't build ScaLAPACK extensions


CMake Warning at src/linear_algebra/CMakeLists.txt:283 (message):
  without ScaLAPACK, Open MPI, LAPACK, or a BLAS


CMake Warning at src/linear_algebra/CMakeLists.txt:284 (message):
  or without MPI_CXX


CMake Warning at src/linear_algebra/CMakeLists.txt:285 (message):
  or without MPI_Fortran interfaces.


CMake Warning at src/linear_algebra/CMakeLists.txt:286 (message):
  This is only a warning and is not a problem unless


CMake Warning at src/linear_algebra/CMakeLists.txt:287 (message):
  you want the libdense_linear_algebra.so plugin to be built.


CMake Warning at src/linear_algebra/CMakeLists.txt:288 (message):


CMake Warning at src/linear_algebra/CMakeLists.txt:289 (message):
  If you intended to built it, the following will help you debug what is
  missing.


CMake Warning at src/linear_algebra/CMakeLists.txt:290 (message):
  -------------------------------------------------


CMake Warning at src/linear_algebra/CMakeLists.txt:291 (message):
  Debug: MPI_Fortran_COMPLIER is /usr/local/bin/mpif77


CMake Warning at src/linear_algebra/CMakeLists.txt:292 (message):
  Debug: CMAKE_Fortran_COMPLIER is /usr/local/bin/gfortran


CMake Warning at src/linear_algebra/CMakeLists.txt:293 (message):
  Debug: CMAKE_CXX_COMPILER is /usr/bin/c++


CMake Warning at src/linear_algebra/CMakeLists.txt:294 (message):
  Debug: SCALAPACK_FOUND is


CMake Warning at src/linear_algebra/CMakeLists.txt:295 (message):
  Debug: LAPACK_FOUND is TRUE


CMake Warning at src/linear_algebra/CMakeLists.txt:296 (message):
  Debug: BLAS_FOUND is TRUE


CMake Warning at src/linear_algebra/CMakeLists.txt:297 (message):
  Debug: MPI_CXX_FOUND is TRUE


CMake Warning at src/linear_algebra/CMakeLists.txt:298 (message):
  Debug: MPI_Fortran_FOUND is TRUE


CMake Warning at src/linear_algebra/CMakeLists.txt:299 (message):
  ----------------------------------------------------


-- Configuring /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/doc/api/Doxyfile.in as /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final/doc/api/Doxyfile
-- Documentation will not be generated automatically. Use -DSCIDB_DOC_TYPE={NONE,FULL,API} to set type of generated documentation. Use make doc to generate FULL documentation manually.
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/kamilgrymuza/work/deps/scidb-12.10.0.5777-Cheshire-final

and scalapack is installed (last lines of output from make install):

Install the project...
-- Install configuration: ""
-- Installing: /usr/local/lib/libscalapack.a
-- Up-to-date: /usr/local/lib/pkgconfig/scalapack.pc
-- Up-to-date: /usr/local/lib/cmake/scalapack-2.0.2/scalapack-config.cmake
-- Up-to-date: /usr/local/lib/cmake/scalapack-2.0.2/scalapack-config-version.cmake
-- Up-to-date: /usr/local/lib/cmake/scalapack-2.0.2/scalapack-targets.cmake
-- Up-to-date: /usr/local/lib/cmake/scalapack-2.0.2/scalapack-targets-noconfig.cmake

What’s important there is that other libraries are picked up correctly from /usr/local/lib so it’s definitely a location being searched for libraries.

Python binding broken on Mac?

Another interesting bit of the cmake output is this:

Python binding broken on Mac! Forcing WITHOUT_PYTHON_BINDING=1

This would suggest that Mac OS X was considered at some point? But if so (or if not) could someone please share some information about what exactly is broken?

Summary

Therefore the question is whether the development team (I’m aware not big) tried running on Mac OS X and whether you guys consider this a platform you’d like to support? Perhaps there are some elementary issues which we’re not aware of ? I’m not complaining or anything like that, just asking for clues and information … I understand you’re doing what you can and perhaps we can help? SciDB looks interesting, and clearly there are people who’d like to test it on a Mac which is a development platform of choice for many people.

So, I’m sorry for a lengthy post, but I wanted to provide as much information as possible. If someone could please shed some light on those issues I think it would be for the benefit of many people running Mac OS X. I’m willing to provide additional information and also help test SciDB and any ideas anybody might have about getting it to work.

Thank you for your time
Kamil Grymuza