Compiling scidb-12.3.0.4813-Snowdrop-final on OSX


#1

Folks:

I attended the SciDB tutorial at XLDB, and am interested in doing some experiments with user defined operators, but I’m getting a fair number of compiler errors on my Mac, for example:

[ 3%] Building CXX object src/system/CMakeFiles/system_lib.dir/Cluster.cpp.o In file included from /Users/tshead/src/scidb-12.3.0.4813-Snowdrop-final/include/array/MemArray.h:37, from /Users/tshead/src/scidb-12.3.0.4813-Snowdrop-final/include/query/Operator.h:55, from /Users/tshead/src/scidb-12.3.0.4813-Snowdrop-final/src/query/QueryProcessor.h:46, from /Users/tshead/src/scidb-12.3.0.4813-Snowdrop-final/src/network/NetworkManager.h:41, from /Users/tshead/src/scidb-12.3.0.4813-Snowdrop-final/src/system/Cluster.cpp:32: /Users/tshead/src/scidb-12.3.0.4813-Snowdrop-final/include/query/Query.h:64: error: ‘bool boost::operator==(const boost::shared_ptr<scidb::SystemCatalog::LockDesc>&, const boost::shared_ptr<scidb::SystemCatalog::LockDesc>&)’ is not declared in ‘boost’ /Users/tshead/src/scidb-12.3.0.4813-Snowdrop-final/include/query/Query.h:67: error: ‘bool boost::operator!=(const boost::shared_ptr<scidb::SystemCatalog::LockDesc>&, const boost::shared_ptr<scidb::SystemCatalog::LockDesc>&)’ is not declared in ‘boost’

… which seems like it would have to be a boost version issue - anyone have any guidance?

Many thanks,
Tim Shead


#2

Following-up, I’ve addressed all of the compile-time issues, but I’m stumped by the following link error:

Linking CXX executable ../bin/scidb Undefined symbols: "pqxx::internal::freemem_result_data(pqxx::internal::result_data*)", referenced from: scidb::SystemCatalog::getArrayDesc(unsigned long long)in libcatalog_lib.a(SystemCatalog.cpp.o) ... and many more references ...

Building, it is linking against pqxx:

$ make VERBOSE=1

cd /Users/tshead/build/scidb/src && /opt/local/bin/cmake -E cmake_link_script CMakeFiles/scidb.dir/link.txt --verbose=1
/usr/bin/c++ -pedantic -W -Wextra -Wall -Wno-strict-aliasing -Wno-long-long -Wno-unused-parameter -fPIC -D__STDC_FORMAT_MACROS -Wno-system-headers -isystem /opt/local/include/ -isystem /usr/local/include/ -D__STDC_LIMIT_MACROS -DPROJECT_ROOT="/Users/tshead/src/scidb-12.3.0.4813-Snowdrop-final/" -O2 -g -DNDEBUG -ggdb3 -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/scidb.dir/network/entry.cpp.o -o …/bin/scidb -L/opt/local/lib network/libnetwork_lib.a util/libutil_lib.a system/libsystem_lib.a query/libqproc_lib.a array/libarray_lib.a smgr/io/libio_lib.a query/libscalar_proc_lib.a query/ops/libops_lib.a network/libnetwork_lib.a query/libqproc_lib.a smgr/io/libio_lib.a query/libscalar_proc_lib.a query/ops/libops_lib.a network/proto/libscidb_msg_lib.a /opt/local/lib/libprotobuf.dylib system/libsystem_lib.a array/libarray_lib.a system/catalog/libcatalog_lib.a -lpqxx /opt/local/lib/postgresql83/libpq.dylib smgr/delta/libdelta_lib.a …/extern/bsdiff/libbsdiff.a util/libutil_lib.a /opt/local/lib/liblog4cxx.dylib …/extern/lib_json/libjson_lib.a smgr/compression/libcompression_lib.a /opt/local/lib/libboost_system-mt.dylib /opt/local/lib/libboost_program_options-mt.dylib /opt/local/lib/libboost_serialization-mt.dylib /opt/local/lib/libboost_regex-mt.dylib /opt/local/lib/libboost_filesystem-mt.dylib /opt/local/lib/libboost_thread-mt.dylib /usr/lib/libz.dylib /usr/lib/libbz2.dylib

Any thoughts?

Cheers,
Tim


#3

Following-up again, I was able to solve this problem by explicitly linking against /opt/local/lib/libpqxx.a. I suspect that the -lpqxx link line probably links against libpqxx.dylib by default, which probably doesn’t export the internal symbols.

So, what’s the best way to submit some patches? I only ran across one platform-specific issue (mallopt() is Linux-specific). Everything else was a simple C++ coding or linking issue. Would be nice to get these OSX fixes committed upstream … while I’m at it, when will 12.7 be released, and is there any way to get access (even read-only) to a repository, to ease patch creation?

Cheers,
Tim


#4

Hi Tim,

Thank you for providing this information and volunteering this effort!

Here’s the state of affairs right now:

  • we’re a small team and we haven’t opened ourselves up to external contributors yet - though it’s definitely a goal
  • we’re about to wrap up release 12.10 (formerly 12.7), currently in bugfix and final test mode; the release will be out in October.
  • once we start working on the next release, we are likely going to switch source control systems; using svn now, considering switching to git

So, please bear with us as we get this version released and get our source-control affairs in order. As we do that, we will work on a process to accept external contributions. Does this work for you?

Also, if you haven’t already, take a look at viewtopic.php?f=18&t=557. There’s an EC2 AMI image that you can use. It’s both - a way to easily launch SciDB in EC2 as well as a preview of 12.10. Just another option for now…

  • Alex Poliakov

#5

Hi, Tim,

may I ask how you solved the first two issues? I’m having the same problem when installing SciDB12.3 to my Ubuntu12.04


#6

Alex: Fair enough, I’ll try to be patient :wink:


#7

amelie:

Disclaimer: this is very rough hacking to see if I could get things to build, real upstream changes will need a little more work. Broadly speaking, I: commented-out some forward-declared templates, tweaked an #include, commented-out calls to mallopt(), and commented-out a catch statement that referenced an exception class that my pqxx library doesn’t seem to have. For the link issue, I modified src/CMakeFiles/scidb.dir/link.txt in my build directory, replacing -lpqxx with /opt/local/lib/libpqxx.a. This is an even bigger hack, since it will get overwritten the next time I run CMake … I just didn’t feel like modifying the CMakeList.txt files.

Here’s a diff:

[code]$ diff -ru scidb-12.3.0.4813-Snowdrop-final-original/ scidb-12.3.0.4813-Snowdrop-final
diff -ru scidb-12.3.0.4813-Snowdrop-final-original/include/query/Query.h scidb-12.3.0.4813-Snowdrop-final/include/query/Query.h
— scidb-12.3.0.4813-Snowdrop-final-original/include/query/Query.h 2012-03-06 06:14:42.000000000 -0700
+++ scidb-12.3.0.4813-Snowdrop-final/include/query/Query.h 2012-09-25 11:20:47.000000000 -0600
@@ -54,6 +54,7 @@
#include “system/SystemCatalog.h”
#include <util/WorkQueue.h>

+/*
namespace boost
{
template<>
@@ -66,6 +67,7 @@
bool operator!= (boost::shared_ptrscidb::SystemCatalog::LockDesc const& l,
boost::shared_ptrscidb::SystemCatalog::LockDesc const& r);
} // namespace boost
+*/

namespace std
{
diff -ru scidb-12.3.0.4813-Snowdrop-final-original/include/system/Cluster.h scidb-12.3.0.4813-Snowdrop-final/include/system/Cluster.h
— scidb-12.3.0.4813-Snowdrop-final-original/include/system/Cluster.h 2012-03-06 06:14:42.000000000 -0700
+++ scidb-12.3.0.4813-Snowdrop-final/include/system/Cluster.h 2012-09-25 11:23:56.000000000 -0600
@@ -31,6 +31,7 @@
#ifndef CLUSTER_H
#define CLUSTER_H

+#include <boost/shared_ptr.hpp>
#include
#include
#include <util/Singleton.h>
@@ -118,6 +119,7 @@

} // namespace scidb

+/*
namespace boost
{
template<>
@@ -133,6 +135,7 @@
boost::shared_ptr const& r);

} // namespace boost
+*/

namespace std
{
diff -ru scidb-12.3.0.4813-Snowdrop-final-original/src/network/entry.cpp scidb-12.3.0.4813-Snowdrop-final/src/network/entry.cpp
— scidb-12.3.0.4813-Snowdrop-final-original/src/network/entry.cpp 2012-04-09 00:47:56.000000000 -0600
+++ scidb-12.3.0.4813-Snowdrop-final/src/network/entry.cpp 2012-09-25 11:41:54.000000000 -0600
@@ -44,7 +44,7 @@
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/resource.h>
-#include <malloc.h>
+#include <sys/malloc.h>

#include “network/NetworkManager.h”
#include “system/SciDBConfigOptions.h”
@@ -129,6 +129,7 @@
}
}

+/*
int largeMemLimit = cfg->getOption(CONFIG_LARGE_MEMALLOC_LIMIT);
if (largeMemLimit>0 && (0==mallopt(M_MMAP_MAX, largeMemLimit))) {

@@ -140,7 +141,8 @@

    LOG4CXX_WARN(logger, "Failed to set small-memalloc-size");
}

+*/
+
boost::shared_ptr messagesJobQueue = boost::make_shared();

// Here we can play with thread number

diff -ru scidb-12.3.0.4813-Snowdrop-final-original/src/system/catalog/SystemCatalog.cpp scidb-12.3.0.4813-Snowdrop-final/src/system/catalog/SystemCatalog.cpp
— scidb-12.3.0.4813-Snowdrop-final-original/src/system/catalog/SystemCatalog.cpp 2012-03-13 21:33:39.000000000 -0600
+++ scidb-12.3.0.4813-Snowdrop-final/src/system/catalog/SystemCatalog.cpp 2012-09-25 11:38:46.000000000 -0600
@@ -2676,11 +2676,13 @@
tr.commit();
invalidateArrayCache(old_array_name);
}
+/*
catch (const unique_violation& e)
{
LOG4CXX_ERROR(logger, “SystemCatalog::renameArray: unique constraint violation:”<< e.what());
throw SYSTEM_EXCEPTION(SCIDB_SE_SYSCAT, SCIDB_LE_ARRAY_ALREADY_EXIST) << new_array_name;
}
+*/
catch (const sql_error &e)
{
throw SYSTEM_EXCEPTION(SCIDB_SE_SYSCAT, SCIDB_LE_PG_QUERY_EXECUTION_FAILED) << e.query[/code]

Now to setup my configuration and get it running.

Cheers,
Tim


#8

Thanks, Tim,

This fix my compiling problem too!
Just out of curiosity, how do you figure out where to modify?
And do you think there may be some run time issues because of the commented codes?


#9

amelie:

Glad to hear it. The commented-out calls to mallopt() could definitely have an effect at runtime, but in your case you could put them back in since you’re building on Linux. The commented-out catch statement should be harmless, it’s just providing some extra log output for a specific error condition, one which can’t occur with my version of pqxx (again, you might be able to put it back for your version). The rest of the changes should have no effect at runtime. I’m not sure why those forward-declared templates were there to begin with, I don’t think they’re required by the language, so perhaps a workaround for a compiler bug.

Cheers,
Tim