Host key verification error


#1

Hi.

I am trying to run the load.sh shell script provided with the orderbook example: https://github.com/Paradigm4/orderbook-example. I am getting an error related Host Key Verification.

[code]./load.sh EQY_US_ALL_ARCA_BOOK_20130404.csv.gz
/opt/scidb/14.12/bin:/opt/scidb/14.12/share/scidb:/opt/scidb/14.12/bin:/opt/scidb/14.12/share/scidb:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/scidb/14.12/bin
Computing type-pattern from load schema.
Getting SciDB configuration information.
This SciDB installation has 8 instance(s).
Creating CSV fragment FIFOs.
Creating DLF fragment FIFOs.

ERROR

Failed to create DLF fragment: “/home/scidb/scidb_data/000/0/stdin.csv.kRHrNe.dlf”.
Host key verification failed.

##############################

Removing CSV fragmemt FIFOs.
Failure: Error Encountered.

Computing type-pattern from load schema.
Getting SciDB configuration information.
This SciDB installation has 8 instance(s).
Creating CSV fragment FIFOs.
Creating DLF fragment FIFOs.

ERROR

Failed to create DLF fragment: “/home/scidb/scidb_data/000/0/stdin.csv.SY503V.dlf”.
Host key verification failed.

##############################

Removing CSV fragmemt FIFOs.
Failure: Error Encountered.

Computing type-pattern from load schema.
Getting SciDB configuration information.
This SciDB installation has 8 instance(s).
Creating CSV fragment FIFOs.
Creating DLF fragment FIFOs.

ERROR

Failed to create DLF fragment: “/home/scidb/scidb_data/000/0/stdin.csv.oh3Ihm.dlf”.
Host key verification failed.

##############################

Removing CSV fragmemt FIFOs.
Failure: Error Encountered.

UserQueryException in file: src/query/parser/Translator.cpp function: createArrayReferenceParam line: 677
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array ‘flat’ does not exist.
remove(flat)
^^^^
UserQueryException in file: src/query/parser/Translator.cpp function: passAFLOperator line: 535
Error id: scidb::SCIDB_SE_SYNTAX::SCIDB_LE_WRONG_OPERATOR_ARGUMENTS_COUNT
Error description: Query syntax error. Operator ‘create_array’ expected 3 argument(s) but was passed 2.
create_array(flat, type:char,sequence:int64,ref:int64,exchange:char,ordertype:char,size:int64,symbol:string,price:double,seconds:int64,milliseconds:int64,system:char,qid:string [i=0:*,10000,0])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UserQueryException in file: src/query/parser/Translator.cpp function: createArrayReferenceParam line: 677
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array ‘load_a’ does not exist.
insert(load_a, flat)
^^^^^^

real 0m0.010s
user 0m0.004s
sys 0m0.004s
UserQueryException in file: src/query/parser/Translator.cpp function: createArrayReferenceParam line: 677
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array ‘flat’ does not exist.
aggregate(flat, count())
^^^^
Count of add orders is
UserQueryException in file: src/query/parser/Driver.cpp function: fail line: 146
Error id: scidb::SCIDB_SE_PARSER::SCIDB_LE_QUERY_PARSING_ERROR
Error description: Error during query parsing. Query parser failed with error ‘syntax error’.
insert(cast(redimension(apply(load_m,j,i+), type:char,sequence:int64,ref:int64,exchange:char,ordertype:char,size:int64,symbol:string,price:double,seconds:int64,milliseconds:int64,system:char,qid:string [j=0:
,10000,0]),type:char,sequence:int64,ref:int64,exchange:char,ordertype:char,size:int64,symbol:string,price:double,seconds:int64,milliseconds:int64,system:char,qid:string [i=0:*,10000,0]), flat)
^

real 0m0.010s
user 0m0.003s
sys 0m0.006s
UserQueryException in file: src/query/parser/Translator.cpp function: createArrayReferenceParam line: 677
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array ‘flat’ does not exist.
aggregate(flat, count())
^^^^
Count of add + modify orders is
UserQueryException in file: src/query/parser/Driver.cpp function: fail line: 146
Error id: scidb::SCIDB_SE_PARSER::SCIDB_LE_QUERY_PARSING_ERROR
Error description: Error during query parsing. Query parser failed with error ‘syntax error’.
insert(cast(redimension(apply(load_d,j,i+,price,double(0), size, int64(0)), type:char,sequence:int64,ref:int64,exchange:char,ordertype:char,size:int64,symbol:string,price:double,seconds:int64,milliseconds:int64,system:char,qid:string [j=0:
,10000,0]),type:char,sequence:int64,ref:int64,exchange:char,ordertype:char,size:int64,symbol:string,price:double,seconds:int64,milliseconds:int64,system:char,qid:string [i=0:*,10000,0]), flat)
^

real 0m0.009s
user 0m0.006s
sys 0m0.003s
UserQueryException in file: src/query/parser/Translator.cpp function: createArrayReferenceParam line: 677
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array ‘flat’ does not exist.
aggregate(flat, count(*))
^^^^
Count of add + modify + delete orders is
UserQueryException in file: src/query/parser/Translator.cpp function: createArrayReferenceParam line: 677
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array ‘load_a’ does not exist.
remove(load_a)
^^^^^^
UserQueryException in file: src/query/parser/Translator.cpp function: createArrayReferenceParam line: 677
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array ‘load_d’ does not exist.
remove(load_d)
^^^^^^
UserQueryException in file: src/query/parser/Translator.cpp function: createArrayReferenceParam line: 677
Error id: scidb::SCIDB_SE_QPROC::SCIDB_LE_ARRAY_DOESNT_EXIST
Error description: Query processor error. Array ‘load_m’ does not exist.
remove(load_m)
^^^^^^
Done loading raw data. Proceed to the ./redim.sh script
[/code]

I have checked that I am running a bash shell (I assume this is what I want):

echo $SHELL /bin/bash

The paths look okay to me:

$PATH bash: /opt/scidb/14.12/bin:/opt/scidb/14.12/share/scidb:/opt/scidb/14.12/bin:/opt/scidb/14.12/share/scidb:/opt/scidb/14.12/bin:/opt/scidb/14.12/share/scidb:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games: No such file or directory

I found a related topic http://www.scidb.org/forum/viewtopic.php?f=11&t=1471. It is entirely likely I am missing some important point here, but this discussion did not to help me. I can’t think of where to go from here.


#2

A simple way around is this:

# reset all the host keys for scidb and collect them ONCE
sudo rm ~scidb/.ssh/known_host
sudo -u scidb bash -c "ssh-keyscan host1 host2 host3 > ~scidb/.ssh/known_hosts"

That is if you trust that your host keys are ok

Also you could create a config file:

# reset all the host keys for scidb and ignore future changes forever
sudo rm ~scidb/.ssh/known_host
cat <<EOF>~scidb/.ssh/known_hosts
## default to NO authentication for remote hosts
Host *
    UserKnownHostsFile /dev/null
    StrictHostKeyChecking no
EOF

#3

Hi,

The loadcsv.py script requires passwordless ssh access to all the cluster nodes. Based on

askubuntu.com/questions/45679/ss … iled-error

I think the problem may be with your ~/.ssh/known_hosts file. As the link says, the -R option can remove stale host keys.

Basically you should be able to run ssh scidbuser@host date to all hosts in the cluster without being prompted for a password. If you get the “host not in your known_hosts file, add it?” message, do add it. Then you should be OK.

I also notice that the orderbook example is a little stale, in that the create_array() operator deprecated in 14.12 in favor of using AQL “create array …” statements. In particular there was an undocumented change to the operator, and it was decided to hide its existence in favor of AQL. So you could convert the script to instead use AQL to create arrays, or (undocumented, beware!) you can add a third create_array() bool parameter of false to get the previous behavior.


#4

Hi Mike,

Somehow I am not able to make this work. I have tried a number of ideas from different ubuntu posts/discussion groups, but the host key verification error refuses to go away. I am not an ssh expert, alas.

I am running the scidb 14.12 on an ec2 m3.2xlarge cluster. I have been assuming that in this configuration the coordinator and worker servers are one and the same. If not, I would need to find the names of all the servers in the cluster. My lack of success makes me wonder if this is the problem.

Working from the single hostname assumption I tried the simplest approach first which should remove the offending key. I get the host name and then remove the key:

> hostname
ip-172-31-1-255
> ssh-keygen -R ip-172-31-1-255
/home/scidb/.ssh/known_hosts updated.
Original contents retained as /home/scidb/.ssh/known_hosts.old

This looks promising, but no joy. Running ./load.sh EQY_US_ALL_ARCA_BOOK_20130404.csv.gz produces the host key verification error.

Next, I tried something more complicated approach which should direct ubuntu to look in /dev/null for keys, bypassing the process:

This proceedure starts a new shell, supposedly with the ssh authentication bypased. However, when I run ./load.sh EQY_US_ALL_ARCA_BOOK_20130404.csv.gz I get the same host key verification error once again.

Thanks also for the tip about updating the scidp query code. I changed the offending line in the bash script to:

Once the ssh issue is resolved I can see if that works or not:)

Regards


#5

Here’s a thought

Consider editing (on the outgoing / coordinator side)
/etc/ssh/ssh_config

Add/uncomment this line:
StrictHostKeyChecking no

That turns off the whole host key checking process.


#6

Thanks Alex. I updated the /etc/ssh/ssh_config file and the script now runs!