Errorless compilation but some tests do not pass

[nadir199]

Issue summary

When I compile Caffe cloned from the official repository, the compilation is successful, but when trying to compile SkimCaffe, I don't get any errors while compiling (only some warnings), but when I run make runtest, some tests pass, but some don't.

nadd@nadd-GL552VW:~/Tools/Compile/SkimCaffe$ make runtest
make -C src/libxsmm AVX=2 OPT=3 DBG=0 FC=
make[1]: Entering directory '/home/nadd/Tools/Compile/SkimCaffe/src/libxsmm'
make[1]: Nothing to be done for 'libxsmm'.
make[1]: Leaving directory '/home/nadd/Tools/Compile/SkimCaffe/src/libxsmm'
make -C src/SpMP DBG=0
make[1]: Entering directory '/home/nadd/Tools/Compile/SkimCaffe/src/SpMP'
make[1]: 'libspmp.a' is up to date.
make[1]: Leaving directory '/home/nadd/Tools/Compile/SkimCaffe/src/SpMP'
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
icpc -shared -o .build_release/lib/libcaffe.so.1.0.0-rc3 .build_release/src/caffe/proto/caffe.pb.o .build_release/src/caffe/data_transformer.o .build_release/src/caffe/layer_factory.o .build_release/src/caffe/net.o .build_release/src/caffe/data_reader.o .build_release/src/caffe/internal_thread.o .build_release/src/caffe/util/math_functions.o .build_release/src/caffe/util/math_functions_intel.o .build_release/src/caffe/util/im2col.o .build_release/src/caffe/util/signal_handler.o .build_release/src/caffe/util/benchmark.o .build_release/src/caffe/util/db_lmdb.o .build_release/src/caffe/util/cudnn.o .build_release/src/caffe/util/upgrade_proto.o .build_release/src/caffe/util/db.o .build_release/src/caffe/util/hdf5.o .build_release/src/caffe/util/mmio.o .build_release/src/caffe/util/blocking_queue.o .build_release/src/caffe/util/libxsmm_spmv.o .build_release/src/caffe/util/io.o .build_release/src/caffe/util/db_leveldb.o .build_release/src/caffe/util/cpu_info.o .build_release/src/caffe/util/insert_splits.o .build_release/src/caffe/solvers/adagrad_solver.o .build_release/src/caffe/solvers/nesterov_solver.o .build_release/src/caffe/solvers/rmsprop_solver.o .build_release/src/caffe/solvers/adadelta_solver.o .build_release/src/caffe/solvers/sgd_solver.o .build_release/src/caffe/solvers/adam_solver.o .build_release/src/caffe/syncedmem.o .build_release/src/caffe/layer.o .build_release/src/caffe/common.o .build_release/src/caffe/layers/mkl_relu_layer.o .build_release/src/caffe/layers/softmax_loss_layer.o .build_release/src/caffe/layers/bnll_layer.o .build_release/src/caffe/layers/euclidean_loss_layer.o .build_release/src/caffe/layers/hinge_loss_layer.o .build_release/src/caffe/layers/cudnn_lrn_layer.o .build_release/src/caffe/layers/cudnn_lcn_layer.o .build_release/src/caffe/layers/im2col_layer.o .build_release/src/caffe/layers/window_data_layer.o .build_release/src/caffe/layers/base_conv_layer.o .build_release/src/caffe/layers/mkl_convolution_layer.o .build_release/src/caffe/layers/cudnn_pooling_layer.o .build_release/src/caffe/layers/sigmoid_layer.o .build_release/src/caffe/layers/inner_product_layer.o .build_release/src/caffe/layers/silence_layer.o .build_release/src/caffe/layers/pooling_layer.o .build_release/src/caffe/layers/deconv_layer.o .build_release/src/caffe/layers/embed_layer.o .build_release/src/caffe/layers/eltwise_layer.o .build_release/src/caffe/layers/parameter_layer.o .build_release/src/caffe/layers/conv_relu_pool_layer.o .build_release/src/caffe/layers/relu_layer.o .build_release/src/caffe/layers/mkl_lrn_layer.o .build_release/src/caffe/layers/sigmoid_cross_entropy_loss_layer.o .build_release/src/caffe/layers/filter_layer.o .build_release/src/caffe/layers/conv_relu_pool_lrn_layer.o .build_release/src/caffe/layers/conv_relu_layer.o .build_release/src/caffe/layers/elu_layer.o .build_release/src/caffe/layers/inner_product_relu_dropout_layer.o .build_release/src/caffe/layers/contrastive_loss_layer.o .build_release/src/caffe/layers/reduction_layer.o .build_release/src/caffe/layers/input_layer.o .build_release/src/caffe/layers/concat_layer.o .build_release/src/caffe/layers/cudnn_conv_layer.o .build_release/src/caffe/layers/bias_layer.o .build_release/src/caffe/layers/absval_layer.o .build_release/src/caffe/layers/mkl_pooling_layer.o .build_release/src/caffe/layers/recurrent_layer.o .build_release/src/caffe/layers/multinomial_logistic_loss_layer.o .build_release/src/caffe/layers/data_layer.o .build_release/src/caffe/layers/image_data_layer.o .build_release/src/caffe/layers/mvn_layer.o .build_release/src/caffe/layers/memory_data_layer.o .build_release/src/caffe/layers/lstm_unit_layer.o .build_release/src/caffe/layers/dropout_layer.o .build_release/src/caffe/layers/mkl_eltwise_layer.o .build_release/src/caffe/layers/split_layer.o .build_release/src/caffe/layers/argmax_layer.o .build_release/src/caffe/layers/mkl_batch_norm_layer.o .build_release/src/caffe/layers/conv_layer.o .build_release/src/caffe/layers/reshape_layer.o .build_release/src/caffe/layers/mkl_concat_layer.o .build_release/src/caffe/layers/spp_layer.o .build_release/src/caffe/layers/accuracy_layer.o .build_release/src/caffe/layers/exp_layer.o .build_release/src/caffe/layers/crop_layer.o .build_release/src/caffe/layers/cudnn_sigmoid_layer.o .build_release/src/caffe/layers/power_layer.o .build_release/src/caffe/layers/tanh_layer.o .build_release/src/caffe/layers/softmax_layer.o .build_release/src/caffe/layers/cudnn_softmax_layer.o .build_release/src/caffe/layers/prelu_layer.o .build_release/src/caffe/layers/infogain_loss_layer.o .build_release/src/caffe/layers/cudnn_tanh_layer.o .build_release/src/caffe/layers/lstm_layer.o .build_release/src/caffe/layers/threshold_layer.o .build_release/src/caffe/layers/flatten_layer.o .build_release/src/caffe/layers/rnn_layer.o .build_release/src/caffe/layers/scale_layer.o .build_release/src/caffe/layers/cudnn_relu_layer.o .build_release/src/caffe/layers/batch_norm_layer.o .build_release/src/caffe/layers/slice_layer.o .build_release/src/caffe/layers/dummy_data_layer.o .build_release/src/caffe/layers/winograd_layer.o .build_release/src/caffe/layers/hdf5_data_layer.o .build_release/src/caffe/layers/hdf5_output_layer.o .build_release/src/caffe/layers/mkl_split_layer.o .build_release/src/caffe/layers/relu_pooling_layer.o .build_release/src/caffe/layers/base_data_layer.o .build_release/src/caffe/layers/log_layer.o .build_release/src/caffe/layers/neuron_layer.o .build_release/src/caffe/layers/batch_reindex_layer.o .build_release/src/caffe/layers/loss_layer.o .build_release/src/caffe/layers/lrn_layer.o .build_release/src/caffe/layers/tile_layer.o .build_release/src/caffe/parallel.o .build_release/src/caffe/solver.o .build_release/src/caffe/mkl_memory.o .build_release/src/caffe/blob.o -Wl,-soname,libcaffe.so.1.0.0-rc3 -Wl,-rpath,$ORIGIN/../lib -qopenmp -pthread -fPIC -DCAFFE_VERSION=1.0.0-rc3 -DNDEBUG -O3 -DUSE_LEVELDB -DUSE_LMDB -DMKL2017_SUPPORTED -DCPU_ONLY -DUSE_MKL -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/local/include -Isrc -Isrc/libxsmm/include -I/usr/local/cuda/include -I/usr/include/hdf5/serial/ -I.build_release/src -I./src -I./include -I/opt/intel/mkl/include -I/opt/intel/mkl/include/fftw -Wall -Wno-sign-compare -L/usr/lib -L/usr/local/lib -L/usr/lib -Lsrc/SpMP -Lsrc/libxsmm/lib -L/usr/local/cuda/lib64 -L/usr/lib/x86_64-linux-gnu/hdf5/serial -L/opt/intel/mkl/lib -L/opt/intel/mkl/lib/intel64 -L.build_release/lib -lglog -lgflags -lprotobuf -lboost_system -lboost_filesystem -lm -lhdf5_hl -lhdf5 -lspmp -lxsmm -lleveldb -lsnappy -llmdb -lboost_thread -lstdc++ -lmkl_rt
.build_release/tools/caffe
freq = 2.592e+09
caffe: command line brew
usage: caffe

commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time

Flags from tools/caffe.cpp:
-forward_only (Optional; Execute only forward pass) type: bool
default: false
-gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
'-gpu all' to run on all available GPUs. The effective training batch
size is multiplied by the number of devices.) type: string default: ""
-iterations (The number of iterations to run.) type: int32 default: 50
-level (Optional; network level.) type: int32 default: 0
-model (The model definition protocol buffer text file.) type: string
default: ""
-phase (Optional; network phase (TRAIN or TEST). Only used for 'time'.)
type: string default: ""
-sighup_effect (Optional; action to take when a SIGHUP signal is received:
snapshot, stop or none.) type: string default: "snapshot"
-sigint_effect (Optional; action to take when a SIGINT signal is received:
snapshot, stop or none.) type: string default: "stop"
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: ""
-solver (The solver definition protocol buffer text file.) type: string
default: ""
-stage (Optional; network stages (not to be confused with phase), separated
by ','.) type: string default: ""
-weights (Optional; the pretrained weights to initialize finetuning,
separated by ','. Cannot be set simultaneously with snapshot.)
type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle --gtest_filter="-GPU"
Note: Google Test filter = -GPU
Note: Randomizing tests' orders with a seed of 12730 .
[==========] Running 998 tests from 141 test cases.
[----------] Global test environment set-up.
[----------] 5 tests from EmbedLayerTest/1, where TypeParam = caffe::CPUDevice
[ RUN ] EmbedLayerTest/1.TestForward
[ OK ] EmbedLayerTest/1.TestForward (2 ms)
[ RUN ] EmbedLayerTest/1.TestGradient
[ OK ] EmbedLayerTest/1.TestGradient (30 ms)
[ RUN ] EmbedLayerTest/1.TestForwardWithBias
[ OK ] EmbedLayerTest/1.TestForwardWithBias (10 ms)
[ RUN ] EmbedLayerTest/1.TestGradientWithBias
[ OK ] EmbedLayerTest/1.TestGradientWithBias (51 ms)
[ RUN ] EmbedLayerTest/1.TestSetUp
[ OK ] EmbedLayerTest/1.TestSetUp (0 ms)
[----------] 5 tests from EmbedLayerTest/1 (93 ms total)

[----------] 1 test from CPUStochasticPoolingLayerTest/0, where TypeParam = float
[ RUN ] CPUStochasticPoolingLayerTest/0.TestSetup
[ OK ] CPUStochasticPoolingLayerTest/0.TestSetup (0 ms)
[----------] 1 test from CPUStochasticPoolingLayerTest/0 (0 ms total)

[----------] 1 test from MultinomialLogisticLossLayerTest/0, where TypeParam = float
[ RUN ] MultinomialLogisticLossLayerTest/0.TestGradientCPU
[ OK ] MultinomialLogisticLossLayerTest/0.TestGradientCPU (1 ms)
[----------] 1 test from MultinomialLogisticLossLayerTest/0 (1 ms total)

[----------] 11 tests from RandomNumberGeneratorTest/1, where TypeParam = double
[ RUN ] RandomNumberGeneratorTest/1.TestRngGaussianTimesBernoulli
[ OK ] RandomNumberGeneratorTest/1.TestRngGaussianTimesBernoulli (0 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngBernoulli
[ OK ] RandomNumberGeneratorTest/1.TestRngBernoulli (0 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngUniform2
[ OK ] RandomNumberGeneratorTest/1.TestRngUniform2 (0 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngBernoulli2
[ OK ] RandomNumberGeneratorTest/1.TestRngBernoulli2 (0 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngGaussian
[ OK ] RandomNumberGeneratorTest/1.TestRngGaussian (0 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngUniformTimesBernoulli
[ OK ] RandomNumberGeneratorTest/1.TestRngUniformTimesBernoulli (0 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngUniformTimesUniform
[ OK ] RandomNumberGeneratorTest/1.TestRngUniformTimesUniform (1 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngGaussianTimesGaussian
[ OK ] RandomNumberGeneratorTest/1.TestRngGaussianTimesGaussian (0 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngBernoulliTimesBernoulli
[ OK ] RandomNumberGeneratorTest/1.TestRngBernoulliTimesBernoulli (0 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngUniform
[ OK ] RandomNumberGeneratorTest/1.TestRngUniform (0 ms)
[ RUN ] RandomNumberGeneratorTest/1.TestRngGaussian2
[ OK ] RandomNumberGeneratorTest/1.TestRngGaussian2 (0 ms)
[----------] 11 tests from RandomNumberGeneratorTest/1 (1 ms total)

[----------] 26 tests from NetTest/1, where TypeParam = caffe::CPUDevice
[ RUN ] NetTest/1.TestHasBlob
[ OK ] NetTest/1.TestHasBlob (2 ms)
[ RUN ] NetTest/1.TestHasLayer
[ OK ] NetTest/1.TestHasLayer (1 ms)
[ RUN ] NetTest/1.TestComboLossWeight
F1029 04:22:19.551991 1528 inner_product_layer.cpp:215] Not Implemented Yet
*** Check failure stack trace: ***
@ 0x7f2cd181b0cd google::LogMessage::Fail()
@ 0x7f2cd181cf33 google::LogMessage::SendToLog()
@ 0x7f2cd181ac28 google::LogMessage::Flush()
@ 0x7f2cd181d999 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f2cce80cb2a caffe::InnerProductLayer<>::Forward_cpu()
@ 0x7f2cce67f43a caffe::Net<>::Forward()
@ 0x89edc1 caffe::Net<>::ForwardBackward()
@ 0x89a819 caffe::NetTest_TestComboLossWeight_Test<>::TestBody()
@ 0xd17eee testing::Test::Run()
@ 0xd17aa0 testing::TestInfo::Run()
@ 0xd17772 testing::TestCase::Run()
@ 0xd16bfc testing::internal::UnitTestImpl::RunAllTests()
@ 0xd13466 testing::UnitTest::Run()
@ 0x45bdbd main
@ 0x7f2ccd86db97 __libc_start_main
@ 0x45bc6a _start
Aborted (core dumped)
Makefile:561: recipe for target 'runtest' failed
make: *** [runtest] Error 134

Steps to reproduce

Try to compile SkimCaffe:
In SkimCaffe directory, execute :
make libxsmm
make all
make test
make runtest

Your system configuration

Operating system: Ubuntu 18
Compiler: icpc
CUDA version (if applicable): No (CPU ONLY)
CUDNN version (if applicable):
BLAS: mkl
Python or MATLAB version (for pycaffe and matcaffe respectively):

Thank you.