训练正常，测试推理时检测不到任何东西

[chuck0518]

I0413 18:42:17.436262 23758 solver.cpp:253] Iteration 20510 (0.360757 iter/s, 27.7195s/10 iters), loss = 6.13863
I0413 18:42:17.436317 23758 solver.cpp:272] Train net output #0: det_loss1 = 0.341735 (* 1 = 0.341735 loss)
I0413 18:42:17.436328 23758 solver.cpp:272] Train net output #1: det_loss2 = 3.03929 (* 1 = 3.03929 loss)
I0413 18:42:17.436337 23758 sgd_solver.cpp:121] Iteration 20510, lr = 0.0005
I0413 18:42:20.734402 23758 yolov3_layer.cpp:532] noobj: 0.00250922 obj: 0.476783 iou: 0.761839 cat: 0.941965 recall: 0.991667 recall75: 0.596111 count: 5
I0413 18:42:20.778740 23758 yolov3_layer.cpp:532] noobj: 0.0030255 obj: 0.461223 iou: 0.589617 cat: 0.919322 recall: 0.692824 recall75: 0.351311 count: 25
I0413 18:42:26.167347 23758 yolov3_layer.cpp:532] noobj: 0.00195143 obj: 0.640569 iou: 0.791826 cat: 0.992684 recall: 0.989796 recall75: 0.766667 count: 3
I0413 18:42:26.212055 23758 yolov3_layer.cpp:532] noobj: 0.00242032 obj: 0.427725 iou: 0.554173 cat: 0.901831 recall: 0.621256 recall75: 0.318708 count: 23
I0413 18:42:30.826737 23758 yolov3_layer.cpp:532] noobj: 0.00174544 obj: 0.444056 iou: 0.783672 cat: 0.972636 recall: 0.980769 recall75: 0.724359 count: 2
I0413 18:42:30.862072 23758 yolov3_layer.cpp:532] noobj: 0.0034129 obj: 0.412179 iou: 0.529127 cat: 0.856678 recall: 0.615313 recall75: 0.24171 count: 22
I0413 18:42:35.728987 23758 yolov3_layer.cpp:532] noobj: 0.00224608 obj: 0.483906 iou: 0.788536 cat: 0.905526 recall: 0.986667 recall75: 0.771111 count: 2
I0413 18:42:35.751526 23758 yolov3_layer.cpp:532] noobj: 0.00365666 obj: 0.49535 iou: 0.535215 cat: 0.864543 recall: 0.610597 recall75: 0.327094 count: 25
I0413 18:42:41.524915 23758 yolov3_layer.cpp:532] noobj: 0.00218737 obj: 0.43209 iou: 0.792419 cat: 0.970947 recall: 0.979167 recall75: 0.674405 count: 4

@eric612 博主，您好，训练时loss正常（如上），推理时使用detect.py检测不到任何物体，有尝试将test.prototxt的data改为deploy的data层,去除eval层拿来检测，以及将去除过的test.prototxt去bn，也是检测不到任何物体。类别相应参数也是正确的，例如75改为21（20类改为2类），预处理部分应该也没错。博主看下问题出在哪？

[chuck0518]

I0417 01:14:06.763319 10804 solver.cpp:443] Iteration 4000, Testing net (#0)
I0417 01:14:06.763365 10804 net.cpp:679] Ignoring source layer label_data_1_split
I0417 01:14:06.763638 10804 net.cpp:679] Ignoring source layer Yolov3Loss1
I0417 01:14:06.763650 10804 net.cpp:679] Ignoring source layer Yolov3Loss2
I0417 01:14:12.269546 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:14:24.775182 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:14:37.382987 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:14:49.903826 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:15:02.552605 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:15:08.886791 10804 solver.cpp:550] class1: 0.426832
I0417 01:15:08.887163 10804 solver.cpp:550] class2: 0.100249
I0417 01:15:08.887171 10804 solver.cpp:556] Test net output #0: detection_eval = 0.263541
I0417 01:15:11.730825 10804 solver.cpp:253] Iteration 4000 (0.106753 iter/s, 93.674s/10 iters), loss = 5.64835
I0417 01:15:11.730870 10804 solver.cpp:272] Train net output #0: det_loss1 = 0.923485 (* 1 = 0.923485 loss)
I0417 01:15:11.730876 10804 solver.cpp:272] Train net output #1: det_loss2 = 4.81016 (* 1 = 4.81016 loss)
I0417 01:15:11.730882 10804 sgd_solver.cpp:121] Iteration 4000, lr = 0.0005
I0417 01:15:13.953441 10804 yolov3_layer.cpp:532] noobj: 0.00172454 obj: 0.239341 iou: 0.735912 cat: 0.968337 recall: 0.958974 recall75: 0.5 count: 3
I0417 01:15:13.996486 10804 yolov3_layer.cpp:532] noobj: 0.00228207 obj: 0.222057 iou: 0.58847 cat: 0.837299 recall: 0.70797 recall75: 0.244564 count: 18
I0417 01:15:19.156597 10804 yolov3_layer.cpp:532] noobj: 0.00165788 obj: 0.276557 iou: 0.719697 cat: 0.865085 recall: 0.982143 recall75: 0.465476 count: 3
I0417 01:15:19.208161 10804 yolov3_layer.cpp:532] noobj: 0.00228386 obj: 0.257899 iou: 0.555846 cat: 0.839862 recall: 0.649744 recall75: 0.244544 count: 21
I0417 01:15:23.632382 10804 yolov3_layer.cpp:532] noobj: 0.00153054 obj: 0.216947 iou: 0.760579 cat: 0.942966 recall: 0.985714 recall75: 0.566667 count: 2
I0417 01:15:23.646881 10804 yolov3_layer.cpp:532] noobj: 0.00309506 obj: 0.241489 iou: 0.527233 cat: 0.790365 recall: 0.543724 recall75: 0.238325 count: 21
I0417 01:15:28.743680 10804 yolov3_layer.cpp:532] noobj: 0.00134971 obj: 0.233779 iou: 0.743891 cat: 0.905655 recall: 0.989796 recall75: 0.556122 count: 2
I0417 01:15:28.775761 10804 yolov3_layer.cpp:532] noobj: 0.00305693 obj: 0.286601 iou: 0.511557 cat: 0.835649 recall: 0.546304 recall75: 0.189129 count: 21
I0417 01:15:34.059072 10804 yolov3_layer.cpp:532] noobj: 0.00146028 obj: 0.194867 iou: 0.728285 cat: 0.911091 recall: 0.964744 recall75: 0.439011 count: 3
I0417 01:15:34.087019 10804 yolov3_layer.cpp:532] noobj: 0.00284167 obj: 0.263691 iou: 0.537525 cat: 0.798906 recall: 0.618155 recall75: 0.203408 count: 21
I0417 01:15:40.074692 10804 yolov3_layer.cpp:532] noobj: 0.00172411 obj: 0.170228 iou: 0.706618 cat: 0.864403 recall: 0.971154 recall75: 0.408272 count: 4
I0417 01:15:40.091838 10804 yolov3_layer.cpp:532] noobj: 0.0023063 obj: 0.227156 iou: 0.52783 cat: 0.831536 recall: 0.586209 recall75: 0.178665 count: 21
I0417 01:15:41.205112 10804 solver.cpp:253] Iteration 4010 (0.339281 iter/s, 29.4741s/10 iters), loss = 6.29814
I0417 01:15:41.205164 10804 solver.cpp:272] Train net output #0: det_loss1 = 0.453541 (* 1 = 0.453541 loss)
I0417 01:15:41.205171 10804 solver.cpp:272] Train net output #1: det_loss2 = 5.71122 (* 1 = 5.71122 loss)
I0417 01:15:41.205179 10804 sgd_solver.cpp:121] Iteration 4010, lr = 0.0005
I0417 01:15:46.354162 10804 yolov3_layer.cpp:532] noobj: 0.00223973 obj: 0.248727 iou: 0.744506 cat: 0.94056 recall: 0.975556 recall75: 0.53254 count: 4
I0417 01:15:46.368454 10804 yolov3_layer.cpp:532] noobj: 0.00255317 obj: 0.264948 iou: 0.559019 cat: 0.804919 recall: 0.688048 recall75: 0.196554 count: 20
I0417 01:15:52.485841 10804 yolov3_layer.cpp:532] noobj: 0.0016891 obj: 0.279516 iou: 0.757101 cat: 0.939476 recall: 0.962302 recall75: 0.539683 count: 4
I0417 01:15:52.531217 10804 yolov3_layer.cpp:532] noobj: 0.00184066 obj: 0.212435 iou: 0.539708 cat: 0.862308 recall: 0.611862 recall75: 0.191591 count: 19
I0417 01:15:59.140897 10804 yolov3_layer.cpp:532] noobj: 0.00158328 obj: 0.177992 iou: 0.750056 cat: 0.976023 recall: 0.967857 recall75: 0.579762 count: 4
I0417 01:15:59.160989 10804 yolov3_layer.cpp:532] noobj: 0.00162256 obj: 0.215766 iou: 0.546512 cat: 0.84008 recall: 0.633807 recall75: 0.224326 count: 17
I0417 01:16:05.295040 10804 yolov3_layer.cpp:532] noobj: 0.00135174 obj: 0.172353 iou: 0.708307 cat: 0.915387 recall: 0.869231 recall75: 0.448718 count: 3
I0417 01:16:05.330834 10804 yolov3_layer.cpp:532] noobj: 0.00180614 obj: 0.233261 iou: 0.578229 cat: 0.841397 recall: 0.662688 recall75: 0.275352 count: 18
I0417 01:16:11.471474 10804 yolov3_layer.cpp:532] noobj: 0.00170454 obj: 0.145291 iou: 0.720401 cat: 0.957166 recall: 0.956667 recall75: 0.499762 count: 4
I0417 01:16:11.546625 10804 yolov3_layer.cpp:532] noobj: 0.00216869 obj: 0.245605 iou: 0.575199 cat: 0.82772 recall: 0.672193 recall75: 0.245107 count: 19
I0417 01:16:16.386481 10804 solver.cpp:253] Iteration 4020 (0.284243 iter/s, 35.1811s/10 iters), loss = 5.28766
I0417 01:16:16.386528 10804 solver.cpp:272] Train net output #0: det_loss1 = 1.1634 (* 1 = 1.1634 loss)
I0417 01:16:16.386533 10804 solver.cpp:272] Train net output #1: det_loss2 = 2.83057 (* 1 = 2.83057 loss)
I0417 01:16:16.386539 10804 sgd_solver.cpp:121] Iteration 4020, lr = 0.0005
I0417 01:16:16.990921 10804 yolov3_layer.cpp:532] noobj: 0.0020566 obj: 0.164882 iou: 0.70567 cat: 0.929537 recall: 0.927407 recall75: 0.423704 count: 4
I0417 01:16:17.016185 10804 yolov3_layer.cpp:532] noobj: 0.00201062 obj: 0.231504 iou: 0.544762 cat: 0.867614 recall: 0.608327 recall75: 0.213358 count: 18
I0417 01:16:22.432796 10804 yolov3_layer.cpp:532] noobj: 0.0021696 obj: 0.233608 iou: 0.730122 cat: 0.899389 recall: 0.958242 recall75: 0.51337 count: 4
I0417 01:16:22.476905 10804 yolov3_layer.cpp:532] noobj: 0.00314332 obj: 0.285354 iou: 0.584276 cat: 0.85679 recall: 0.649635 recall75: 0.304402 count: 19
I0417 01:16:27.626332 10804 yolov3_layer.cpp:532] noobj: 0.00166812 obj: 0.251183 iou: 0.720748 cat: 0.927985 recall: 0.923077 recall75: 0.551282 count: 2
I0417 01:16:27.656478 10804 yolov3_layer.cpp:532] noobj: 0.00283306 obj: 0.300209 iou: 0.525543 cat: 0.816252 recall: 0.584902 recall75: 0.204652 count: 18
I0417 01:16:34.376063 10804 yolov3_layer.cpp:532] noobj: 0.00204174 obj: 0.276532 iou: 0.726224 cat: 0.846325 recall: 0.958889 recall75: 0.533889 count: 4
I0417 01:16:34.412142 10804 yolov3_layer.cpp:532] noobj: 0.00180618 obj: 0.263592 iou: 0.564033 cat: 0.82118 recall: 0.662399 recall75: 0.266307 count: 18
I0417 01:16:39.552242 10804 yolov3_layer.cpp:532] noobj: 0.00156027 obj: 0.268313 iou: 0.70862 cat: 0.91409 recall: 0.955357 recall75: 0.35119 count: 3
I0417 01:16:39.592263 10804 yolov3_layer.cpp:532] noobj: 0.00321347 obj: 0.264003 iou: 0.478043 cat: 0.80798 recall: 0.502037 recall75: 0.188421 count: 21
I0417 01:16:44.686738 10804 yolov3_layer.cpp:532] noobj: 0.00148813 obj: 0.18309 iou: 0.717419 cat: 0.943621 recall: 0.986111 recall75: 0.375 count: 3
I0417 01:16:44.705237 10804 yolov3_layer.cpp:532] noobj: 0.00314259 obj: 0.233115 iou: 0.48985 cat: 0.798157 recall: 0.532856 recall75: 0.20255 count: 23
...
...
...
I0417 01:40:31.351167 10804 solver.cpp:443] Iteration 4500, Testing net (#0)
I0417 01:40:31.351199 10804 net.cpp:679] Ignoring source layer label_data_1_split
I0417 01:40:31.351378 10804 net.cpp:679] Ignoring source layer Yolov3Loss1
I0417 01:40:31.351384 10804 net.cpp:679] Ignoring source layer Yolov3Loss2
I0417 01:40:42.772285 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:40:55.371289 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:41:08.256628 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:41:20.853245 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:41:33.417235 10804 blocking_queue.cpp:50] Waiting for data
I0417 01:41:36.257005 10804 solver.cpp:253] Iteration 4500 (0.108314 iter/s, 92.3242s/10 iters), loss = 6.04756
I0417 01:41:36.257053 10804 solver.cpp:272] Train net output #0: det_loss1 = 0.192515 (* 1 = 0.192515 loss)
I0417 01:41:36.257061 10804 solver.cpp:272] Train net output #1: det_loss2 = 4.32297 (* 1 = 4.32297 loss)
I0417 01:41:36.257066 10804 sgd_solver.cpp:121] Iteration 4500, lr = 0.0005
I0417 01:41:36.821374 10804 yolov3_layer.cpp:532] noobj: 0.00194495 obj: 0.239689 iou: 0.74027 cat: 0.936015 recall: 1 recall75: 0.399359 count: 3
I0417 01:41:36.837600 10804 yolov3_layer.cpp:532] noobj: 0.0031834 obj: 0.244109 iou: 0.543412 cat: 0.828508 recall: 0.635931 recall75: 0.192784 count: 21
I0417 01:41:41.622542 10804 yolov3_layer.cpp:532] noobj: 0.00154359 obj: 0.258396 iou: 0.744503 cat: 0.949754 recall: 0.967262 recall75: 0.733333 count: 3
I0417 01:41:41.636816 10804 yolov3_layer.cpp:532] noobj: 0.00343409 obj: 0.298613 iou: 0.554002 cat: 0.798239 recall: 0.625694 recall75: 0.208962 count: 20
I0417 01:41:47.171376 10804 yolov3_layer.cpp:532] noobj: 0.00180637 obj: 0.269898 iou: 0.755353 cat: 0.959803 recall: 0.983333 recall75: 0.593175 count: 3
I0417 01:41:47.211905 10804 yolov3_layer.cpp:532] noobj: 0.00229508 obj: 0.256093 iou: 0.585642 cat: 0.781785 recall: 0.674531 recall75: 0.294409 count: 19
I0417 01:41:51.997382 10804 yolov3_layer.cpp:532] noobj: 0.00155116 obj: 0.327347 iou: 0.754532 cat: 0.949671 recall: 0.948718 recall75: 0.647436 count: 2
I0417 01:41:52.007910 10804 yolov3_layer.cpp:532] noobj: 0.00246271 obj: 0.284302 iou: 0.565202 cat: 0.847037 recall: 0.637224 recall75: 0.304283 count: 18
I0417 01:41:57.120379 10804 yolov3_layer.cpp:532] noobj: 0.00136798 obj: 0.0937484 iou: 0.688966 cat: 0.86048 recall: 0.892857 recall75: 0.270578 count: 3
I0417 01:41:57.167546 10804 yolov3_layer.cpp:532] noobj: 0.0024238 obj: 0.25775 iou: 0.517224 cat: 0.813302 recall: 0.579652 recall75: 0.226081 count: 18
I0417 01:42:03.212220 10804 yolov3_layer.cpp:532] noobj: 0.00198894 obj: 0.254665 iou: 0.740418 cat: 0.972942 recall: 0.968254 recall75: 0.50328 count: 4
I0417 01:42:03.234072 10804 yolov3_layer.cpp:532] noobj: 0.00204909 obj: 0.28225 iou: 0.534509 cat: 0.831194 recall: 0.611214 recall75: 0.24834 count: 19
I0417 01:42:05.228647 10804 solver.cpp:253] Iteration 4510 (0.345168 iter/s, 28.9714s/10 iters), loss = 5.45083
I0417 01:42:05.228718 10804 solver.cpp:272] Train net output #0: det_loss1 = 0.963693 (* 1 = 0.963693 loss)
I0417 01:42:05.228727 10804 solver.cpp:272] Train net output #1: det_loss2 = 5.47839 (* 1 = 5.47839 loss)

@eric612 博主，你好，自查了一下问题。以上是训练log，迭代4000次结果正常，eval能给出相应的detection_eval，推理时目标框结果正常。迭代到4500次忽然detection_eval显示不了，代表测试图片一张也检测不出来，然而validation loss还在下降，推理时也出不来结果，推理至conv39数据全部为nan，conv38正常，感觉很奇怪，好像哪里出现了bug