-
Notifications
You must be signed in to change notification settings - Fork 205
1602AnsselCNN
Petr Baudis edited this page May 12, 2016
·
6 revisions
CNN is a popular model that seems like it could work better on smaller datasets.
Setting for configuration "1cnn":
{"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{1: 1, 2: 0.5, 3: 0.5, 4: 0.5, 5: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.8", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xb816668>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "2.5", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xb816f50>"}
Compared to RNN, note the Ddim=1, as well as high internal dropout 4/5.
64x ay_1cnn - 0.331514 (95% [0.323768, 0.339260]):
10720379.arien.ics.muni.cz.ay_1cnn etc.
[0.345167, 0.292328, 0.329627, 0.335328, 0.355619, 0.335739, 0.302157, 0.309687, 0.349448, 0.364152, 0.357973, 0.377934, 0.342250, 0.294438, 0.308203, 0.349373, 0.355517, 0.311810, 0.303926, 0.244570, 0.346717, 0.364751, 0.339015, 0.348370, 0.237564, 0.339533, 0.323730, 0.323730, 0.362342, 0.344231, 0.318624, 0.348433, 0.325957, 0.318332, 0.361780, 0.358816, 0.317717, 0.246553, 0.308939, 0.328769, 0.227277, 0.328469, 0.333378, 0.327592, 0.363001, 0.352183, 0.334728, 0.323741, 0.336934, 0.324669, 0.308410, 0.357007, 0.367720, 0.328169, 0.357388, 0.349632, 0.358994, 0.327276, 0.358028, 0.317978, 0.351947, 0.335090, 0.372716, 0.345399, ]
32x pact=tanh ay_1cnn_t - 0.353034 (95% [0.345650, 0.360418]):
10720471.arien.ics.muni.cz.ay_1cnn_t etc.
[0.377687, 0.343416, 0.347380, 0.344893, 0.325351, 0.317631, 0.357307, 0.371562, 0.353174, 0.350956, 0.351545, 0.310054, 0.363473, 0.353009, 0.372604, 0.380763, 0.337578, 0.343555, 0.360854, 0.375445, 0.302055, 0.349213, 0.367814, 0.342827, 0.339200, 0.332919, 0.370010, 0.370502, 0.349310, 0.368820, 0.378530, 0.387647, ]
(good idea!)
32x ay_1cnn_tdot - 0.281713 (95% [0.274802, 0.288623]):
10720495.arien.ics.muni.cz.ay_1cnn_tdot etc.
[0.287812, 0.287236, 0.296515, 0.300662, 0.274575, 0.272433, 0.266690, 0.288229, 0.271385, 0.286212, 0.254792, 0.260062, 0.318057, 0.329208, 0.254762, 0.298667, 0.261490, 0.304788, 0.285137, 0.289728, 0.279296, 0.281601, 0.273979, 0.235686, 0.268907, 0.293917, 0.278176, 0.288217, 0.293680, 0.297488, 0.282550, 0.252868, ]
(wow, bad idea! it's not clear why, but mlp scorer is really required here)
32x mlpsum=absdiff ay_1cnn_tabs - 0.354809 (95% [0.345751, 0.363867]):
10726809.arien.ics.muni.cz.ay_1cnn_tabs etc.
[0.391494, 0.333033, 0.345339, 0.384590, 0.367768, 0.335566, 0.321510, 0.362905, 0.346020, 0.350228, 0.293495, 0.363916, 0.317735, 0.377005, 0.368320, 0.337578, 0.359678, 0.363597, 0.356083, 0.350491, 0.349202, 0.430688, 0.340494, 0.348994, 0.329083, 0.348390, 0.369493, 0.357985, 0.372274, 0.365412, 0.327036, 0.388482, ]
(whether we use elementwise sum or absolute distance doesn't matter)
16x baseline for RandomSearch ay_1cnn_i12d12t - 0.342397 (95% [0.326954, 0.357839]):
10745811.arien.ics.muni.cz.ay_1cnn_i12d12t etc.
[0.302120, 0.332215, 0.323852, 0.401458, 0.353206, 0.306473, 0.291422, 0.338478, 0.381578, 0.351939, 0.364919, 0.349142, 0.346782, 0.376348, 0.338148, 0.320265, ]
TODO what about no dropout?
Then, some automated tuning with an updated configuration:
rs = RandomSearch(modelname+'_rlog.txt',
dropout=[1/2], inp_e_dropout=[1/2], l2reg=[1e-6, 1e-5, 1e-4],
cnnact=['tanh', 'tanh', 'relu'], cnninit=['glorot_uniform'],
cdim={1: [0, 0, 1/2, 1], 2: [0, 0, 1/2, 1], 3: [0, 0, 1/2, 1], 4: [0, 0, 1/2, 1], 5: [0, 0, 1/2, 1]},
project=[True], pdim=[1/2, 1, 2, 3], pact=['tanh', 'linear'],
ptscorer=[B.dot_ptscorer, B.mlp_ptscorer], Ddim=[1, 2])
Most notably, this has internal dropout as 1/2 instead of 4/5.
-598dad6be14a6fef 0.134156 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 2: 1, 4: 1, 5: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "3", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
7d3dca8017bf9f26 0.201960 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{2: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "2", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-39208e977c111d6b 0.207316 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{3: 0.5, 4: 0.5}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "0.5", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-4d8afb79a6ad0968 0.217822 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 3: 1}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "2", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
79f688fddb66b500 0.230505 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{3: 1}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "1", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
30b2d581957ac15c 0.231981 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{4: 0.5}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "1", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
543962d2a61deca8 0.235282 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{3: 1, 4: 0.5, 5: 1}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "2", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-474745b54b99d40 0.238087 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "0.5", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-4eb9848672496e5 0.242066 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{2: 0.5, 4: 0.5}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "0.5", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
43234b75bcdcfdd6 0.252250 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{2: 0.5, 5: 1}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "3", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-4bb652ea8182ce87 0.252930 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{4: 1}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "1", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-6ca3c401f05eabd3 0.270313 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{2: 1, 3: 0.5}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "3", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-e0accf573ab767b 0.276622 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{1: 1, 5: 1}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "0.5", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
4f528f29f445352f 0.283032 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 5: 0.5}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "3", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
1e44b0c5739baf94 0.288885 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{2: 0.5, 4: 1, 5: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "1", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
3c957828263d368b 0.292239 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{3: 0.5, 4: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "0.5", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-39def48ea7332c6b 0.292294 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 2: 0.5, 4: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "1", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
49d04fde6cc62a18 0.295274 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{4: 0.5, 5: 1}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "0.5", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-796af35956767cfe 0.295440 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{5: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "2", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
3c1776b34e656334 0.297491 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{1: 1, 3: 0.5, 4: 0.5, 5: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "1", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
75102ee8adb1f4cd 0.300933 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{1: 1, 3: 1}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "2", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
5a98aacb621d0eaa 0.302335 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 2: 1, 3: 0.5, 4: 1}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "0.5", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-8748f3ff54d576f 0.303476 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{1: 1, 3: 0.5, 5: 1}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "1", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-d8a85921c3f866f 0.309065 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{2: 0.5, 5: 1}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "3", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
5c97ba0e7c23e509 0.321591 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{2: 1, 3: 0.5, 5: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "1", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-5404cb26e940ad82 0.324566 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{3: 1, 4: 1, 5: 1}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "3", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
-5fa4d80c206c8ca7 0.343297 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{1: 1, 2: 0.5, 5: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "0.5", "project": "True", "ptscorer": "<function dot_ptscorer at 0xbf3ec80>"}
-71279dd55ca926a 0.355385 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{2: 0.5, 4: 1, 5: 1}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "1", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
503cd7bb6fd49245 0.367062 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 2: 1, 3: 1, 4: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "3", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
467868236965d0a5 0.372199 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 2: 0.5, 3: 1, 4: 1}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "0.5", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
7188642902582036 0.374550 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{2: 0.5, 4: 1, 5: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "2", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
229392ffab80bd13 0.375639 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 2: 0.5}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-06", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "linear", "pdim": "1", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
16c8df5f2c26b5a8 0.381620 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 2: 0.5, 3: 0.5, 4: 0.5}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "2", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
-13a6ee4b64d79268 0.383439 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 2: 1, 4: 0.5}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "1", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
-1cb732fdb8352599 0.385715 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{2: 1, 3: 0.5, 4: 1, 5: 1}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "2", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
-7132c362501f0009 0.395977 {"Ddim": "2", "balance_class": "False", "batch_size": "160", "cdim": "{2: 0.5, 4: 0.5, 5: 1}", "cnnact": "tanh", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "0.0001", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "3", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
1ad9035c69d97ee1 0.443827 {"Ddim": "1", "balance_class": "False", "batch_size": "160", "cdim": "{1: 0.5, 2: 1, 3: 1, 5: 0.5}", "cnnact": "relu", "cnninit": "glorot_uniform", "dropout": "0.5", "e_add_flags": "True", "embdim": "300", "epoch_fract": "0.25", "inp_e_dropout": "0.5", "inp_w_dropout": "0", "l2reg": "1e-05", "loss": "<function ranknet at 0xbf45410>", "mlpsum": "sum", "nb_epoch": "16", "pact": "tanh", "pdim": "2", "project": "True", "ptscorer": "<function mlp_ptscorer at 0xbf3ed70>"}
16x ay_1cnn_i12d12t_crelu - 0.360639 (95% [0.348838, 0.372440]):
10745827.arien.ics.muni.cz.ay_1cnn_i12d12t_crelu etc.
[0.383331, 0.354686, 0.377177, 0.337458, 0.325115, 0.338724, 0.367645, 0.407843, 0.334177, 0.389462, 0.365057, 0.361076, 0.365694, 0.333640, 0.372650, 0.356492, ]
16x ay_1cnn_i12d12tp2_crelu - 0.363154 (95% [0.348820, 0.377488]):
10745843.arien.ics.muni.cz.ay_1cnn_i12d12tp2_crelu etc.
[0.375069, 0.401165, 0.348604, 0.365775, 0.328540, 0.335079, 0.325679, 0.410980, 0.394049, 0.392161, 0.365593, 0.335101, 0.337968, 0.354856, 0.388777, 0.351067, ]
16x ay_1cnn_i12d12tp2_crelu1211110012 (grid search result) - 0.376768 (95% [0.365899, 0.387637]):
10745859.arien.ics.muni.cz.ay_1cnn_i12d12tp2_crelu1211110012 etc.
[0.363973, 0.387154, 0.357717, 0.331872, 0.382500, 0.360163, 0.413687, 0.393420, 0.344466, 0.398210, 0.377632, 0.376403, 0.382119, 0.388495, 0.396283, 0.374189, ]
16x ay_1cnn_i12d12tp2_ct1211110012 - 0.340778 (95% [0.331372, 0.350184]):
10745875.arien.ics.muni.cz.ay_1cnn_i12d12tp2_ct1211110012 etc.
[0.331238, 0.304096, 0.339431, 0.358262, 0.339176, 0.325519, 0.332444, 0.343566, 0.334320, 0.364965, 0.369329, 0.347300, 0.319419, 0.371460, 0.336344, 0.335581, ]
16x ay_1cnn_i12d12tp2_crelu0031000000 - 0.342156 (95% [0.324243, 0.360069]):
10742018.arien.ics.muni.cz.ay_1cnn_i12d12tp2_crelu0031000000 etc.
[0.370126, 0.406253, 0.307128, 0.300053, 0.313325, 0.330850, 0.316754, 0.346204, 0.385596, 0.336538, 0.401895, 0.357748, 0.315338, 0.296303, 0.345616, 0.344768, ]
defaults 2cnn cnn-7ed76a006e2678d5 - val mrr 0.632769
no dropout 2cnn cnn-16638a423c5805db - val mrr 0.694954
p1dot 2cnn cnn-1fe3d2dfa9f40a61 - val mrr 0.718287
Let's check the effect of no-dropout and no-siamese modes in the context of prescoring.
al_2cnn - 0.348247 ±0.006217
8x R_al_2cnn_d0 - 0.395451 (95% [0.383459, 0.407444]):
11240965.arien.ics.muni.cz.R_al_2cnn_d0 etc.
[0.404426, 0.397346, 0.396247, 0.362152, 0.388980, 0.397135, 0.403238, 0.414086, ]
16x R_al_2cnn_preBM25P20 - 0.463438 (95% [0.460091, 0.466785]):
16x no-dropout R_al_2cnn_preBM25P20_i0d0w0 - 0.480827 (95% [0.476222, 0.485432]):
11226140.arien.ics.muni.cz.R_al_2cnn_preBM25P20_i0d0w0 etc.
[0.463730, 0.480296, 0.491250, 0.480813, 0.475103, 0.487560, 0.480675, 0.488414, 0.476012, 0.484290, 0.466829, 0.484960, 0.483059, 0.496134, 0.468623, 0.485486, ]
8x non-siamese R_al_2cnnS_preBM25p20_i0d0w0 - 0.493650 (95% [0.484262, 0.503038]):
11236030.arien.ics.muni.cz.R_al_2cnnS_preBM25p20_i0d0w0 etc.
[0.490499, 0.480853, 0.492547, 0.483200, 0.508808, 0.500322, 0.481751, 0.511221, ]
16x (also without pruning) R_ay_2cnn_preBM25f - 0.483106 (95% [0.472069, 0.494143]):
16x R_ay_2cnn_preBM25P20_i0d0w0 - 0.529302 (95% [0.513541, 0.545063]):
11233395.arien.ics.muni.cz.R_ay_2cnn_preBM25P20_i0d0w0 etc.
[0.560938, 0.539073, 0.447483, 0.549825, 0.523195, 0.541627, 0.535003, 0.466559, 0.529262, 0.524238, 0.544389, 0.526275, 0.559718, 0.547152, 0.536153, 0.537940, ]
aw_2cnn - 0.841343 ±0.005409
16x R_aw_2cnn_d0 - 0.855211 (95% [0.848311, 0.862111]):
11240966.arien.ics.muni.cz.R_aw_2cnn_d0 etc.
[0.871795, 0.866923, 0.838974, 0.858876, 0.854359, 0.846886, 0.871410, 0.851282, 0.834615, 0.874872, 0.841218, 0.838974, 0.870513, 0.844615, 0.859091, 0.858974, ]
16x R_aw_2cnn_preBM25P20 - 0.863481 (95% [0.855863, 0.871100]):
16x R_aw_2cnn_preBM25P20_i0d0w0 - 0.872277 (95% [0.865844, 0.878710]):
11233397.arien.ics.muni.cz.R_aw_2cnn_preBM25P20_i0d0w0 etc.
[0.870513, 0.850916, 0.857070, 0.892308, 0.876923, 0.865385, 0.858974, 0.877839, 0.880769, 0.875018, 0.876923, 0.873223, 0.885897, 0.855275, 0.891453, 0.867949, ]
Conclusion: No-dropout is clearly a good idea! No-siamese doesn't seem obviously desirable.