TimeDistributed wrapping causing issues

[voletiv]

Hi,
I want to wrap TimeDistributed over a built resnet. So I tried doing it in my model like this:

TimeDistributed(ResnetBuilder.build_resnet_18(mouth_input_shape, mouth_features_dim),
                                              input_shape=(TIME_STEPS, *mouth_input_shape))

However, while fitting my model, it is giving me the following error:

2017-11-17 18:35:58.790402: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
         [[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-11-17 18:35:58.790456: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
         [[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-11-17 18:35:58.790835: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
         [[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-11-17 18:35:58.791022: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
         [[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
2017-11-17 18:35:58.791086: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
         [[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
Traceback (most recent call last):
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
    return fn(*args)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
    status, run_metadata)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
         [[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
         [[Node: _arg_input_4_0_5/_850 = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", 
send_device_incarnation=1, tensor_name="edge_10815__arg_input_4_0_5", _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_input_4_0_5)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 11, in <module>                                                                                                                                                              [0/1869]
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/keras/engine/training.py", line 2077, in fit_generator
    class_weight=class_weight)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/keras/engine/training.py", line 1797, in train_on_batch
    outputs = self.train_function(ins)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2332, in __call__
    **self.session_kwargs)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
         [[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
         [[Node: _arg_input_4_0_5/_850 = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", 
send_device_incarnation=1, tensor_name="edge_10815__arg_input_4_0_5", _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_input_4_0_5)]]

Caused by op 'batch_normalization_1/keras_learning_phase', defined at:
  File "<stdin>", line 1, in <module>
  File "/shared/fusor/home/voleti.vikram/lipreading-in-the-wild-experiments/assessor/assessor_model.py", line 40, in my_assessor_model
    mouth_feature_model = TimeDistributed(ResnetBuilder.build_resnet_18(mouth_input_shape, mouth_features_dim),
  File "/shared/fusor/home/voleti.vikram/lipreading-in-the-wild-experiments/assessor/resnet.py", line 273, in build_resnet_18
    return ResnetBuilder.build(input_shape, num_outputs, basic_block, [2, 2, 2, 2])
  File "/shared/fusor/home/voleti.vikram/lipreading-in-the-wild-experiments/assessor/resnet.py", line 248, in build
    conv1 = _conv_bn_relu(filters=64, kernel_size=(7, 7), strides=(2, 2))(input)
  File "/shared/fusor/home/voleti.vikram/lipreading-in-the-wild-experiments/assessor/resnet.py", line 81, in f
    return _bn_relu(conv)
  File "/shared/fusor/home/voleti.vikram/lipreading-in-the-wild-experiments/assessor/resnet.py", line 62, in _bn_relu
    norm = BatchNormalization(axis=CHANNEL_AXIS)(input)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/keras/engine/topology.py", line 603, in __call__
    output = self.call(inputs, **kwargs)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/keras/layers/normalization.py", line 190, in call
    training=training)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2712, in in_train_phase
    training = learning_phase()
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 120, in learning_phase
    name='keras_learning_phase')
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1599, in placeholder
    return gen_array_ops._placeholder(dtype=dtype, shape=shape, name=name)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3091, in _placeholder
    "Placeholder", dtype=dtype, shape=shape, name=name)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/users/voleti.vikram/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'batch_normalization_1/keras_learning_phase' with dtype bool
         [[Node: batch_normalization_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=<unknown>, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
         [[Node: _arg_input_4_0_5/_850 = _Send[T=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", 
send_device_incarnation=1, tensor_name="edge_10815__arg_input_4_0_5", _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_input_4_0_5)]]

It is possibly because learning_phase needs to be set to 1. But I tried doing that and it still doesn't work. Please help.