Impact of pre-trained weights ?

[leolln]

Hello !
My use case is as follows:

I'm working on images from the Cityscapes dataset that I'm trying to segment.

I have a tensorflow train dataset with around 60 images. Resized to 512 px.
I'm trying to compare the performance of my network with and without pre trained weigths and the impact is very limited I'm struggling to understand why.

BACKBONE = 'vgg16'
BATCH_SIZE = 4
CLASSES = CLASSES
LR = 0.0001
EPOCHS = 5
NUM_TRAIN= 60
NUM_VAL = 20
tf.keras.backend.clear_session()
preprocess_input = sm.get_preprocessing(BACKBONE)

n_classes = 1 if len(CLASSES) == 1 else (len(CLASSES))  # case for binary and multiclass segmentation
activation = 'sigmoid' if n_classes == 1 else 'softmax'

model = sm.Unet(BACKBONE, classes=n_classes, activation=activation,encoder_weights='imagenet')
model.summary(show_trainable=True)
optim = keras.optimizers.legacy.Adam(LR)
# train model
mlflow.tensorflow.autolog()
run_description='VGG16 Weights Image Net, 60 train 20 val'
with mlflow.start_run(description=run_description) as run:
    history = model.fit(
        train_dataset_tf, 
        steps_per_epoch=train_dataset_tf.cardinality().numpy(), 
        epochs=EPOCHS, 
        callbacks=callbacks, 
        validation_data=val_dataset_tf, 
        validation_steps=val_dataset_tf.cardinality().numpy()
    )

When I use encoder_weights='imagenet' I got
Last Train IOU : 0.15
Last Val IOU : 0.04
Mean Test IOU : 0.15

When I use encoder_weights=None I got
Last Train IOU : 0.31
Last Val IOU : 0.09
Mean Test IOU : 0.27

Is there something I am missing ? How can I check that the pre-trained weights are correctly loaded / used ?

Thank you very much !

[leolln]

Detailed résults :

Freeze_encoder = False :

encoder_weights='imagenet'
Last Train IOU : 0.26
Last Val IOU : 0.08
Mean Test IOU : 0.15

encoder_weights=None
Last Train IOU : 0.31
Last Val IOU : 0.09
Mean Test IOU : 0.27

Encoder_freeze = True,

encoder_weights='imagenet'

Train IOU : 0.11

Val IOU : 0.04

Test : 0.08

encoder_weights='None'

Train : 0.3

Val : 0.05

Test : 0.14

[Raito03]

when making encoder_freezes = True , it freezes the weights from being changed any further (its static)
when making encoder_weights = "None" it does random initialisation , so randomly setting the weights , which can range either in the high range or in the lower range

so now when you make encoder_weights = None and encoder_freeze = True, it may have freezed the weight values where the Random weights performed better than the imagenet but this random initialisation varies at every run command so yeah.