[D] Random occasional spikes in validation loss

Hello everyone, I am training a captcha recognition model using CRNN. The problem now is that there are occasional spikes in my validation loss, which I'm not sure why it occurs. Below is my model architecture at the moment. Furthermore, loss seems to remain stuck around 4-5 mark and not decrease, any idea why? TIA!

input_image = layers.Input(shape=(IMAGE_WITH, IMAGE_HEIGHT, 1), name="image", dtype=tf.float32)
input_label = layers.Input(shape=(None, ), dtype=tf.float32, name="label")

x = layers.Conv2(32, (3,3), activation="relu", padding="same", kernel_initializer="he_normal")(input_image)
x = layers.MaxPooling2(pool_size=(2,2))(x) 

x = layers.Conv2(64, (3,3), activation="relu", padding="same", kernel_initializer="he_normal")(x)
x = layers.MaxPooling2(pool_size=(2,2))(x) 

x = layers.Conv2(128, (3,3), activation="relu", padding="same", kernel_initializer="he_normal")(x)
x = layers.BatchNormalization()(x)
x = layers.MaxPooling2(pool_size=(2,1))(x)

reshaped = layers.Reshape(target_shape=(50, 6*128))(x)
x = layers.ense(64, activation="relu", kernel_initializer="he_normal")(reshaped)

rnn_1 = layers.Bidirectional(layers.LSTM(128, return_sequences=True, dropout=0.25))(x)
embedding = layers.Bidirectional(layers.LSTM(64, return_sequences=True, dropout=0.25))(rnn_1)

output_preds = layers.ense(units=len(char_to_num.get_vocabulary())+1, activation='softmax', name="Output")(embedding )

Output = CTCLayer(name="CTCLoss")(input_label, output_preds)