[FIXED] Issue create multi input model Keras

Issue

The model of this image is what I need.

But when I use netron.app to get model structure, I found that my code seems to be wrong.

It seems I combined two images into a six-channel and trained it.

conv_base = tf.keras.applications.InceptionResNetV2(weights=None,include_top=False , 
input_broc = tf.keras.Input(shape=(299, 299, 3))
input_temp = tf.keras.Input(shape=(299, 299, 3))

x_broc = conv_base(input_broc)
x_temp = conv_base(input_temp)
x_broc = tf.keras.layers.GlobalAveragePooling2D()(x_broc)
x_temp = tf.keras.layers.GlobalAveragePooling2D()(x_temp)
x = tf.keras.layers.Concatenate()([x_broc, x_temp])
x = tf.keras.layers.Dense(7, activation='softmax')(x)

model = tf.keras.Model(inputs=[input_broc, input_temp], outputs=x)

Model: "model_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_5 (InputLayer)            [(None, 299, 299, 3) 0                                            
__________________________________________________________________________________________________
input_6 (InputLayer)            [(None, 299, 299, 3) 0                                            
__________________________________________________________________________________________________
inception_resnet_v2 (Functional (None, 8, 8, 1536)   54336736    input_5[0][0]                    
                                                                 input_6[0][0]                    
__________________________________________________________________________________________________
global_average_pooling2d_2 (Glo (None, 1536)         0           inception_resnet_v2[0][0]        
__________________________________________________________________________________________________
global_average_pooling2d_3 (Glo (None, 1536)         0           inception_resnet_v2[1][0]        
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 3072)         0           global_average_pooling2d_2[0][0] 
                                                                 global_average_pooling2d_3[0][0] 
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 7)            21511       concatenate_1[0][0]              
==================================================================================================

custom datagenerator return it

return [X1, X2], y

Solution

It didn't concatenate both images channel-wise, but both input pass through the same Inception model and their outputs are then concatenated. What you want is two different instances of Inception.

Like this:

import tensorflow as tf

backbone1 = tf.keras.applications.InceptionResNetV2(
    weights=None, include_top=False)
backbone1._name = 'backbone1'
backbone2 = tf.keras.applications.InceptionResNetV2(
    weights=None, include_top=False
)
backbone2._name = 'backbone2'

input_broc = tf.keras.Input(shape=(299, 299, 3))
input_temp = tf.keras.Input(shape=(299, 299, 3))

x_broc = backbone1(input_broc)
x_temp = backbone2(input_temp)
x_broc = tf.keras.layers.GlobalAveragePooling2D()(x_broc)
x_temp = tf.keras.layers.GlobalAveragePooling2D()(x_temp)
x = tf.keras.layers.Concatenate()([x_broc, x_temp])
x = tf.keras.layers.Dense(7, activation='softmax')(x)

model = tf.keras.Model(inputs=[input_broc, input_temp], outputs=x)

Answered By - Nicolas Gervais