如何在softmax之前获得模型的输出，而不改变模型架构？

Question

我有一个经过训练的

sequential

keras

模型。

最后一层是带有softmax激活函数的Dense层：

model = keras.models.Sequential()
model.add(...)
model.add(...)
model.add(...)
model.add(keras.layers.Dense(50, activation='softmax'))

如何在

softmax

之前获得模型的输出，而不更改模型架构？ （我已经训练了模型，我无法更改或训练）。

我尝试过：

probs = model.predict(X_train)
logits = probs - np.log(np.sum(np.exp(probs), axis=-1, keepdims=True))

但似乎如果我在 logtis 上运行 softmax，它会给出与

probs

不同的结果：

def softmax(x):
    e_x = np.exp(x - np.max(x))  
    return e_x / e_x.sum(axis=1, keepdims=True)

probabilities = softmax(logits)

Answer 1

您可以通过以下方式将softmax反转回logits。

# options 1
# ref. https://stackoverflow.com/a/64668809/9215780
def inv_softmax(x, C):
    logits = np.log(x) + C
    return logits

# options 2
def inv_softmax(x):
    logits = np.log(x / (1 - x))
    return logits 

这里有一些虚拟代码来测试它们。

num_class=50
inputs = Input(shape=(5,))
x = Dense(128, activation='relu')(inputs)
x = Dense(64, activation='relu')(x)
x = Dense(32, activation='relu')(x)
outputs = Dense(num_class, activation='softmax')(x)
model = Model(inputs=inputs, outputs=outputs)

a = tf.ones(shape=(2, 5))
y_pred_prob = model(a)
y_pred_prob.shape # TensorShape([2, 50])

# following option 1
logits = inv_softmax(y_pred_prob, num_class)
logits.shape # (2, 50)

y_pred_prob_reproduce = tf.nn.softmax(logits)
y_pred_prob_reproduce.shape # TensorShape([2, 50])

y_pred_prob[0]
<tf.Tensor: shape=(50,), dtype=float32, numpy=
array([0.02209216, 0.01324271, 0.01495313, 0.0182846 , 0.02364523,
       0.01963637, 0.02066819, 0.02271825, 0.02229412, 0.01854686,
       0.01951347, 0.02007069, 0.02835885, 0.01483266, 0.02553979,
       0.01616779, 0.01538332, 0.01937215, 0.01792852, 0.01752241,
       0.02167817, 0.01575256, 0.0232809 , 0.0204947 , 0.01880379,
       0.01848676, 0.0199989 , 0.02911243, 0.02096296, 0.02170451,
       0.02149592, 0.02127673, 0.01858926, 0.02001583, 0.01901014,
       0.01976348, 0.01502533, 0.01940756, 0.01502022, 0.02546986,
       0.02210576, 0.01966349, 0.01942356, 0.02224619, 0.02430816,
       0.0187437 , 0.01451708, 0.02327427, 0.01841178, 0.02118474],
      dtype=float32)>

y_pred_prob_reproduce[0]
<tf.Tensor: shape=(50,), dtype=float32, numpy=
array([0.02209212, 0.01324273, 0.01495312, 0.01828457, 0.02364521,
       0.0196364 , 0.02066822, 0.02271822, 0.02229412, 0.01854688,
       0.01951349, 0.02007072, 0.02835883, 0.01483268, 0.02553976,
       0.01616781, 0.0153833 , 0.01937212, 0.01792852, 0.01752243,
       0.02167813, 0.01575255, 0.02328094, 0.02049471, 0.01880377,
       0.01848677, 0.01999888, 0.02911241, 0.02096297, 0.02170452,
       0.02149589, 0.02127676, 0.01858924, 0.02001582, 0.01901012,
       0.01976348, 0.01502534, 0.01940755, 0.01502024, 0.02546991,
       0.02210577, 0.01966346, 0.01942355, 0.0222462 , 0.02430819,
       0.01874369, 0.01451708, 0.02327428, 0.01841175, 0.02118476],
      dtype=float32)>


tolerance = 1e-6
is_equal = np.allclose(
    y_pred_prob, y_pred_prob_reproduce, atol=tolerance
) # OK