Logits 详解：与 Softmax 关系、使用场景及注意事项（中英双语）

我们将 Logits 通过 Softmax 转化为概率分布：

P(A) = exp(2.0) / (exp(2.0) + exp(1.0) + exp(0.1))
P(B) = exp(1.0) / (exp(2.0) + exp(1.0) + exp(0.1))
P(C) = exp(0.1) / (exp(2.0) + exp(1.0) + exp(0.1))

计算每项的指数值：

• exp(2.0) ≈ 7.39
• exp(1.0) ≈ 2.72
• exp(0.1) ≈ 1.11

归一化：

• P(A) = 7.39 / (7.39 + 2.72 + 1.11) ≈ 0.66
• P(B) = 2.72 / (7.39 + 2.72 + 1.11) ≈ 0.24
• P(C) = 1.11 / (7.39 + 2.72 + 1.11) ≈ 0.10

最终概率分布：P = [0.66, 0.24, 0.10]

解释

• 类别 A（新闻）的概率最高，模型认为输入句子最有可能属于新闻类别。
• 类别 B 和 C 的概率较低，模型对这些类别的置信度较弱。

4. Logits 的使用场景和注意事项

(1) 使用场景

• 分类任务：Logits 是交叉熵损失的输入，计算预测类别的损失。
• 推理阶段：可以直接使用 Logits 的最大值对应的类别（无需计算 Softmax），因为 Softmax 不会改变 Logits 的排序。

(2) 注意事项

1. 数值稳定性： Logits 值过大或过小可能导致溢出。
   • 解决方法：在 Softmax 计算时，减去 Logits 的最大值： P(y_i) = exp(z_i - max(z)) / sum(exp(z_j - max(z))) 这不会改变最终的概率分布结果，但能避免溢出。
2. 对比损失（Contrastive Loss）： 在一些高级任务中，Logits 被用来计算对比学习的损失函数，直接比较不同样本的相似性。
3. 解释能力： 虽然 Logits 本身不是概率，但可以看作是模型的'原始信号'，值越高表示模型对该类别的'倾向性'越强。

5. 总结与洞见

1. Logits 是模型输出的核心中间结果：它们直接反映了模型对各个类别的偏好，但未归一化为概率。
2. Softmax 将 Logits 转化为概率分布：提供清晰的概率解释，便于模型评估和决策。
3. 数值稳定性和高效计算是关键：通过合理的数值操作（如减去最大值），可以确保计算稳定性。
4. 实际应用中的选择：在推理阶段，可以直接使用 Logits 最大值对应的类别进行预测，避免额外的 Softmax 开销。

理解 Logits 和它们与 Softmax 的关系，不仅能帮助我们更好地优化模型，还能让我们在实际应用中更高效地设计系统。

What Are Logits? A Comprehensive Guide

In deep learning, logits refer to the raw output values produced by the final layer of a model before applying any normalization, such as Softmax. Understanding logits is crucial as they serve as the foundation for transforming model outputs into probabilities and making predictions.

This article covers:

1. Definition and Purpose of Logits
2. Relationship Between Logits and Softmax
3. Practical Example: From Logits to Probability Distributions
4. Applications and Considerations of Logits
5. Key Takeaways and Insights

1. Definition and Purpose of Logits

What Are Logits?

Logits are the raw scores output by the model's final layer (often a dense layer). They represent the unnormalized confidence of the model for each class. Logits are not probabilities—they can be positive, negative, or extremely large/small numbers.

• Shape: If the model predicts C classes for a single input, the logits are a vector of size [C].
• Properties:
  • They do not satisfy the constraints of probabilities (e.g., summing to 1 or being between 0 and 1).
  • They are a direct representation of the model's tendency or 'preference' for each class.

Why Are Logits Important?

Logits are an intermediate result in a model's prediction pipeline. They are transformed into probabilities through activation functions like Softmax, which are then used for decision-making or loss computation.

2. Relationship Between Logits and Softmax

The Softmax Function

Softmax is a mathematical function that transforms logits into probabilities. It is defined as:

P(y_i) = exp(z_i) / sum(exp(z_j))

Where:

• z_i: Logit for class i.
• C: Total number of classes.
• P(y_i): Normalized probability for class i, satisfying: sum(P(y_i)) = 1

Difference Between Logits and Probabilities

1. Logits (Unnormalized Scores):
   • Range: Any real number ((−∞) to (+∞)).
   • No probabilistic interpretation.
2. Softmax Output (Probabilities):
   • Range: [0, 1].
   • Represents the model's confidence in each class, summing to 1.

3. Practical Example: From Logits to Probabilities

Let's take a text classification task where the model predicts the category of a sentence. Suppose the task has three categories:

• Class A: News
• Class B: Entertainment
• Class C: Technology

Input: The sentence "The new smartphone has amazing features." Model's Logits Output: logits = [2.0, 1.0, 0.1]

Step 1: Apply Softmax

To convert logits to probabilities, apply the Softmax function:

P(A) = exp(2.0) / (exp(2.0) + exp(1.0) + exp(0.1))
P(B) = exp(1.0) / (exp(2.0) + exp(1.0) + exp(0.1))
P(C) = exp(0.1) / (exp(2.0) + exp(1.0) + exp(0.1))

Step 2: Compute the Exponentials

• exp(2.0) ≈ 7.39
• exp(1.0) ≈ 2.72
• exp(0.1) ≈ 1.11

Step 3: Normalize

The total sum of exponentials is: sum = 7.39 + 2.72 + 1.11 ≈ 11.22

The probabilities for each class are:

• P(A) = 7.39 / 11.22 ≈ 0.66
• P(B) = 2.72 / 11.22 ≈ 0.24
• P(C) = 1.11 / 11.22 ≈ 0.10

Interpretation

• The model assigns the highest probability to Class A (News), indicating the sentence is most likely about news.
• Class B (Entertainment) and Class C (Technology) have lower probabilities, reflecting weaker confidence in those predictions.

4. Applications and Considerations of Logits

Applications

1. Classification Tasks: Logits are used as input to loss functions like cross-entropy, which compares the logits (or their normalized probabilities) with ground truth labels.
2. Inference: During inference, instead of computing probabilities, we can directly use the index of the largest logit value for the predicted class: Predicted Class = argmax(logits) This avoids unnecessary computation and yields the same result as Softmax.

Considerations

1. Numerical Stability: Large or small logits can cause numerical overflow or underflow during Softmax computation. To mitigate this, subtract the maximum logit from all logits before applying Softmax: P(y_i) = exp(z_i - max(z)) / sum(exp(z_j - max(z))) This adjustment ensures stable calculations without affecting the final probabilities.
2. Gradient Behavior: The magnitude of logits affects gradients during backpropagation, influencing model training dynamics. Proper initialization and regularization can help manage this.
3. Interpretability: Logits are not human-readable probabilities but provide insight into how confident the model is about different classes before normalization.

5. Key Takeaways and Insights

1. Logits Are the Raw Model Outputs: They represent unnormalized scores indicating the model's inclination toward different classes.
2. Softmax Converts Logits to Probabilities: This transformation is essential for interpreting model predictions and training with probability-based loss functions.
3. Numerical Stability Is Critical: Subtracting the maximum logit during Softmax computation avoids overflow and ensures robust results.
4. Efficiency in Inference: For classification tasks, the maximum logit directly gives the predicted class, eliminating the need for Softmax in inference pipelines.

By understanding logits and their transformation into probabilities, you gain deeper insights into the inner workings of deep learning models and how they make predictions. With practical examples and careful considerations, logits can be harnessed effectively for various machine learning tasks.