pytorch（16）损失函数（二）

5和6是在数据回归中用的较多的损失函数

5. nn.L1Loss

功能：计算inputs与target之差的绝对值

代码：

nn.L1Loss(reduction='mean')

公式：

\[l_n = |x_n-y_n|
\]

6. nn.MSELoss

功能：计算inputs与target之差的平方

代码：

nn.MSELoss(reduction='mean')

主要参数：reduction:计算模式,none/sum/mean

公式：

\[l_n = (x_n - y_n)^2
\]

7. SmoothL1Loss

功能：平滑的L1Loss

代码：

nn.SmoothL1Loss(size_average=None,reduce=None,reduction='mean')
$$f(x)=
\begin{cases}
0& \text{x=0}\\
1& \text{x!=0}
\end{cases}$$

\[loss(x,y)=\frac{1}{n}\sum_{i}{z_i}\\
z_i = \begin{cases}
0.5(x_i - y_i)^2& \text{,if|x_i - y_i|}<1\\
|x_i-y_i|-0.5& \text{,otherwise} \end{cases}
\]

8. PoissonNLLLoss

功能：泊松分布的负对数似然损失函数

主要参数：

• log_input:输入是否为对数形式，决定计算公式
• full:计算所有Loss，默认为False
• eps:修正项，避免log(input)为nan
  
  代码：

nn.PoissonNLLLoss(log_input=True,full=False,eps=1e-08,reduction='mean') 

log_input = True
loss(input,target)=exp(input)-target*input

log_input = False
loss(input,target)=input-target*log(input+eps)

9. nn.KLDivLoss

功能：计算KLD(divergence),KL散度，相对熵

注意事项：需提前将输入计算log-probabilities,如通过nn.logsoftmax()，也就是输入一个概率，概率取值的区间是(0,1)。因为在公式中有Log，因此要求我们在输入中就给log函数进行运算。

主要参数：

-reduce：none,sum,mean,batchmean.

batchmean-batchsize维度求平均值

nn.KLDivLoss(reduction='mean')

\[D_{KL}(P||Q)=E_{x\sim p}\Big[log \frac{P(x)}{Q(x)} \Big]=E_{x\sim p}[log P(x)-log Q(x)]\\
=\sum_{i=1}^{N}P(x_i)(logP(x_i)-logQ(x_i))
\]

\[l_n = y_n(log_{yn}-x_n)
\]

10. nn.MarginRankingLoss

功能：计算两个向量之间的相似度，用于排序任务

特别说明：该方法计算两组数据之间的差异，返回一个n*n的loss的矩阵

主要参数：

• margin:边界值，x1与x2之间的差异值
• reduction:计算模式，可为none/sum/mean
  
  y=1时，希望x1比x2大，当x1>x2时，不产生Loss
  
  y=-1时，希望x2比x1大，当x2>x1时，不产生Loss

nn.MarginRankingLoss(margin=0.0,size_average=None,reduce=None,reduction='mean')

\[loss(x,y)=max(0,-y*(x1-x2)+margin)
\]

11. nn.MultiLabelMarginLoss

功能：多标签边界损失函数。

比如一个样本有多个类别。

距离：四分类任务，样本x属于0类和3类

标签：[0,3-1,-1]，而不是[1,0,0,1]

主要参数：

• reduction:计算模式，可为none/sum/mean

nn.MultiLabelMarginLoss(size_average=None,reduce=None,reduction='mean')

\[loss(x,y) = \sum_{ij}\frac{max(0,1-(x[y[i]])-x[i])}{x.size(0)}\\
where\ i == 0 \to x.size(0),j==0 \to y.size(0),y[j]\geq 0 ,and\ i \neq y[j] for\ all\ i\ and\ j.
\]

分母是x的大小，是输出向量的神经元的个数。

12. nn.SoftMarginLoss

功能：计算二分类的logistic损失

主要参数

• reduction:计算模式，可为none,sum,mean

nn.SoftMarginLoss(size_average=None,reduce=None,reduction='mean')

\[loss(x,y)=\sum_{i}\frac{log(1+exp(-y[i]*x[i]))}{x.nelement()}
\]

13. nn.MultiLabelSoftMarginLoss

功能：SoftMarginLoss多标签版本

主要参数：这里的标签是01。

• weight:各类别的Loss设置权值
• reduction:计算模式，可为none,sum,mean

nn.MultiLabelSoftMarginLoss(weight=None,size_average=None,reduce=None,reduction='mean')

\[loss(x,y)=-\frac{1}{C}*\sum_i y[i]*log((1+exp(-x[i]))^{-1})
+(1-y[i])*log\Big(\frac{exp(-x[i])}{(1+exp(-x[i]))} \Big)
\]

14. nn.MultiMarginLoss

功能：计算多分类的折页损失

主要参数：

• p:可选1或2
• weight:各类别的Loss设置权值
• margin:边界值
• reduction:计算模式，可为none,sum,mean

nn.MultiMarginLoss(p=1,margin=1.0,weight=None,size_average=None,reduce=None,reduction='mean')

\[loss(x,y)=\frac{\sum_(i)max(0,margin-x[y]+x[i])^p}{x.size(0)}\\
where\ x \in \{0,...,x.size(0)-1 \},y\in \{0,...,y.size(0)-1\},0\leq y[j] \leq x.size(0)-1,and\ i\neq y[j]\ for \ all\ i\ and\ i.
\]

15. nn.TripletMarginLoss

功能：计算三元组损失，人脸验证中常用

主要参数：

• p :范数的阶，默认为2
• margin:边界值
• reduction:计算模式，none,sum,mean

nn.TripletMarginLoss(margin=1.0,p=2.0,eps=1e-06,swap=False,size_average=None,reduce=None,reduction='mean')

\[L(a,p,n)=max\{d(a_i,p_i)-d(a_i,n_i)+margin,0 \} \\
d(x_i,y_i)=||x_i-y_i||_p
\]

16. nn.HingeEmbeddingLoss

功能：计算两个输入的相似性，常用于非线性embedding和半监督学习

特别注意：输入x应为两个输入之差的绝对值

主要参数：

• margin:边界值
• reduction:计算模式，可为none,sum,mean

nn.HingeEmbeddingLoss(margin=1.0,size_average=None,reduce=None,reduction='mean')

\[l_n=\begin{cases}
x_n& \text{,if y_n=1,}\\
max\{0,\Delta -x_n\}& \text{,if y_n=-1}\end{cases}
\]

17. nn.CosineEmbeddingLoss

功能：采用余弦相似度计算两个输入的相似性

主要参数：

• margin:可取值[-1,1],推荐为[0,0.5]
• reduction:计算模式，可为none,sum,mean

nn.CosineEmbeddingLoss(margin=0.0,size_average=None,reduce=None,reduction='mean')

\[loss(x,y)=\begin{cases}
1-cos(x_1,x_2)& \text{if, y=1}\\
max(0,cos(x_1,x_2)-margin)& \text{if, y=-1}\end{cases}
\]

\[cos(\theta)=\frac{A \cdot B}{||A||||b||}=\frac{\sum_{i=1}^{n}A_i\times B_i}{\sqrt{\sum_{i=1}^{n}(A_i)^2}\times \sqrt{\sum_{i=1}^{n}(B_i)^2}}
\]

18. nn.CTCLoss

功能：计算CTC损失，解决时序类数据的分类

Connetionist Temporal Classification

主要参数：

• blank:blank label
• zero_infinity:无穷大的值或梯度置0
• reduction:计算模式，可为none,sum,mean

torch.nn.CTCLoss(blank=0,reduction='mean',zero_infinity=False)

import torch
import torch.nn as nn
import os
import matplotlib.pyplot as plt
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
torch.manual_seed(2)
# ===========================nn.L1Loss===========
# flag = True
flag = False
if flag:
    x1_data = torch.ones((2, 2))
    x1_label = torch.ones((2, 2)) * 3

    loss1 = nn.L1Loss(reduction='none')
    loss_data = loss1(x1_data, x1_label)
    print("x1_data:{}\nx1_label:{}\nloss:{}".format(x1_data,x1_label,loss_data))

# ===========================nn.MSELoss===========
# flag = True
flag = False
if flag:
    x1_data = torch.ones((2, 2))
    x1_label = torch.ones((2, 2)) * -1

    loss1 = nn.MSELoss(reduction='none')
    loss_data = loss1(x1_data, x1_label)
    print("x1_data:{}\nx1_label:{}\nloss:{}".format(x1_data,x1_label,loss_data))

# ===========================nn.SmoothLoss===========
# flag = True
flag = False
if flag:
    x1_data = torch.linspace(-3, 3, steps=1000)
    x1_label = torch.zeros_like(x1_data)

    lossmoth1 = nn.SmoothL1Loss(reduction='none')
    loss_datamoth1 = lossmoth1(x1_data, x1_label)

    loss1 = nn.L1Loss(reduction='none')
    loss_data = loss1(x1_data, x1_label)

    plt.plot(x1_data.numpy(), loss_data.numpy(), label='L1Loss')
    plt.plot(x1_data.numpy(), loss_datamoth1.numpy(), label='SmoothL1Loss')
    plt.xlabel('xi-yi')
    plt.ylabel('loss value')
    plt.legend()
    plt.grid()
    plt.show()

# ===========================nn.PoissonNLLLoss===========
# flag = True
flag = False
if flag:
    x1_data = torch.randn((2, 2))
    x1_label = torch.randn((2, 2))*2+0.6

    loss1 = nn.PoissonNLLLoss(log_input=False, reduction='none',eps=1e-02)
    loss_data = loss1(x1_data, x1_label)

    print(x1_data)
    print(x1_label)
    print(loss_data)

    # computebyhan = torch.exp(x1_data)-x1_label*x1_data
    # print(computebyhan)

    computebyhan1 = x1_data - x1_label*torch.log(x1_data+1e-02)
    print(computebyhan1)

# ===========================nn.KLDivLoss===========
# flag = True
flag = False
if flag:
    x1_data = torch.tensor([[0.5, 0.4, 0.1], [0.2, 0.4, 0.4], [0.1, 0.5, 0.4], [0.1, 0.8, 0.1]])
    x1_datalog = torch.log(x1_data)
    x1_label = torch.tensor([[0.7, 0.2, 0.1], [0.1, 0.4, 0.5], [0.4, 0.2, 0.4], [0.3, 0.4, 0.3]],dtype=torch.float)

    lossd = nn.KLDivLoss(reduction='none')
    loss_data = lossd(x1_data, x1_label)
    print(loss_data)

    ln = x1_label*(torch.log(x1_label)-x1_data)
    print(ln)

# ===========================nn.MarginRankingLoss===========
# flag = True
flag = False
if flag:
    x1 = torch.tensor([[1], [2], [3]], dtype=torch.float)
    x2 = torch.tensor([[2], [2], [2]], dtype=torch.float)

    labeld = torch.tensor([1, 1, -1], dtype=torch.float)

    loss_f = nn.MarginRankingLoss(margin=0, reduction='none')

    loss = loss_f(x1, x2, labeld)
    print(loss)

# ===========================nn.MultiLabelMarginLoss===========
# flag = True
flag = False
if flag:
    x = torch.tensor([[0.4, 02., 0.4, 0.5]])
    y = torch.tensor([[0, 3, -1, -1]], dtype=torch.long)

    lossf = nn.MultiLabelMarginLoss(reduction='none')
    loss = lossf(x, y)

    print(loss)

    x = x[0]
    item1 = (1 - (x[0]-x[1]))+(1-(x[0]-x[2]))
    item2 = (1 - (x[3]-x[1]))+(1-(x[3]-x[2]))
    loss2 = (item1+item2)/4
    print(loss2)

# ===========================nn.SoftMarginLoss===========
# flag = True
flag = False
if flag:
    x = torch.tensor([[0.4, 0.6], [0.8, 0.2]])
    y = torch.tensor([[-1, 1], [1, -1]], dtype=torch.long)

    lossf = nn.SoftMarginLoss(reduction='none')
    loss = lossf(x, y)

    print(loss)

    idx = 0
    itemx = x[idx, idx]
    itemy = y[idx, idx]
    loss = torch.log(1+torch.exp(-itemy*itemx))
    print(loss)

# ===========================nn.MultiLabelSoftMarginLoss===========
# flag = True
flag = False
if flag:
    x = torch.tensor([[0.4, 0.6, 0.8, 0.2]])
    y = torch.tensor([[0, 1, 1, 0]], dtype=torch.long)

    lossf = nn.MultiLabelSoftMarginLoss(reduction='none')
    loss = lossf(x, y)

    print(loss)

    sum=0
    for i in range(4):
        if(y[0,i]==1):
            sum+=y[0, i]*torch.log((1+torch.exp(-x[0,i]))**(-1))
        else:
            sum+=torch.log(torch.exp(-x[0,i])/(1+torch.exp(-x[0,i])))
    sum=sum/4
    print(sum)

# ===========================nn.MultiMarginLoss===========
# ---------------------------------------------- 14 Multi Margin Loss -----------------------------------------
flag = 0
# flag = 1
if flag:

    x = torch.tensor([[0.1, 0.2, 0.7], [0.2, 0.5, 0.3]])
    y = torch.tensor([1, 2], dtype=torch.long)

    loss_f = nn.MultiMarginLoss(reduction='none')

    loss = loss_f(x, y)

    print("Multi Margin Loss: ", loss)

# --------------------------------- compute by hand
flag = 0
# flag = 1
if flag:

    x = x[0]
    margin = 1

    i_0 = margin - (x[1] - x[0])
    # i_1 = margin - (x[1] - x[1])
    i_2 = margin - (x[1] - x[2])

    loss_h = (i_0 + i_2) / x.shape[0]

    print(loss_h)

# ---------------------------------------------- 15 Triplet Margin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:
    anchor = torch.tensor([[1.]])
    pos = torch.tensor([[2.]])
    neg = torch.tensor([[0.5]])

    lossd = nn.TripletMarginLoss(margin=1, p=1)

    loss = lossd(anchor, pos, neg)
    print(loss)

    ap = abs(pos-anchor)
    an = abs(anchor-neg)
    l = ap-an+1
    print(l)

# ---------------------------------------------- 16 Hinge Embedding Loss -----------------------------------------
flag = 0
# flag = 1
if flag:

    inputs = torch.tensor([[1., 0.8, 0.5]])
    target = torch.tensor([[1, 1, -1]])

    loss_f = nn.HingeEmbeddingLoss(margin=1, reduction='none')

    loss = loss_f(inputs, target)

    print("Hinge Embedding Loss", loss)

# --------------------------------- compute by hand
flag = 0
# flag = 1
if flag:
    margin = 1.
    loss = max(0, margin - inputs.numpy()[0, 2])

    print(loss)

# ---------------------------------------------- 17 Cosine Embedding Loss -----------------------------------------
flag = 0
# flag = 1
if flag:

    x1 = torch.tensor([[0.3, 0.5, 0.7], [0.3, 0.5, 0.7]])
    x2 = torch.tensor([[0.1, 0.3, 0.5], [0.1, 0.3, 0.5]])

    target = torch.tensor([[1, -1]], dtype=torch.float)

    loss_f = nn.CosineEmbeddingLoss(margin=0., reduction='none')

    loss = loss_f(x1, x2, target)

    print("Cosine Embedding Loss", loss)

# --------------------------------- compute by hand
flag = 0
# flag = 1
if flag:
    margin = 0.

    def cosine(a, b):
        numerator = torch.dot(a, b)
        denominator = torch.norm(a, 2) * torch.norm(b, 2)
        return float(numerator/denominator)

    l_1 = 1 - (cosine(x1[0], x2[0]))

    l_2 = max(0, cosine(x1[0], x2[0]))

    print(l_1, l_2)