pytorch中的学习率调整函数
参考：pytorch/docs/master/optim.html#how-to-adjust-learning-rate
torch.optim.lr_scheduler提供了⼏种⽅法来根据迭代的数量来调整学习率
⾃⼰⼿动定义⼀个学习率衰减函数：
def adjust_learning_rate(optimizer, epoch, lr):
"""Sets the learning rate to the initial LR decayed by 10 every 2 epochs"""
lr *= (0.1 ** (epoch // 2))
for param_group in optimizer.param_groups:
param_group['lr'] = lr
optimizer通过param_group来管理参数组。param_group中保存了参数组及其对应的学习率，动量等等
使⽤：
model = AlexNet(num_class=2)
optimizer = optim.SGD(params = model.parameters(), lr=10)
plt.figure()
x = list(range(10))
y = []
lr_init = optimizer.param_groups[0]['lr']
for epoch in range(10):
adjust_learning_rate(optimizer, epoch, lr_init)
羽毛球拍品牌排名lr = optimizer.param_groups[0]['lr']
print(epoch, lr)
y.append(lr)
plt.plot(x,y)
返回：
010.0
110.0
21.0
31.0
40.10000000000000002
50.10000000000000002
60.010000000000000002
70.010000000000000002
80.0010000000000000002
90.0010000000000000002
如图：
举例先导⼊所需的库：
import torch
import torch.optim as optim
from torch.optim import lr_scheduler
dels import AlexNet
import matplotlib.pyplot as plt
1.LambdaLR
CLASS torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda, last_epoch=-1)
将每个参数组的学习率设置为初始lr乘以给定函数。当last_epoch=-1时，将初始lr设置为lr。
参数：
optimizer () – 封装好的优化器
怎么去锈
lr_lambda (function or ) –当是⼀个函数时，需要给其⼀个整数参数，使其计算出⼀个乘数因⼦,⽤于调整学习率，通常该输⼊参数是epoch数⽬；或此类函数的列表，根据在optimator.param_groups中的每组的长度决定lr_lambda的函数个数，如下报错。
last_epoch () – 最后⼀个迭代epoch的索引. Default: -1.画画课
如：
optimizer = optim.SGD(params = model.parameters(), lr=0.05)
lambda1 = lambda epoch:epoch // 10 #根据epoch计算出与lr相乘的乘数因⼦为epoch//10的值
lambda2 = lambda epoch:0.95 ** epoch #根据epoch计算出与lr相乘的乘数因⼦为0.95 ** epoch的值
scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
报错：
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-2-c02d2d9ffc0d> in <module>
4 lambda1 = lambda epoch:epoch // 10
5 lambda2 = lambda epoch:0.95 ** epoch
----> 6 scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
7 plt.figure()
8 x = list(range(40))
/anaconda3/envs/deeplearning/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in __init__(lf, optimizer, lr_lambda, last_epoch) 83if len(lr_lambda) != len(optimizer.param_groups):
84 rai ValueError("Expected {} lr_lambdas, but got {}".format(
---> 85 len(optimizer.param_groups), len(lr_lambda)))
86 lf.lr_lambdas = list(lr_lambda)
87 lf.last_epoch = last_epoch
ValueError: Expected 1 lr_lambdas, but got 2
说明这⾥只需要⼀个lambda函数
举例：
1)使⽤的是lambda2
model = AlexNet(num_class=2)
optimizer = optim.SGD(params = model.parameters(), lr=0.05)
#下⾯是两种lambda函数
#epoch=0到9时，epoch//10=0，所以这时的lr = 0.05*0=0
#epoch=10到19时，epoch//10=1，所以这时的lr = 0.05*1=0.05
lambda1 = lambda epoch:epoch // 10
#当epoch=0时，lr = lr * (0.2**0)=0.05；当epoch=1时，lr = lr * (0.2**1)=0.01
lambda2 = lambda epoch:0.2 ** epoch
scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda2)
plt.figure()
x = list(range(40))
y = []
for epoch in range(40):
scheduler.step()
lr = _lr()
print(epoch, _lr()[0])
y._lr()[0])
plt.plot(x,y)
返回：
00.05
10.010000000000000002
20.0020000000000000005
30.00040000000000000013
48.000000000000002e-05
51.6000000000000006e-05
63.2000000000000015e-06
76.400000000000002e-07
81.2800000000000006e-07
92.5600000000000014e-08
105.120000000000003e-09
111.0240000000000006e-09
122.0480000000000014e-10
134.096000000000003e-11
148.192000000000007e-12
151.6384000000000016e-12
163.276800000000003e-13
176.553600000000007e-14
181.3107200000000014e-14
192.621440000000003e-15
205.242880000000006e-16
211.0485760000000013e-16
222.0971520000000027e-17
234.194304000000006e-18
248.388608000000012e-19
251.6777216000000025e-19
263.355443200000005e-20
276.71088640000001e-21
281.3421772800000022e-21
292.6843545600000045e-22
305.368709120000009e-23
311.0737418240000018e-23
322.147483648000004e-24
334.294967296000008e-25
348.589934592000016e-26
351.7179869184000033e-26
363.435973836800007e-27
376.871947673600015e-28
381.3743895347200028e-28
392.748779069440006e-29
View Code
如图：
也可以写成下⾯的格式：
def lambda_rule(epoch):
lr_l = 0.2 ** epoch
return lr_l
scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda_rule)
2)使⽤的是lambda1函数：
返回：
00.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0
90.0
100.05
110.05
120.05
130.05
140.05
150.05
160.05
170.05
180.05
190.05
200.1
210.1
220.1
230.1
240.1
250.1
260.1
270.1
280.1
290.1
300.15000000000000002
310.15000000000000002
320.15000000000000002
330.15000000000000002
340.15000000000000002
350.15000000000000002
360.15000000000000002
370.15000000000000002
380.15000000000000002
390.15000000000000002
View Code
如图：
其他函数：
load_state_dict(state_dict)
下载调试器状态
参数：
state_dict () – 调试器状态。应为state_dict()调⽤返回的对象.
state_dict()
以字典格式返回调试器的状态
它为lf.__dict__中的每个变量都包含⼀个条⽬，这不是优化器。只有当学习率的lambda函数是可调⽤对象时才会保存它们，⽽当它们是函数或lambdas时则不会保存它们。
2.StepLR
CLASS torch.optim.lr_scheduler.StepLR(optimizer, step_size, gamma=0.1, last_epoch=-1)
每个step_size时间步长后使每个参数组的学习率降低。注意，这种衰减可以与此调度程序外部对学习率的其他更改同时发⽣。当last_epoch=-1时，将初始lr设置为lr。
参数：
optimizer () – 封装的优化器
step_size () – 学习率衰减的周期
gamma () – 学习率衰减的乘数因⼦。Default: 0.1.
last_epoch () – 最后⼀个迭代epoch的索引. Default: -1.
举例：风声台词
model = AlexNet(num_class=2)
optimizer = optim.SGD(params = model.parameters(), lr=0.05)
#即每10次迭代，lr = lr * gamma
scheduler = lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)
plt.figure()
x = list(range(40))
y = []
for epoch in range(40):
scheduler.step()
lr = _lr()
print(epoch, _lr()[0])
y._lr()[0])
plt.plot(x,y)
返回：
00.05
10.05
20.05
30.05
40.05
50.05
60.05
70.05
80.05
90.05
100.005000000000000001
110.005000000000000001
120.005000000000000001
130.005000000000000001
140.005000000000000001
150.005000000000000001
160.005000000000000001
170.005000000000000001
180.005000000000000001
190.005000000000000001
200.0005000000000000001
210.0005000000000000001
220.0005000000000000001
230.0005000000000000001
240.0005000000000000001
250.0005000000000000001
260.0005000000000000001
270.0005000000000000001
280.0005000000000000001
290.0005000000000000001
305.0000000000000016e-05
315.0000000000000016e-05
325.0000000000000016e-05
335.0000000000000016e-05
345.0000000000000016e-05
355.0000000000000016e-05
365.0000000000000016e-05
375.0000000000000016e-05
385.0000000000000016e-05
395.0000000000000016e-05
View Code
快收如图：
3.MultiStepLR
CLASS torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones, gamma=0.1, last_epoch=-1)
当迭代数epoch达到某个⾥程碑时，每个参数组的学习率将被gamma衰减。注意，这种衰减可以与此调度程序外部对学习率的其他更改同时发⽣。当last_epoch=-1时，将初始lr 设置为lr。
参数：
optimizer () – 封装的优化器
milestones () –迭代epochs指数列表. 列表中的值必须是增长的.
gamma () – 学习率衰减的乘数因⼦。Default: 0.1.
last_epoch () – 最后⼀个迭代epoch的索引. Default: -1.
举例：
model = AlexNet(num_class=2)
optimizer = optim.SGD(params = model.parameters(), lr=0.05)
#在指定的epoch值，如[10,15，25，30]处对学习率进⾏衰减，lr = lr * gamma
scheduler = lr_scheduler.MultiStepLR(optimizer, milestones=[10,15,25,30], gamma=0.1)
plt.figure()
x = list(range(40))
y = []
for epoch in range(40):
scheduler.step()
lr = _lr()
print(epoch, _lr()[0])
y._lr()[0])
plt.plot(x,y)
返回：
00.05
10.05
20.05
30.05
40.05
50.05
60.05
70.05
80.05
90.05
100.005000000000000001
110.005000000000000001
120.005000000000000001
130.005000000000000001
140.005000000000000001
150.0005000000000000001
160.0005000000000000001
170.0005000000000000001
180.0005000000000000001
190.0005000000000000001
200.0005000000000000001
210.0005000000000000001
220.0005000000000000001
230.0005000000000000001
滕王阁序诗240.0005000000000000001
255.0000000000000016e-05
265.0000000000000016e-05
275.0000000000000016e-05
285.0000000000000016e-05
295.0000000000000016e-05
305.000000000000001e-06毛蚶怎么读
315.000000000000001e-06
325.000000000000001e-06
335.000000000000001e-06
345.000000000000001e-06
355.000000000000001e-06
365.000000000000001e-06
375.000000000000001e-06
385.000000000000001e-06
395.000000000000001e-06
View Code
如图：
！！！！！有问题，更改：
可能是新版源码进⾏了更改，不能够再使⽤get_lr()来得到当前的lr值了，因为在get_lr()函数中实现了lr
的衰减，这样你step()中会调⽤⼀次get_lr()，然后你为了得到lr⼜掉⽤⼀次get_lr()，这样你就会发现得到的lr结果是衰减两次的结果，改为使⽤get_last_lr()即可
我只看了该函数的get_lr()，其他函数可能也有同样的问题，请根据该说明进⾏相应更改：
议付
import torch.optim as optim
from torch.optim.lr_scheduler import MultiStepLR
from matplotlib import pyplot as plt
dels as models
model = models.vgg16()
optimizer = optim.SGD(params = model.parameters(), lr=0.05)
#在指定的epoch值，如[10,15，25，30]处对学习率进⾏衰减，lr = lr * gamma
scheduler = MultiStepLR(optimizer, milestones=[10,15,25,30], gamma=0.1)
plt.figure()
x = list(range(40))
y = []
for epoch in range(40):
print(epoch)
print("before scheduler step ,lr is ", _last_lr()[0])
y._last_lr()[0]) #在进⾏step()前才是当前step的lr值
scheduler.step() #step()函数实现lr衰减就已经调⽤get_lr()⼀次,get_lr()函数实现lr的衰减
#使⽤get_last_lr()才是单纯访问当前lr的值
print("after scheduler step ,lr is ", _last_lr()[0])
plt.plot(x,y)
plt.show()
结果与上⾯相同
4.ExponentialLR
CLASS torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma, last_epoch=-1)
每个epoch都对每个参数组的学习率进⾏衰减。当last_epoch=-1时，将初始lr设置为lr。
