BatchNorm 动量约定 PyTorch

问题描述

是 batchnorm 动量约定(默认=0.1) 像在其他库中一样正确，例如Tensorflow 默认情况下似乎通常是 0.9 或 0.99?或者，也许我们只是使用了不同的约定?

Is the batchnorm momentum convention (default=0.1) correct as in other libraries e.g. Tensorflow it seems to usually be 0.9 or 0.99 by default? Or maybe we are just using a different convention?

推荐答案

貌似pytorch中的参数化约定和tensorflow中的不同，所以pytorch中的0.1相当于tensorflow中的0.9.

It seems that the parametrization convention is different in pytorch than in tensorflow, so that 0.1 in pytorch is equivalent to 0.9 in tensorflow.

更准确地说:

在张量流中:

running_mean = decay*running_mean + (1-decay)*new_value

在 PyTorch 中:

In PyTorch:

running_mean = (1-decay)*running_mean + decay*new_value

这意味着 PyTorch 中 decay 的值等同于 Tensorflow 中 (1-decay) 的值.

This means that a value of decay in PyTorch is equivalent to a value of (1-decay) in Tensorflow.

这篇关于BatchNorm 动量约定 PyTorch的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！