Feedforward Neural Networks in Depth, Part 1: Forward and Backward Propagations深度解析前馈神经网络，第一部分：前向传播与反向传播

This post is the first of a three-part series in which we set out to derive the mathematics behind feedforward neural networks. They have
本文是三部曲系列的第一篇，旨在推导前馈神经网络背后的数学原理。

• an input and an output layer with at least one hidden layer in between,
  一个输入层和一个输出层，中间至少有一个隐藏层，
• fully-connected layers, which means that each node in one layer connects to every node in the following layer, and
  全连接层，意味着一层中的每个节点都与下一层的所有节点相连，并且
• ways to introduce nonlinearity by means of activation functions.
  通过激活函数引入非线性的方法。

We start with forward propagation, which involves computing predictions and the associated cost of these predictions.
我们首先进行前向传播，这涉及计算预测值及其相应的成本。

Forward Propagation 前向传播

Settling on what notations to use is tricky since we only have so many letters in the Roman alphabet. As you browse the Internet, you will likely find derivations that have used different notations than the ones we are about to introduce. However, and fortunately, there is no right or wrong here; it is just a matter of taste. In particular, the notations used in this series take inspiration from Andrew Ng’s Standard notations for Deep Learning. If you make a comparison, you will find that we only change a couple of the details.
选择使用哪些符号颇具挑战，因为罗马字母数量有限。浏览互联网时，你可能会发现与我们即将介绍的符号不同的衍生用法。然而，幸运的是，这里并无对错之分，仅关乎个人偏好。特别是，本系列所采用的符号灵感源自 Andrew Ng 的深度学习标准符号。若进行对比，你会发现我们仅对细节做了少许改动。

Now, whatever we come up with, we have to support
现在，无论我们提出什么，我们都必须支持

• multiple layers, 多层结构
• several nodes in each layer,
  每层中的若干节点，
• various activation functions,
  各种激活函数
• various types of cost functions, and
  各种类型的成本函数，以及
• mini-batches of training examples.
  训练示例的小批量。

As a result, our definition of a node ends up introducing a fairly large number of notations:
因此，我们对节点的定义最终引入了相当多的符号表示：

$$\begin{array}{}\text{(1)}& z_{j,i}^{[l]}& =\sum _k{w}_{j,k}^{[l]}{a}_{k,i}^{[l-1]}+b_j^{[l]},\text{(2)}& a_{j,i}^{[l]}& =g_j^{[l]}(z_{1,i}^{[l]},\dots ,z_{j,i}^{[l]},\dots ,z_{n^{[l]},i}^{[l]}).\end{array}$$

Does the node definition look intimidating to you at first glance? Do not worry. Hopefully, it will make more sense once we have explained the notations, which we shall do next:
节点定义乍看之下是否让你感到畏惧？不必担心。希望在我们解释了符号之后，它将变得更加清晰，接下来我们将进行这一步骤：

Entity 实体	Description 描述
$l$	The current layer $l=1,\dots ,L$, where $L$ is the number of layers that have weights and biases. We use $l=0$ and $l=L$ to denote the input and output layers. 当前层 $l=1,\dots ,L$ ，其中 $L$ 表示具有权重和偏置的层数。我们使用 $l=0$ 和 $l=L$ 来表示输入层和输出层。
$n^{[l]}$	The number of nodes in the current layer. 当前层的节点数。
$n^{[l-1]}$	The number of nodes in the previous layer. 前一层中的节点数量。
$j$	The $j$th node of the current layer, $j=1,\dots ,n^{[l]}$. 当前层的第 $j$ 个节点， $j=1,\dots ,n^{[l]}$ 。
$k$	The $k$th node of the previous layer, $k=1,\dots ,n^{[l-1]}$. 前一层的第 $k$ 个节点， $k=1,\dots ,n^{[l-1]}$ 。
$i$	The current training example $i=1,\dots ,m$, where $m$ is the number of training examples. 当前训练示例 $i=1,\dots ,m$ ，其中 $m$ 表示训练示例的数量。
$z_{j,i}^{[l]}$	A weighted sum of the activations of the previous layer, shifted by a bias. 前一层激活值的加权和，经偏置调整。
$w_{j,k}^{[l]}$	A weight that scales the $k$th activation of the previous layer. 一个权重，用于调整前一层第 $k$ 个激活值的尺度。
$b_j^{[l]}$	A bias in the current layer. 当前层的偏差。
$a_{j,i}^{[l]}$	An activation in the current layer. 当前层的激活。
$a_{k,i}^{[l-1]}$	An activation in the previous layer. 上一层的激活。
$g_j^{[l]}$	An activation function $g_j^{[l]}:{\mathbb{R}}^{n^{[l]}}\to \mathbb{R}$ used in the current layer. 当前层使用的激活函数 $g_j^{[l]}:{\mathbb{R}}^{n^{[l]}}\to \mathbb{R}$ 。

To put it concisely, a node in the current layer depends on every node in the previous layer, and the following visualization can help us see that more clearly:
简而言之，当前层的节点依赖于前一层的每个节点，下面的可视化图将帮助我们更清晰地理解这一点：

Figure 1: A node in the current layer.
图 1：当前层中的一个节点。

Moreover, a node in the previous layer affects every node in the current layer, and with a change in highlighting, we will also be able to see that more clearly:
此外，前一层的节点会影响当前层的每个节点，通过改变突出显示方式，我们也将更清楚地看到这一点：

Figure 2: A node in the previous layer.
图 2：前一层中的一个节点。

In the future, we might want to write an implement from scratch in, for example, Python. To take advantage of the heavily optimized versions of vector and matrix operations that come bundled with libraries such as NumPy, we need to vectorize $\text{(1)}$ and $\text{(2)}$.
未来，我们可能希望从头开始编写一个实现，例如使用 Python。为了充分利用与 NumPy 等库捆绑在一起的向量和矩阵操作的高度优化版本，我们需要对 $\text{(1)}$ 和 $\text{(2)}$ 进行向量化。

To begin with, we vectorize the nodes:
首先，我们对节点进行向量化：

$$\begin{array}{rl}\left[\begin{array}{c}{z}_{1,i}^{[l]}\\ ⋮\\ z_{j,i}^{[l]}\\ ⋮\\ z_{n^{[l]},i}^{[l]}\end{array}\right]& =\left[\begin{array}{ccccc}{w}_{1,1}^{[l]}& \dots & w_{1,k}^{[l]}& \dots & w_{1,n^{[l-1]}}^{[l]}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ w_{j,1}^{[l]}& \dots & w_{j,k}^{[l]}& \dots & w_{j,n^{[l-1]}}^{[l]}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ w_{n^{[l]},1}^{[l]}& \dots & w_{n^{[l]},k}^{[l]}& \dots & w_{n^{[l]},n^{[l-1]}}^{[l]}\end{array}\right]\left[\begin{array}{c}{a}_{1,i}^{[l-1]}\\ ⋮\\ a_{k,i}^{[l-1]}\\ ⋮\\ a_{n^{[l-1]},i}^{[l-1]}\end{array}\right]+\left[\begin{array}{c}{b}_1^{[l]}\\ ⋮\\ b_j^{[l]}\\ ⋮\\ b_{n^{[l]}}^{[l]}\end{array}\right],\\ \left[\begin{array}{c}{a}_{1,i}^{[l]}\\ ⋮\\ a_{j,i}^{[l]}\\ ⋮\\ a_{n^{[l]},i}^{[l]}\end{array}\right]& =\left[\begin{array}{c}{g}_1^{[l]}(z_{1,i}^{[l]},\dots ,z_{j,i}^{[l]},\dots ,z_{n^{[l]},i}^{[l]})\\ ⋮\\ g_j^{[l]}(z_{1,i}^{[l]},\dots ,z_{j,i}^{[l]},\dots ,z_{n^{[l]},i}^{[l]})\\ ⋮\\ g_{n^{[l]}}^{[l]}(z_{1,i}^{[l]},\dots ,z_{j,i}^{[l]},\dots ,z_{n^{[l]},i}^{[l]})\end{array}\right],\end{array}$$

which we can write as
我们可以将其写作

$$\begin{array}{}\text{(3)}& {\mathbf{z}}_{:,i}^{[l]}& ={\mathbf{W}}^{[l]}{\mathbf{a}}_{:,i}^{[l-1]}+{\mathbf{b}}^{[l]},\text{(4)}& {\mathbf{a}}_{:,i}^{[l]}& ={\mathbf{g}}^{[l]}({\mathbf{z}}_{:,i}^{[l]}),\end{array}$$

where ${\mathbf{z}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$, ${\mathbf{W}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times n^{[l-1]}}$, ${\mathbf{b}}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$, ${\mathbf{a}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$, ${\mathbf{a}}_{:,i}^{[l-1]}\in {\mathbb{R}}^{n^{[l-1]}}$, and lastly, ${\mathbf{g}}^{[l]}:{\mathbb{R}}^{n^{[l]}}\to {\mathbb{R}}^{n^{[l]}}$. We have used a colon to clarify that ${\mathbf{z}}_{:,i}^{[l]}$ is the $i$th column of ${\mathbf{Z}}^{[l]}$, and so on.
其中 ${\mathbf{z}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$ ， ${\mathbf{W}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times n^{[l-1]}}$ ， ${\mathbf{b}}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$ ， ${\mathbf{a}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$ ， ${\mathbf{a}}_{:,i}^{[l-1]}\in {\mathbb{R}}^{n^{[l-1]}}$ ，最后是 ${\mathbf{g}}^{[l]}:{\mathbb{R}}^{n^{[l]}}\to {\mathbb{R}}^{n^{[l]}}$ 。我们使用冒号来明确指出 ${\mathbf{z}}_{:,i}^{[l]}$ 是 ${\mathbf{Z}}^{[l]}$ 的第 $i$ 列，依此类推。

Next, we vectorize the training examples:
接下来，我们将训练样本向量化：

$$\begin{array}{rl}\text{(5)}& {\mathbf{Z}}^{[l]}& =\left[\begin{array}{ccccc}{\mathbf{z}}_{:,1}^{[l]}& \dots & {\mathbf{z}}_{:,i}^{[l]}& \dots & {\mathbf{z}}_{:,m}^{[l]}\end{array}\right]& ={\mathbf{W}}^{[l]}\left[\begin{array}{ccccc}{\mathbf{a}}_{:,1}^{[l-1]}& \dots & {\mathbf{a}}_{:,i}^{[l-1]}& \dots & {\mathbf{a}}_{:,m}^{[l-1]}\end{array}\right]+\left[\begin{array}{ccccc}{\mathbf{b}}^{[l]}& \dots & {\mathbf{b}}^{[l]}& \dots & {\mathbf{b}}^{[l]}\end{array}\right]\\ & ={\mathbf{W}}^{[l]}{\mathbf{A}}^{[l-1]}+\mathrm{broadcast}({\mathbf{b}}^{[l]}),\\ \text{(6)}& {\mathbf{A}}^{[l]}& =\left[\begin{array}{ccccc}{\mathbf{a}}_{:,1}^{[l]}& \dots & {\mathbf{a}}_{:,i}^{[l]}& \dots & {\mathbf{a}}_{:,m}^{[l]}\end{array}\right],\end{array}$$

where ${\mathbf{Z}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times m}$, ${\mathbf{A}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times m}$, and ${\mathbf{A}}^{[l-1]}\in {\mathbb{R}}^{n^{[l-1]}\times m}$. In addition, have a look at the NumPy documentation if you want to read a well-written explanation of broadcasting.
其中 ${\mathbf{Z}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times m}$ 、 ${\mathbf{A}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times m}$ 和 ${\mathbf{A}}^{[l-1]}\in {\mathbb{R}}^{n^{[l-1]}\times m}$ 。此外，若想阅读有关广播的详尽解释，可参阅 NumPy 文档。

We would also like to establish two additional notations:
我们还希望建立两个额外的符号表示：

$$\begin{array}{}\text{(7)}& {\mathbf{A}}^{[0]}& =\mathbf{X},\text{(8)}& {\mathbf{A}}^{[L]}& =\hat{\mathbf{Y}},\end{array}$$

where $\mathbf{X}\in {\mathbb{R}}^{n^{[0]}\times m}$ denotes the inputs and $\hat{\mathbf{Y}}\in {\mathbb{R}}^{n^{[L]}\times m}$ denotes the predictions/outputs.
其中 $\mathbf{X}\in {\mathbb{R}}^{n^{[0]}\times m}$ 表示输入， $\hat{\mathbf{Y}}\in {\mathbb{R}}^{n^{[L]}\times m}$ 表示预测/输出。

Finally, we are ready to define the cost function:
最后，我们准备定义成本函数：

$$\begin{array}{}\text{(9)}& J=f(\hat{\mathbf{Y}},\mathbf{Y})=f({\mathbf{A}}^{[L]},\mathbf{Y}),\end{array}$$

where $\mathbf{Y}\in {\mathbb{R}}^{n^{[L]}\times m}$ denotes the targets and $f:{\mathbb{R}}^{2n^{[L]}}\to \mathbb{R}$ can be tailored to our needs.
其中 $\mathbf{Y}\in {\mathbb{R}}^{n^{[L]}\times m}$ 表示目标，而 $f:{\mathbb{R}}^{2n^{[L]}}\to \mathbb{R}$ 可根据我们的需求进行定制。

We are done with forward propagation! Next up: backward propagation, also known as backpropagation, which involves computing the gradient of the cost function with respect to the weights and biases.
我们已完成前向传播！接下来：反向传播，亦称反向传播算法，涉及计算成本函数相对于权重和偏置的梯度。

Backward Propagation 反向传播

We will make heavy use of the chain rule in this section, and to understand better how it works, we first apply the chain rule to the following example:
本节中我们将大量运用链式法则，为了更好地理解其工作原理，我们首先将链式法则应用于以下示例：

$$\begin{array}{}\text{(10)}& u_i& =g_i(x_1,\dots ,x_j,\dots ,x_n),\text{(11)}& y_k& =f_k(u_1,\dots ,u_i,\dots ,u_m).\end{array}$$

Note that $x_j$ may affect $u_1,\dots ,u_i,\dots ,u_m$, and $y_k$ may depend on $u_1,\dots ,u_i,\dots ,u_m$; thus,
请注意， $x_j$ 可能影响 $u_1,\dots ,u_i,\dots ,u_m$ ，而 $y_k$ 可能依赖于 $u_1,\dots ,u_i,\dots ,u_m$ ；因此，

$$\begin{array}{}\text{(12)}& \frac{\partial y_k}{\partial x_j}=\sum _i\frac{\partial y_k}{\partial u_i}\frac{\partial u_i}{\partial x_j}.\end{array}$$

Great! If we ever get stuck trying to compute or understand some partial derivative, we can always go back to $\text{(10)}$, $\text{(11)}$, and $\text{(12)}$. Hopefully, these equations will provide the clues necessary to move forward. However, be extra careful not to confuse the notation used for the chain rule example with the notation we use elsewhere in this series. The overlap is unintentional.
太好了！如果在计算或理解某些偏导数时遇到困难，我们总可以回过头来参考 $\text{(10)}$ 、 $\text{(11)}$ 和 $\text{(12)}$ 。希望这些方程能提供推进所需的线索。但请特别注意，不要将链式法则示例中使用的符号与我们在此系列其他地方使用的符号混淆，这种重叠是无意的。

Now, let us concentrate on the task at hand:
现在，让我们专注于当前的任务：

$$\begin{array}{}\text{(13)}& \frac{\partial J}{\partial w_{j,k}^{[l]}}& =\sum _i\frac{\partial J}{\partial z_{j,i}^{[l]}}\frac{\partial z_{j,i}^{[l]}}{\partial w_{j,k}^{[l]}}=\sum _i\frac{\partial J}{\partial z_{j,i}^{[l]}}{a}_{k,i}^{[l-1]},\text{(14)}& \frac{\partial J}{\partial b_j^{[l]}}& =\sum _i\frac{\partial J}{\partial z_{j,i}^{[l]}}\frac{\partial z_{j,i}^{[l]}}{\partial b_j^{[l]}}=\sum _i\frac{\partial J}{\partial z_{j,i}^{[l]}}.\end{array}$$

Vectorization results in
向量化结果为

$$\begin{array}{rl}& \left[\begin{array}{ccccc}{\displaystyle \frac{\partial J}{\partial w_{1,1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial w_{1,k}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial w_{1,n^{[l-1]}}^{[l]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial J}{\partial w_{j,1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial w_{j,k}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial w_{j,n^{[l-1]}}^{[l]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial J}{\partial w_{n^{[l]},1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial w_{n^{[l]},k}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial w_{n^{[l]},n^{[l-1]}}^{[l]}}}\end{array}\right]\\ & =\left[\begin{array}{ccccc}{\displaystyle \frac{\partial J}{\partial z_{1,1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{1,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{1,m}^{[l]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{j,1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{j,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{j,m}^{[l]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},i}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},m}^{[l]}}}\end{array}\right]\\ & \phantom{=}\cdot \left[\begin{array}{ccccc}{a}_{1,1}^{[l-1]}& \dots & a_{k,1}^{[l-1]}& \dots & a_{n^{[l-1]},1}^{[l-1]}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ a_{1,i}^{[l-1]}& \dots & a_{k,i}^{[l-1]}& \dots & a_{n^{[l-1]},i}^{[l-1]}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ a_{1,m}^{[l-1]}& \dots & a_{k,m}^{[l-1]}& \dots & a_{n^{[l-1]},m}^{[l-1]}\end{array}\right],\\ & \left[\begin{array}{c}{\displaystyle \frac{\partial J}{\partial b_1^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial b_j^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial b_{n^{[l]}}^{[l]}}}\end{array}\right]=\left[\begin{array}{c}{\displaystyle \frac{\partial J}{\partial z_{1,1}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{j,1}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},1}^{[l]}}}\end{array}\right]+\cdots +\left[\begin{array}{c}{\displaystyle \frac{\partial J}{\partial z_{1,i}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{j,i}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},i}^{[l]}}}\end{array}\right]+\cdots +\left[\begin{array}{c}{\displaystyle \frac{\partial J}{\partial z_{1,m}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{j,m}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},m}^{[l]}}}\end{array}\right],\end{array}$$

which we can write as
我们可以将其写作

$$\begin{array}{}\text{(15)}& \frac{\partial J}{\partial {\mathbf{W}}^{[l]}}& =\sum _i\frac{\partial J}{\partial {\mathbf{z}}_{:,i}^{[l]}}{{\mathbf{a}}_{:,i}^{[l-1]}}^{⊺}=\frac{\partial J}{\partial {\mathbf{Z}}^{[l]}}{{\mathbf{A}}^{[l-1]}}^{⊺},\text{(16)}& \frac{\partial J}{\partial {\mathbf{b}}^{[l]}}& =\sum _i\frac{\partial J}{\partial {\mathbf{z}}_{:,i}^{[l]}}=\underset{\text{column vector}}{\underbrace{\sum _{\text{axis}=1}\frac{\partial J}{\partial {\mathbf{Z}}^{[l]}}}},\end{array}$$

where $\partial J/\partial {\mathbf{z}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$, $\partial J/\partial {\mathbf{Z}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times m}$, $\partial J/\partial {\mathbf{W}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times n^{[l-1]}}$, and $\partial J/\partial {\mathbf{b}}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$.
其中 $\partial J/\partial {\mathbf{z}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$ ， $\partial J/\partial {\mathbf{Z}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times m}$ ， $\partial J/\partial {\mathbf{W}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times n^{[l-1]}}$ ，以及 $\partial J/\partial {\mathbf{b}}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$ 。

Looking back at $\text{(13)}$ and $\text{(14)}$, we see that the only unknown entity is $\partial J/\partial z_{j,i}^{[l]}$. By applying the chain rule once again, we get
回顾 $\text{(13)}$ 和 $\text{(14)}$ ，我们发现唯一未知的实体是 $\partial J/\partial z_{j,i}^{[l]}$ 。再次应用链式法则，我们得到

$$\begin{array}{}\text{(17)}& \frac{\partial J}{\partial z_{j,i}^{[l]}}=\sum _p\frac{\partial J}{\partial a_{p,i}^{[l]}}\frac{\partial a_{p,i}^{[l]}}{\partial z_{j,i}^{[l]}},\end{array}$$

where $p=1,\dots ,n^{[l]}$. 源文本：where $p=1,\dots ,n^{[l]}$ . 翻译文本：在$p=1,\dots ,n^{[l]}$处。

Next, we present the vectorized version:
接下来，我们展示向量化版本：

$$\left[\begin{array}{c}{\displaystyle \frac{\partial J}{\partial z_{1,i}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{j,i}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},i}^{[l]}}}\end{array}\right]=\left[\begin{array}{ccccc}{\displaystyle \frac{\partial a_{1,i}^{[l]}}{\partial z_{1,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial a_{j,i}^{[l]}}{\partial z_{1,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial a_{n^{[l]},i}^{[l]}}{\partial z_{1,i}^{[l]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial a_{1,i}^{[l]}}{\partial z_{j,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial a_{j,i}^{[l]}}{\partial z_{j,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial a_{n^{[l]},i}^{[l]}}{\partial z_{j,i}^{[l]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial a_{1,i}^{[l]}}{\partial z_{n^{[l]},i}^{[l]}}}& \dots & {\displaystyle \frac{\partial a_{j,i}^{[l]}}{\partial z_{n^{[l]},i}^{[l]}}}& \dots & {\displaystyle \frac{\partial a_{n^{[l]},i}^{[l]}}{\partial z_{n^{[l]},i}^{[l]}}}\end{array}\right]\left[\begin{array}{c}{\displaystyle \frac{\partial J}{\partial a_{1,i}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial a_{j,i}^{[l]}}}\\ ⋮\\ {\displaystyle \frac{\partial J}{\partial a_{n^{[l]},i}^{[l]}}}\end{array}\right],$$

which compresses into 压缩成

$$\begin{array}{}\text{(18)}& \frac{\partial J}{\partial {\mathbf{z}}_{:,i}^{[l]}}=\frac{\partial {\mathbf{a}}_{:,i}^{[l]}}{\partial {\mathbf{z}}_{:,i}^{[l]}}\frac{\partial J}{\partial {\mathbf{a}}_{:,i}^{[l]}},\end{array}$$

where $\partial J/\partial {\mathbf{a}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$ and $\partial {\mathbf{a}}_{:,i}^{[l]}/\partial {\mathbf{z}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times n^{[l]}}$.
在 $\partial J/\partial {\mathbf{a}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}}$ 和 $\partial {\mathbf{a}}_{:,i}^{[l]}/\partial {\mathbf{z}}_{:,i}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times n^{[l]}}$ 处。

We have already encountered
我们已有所接触

$$\begin{array}{}\text{(19)}& \frac{\partial J}{\partial {\mathbf{Z}}^{[l]}}=\left[\begin{array}{ccccc}{\displaystyle \frac{\partial J}{\partial {\mathbf{z}}_{:,1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial {\mathbf{z}}_{:,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial {\mathbf{z}}_{:,m}^{[l]}}}\end{array}\right],\end{array}$$

and for the sake of completeness, we also clarify that
为了完整性起见，我们也明确指出

$$\begin{array}{}\text{(20)}& \frac{\partial J}{\partial {\mathbf{A}}^{[l]}}=\left[\begin{array}{ccccc}{\displaystyle \frac{\partial J}{\partial {\mathbf{a}}_{:,1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial {\mathbf{a}}_{:,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial {\mathbf{a}}_{:,m}^{[l]}}}\end{array}\right],\end{array}$$

where $\partial J/\partial {\mathbf{A}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times m}$. 源文本：where $\partial J/\partial {\mathbf{A}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times m}$ . 翻译文本：在$\partial J/\partial {\mathbf{A}}^{[l]}\in {\mathbb{R}}^{n^{[l]}\times m}$处。

On purpose, we have omitted the details of $g_j^{[l]}(z_{1,i}^{[l]},\dots ,z_{j,i}^{[l]},\dots ,z_{n^{[l]},i}^{[l]})$; consequently, we cannot derive an analytic expression for $\partial a_{j,i}^{[l]}/\partial z_{j,i}^{[l]}$, which we depend on in $\text{(17)}$. However, since the second post of this series will be dedicated to activation functions, we will instead derive $\partial a_{j,i}^{[l]}/\partial z_{j,i}^{[l]}$ there.
出于特定目的，我们省略了 $g_j^{[l]}(z_{1,i}^{[l]},\dots ,z_{j,i}^{[l]},\dots ,z_{n^{[l]},i}^{[l]})$ 的细节；因此，我们无法推导出依赖于 $\text{(17)}$ 的 $\partial a_{j,i}^{[l]}/\partial z_{j,i}^{[l]}$ 的解析表达式。然而，由于本系列第二篇文章将专门讨论激活函数，我们将在那里推导出 $\partial a_{j,i}^{[l]}/\partial z_{j,i}^{[l]}$ 。

Furthermore, according to $\text{(17)}$, we see that $\partial J/\partial z_{j,i}^{[l]}$ also depends on $\partial J/\partial a_{j,i}^{[l]}$. Now, it might come as a surprise, but $\partial J/\partial a_{j,i}^{[l]}$ has already been computed when we reach the $l$th layer during backward propagation. How did that happen, you may ask. The answer is that every layer paves the way for the previous layer by also computing $\partial J/\partial a_{k,i}^{[l-1]}$, which we shall do now:
此外，根据 $\text{(17)}$ ，我们发现 $\partial J/\partial z_{j,i}^{[l]}$ 也依赖于 $\partial J/\partial a_{j,i}^{[l]}$ 。现在，这可能令人惊讶，但在反向传播过程中到达 $l$ 层时， $\partial J/\partial a_{j,i}^{[l]}$ 已经计算完毕。你可能会问，这是如何发生的？答案是，每一层都为前一层铺平道路，同时也计算 $\partial J/\partial a_{k,i}^{[l-1]}$ ，我们现在就来进行这一步：

$$\begin{array}{}\text{(21)}& \frac{\partial J}{\partial a_{k,i}^{[l-1]}}=\sum _j\frac{\partial J}{\partial z_{j,i}^{[l]}}\frac{\partial z_{j,i}^{[l]}}{\partial a_{k,i}^{[l-1]}}=\sum _j\frac{\partial J}{\partial z_{j,i}^{[l]}}{w}_{j,k}^{[l]}.\end{array}$$

As usual, our next step is vectorization:
如常，我们的下一步是向量化：

$$\begin{array}{rl}& \left[\begin{array}{ccccc}{\displaystyle \frac{\partial J}{\partial a_{1,1}^{[l-1]}}}& \dots & {\displaystyle \frac{\partial J}{\partial a_{1,i}^{[l-1]}}}& \dots & {\displaystyle \frac{\partial J}{\partial a_{1,m}^{[l-1]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial J}{\partial a_{k,1}^{[l-1]}}}& \dots & {\displaystyle \frac{\partial J}{\partial a_{k,i}^{[l-1]}}}& \dots & {\displaystyle \frac{\partial J}{\partial a_{k,m}^{[l-1]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial J}{\partial a_{n^{[l-1]},1}^{[l-1]}}}& \dots & {\displaystyle \frac{\partial J}{\partial a_{n^{[l-1]},i}^{[l-1]}}}& \dots & {\displaystyle \frac{\partial J}{\partial a_{n^{[l-1]},m}^{[l-1]}}}\end{array}\right]\\ & =\left[\begin{array}{ccccc}{w}_{1,1}^{[l]}& \dots & w_{j,1}^{[l]}& \dots & w_{n^{[l]},1}^{[l]}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ w_{1,k}^{[l]}& \dots & w_{j,k}^{[l]}& \dots & w_{n^{[l]},k}^{[l]}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ w_{1,n^{[l-1]}}^{[l]}& \dots & w_{j,n^{[l-1]}}^{[l]}& \dots & w_{n^{[l]},n^{[l-1]}}^{[l]}\end{array}\right]\\ & \phantom{=}\cdot \left[\begin{array}{ccccc}{\displaystyle \frac{\partial J}{\partial z_{1,1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{1,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{1,m}^{[l]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{j,1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{j,i}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{j,m}^{[l]}}}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},1}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},i}^{[l]}}}& \dots & {\displaystyle \frac{\partial J}{\partial z_{n^{[l]},m}^{[l]}}}\end{array}\right],\end{array}$$

which we can write as
我们可以将其写作

$$\begin{array}{}\text{(22)}& \frac{\partial J}{\partial {\mathbf{A}}^{[l-1]}}={{\mathbf{W}}^{[l]}}^{⊺}\frac{\partial J}{\partial {\mathbf{Z}}^{[l]}},\end{array}$$

where $\partial J/\partial {\mathbf{A}}^{[l-1]}\in {\mathbb{R}}^{n^{[l-1]}\times m}$. 源文本：where $\partial J/\partial {\mathbf{A}}^{[l-1]}\in {\mathbb{R}}^{n^{[l-1]}\times m}$ . 翻译文本：在$\partial J/\partial {\mathbf{A}}^{[l-1]}\in {\mathbb{R}}^{n^{[l-1]}\times m}$处。

Summary 摘要

Forward propagation is seeded with ${\mathbf{A}}^{[0]}=\mathbf{X}$ and evaluates a set of recurrence relations to compute the predictions ${\mathbf{A}}^{[L]}=\hat{\mathbf{Y}}$. We also compute the cost $J=f(\hat{\mathbf{Y}},\mathbf{Y})=f({\mathbf{A}}^{[L]},\mathbf{Y})$.
正向传播以 ${\mathbf{A}}^{[0]}=\mathbf{X}$ 为起点，通过一系列递推关系计算出预测值 ${\mathbf{A}}^{[L]}=\hat{\mathbf{Y}}$ 。同时，我们还计算了成本 $J=f(\hat{\mathbf{Y}},\mathbf{Y})=f({\mathbf{A}}^{[L]},\mathbf{Y})$ 。

Backward propagation, on the other hand, is seeded with $\partial J/\partial {\mathbf{A}}^{[L]}=\partial J/\partial \hat{\mathbf{Y}}$ and evaluates a different set of recurrence relations to compute $\partial J/\partial {\mathbf{W}}^{[l]}$ and $\partial J/\partial {\mathbf{b}}^{[l]}$. If not stopped prematurely, it eventually computes $\partial J/\partial {\mathbf{A}}^{[0]}=\partial J/\partial \mathbf{X}$, a partial derivative we usually ignore.
另一方面，反向传播以 $\partial J/\partial {\mathbf{A}}^{[L]}=\partial J/\partial \hat{\mathbf{Y}}$ 为起点，通过计算一组不同的递推关系来求得 $\partial J/\partial {\mathbf{W}}^{[l]}$ 和 $\partial J/\partial {\mathbf{b}}^{[l]}$ 。若不提前终止，它最终会计算出 $\partial J/\partial {\mathbf{A}}^{[0]}=\partial J/\partial \mathbf{X}$ ，这是一个我们通常忽略的偏导数。

Moreover, let us visualize the inputs we use and the outputs we produce during the forward and backward propagations:
此外，让我们设想一下在正向和反向传播过程中使用的输入和产生的输出：

Figure 3: An overview of inputs and outputs.
图 3：输入与输出的概览。

Now, you might have noticed that we have yet to derive an analytic expression for the backpropagation seed $\partial J/\partial {\mathbf{A}}^{[L]}=\partial J/\partial \hat{\mathbf{Y}}$. To recap, we have deferred the derivations that concern activation functions to the second post of this series. Similarly, since the third post will be dedicated to cost functions, we will instead address the derivation of the backpropagation seed there.
现在，您可能已经注意到，我们尚未推导出反向传播种子 $\partial J/\partial {\mathbf{A}}^{[L]}=\partial J/\partial \hat{\mathbf{Y}}$ 的解析表达式。回顾一下，我们将涉及激活函数的推导推迟到本系列的第二篇文章中。同样，由于第三篇文章将专门讨论成本函数，我们将在那里处理反向传播种子的推导。

Last but not least: congratulations! You have made it to the end (of the first post). 🏅
最后但同样重要的是：恭喜！你已经完成了第一篇文章的阅读。🏅