Normal Approximation of Random Gaussian Neural Networks

1. Introduction

This work is part of the literature studying random neural networks (NNs for short), that is, NNs whose biases and weights are random variables. In the context of modern deep learning, the interest in these types of networks is twofold; on the one hand, they naturally constitute a prior in a Bayesian approach, and on the other hand they may represent the initialization of gradient flows in empirical risk minimization. See Roberts et al. (2022) for a general reference on the subject.

Within the boundaries of this topic, many contributions in the literature have been handling the asymptotic Gaussianity of random NNs, as the number of neurons in the hidden layers tends to infinity. A seminal paper is Neal (1996), where the output of a shallow (i.e., having a single hidden layer) random NN, viewed as a stochastic process on the sphere, is shown to converge to a Gaussian process, as the number of neurons in the hidden layer grows large. From that point onward, many sophisticated results have been published for deep (i.e., having more than one hidden layer) random NNs at the large width limit. Early contributions in this direction were from Lee et al. (2018), Matthews et al. (2018), Yaida (2019), and Yang (2019). These achievements were extended in Hanin (2023), where it was proved that the output of a deep random NN, with Gaussian biases and weights, viewed as a random element on the space of continuous functions on a compact set, converges to a Gaussian process as the number of neurons in all the hidden layers tends to infinity. Large and moderate deviations of the output of a deep random NN, with Gaussian biases and weights, were studied in Macci et al. (2024).

Recently, the problem of the quantitative Gaussian approximation of the output of a random NN has received a lot of attention. For instance, Eldan et al. (2021), exploiting Wasserstein distances, provided quantitative versions of the results in Neal (1996) when the activation function was polynomial, ReLU, and hyperbolic tangent. We emphasize that the shallow random NN model considered in Neal (1996) and Eldan et al. (2021) has weights on the outer layer given by Rademacher random variables and weights on the inner layer distributed according to the Gaussian law (see Remark 3.4). Slightly different models were investigated in Klukowski (2022) and Cammarota et al. (2024). Indeed, as the number of neurons on the hidden layer grew large, Klukowski (2022) provided a quantitative functional central limit theorem for a shallow random NN model with input variables still on the sphere and weights on the outer layer still given by Rademacher random variables but weights on the inner layer uniformly distributed on the sphere, whereas as the number of neurons on the hidden layer grew large, Cammarota et al. (2024) gave quantitative functional central limit theorems for a shallow random NN model with weights on outer and inner layers distributed according to the Gaussian law and again input variables on the sphere. A quantitative functional central limit theorem for deep random NNs, with input variables on the sphere and Lipschitz continuous activation functions, was proved in Balasubramanian et al. (2024); this paper shares with our work the idea to apply Stein’s method for the Gaussian approximation in the context of deep NNs, albeit in a different mathematical setting.

A significant achievement is provided in Basteri and Trevisan (2024), where, for the first time in the literature, a quantitative proof of the Gaussian behavior of the output of a deep random Gaussian NN (i.e., a random NN whose biases and weights are Gaussian distributed) with a Lipschitz continuous activation function was given; in Basteri and Trevisan (2024), the distance from Gaussianity was measured by means of the 2-Wasserstein metric, which comes from the Monge-Kantorovich problem with quadratic cost. As far as shallow random Gaussian NNs with univariate output is concerned, we mention Bordino et al. (2024), which provided quantitative bounds on the Kolmogorov, the total variation and the 1-Wasserstein distances between the output and a Gaussian random variable, when the activation function was sufficiently smooth and had a sub-polynomial growth. A special mention is deserved for the independently written paper Favaro et al. (2023), where Stein’s method was used to obtain tight probabilistic bounds for various distances between the output (and its derivatives) of a deep random Gaussian NN and a Gaussian random vector. We refer the reader to Remarks 4.2, 6.2, and 6.4 for comparisons between our results and the corresponding achievements in Favaro et al. (2023).

The main contribution of our paper concerns the Gaussian approximation of the output of deep random Gaussian NNs in the convex and 1-Wasserstein distances under mild assumptions on the activation function (which, differently from Basteri and Trevisan (2024), can be non-Lipschitz). A specialization of these results clearly provides approximations for shallow random Gaussian NN with single input and real valued (i.e., univariate) output. However, in this specific case we furnish direct proofs, which (for various technical reasons) give the same rates under more general assumptions on the activation function.

For shallow random Gaussian NNs with univariate output, combining the Stein method for the Gaussian approximation with the integration by parts formula for the Gaussian law, we provide explicit bounds for the Kolmogorov, the total variation and the 1-Wasserstein distances between the output and a Gaussian random variable, under a minimal assumption on the activation function (see Theorem 4.1). Remarkably, we obtain the same rate of convergence as in Bordino et al. (2024) as the number of neurons in the hidden layer grows large, our constants being presumably better than the ones in Bordino et al. (2024) (see Table 1). For deep random Gaussian NNs, the novelty of our results is that we measure the error in the Gaussian approximation of the output in terms of the convex distance (see Theorem 6.1) and of the 1-Wasserstein distance (see Theorem 6.3) for a class of activation functions that strictly includes the family of Lipschitz continuous functions if either the biases are non null or the activation function vanishes at 0 (see Proposition 5.2). Remarkably, for both the convex and the 1-Wasserstein distances the rate of convergence that we obtain is of the same order as the one in Basteri and Trevisan (2024) because the number of neurons in all the hidden layers tend to infinity. The proofs of Theorems 6.1 and 6.3 are based on the Stein method for the multivariate Gaussian approximation and the integration by parts formula for the multivariate Gaussian law. The presence of more than one hidden layer complicates the derivations of the bounds, which rely on a key estimate for the L²-distance between the so-called collective observables and their limiting values (see Theorem 5.1). We emphasize that, when considering the convex distance, an expedient tool is provided by a smoothing lemma that we borrow from Schulte and Yukich (2019).

Table 1. Values of the Constants Given in Theorems 3.1 and 4.1

It is well known that localizing the output of a random NN, that is, having a control over the probability that the output lies in a region of the space (belonging to a large class of measurable sets), is of valuable interest for practitioners. From a theorist’s point of view, the output distribution of a NN is often analytically untractable, and computing the probability that the output belongs to some measurable set results in performing a “heroic” mathematical integration (see, e.g., Roberts et al. 2022, p. 49). Our Theorems 4.1(ii) and 6.1 offer some insights into the localization problem in a simple and efficient way for both the univariate output of a shallow random Gaussian NN and the output of a deep random Gaussian NN, respectively. We refer the reader to Section 7 for some numerical illustrations of this issue.

The paper is organized as follows. In Section 2, we introduce our toolkit such as the integral probability metrics considered in the paper and some preliminaries on the Stein method. In Section 3, first we introduce all of the NNs considered in this work and then give a brief overview of the main results in Basteri and Trevisan (2024) and Bordino et al. (2024), comparing them with our achievements. In Section 4, we give upper bounds for the Kolmogorov, the total variation and the 1-Wasserstein distances between the (univariate) output of a shallow random Gaussian NN and a Gaussian random variable. In Section 5, we prove the aforementioned key estimate on the L²-distance between the collective observables and their limiting values. In Section 6, we furnish explicit upper bounds on the convex and the 1-Wasserstein distances between the output of a deep random Gaussian NN and a Gaussian random vector. Finally, in Section 7, we present some numerical illustrations concerning the above-mentioned issue of the output localization.

2. Preliminaries

In the present section, we introduce some notation, and we recall some results that will be of use throughout the paper.

2.1. Distances Between Probability Measures

In this paper, we consider various distances between probability measures on ${\mathbb{R}}^d,\hspace{0.17em}d\in \mathbb{N}≔\{1,2,\dots \}$: the total variation distance, the convex distance, the Komogorov distance, and the p-Wasserstein distances. Hereon, the symbol $\Vert ·{\Vert }_d$ denotes the Euclidean norm on ${\mathbb{R}}^d$.

Definition 2.1.

The total variation distance between the laws of two ${\mathbb{R}}^d$-valued random vectors $\mathbf{X}$ and $\mathbf{Y}$, written $d_{TV}(\mathbf{X},\mathbf{Y})$, is given by

$$d_{TV}(\mathbf{X},\mathbf{Y})≔\underset{B\in \mathcal{B}({\mathbb{R}}^d)}{\sup }|\mathbb{P}(\mathbf{X}\in B)-\mathbb{P}(\mathbf{Y}\in B)|,$$

where $\mathcal{B}({\mathbb{R}}^d)$ denotes the Borel σ-field on ${\mathbb{R}}^d$.

Definition 2.2.

The convex distance between the laws of two ${\mathbb{R}}^d$-valued random vectors $\mathbf{X}$ and $\mathbf{Y}$, written $d_c(\mathbf{X},\mathbf{Y})$, is given by

$$d_c(\mathbf{X},\mathbf{Y})≔\underset{C\in {\mathcal{C}}_d}{\sup }|\mathbb{P}(\mathbf{X}\in C)-\mathbb{P}(\mathbf{Y}\in C)|,$$

where ${\mathcal{C}}_d$ denotes the collection of all Borel convex sets in ${\mathbb{R}}^d$.

Definition 2.3.

The Kolmogorov distance between the laws of two ${\mathbb{R}}^d$-valued random vectors $\mathbf{X}=(X_1,\dots ,X_d)$ and $\mathbf{Y}=(Y_1,\dots ,Y_d)$, written $d_K(\mathbf{X},\mathbf{Y})$, is given by

$$d_K(\mathbf{X},\mathbf{Y})≔\underset{\mathbf{y}=(y_1,\dots ,y_d)\in {\mathbb{R}}^d}{\sup }|\mathbb{P}(X_1\le y_1,\dots ,X_d\le y_d)-\mathbb{P}(Y_1\le y_1,\dots ,Y_d\le y_d)|.$$

Definition 2.4.

For $p\in [1,+\infty )$, the p-Wasserstein distance between the laws of two ${\mathbb{R}}^d$-valued random vectors $\mathbf{X}$ and $\mathbf{Y}$, written $d_{W_p}(\mathbf{X},\mathbf{Y})$, is given by

$$d_{W_p}(\mathbf{X},\mathbf{Y})≔\underset{(\mathbf{U},\mathbf{V})\in C_{(\mathbf{X},\mathbf{Y})}}{\inf }\mathbb{E}{[\Vert \mathbf{U}-\mathbf{V}{\Vert }_d^p]}^{1/p},$$

where $C_{(\mathbf{X},\mathbf{Y})}$ is the family of all the couplings of $\mathbf{X}$ and $\mathbf{Y}$, that is, the family of all random vectors $(\mathbf{U},\mathbf{V})$ such that $\mathbf{U}$ is distributed as $\mathbf{X}$ and $\mathbf{V}$ is distributed as $\mathbf{Y}$.

Clearly, by Jensen’s inequality, $d_{W_1}\le d_{W_2}$, and it follows directly by the definitions that $d_K\le d_c\le d_{TV}$. Furthermore, for all $s=TV,c,K,W_p$, if $d_s({\mathbf{Y}}_n,\mathbf{Y})\to 0$, as $n\to +\infty$, where ${\mathbf{Y}}_n,\hspace{0.17em}n\in \mathbb{N}$, and $\mathbf{Y}$ are random vectors with values in ${\mathbb{R}}^d$, then ${\mathbf{Y}}_n$ converges in law to $\mathbf{Y}$, as $n\to +\infty$ (see, e.g., Villani 2009 and Nourdin and Peccati 2012).

In view of the Kantorovich-Rubinstein duality (see theorem 5.10 and equation (5.11) in Villani 2009), the 1-Wasserstein distance between the laws of two ${\mathbb{R}}^d$-valued random vectors $\mathbf{X}$ and $\mathbf{Y}$ such that $\max \{\mathbb{E}\Vert \mathbf{X}{\Vert }_d,\mathbb{E}\Vert \mathbf{Y}{\Vert }_d\}<\infty$ satisfies the relation

where ${\mathcal{L}}_d(1)$ is the collection of Lipschitz continuous functions with Lipschitz constant less than or equal to 1.

Because d_c is defined by testing against indicator functions of Borel convex sets rather than arbitrary Borel sets, the convex distance can be expected to be estimated more easily than the total variation distance; moreover, the convex distance looks more flexible than the Kolmogorov distance; for example, it enjoys a number of invariance properties not satisfied by d_K (see Benktus 2003).

As for the relation between the convex distance and the optimal transport metric $d_{W_1}$, it turns out that the convex distance to a fixed centered Gaussian law is bounded from above by a multiple of the square root of the 1-Wasserstein distance. More precisely, one has the following Proposition 2.5, which is proved in Nourdin et al. (2022).

Here and henceforth, we denote by ${\mathbf{N}}_{\mathrm{\Sigma }}=({(N_{\mathrm{\Sigma }})}_1,\dots ,{(N_{\mathrm{\Sigma }})}_d),\hspace{0.17em}d\in \mathbb{N}$, a centered Gaussian vector with invertible covariance matrix $\mathrm{\Sigma }={({\mathrm{\Sigma }}_{ij})}_{1\le i,j\le d}$.

Proposition 2.5.

For any d-dimensional random vector $\mathbf{Y}$, we have

$$d_c(\mathbf{Y},{\mathbf{N}}_{\mathrm{\Sigma }})\le 2\sqrt{2}\mathrm{\Gamma }{(\mathrm{\Sigma })}^{1/2}{d}_{W_1}{(\mathbf{Y},{\mathbf{N}}_{\mathrm{\Sigma }})}^{1/2},$$

where $\mathrm{\Gamma }(\mathrm{\Sigma })$ is the constant defined by

$$\mathrm{\Gamma }(\mathrm{\Sigma })≔\underset{Q,ϵ>0}{\sup }\frac{\mathbb{P}({\mathbf{N}}_{\mathrm{\Sigma }}\in Q^{ϵ})-\mathbb{P}({\mathbf{N}}_{\mathrm{\Sigma }}\in Q)}{ϵ},$$(2)

where Q ranges over all the Borel measurable convex subsets of ${\mathbb{R}}^d$, and $Q^{ϵ}$ denotes the set of all elements of ${\mathbb{R}}^d$ whose Euclidean distance from Q does not exceed ϵ.

We note that $\mathrm{\Gamma }(\mathrm{\Sigma })$ is an isoperimetric constant that satisfies the following relation (see Nazarov 2004):

2.2. The One-Dimensional Stein Equation

Throughout this paper, we denote by $\mathcal{N}(\mu ,\eta )$ the one-dimensional Gaussian law with mean $\mu \in \mathbb{R}$ and variance $\eta >0$ and let $Z\sim \mathcal{N}(0,1)$.

The celebrated Stein equation for the one-dimensional Normal approximation Stein (1972) is given by

where $g:\mathbb{R}\to \mathbb{R}$ is a measurable function such that $\mathbb{E}|g(Z)|<\infty$, and $f_g:\mathbb{R}\to \mathbb{R}$ is unknown. The following lemma holds; see, for example, proposition 3.2.2, theorem 3.3.1, theorem 3.4.2, and proposition 3.5.1 in Nourdin and Peccati (2012). See Chen et al. (2011) for an introduction on the Stein method.

Hereon, for a Lipschitz continuous function $g:{\mathbb{R}}^d\to \mathbb{R}$, we denote by $\text{Lip}(g)$ the Lipschitz constant of g, and for a function $f:{\mathbb{R}}^d\to \mathbb{R}$ we denote by $\Vert f{\Vert }_{\infty }$ the supremum norm of f.

Lemma 2.6.

The following claims hold:

1. For any $y\in \mathbb{R}$, the Stein Equation (4) with $g(w)≔{\mathbf{1}}_{(-\infty ,y]}(w)$ has a unique solution f_g and $\Vert f_g\prime {\Vert }_{\infty }\le 1$.

2. Let $g:\mathbb{R}\to [0,1]$ be a measurable function. Then, there exists a unique solution f_g of the Stein Equation (4) and $\Vert f{\prime }_g{\Vert }_{\infty }\le 2.$

3. Let $g:\mathbb{R}\to \mathbb{R}$ be a Lipschitz continuous function. Then, there exists a unique solution f_g of the Stein Equation (4) and $\Vert f{\prime }_g{\Vert }_{\infty }\le \text{Lip}(g)\sqrt{2/\pi }.$

2.3. The Multidimensional Stein Equation

Throughout this paper, given a sufficiently smooth function $f:{\mathbb{R}}^d\to \mathbb{R}$, we define

Let ${\mathcal{M}}_{d\times d}(\mathbb{R}),\hspace{0.17em}d\in \mathbb{N}$, be the set of d × d real matrices. For a function $f\in C^2({\mathbb{R}}^d)$, we denote by $\text{Hess}\hspace{0.17em}f(\mathbf{y})\in {\mathcal{M}}_{d\times d}(\mathbb{R})$ the Hessian matrix of f at $\mathbf{y}\in {\mathbb{R}}^d$ and by $\Vert ·{\Vert }_{op}$ the operator norm on ${\mathcal{M}}_{d\times d}(\mathbb{R})$, that is, for any $\mathrm{\Gamma }\in {\mathcal{M}}_{d\times d}(\mathbb{R}),\Vert \mathrm{\Gamma }{\Vert }_{\text{op}}≔{\sup }_{\mathbf{y}:\Vert \mathbf{y}{\Vert }_d=1}\Vert \mathrm{\Gamma }\mathbf{y}{\Vert }_d$. We consider the Hilbert-Schmidt inner product and the Hilbert-Schmidt norm on ${\mathcal{M}}_{d\times d}(\mathbb{R})$, which are defined, respectively, by

for every pair of matrices $\mathrm{\Gamma }={({\mathrm{\Gamma }}_{ij})}_{1\le i,j\le j}$ and $\mathrm{\Psi }={({\mathrm{\Psi }}_{ij})}_{1\le i,j\le d}$, where the symbols $\text{Tr}(\mathrm{\Gamma })$ and ${\mathrm{\Gamma }}^{\top }$ denote, respectively, the trace and the transpose of the matrix Γ.

The Stein equation for multivariate Normal approximation is defined as

where $g:{\mathbb{R}}^d\to \mathbb{R}$ is given, f_g is unknown, Σ is the (invertible) covariance matrix of the centered Gaussian vector ${\mathbf{N}}_{\mathrm{\Sigma }}$, and the symbol ${〈·,·〉}_d$ denotes the inner product in ${\mathbb{R}}^d$.

The following lemmas provide solutions to Stein’s Equation (5) under different assumptions on g.

Lemma 2.7.

Let $g\in {\mathcal{L}}_d(1)$. Then, the function

$$f_g(\mathbf{y})≔{\displaystyle {\int }_0^{\infty }\mathbb{E}}[g({\mathbf{N}}_{\mathrm{\Sigma }})-g({\text{e}}^{-t}\mathbf{y}+\sqrt{1-{\text{e}}^{-2t}}{\mathbf{N}}_{\mathrm{\Sigma }})]\hspace{0.17em}\text{d}t,\hspace{1em}\mathbf{y}\in {\mathbb{R}}^d$$

is such that $f_g\in C^2({\mathbb{R}}^d)$, f_g solves (5), f_g satisfies

$$\Vert {\partial }_i{f}_g{\Vert }_{\infty }\le 1,\hspace{1em}\mathit{\text{for}} \mathit{\text{any}} i=1,\dots ,d,$$(6)

and

$$\underset{\mathbf{y}\in {\mathbb{R}}^d}{\sup }\Vert \text{Hess} f_g(\mathbf{y}){\Vert }_{H.S.}\le \sqrt{d}\Vert {\mathrm{\Sigma }}^{-1}{\Vert }_{op}\Vert \mathrm{\Sigma }{\Vert }_{op}^{1/2}.$$

Lemma 2.8.

For $g:{\mathbb{R}}^d\to \mathbb{R}$ measurable and bounded, define the smoothed function

$$g_t(\mathbf{y})≔\mathbb{E}[g(\sqrt{t}{\mathbf{N}}_{\mathrm{\Sigma }}+\sqrt{1-t}\mathbf{y})],\hspace{1em}\mathbf{y}\in {\mathbb{R}}^d,$$(7)

where $t\in (0,1)$ is a smoothing parameter. Then:

1. For any $t\in (0,1)$, the function

   $$f_{t,g}(\mathbf{y})≔\frac{1}{2}{\displaystyle {\int }_t^1\frac{1}{1-s}}\mathbb{E}[g(\sqrt{s}{\mathbf{N}}_{\mathrm{\Sigma }}+\sqrt{1-s}\mathbf{y})-g({\mathbf{N}}_{\mathrm{\Sigma }})]\hspace{0.17em}\text{d}s,\hspace{1em}\mathbf{y}\in {\mathbb{R}}^d$$

   is such that $f_{t,g}\in C^2({\mathbb{R}}^d),\hspace{0.17em}{f}_{t,g}$ solves (5) with g_t in place of g, and

   $$\Vert {\partial }_i{f}_{t,g}{\Vert }_{\infty }\le \Vert g{\Vert }_{\infty }\sqrt{\frac{1-t}{t}}{\displaystyle \sum _{j=1}^d{({\mathrm{\Sigma }}^{-1/2})}_{ji}}\sqrt{{\mathrm{\Sigma }}_{jj}},\hspace{1em}\mathrm{\text{for}} \mathrm{\text{any}} i=1,\dots ,d.$$(8)

2. For any d-dimensional random vector $\mathbf{Y}$, it holds

   $$\underset{g\in {\mathcal{I}}_d}{\sup }\mathbb{E}\Vert \text{Hess}(f_{t,g}(\mathbf{Y})){\Vert }_{H.S.}^2\le \Vert {\mathrm{\Sigma }}^{-1}{\Vert }_{op}^2(d^2{(\log \hspace{0.17em}t)}^2{d}_c(\mathbf{Y},{\mathbf{N}}_{\mathrm{\Sigma }})+530d^{17/6}),\hspace{1em}\mathit{\text{for}} \mathit{\text{any}} t\in (0,1).$$

See proposition 4.3.2 in Nourdin and Peccati (2012) for Lemma 2.7; in particular, in Nourdin and Peccati (2012) it is noticed that

which, combined with the fact that $g\in {\mathcal{L}}_d(1)$, gives the bound (6); see Schulte and Yukich (2019) p. 12, and proposition 2.3 for lemma 2.8 and lemma 3.6(ii) in Torrisi (2023) for the bound (8).

2.4. The Smoothing Lemma and the Integration by Parts Formula for Gaussian Random Vectors

We state a remarkable smoothing lemma for the convex distance proved in Schulte and Yukich (2019); see lemma 2.2 therein. It plays a crucial role in the proof of the Normal approximation of the output of a deep random Gaussian NN in the metric d_c; see Theorem 6.1.

Lemma 2.9.

Let $\mathbf{Y}$ be a d-dimensional random vector. Then, for any $t\in (0,1)$,

$$d_c(\mathbf{Y},{\mathbf{N}}_{\mathrm{\Sigma }})\le \frac{4}{3}\underset{C\in {\mathcal{C}}_d}{\sup }|\mathbb{P}(\sqrt{t}{\mathbf{N}}_{\mathrm{\Sigma }}+\sqrt{1-t}\mathbf{Y}\in C)-\mathbb{P}(\sqrt{t}{\mathbf{N}}_{\mathrm{\Sigma }}+\sqrt{1-t}{\mathbf{N}}_{\mathrm{\Sigma }}\in C)|+\frac{20d}{\sqrt{2}}\frac{\sqrt{t}}{1-t},$$

where ${\mathcal{C}}_d$ is given in Definition 2.2.

We recall the Gaussian integration by parts formula (we refer the reader to exercise 3.1.4 in Nourdin and Peccati (2012) for the Part (i) of Lemma 2.10 and to exercise 3.1.5 in Nourdin and Peccati (2012) for the Part (ii) of Lemma 2.10).

Lemma 2.10.

The following claims hold:

1. $N\sim \mathcal{N}(\mu ,\eta ),\hspace{0.17em}\eta >0$, if and only if, for any differentiable function $g:\mathbb{R}\to \mathbb{R}$ such that $\mathbb{E}|g\prime (N)|<\infty$, we have $\mathbb{E}(N-\mu )g(N)=\eta \mathbb{E}g\prime (N)$.

2. Let $g\in C^1({\mathbb{R}}^d)$ with bounded first partial derivatives. Then,

   $$\mathbb{E}[{(N_{\mathrm{\Sigma }})}_{i}g({\mathbf{N}}_{\mathrm{\Sigma }})]={\displaystyle \sum _{j=1}^d{\mathrm{\Sigma }}_{ij}}\mathbb{E}[{\partial }_{j}g({\mathbf{N}}_{\mathrm{\Sigma }})],\hspace{1em}\mathit{\text{for}} \mathit{\text{any}} i=1,\dots ,d.$$

This relation holds true even if Σ is not positive definite.

3. Random Neural Networks

We let $L\in \mathbb{N}$, we take L + 2 positive integers $n_0,\dots ,n_{L+1}\in \mathbb{N}$, and we fix a function $\sigma :\mathbb{R}\to \mathbb{R}$. A fully connected NN of depth L with input dimension n₀, output dimension $n_{L+1}$, hidden layer widths $n_1,\dots ,n_L$, and nonlinearity σ is a mapping

that is defined by a recursive relation of the formwhere the parameters $b_i^{(\mathcal{\ell })}\in \mathbb{R}$ and $W_{ij}^{(\mathcal{\ell })}\in \mathbb{R}$ are called network biases and weights, respectively. The quantities L and $n_0,\dots ,n_{L+1}$ constitute the so-called network architecture. The function σ is usually called activation function. NNs of this kind will be denoted bywhere ${\mathbf{n}}_L≔(n_1,\dots ,n_L)$, $\mathbf{b}≔(b_i^{(\mathcal{\ell })})$ and $\mathbf{W}≔(W_{ij}^{(\mathcal{\ell })})$.

We say that the neural network $\text{NN}(L,n_0,n_{L+1},{\mathbf{n}}_L,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$ is a (fully connected and) deep random Gaussian neural network denoted by

if $\sigma :\mathbb{R}\to \mathbb{R}$ is measurable and $b_i^{(\mathcal{\ell })},W_{ij}^{(\mathcal{\ell })},\hspace{0.17em}i=1,\dots ,n_{\mathcal{\ell }},\hspace{0.17em}j=1,\dots ,n_{\mathcal{\ell }-1},\hspace{0.17em}\mathcal{\ell }=1,\dots ,L+1$, are independent random variables withfor constants $C_b\ge 0$ and $C_W>0$.

NNs of depth L = 1 are called shallow NNs. We will denote shallow NNs (respectively, shallow random Gaussian NNs) by $\text{NN}(1,n_0,n_2,n_1,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$ (respectively, by $\text{GNN}(1,n_0,n_2,n_1,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$).

Throughout this paper, we will also consider NNs with univariate output, that is, NNs with $n_{L+1}=1$.

Consider a deep random Gaussian neural network $\text{GNN}(L,n_0,n_{L+1},{\mathbf{n}}_L,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$. It turns out that the random variables $z_i^{(1)}=z_i^{(1)}(\mathbf{x}),\hspace{0.17em}i=1,\dots ,n_1$, are independent and identically distributed with

For $\mathcal{\ell }=1,\dots ,L$, let ${\mathcal{F}}_{\mathcal{\ell }}$ be the σ-field generated by the random variables

By construction, for any fixed $\mathcal{\ell }\in \{2,\dots ,L+1\}$, given ${\mathcal{F}}_{\mathcal{\ell }-1}$, the random variables $z_i^{(\mathcal{\ell })}=z_i^{(\mathcal{\ell })}(\mathbf{x}),\hspace{0.17em}i=1,\dots ,n_{\mathcal{\ell }}$, are independent and Gaussian (as linear combination of independent Gaussian random variables). A straightforward computation yields

and

Setting ${\mathbf{n}}_{\mathcal{\ell }}=(n_1,\dots ,n_{\mathcal{\ell }}),\hspace{0.17em}\mathcal{\ell }=1,\dots ,L$, we define the quantities

andwhere

The random variable ${\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}$ is often referred to as collective observable at layer $\mathcal{\ell }$; see, for example, Roberts et al. (2022) and Hanin (2023).

3.1. Some Related Literature

Consider the output ${\mathbf{z}}^{(L+1)}≔(z_1^{(L+1)},\dots ,z_{n_{L+1}}^{(L+1)})$ of a deep random Gaussian neural network $\text{GNN}(L,n_0,n_{L+1},{\mathbf{n}}_L,\sigma ;\mathbf{x};\mathbf{b},\mathbf{W})$, and let

be a centered $n_{L+1}$-dimensional Gaussian random vector with covariance matrixwhere ${\text{Id}}_{n_{L+1}}$ is the identity matrix of ${\mathcal{M}}_{n_{L+1}\times n_{L+1}}(\mathbb{R})$. Here and henceforth, we suppose that the quantities $C_b,\Vert \mathbf{x}\Vert ,\sigma (0)$, where $\mathbf{x}$ is the input of the network, are not all simultaneously equal to zero. This causes no loss of generality, because otherwise ${\mathbf{z}}^{(L+1)}=\mathbf{z}=0$ almost surely.

It follows from theorem 1.2 in Hanin (2023) (which indeed, more generally, establishes a functional weak convergence) that, if σ is continuous and polynomially bounded, then,

The following result for shallow random Gaussian NNs was proved in Bordino et al. (2024); see theorem 3.2 therein.

Theorem 3.1.

Let $\text{GNN}(1,n_0,1,n_1,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$ be a shallow random Gaussian NN with univariate output. If

$$\sigma \in C^2(\mathbb{R}) \mathit{\text{and}} \max \{|\sigma (x)|,|\sigma \prime (x)|,|\sigma ″(x)|\}\le r_1+r_2|x{|}^{\gamma },\hspace{0.17em}x\in \mathbb{R}$$(14)

for some $r_1,r_2,\gamma \ge 0$, then

$$d_s(z^{(2)},z)\le c_s\Vert r_1+r_2|{\mathfrak{s}}_{0}Z{|}^{\gamma }{\Vert }_{L^4}^2\sqrt{{\mathfrak{s}}_0^2+{\mathfrak{s}}_0^4(2+\sqrt{3\left(1+2{\mathfrak{s}}_0^2+3{\mathfrak{s}}_0^4\right)})}\times \frac{1}{\sqrt{n_1}},$$

where $s=TV,K,W_1$ and

$$c_{TV}≔\frac{4}{{\mathfrak{s}}_1^2},\hspace{1em}{c}_K;=\frac{2}{{\mathfrak{s}}_1^2},\hspace{1em}{c}_{W_1}≔\frac{1}{{\mathfrak{s}}_1}\sqrt{8/\pi }.$$

In Section 4, we will give bounds on the quantities $d_s(z^{(2)},z),\hspace{0.17em}s=TV,K,W_1$, of order $1/\sqrt{n_1}$, as $n_1\to \infty$, under a minimal assumption on the activation function (note that Condition (14) excludes the important case of the ReLU function, that is, $\sigma (x)≔x\mathbf{1}\{x\ge 0\}$); see Theorem 4.1. In Section 6, we will give two general bounds on $d_c({\mathbf{z}}^{(L+1)},\mathbf{z})$ and $d_{W_1}({\mathbf{z}}^{(L+1)},\mathbf{z})$ for deep random Gaussian NNs; see Theorems 6.1 and 6.3. When specialized to shallow random Gaussian neural networks $\text{GNN}(1,n_0,n_2,n_1,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$, they provide computable bounds, respectively, on $d_c({\mathbf{z}}^{(2)},\mathbf{z})$ and $d_{W_1}({\mathbf{z}}^{(2)},\mathbf{z})$ of order $1/{\sqrt{n}}_1$, as $n_1\to \infty$, see theorem 3.3 in Bordino et al. (2024) for a related result.

The first result in the literature that quantifies the convergence in distribution (13) with $L\ge 2$ is given in Basteri and Trevisan (2024), where the following theorem has been proven.

Theorem 3.2.

Let $\text{GNN}(L,n_0,n_{L+1},{\mathbf{n}}_L,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$ be a deep random Gaussian NN, and suppose that the activation function σ is Lipschitz continuous. Then,

$$d_{W_2}({\mathbf{z}}^{(L+1)},\mathbf{z})\le \sqrt{n_{L+1}}{\displaystyle \sum _{i=1}^L\frac{C^{(i+1)}{[\text{Lip}(\sigma )\sqrt{C_W}]}^{L-i}}{\sqrt{n_i}}},$$(15)

where, for any $i=1,\dots ,L,\hspace{0.17em}{C}^{(i+1)}$ are explicitly known positive constants, depending upon σ, $\mathbf{x}$, C_b and C_W.

The next corollary is an immediate consequence of Theorem 3.2, the fact that $d_{W_1}\le d_{W_2}$, Proposition 2.5, and (3).

Corollary 3.3.

Let the assumptions and notation of Theorem 3.2 prevail. Then,

$$d_{W_1}({\mathbf{z}}^{(L+1)},\mathbf{z})\le \sqrt{n_{L+1}}{\displaystyle \sum _{i=1}^L\frac{C^{(i+1)}{[\text{Lip}(\sigma )\sqrt{C_W}]}^{L-i}}{\sqrt{n_i}}}$$(16)

and

$$d_c({\mathbf{z}}^{(L+1)},\mathbf{z})\le 2^{\frac{11}{8}}{\pi }^{-\frac{1}{8}}{n}_{L+1}^{\frac{3}{8}}{\left({\displaystyle \sum _{i=1}^L\frac{C^{(i+1)}{[\text{Lip}(\sigma )\sqrt{C_W}]}^{L-i}}{\sqrt{n_i}}}\right)}^{1/2}.$$(17)

Theorems 6.1 and 6.3 will provide, under alternate assumptions on σ (see by Proposition 5.2(ii)), explicit bounds, respectively, on $d_c({\mathbf{z}}^{(L+1)},\mathbf{z})$ and $d_{W_1}({\mathbf{z}}^{(L+1)},\mathbf{z})$ that are of the same order of the bound in (16), as $n_1,\dots ,n_L\to \infty$. In particular, the bound on the convex distance of Theorem 6.1 considerably improves the one in (17). We emphasize that to compare the results in Basteri and Trevisan (2024) with our achievements, we specialized those results to the case of a single input $\mathbf{x}\in {\mathbb{R}}^{n_0}$. In fact, the bound (15) was proven in Basteri and Trevisan (2024) for multiple inputs, and consequently the bound (17) can be stated for multiple inputs too.

Remark 3.4.

Although not strictly related to our results, the pioneering papers Eldan et al. (2021) and Neal (1996) deserve a special mention. In Neal (1996) the author considered a random shallow $NN(1,n_0,1,n_1,\sigma ,\mathbf{x},\mathbf{0},\mathbf{W})$ with univariate output, $b_i^{(1)}≔0$, for all $i=1,\dots ,n_1,\hspace{0.17em}{b}_1^{(2)}≔0$, $W_{ij}^{(1)},\hspace{0.17em}i=1,\dots ,n_1,\hspace{0.17em}j=1,\dots ,n_0$, independent with law $\mathcal{N}(0,1)$ and $W_{1j}^{(2)},\hspace{0.17em}j=1,\dots ,n_1$, independent, identically distributed with law $\mathbb{P}(W_{1j}^{(2)}=\pm 1/\sqrt{n_1})=1/2$ and independent of the random variables $W_{ij}^{(1)}$. It was proven in Neal (1996) that there exists a Gaussian process G on ${\mathbb{S}}^{n_0-1}$ (the unit sphere in ${\mathbb{R}}^{n_0}$) such that the process ${\{z^{(2)}(\mathbf{x})\}}_{\mathbf{x}\in {\mathbb{S}}^{n_0-1}}$ converges in distribution to G, as $n_1\to \infty$. Quantitative versions of this result (for various Wasserstein metrics and some specific choices of σ) are provided in Eldan et al. (2021).

4. Normal Approximation of Shallow Random Gaussian NNs with Univariate Output

The following theorem holds.

Theorem 4.1.

Let $\text{GNN}(1,n_0,1,n_1,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$ be a shallow random Gaussian NN with univariate output, and assume that the activation function σ is such that

$$0<\mathbb{V}\text{ar}(\sigma {({\mathfrak{s}}_{0}Z)}^2)<\infty .$$(18)

Then:

1. $$d_K(z^{(2)},z)\le \frac{C_W\sqrt{\mathbb{V}\text{ar}(\sigma {({\mathfrak{s}}_{0}Z)}^2)}}{C_b+C_W\mathbb{E}\sigma {({\mathfrak{s}}_{0}Z)}^2}\frac{1}{{\sqrt{n}}_1}.$$

2. $$d_{TV}(z^{(2)},z)\le 2\frac{C_W\sqrt{\mathbb{V}\text{ar}(\sigma {({\mathfrak{s}}_{0}Z)}^2)}}{C_b+C_W\mathbb{E}\sigma {({\mathfrak{s}}_{0}Z)}^2}\frac{1}{{\sqrt{n}}_1}.$$

3. $$d_{W_1}(z^{(2)},z)\le \sqrt{2/\pi }\frac{C_W\sqrt{\mathbb{V}\text{ar}(\sigma {({\mathfrak{s}}_{0}Z)}^2)}}{\sqrt{C_b+C_W\mathbb{E}\sigma {({\mathfrak{s}}_{0}Z)}^2}}\frac{1}{{\sqrt{n}}_1}.$$

   Note that the bound on the Kolmogorov distance is better than the one that can be obtained using the relation $d_K\le d_{TV}$.

Remark 4.2.

Remarkably, theorem 3.3 in Favaro et al. (2023) shows that if $\text{GNN}(1,n_0,1,n_1,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$ is a shallow random Gaussian NN with univariate output and the activation function σ is polynomially bounded to order $r\ge 1$ (see definition 2.1 in Favaro et al. (2023)), then there exist two constants $C,C_0>0$ such that

$$\frac{C_0}{n_1}\le \max \{d_{W_1}(z^{(2)},z),d_{TV}(z^{(2)},z)\}\le \frac{C}{n_1}.$$

Although this inequality shows the optimality of the rate $1/n_1$, because the constants are not provided in closed form, it cannot be directly used for the purpose of output localization (see Section 7). Moreover, the assumption (18) on σ does not require any regularity of the activation function.

Proof.

We prove the three bounds (i), (ii), and (iii) separately by the Stein method.

Proof of Part (i).

Consider the Stein Equation (4) with

$$g(w)≔{\mathbf{1}}_{(-\infty ,y]}({\mathfrak{s}}_{1}w).$$

Let f_g be the unique solution of the Stein equation (see Lemma 2.6(i)). Then, for any $y\in \mathbb{R}$,

$$\mathbf{1}\{z^{(2)}\le y\}-\mathbb{P}(z\le y)=f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)-(z^{(2)}/{\mathfrak{s}}_1)f_g(z^{(2)}/{\mathfrak{s}}_1).$$

Taking the expectation, we have

$$\mathbb{P}(z^{(2)}\le y)-\mathbb{P}(z\le y)=\mathbb{E}[f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)-(z^{(2)}/{\mathfrak{s}}_1)f_g(z^{(2)}/{\mathfrak{s}}_1)].$$

By Lemma 2.10(i), we have

$$\begin{array}{l}\mathbb{E}[(z^{(2)}/{\mathfrak{s}}_1)f_g(z^{(2)}/{\mathfrak{s}}_1)\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_1]={\mathfrak{s}}_1^{-2}(C_b+C_W{\mathcal{O}}_{n_1}^{(1)})\mathbb{E}[f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_1]\\ =\mathbb{E}[{\mathfrak{s}}_1^{-2}(C_b+C_W{\mathcal{O}}_{n_1}^{(1)})f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_1],\end{array}$$(19)

where the latter equality follows by the ${\mathcal{F}}_1$-measurability of ${\mathcal{O}}_{n_1}^{(1)}$. Then,

$$\mathbb{P}(z^{(2)}\le y)-\mathbb{P}(z\le y)=\mathbb{E}[f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)(1-{\mathfrak{s}}_1^{-2}(C_b+C_W{\mathcal{O}}_{n_1}^{(1)}))],\hspace{1em}y\in \mathbb{R}.$$

Setting

$$\phi (n_1)≔{\mathfrak{s}}_1^{-2}{C}_W\sqrt{\mathbb{V}\text{ar}({\mathcal{O}}_{n_1}^{(1)})}={\mathfrak{s}}_1^{-2}{C}_W\sqrt{\mathbb{V}\text{ar}(\sigma {(z_1^{(1)})}^2)}/\sqrt{n_1},$$(20)

we have

$$\frac{\mathbb{P}(z^{(2)}\le y)-\mathbb{P}(z\le y)}{\phi (n_1)}=\mathbb{E}\left[f{\prime }_g(z^{(2)}/\nu )V_{n_1}\right],$$(21)

where

$$V_{n_1}≔\frac{1-{\mathfrak{s}}_1^{-2}(C_b+C_W{\mathcal{O}}_{n_1}^{(1)}))}{\phi (n_1)}=-\frac{{\mathcal{O}}_{n_1}^{(1)}-{\mathcal{O}}^{(1)}}{\sqrt{\mathbb{V}\text{ar}({\mathcal{O}}_{n_1}^{(1)})}}=-\frac{\sum _{j=1}^{n_1}\sigma {(z_j^{(1)})}^2-n_1\mathbb{E}\sigma {(z_1^{(1)})}^2}{\sqrt{n_1}\sqrt{\mathbb{V}\text{ar}(\sigma {(z_1^{(1)})}^2)}}.$$

Taking the modulus in (21) and then using that $\Vert f_g\prime {\Vert }_{\infty }\le 1$ uniformly in $y\in \mathbb{R}$ (see Lemma 2.6(i)) and that $\mathbb{E}{V}_{n_1}^2=1$, we have

$$d_K(z^{(2)},z)\le \phi (n_1)\mathbb{E}[|V_{n_1}|]\le \phi (n_1),$$

which, combined with (20), proves the statement.

Proof of Part (ii).

Consider the Stein Equation (4) with $g(w)≔{\mathbf{1}}_B({\mathfrak{s}}_{1}w)$, where $B\subseteq {\mathbb{R}}^d$ is a Borel set. Let f_g be the unique solution of the Stein equation (see Lemma 2.6(ii)). Then,

$${\mathbf{1}}_B(z^{(2)})-\mathbb{E}{\mathbf{1}}_B(z)=f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)-(z^{(2)}/{\mathfrak{s}}_1)f_g(z^{(2)}/{\mathfrak{s}}_1).$$

Taking the expectation and arguing as in (19), we have

$$\mathbb{P}(z^{(2)}\in B)-\mathbb{P}(z\in B)=\mathbb{E}[f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)(1-{\mathfrak{s}}_1^{-2}(C_b+C_W{\mathcal{O}}_{n_1}^{(1)}))].$$

Along similar computations as for (21), we have

$$\frac{\mathbb{P}(z^{(2)}\in B)-\mathbb{P}(z\in B)}{\phi (n_1)}=\mathbb{E}\left[f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)V_{n_1}\right].$$

Taking the modulus on this relation and then using that $\Vert f_g\prime {\Vert }_{\infty }\le 2$ (see Lemma 2.6(ii)) and that $\mathbb{E}{V}_{n_1}^2=1$, we have

$$d_{TV}(z^{(2)},z)\le 2\phi (n_1)\mathbb{E}[|V_{n_1}|]\le 2\phi (n_1),$$

which, combined with (20), proves the statement.

Proof of Part (iii).

Consider the Stein Equation (4) with $g(y)≔h({\mathfrak{s}}_{1}y)$, where $h:\mathbb{R}\to \mathbb{R}$ is Lipschitz continuous with $\text{Lip}(h)\le 1$. Let f_g be the unique solution of the Stein equation (see Lemma 2.6(iii)). Then,

$$h(z^{(2)})-\mathbb{E}h(z)=f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)-(z^{(2)}/{\mathfrak{s}}_1)f_g(z^{(2)}/{\mathfrak{s}}_1).$$

Taking the expectation and arguing as in (19), we have

$$\mathbb{E}h(z^{(2)})-\mathbb{E}h(z)=\mathbb{E}[f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)(1-{\mathfrak{s}}_1^{-2}(C_b+C_W{\mathcal{O}}_{n_1}^{(1)}))].$$

Along similar computations as for (21), we have

$$\frac{\mathbb{E}h(z^{(2)})-\mathbb{E}h(z)}{\phi (n_1)}=\mathbb{E}\left[f{\prime }_g(z^{(2)}/{\mathfrak{s}}_1)V_{n_1}\right].$$

Taking the modulus on this relation and then using that $\Vert f_g\prime {\Vert }_{\infty }\le {\mathfrak{s}}_1\sqrt{2/\pi }$ (see Lemma 2.6(iii)) and that $\mathbb{E}{V}_{n_1}^2=1$, we have

$$d_{W_1}(z^{(2)},z)\le {\mathfrak{s}}_1\sqrt{2/\pi }\phi (n_1)\mathbb{E}[|V_{n_1}|]\le {\mathfrak{s}}_1\sqrt{2/\pi }\phi (n_1),$$

which, combined with (20), proves the statement. □

Note that both Theorem 3.1 and Theorem 4.1 provide bounds on $d_s(z^{(2)},z),\hspace{0.17em}s=TV,K,W_1$, with a common rate $1/\sqrt{n_1}$, but different constants. In Table 1, we compare those constants in a special case. We observe that the constants given by Theorem 4.1 are more effective than those in Bordino et al. (2024). We also note that Condition (18) is satisfied by the ReLU activation function, whereas the assumptions of Theorem 3.1 do not hold for the ReLU.

5. A Key Estimate for the Collective Observables

The next theorem provides an estimate for the L²-norm of the random variable ${\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}-{\mathcal{O}}^{(\mathcal{\ell })},\hspace{0.17em}\mathcal{\ell }=1,\dots ,L$. In Section 6, such an estimate will play a crucial role in the proofs of the results on the Normal approximation of the output of a deep random Gaussian NN both in the convex and in the 1-Wasserstein distances (see Theorems 6.1 and 6.3).

Hereon, we denote by $\Vert Y{\Vert }_{L^2}≔{(\mathbb{E}[|Y{|}^2])}^{1/2}$ the L²-norm of a real-valued random variable Y.

Theorem 5.1.

Let $\text{GNN}(L,n_0,n_{L+1},{\mathbf{n}}_L,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$ be a deep random Gaussian NN. Suppose that the activation function σ is such that:

1. For any $a_1,a_2\ge 0$, there exists a polynomial

   $$P(x)≔{\displaystyle \sum _{k=0}^m{d}_k}{x}^k,$$

   with nonnegative coefficients $d_k=d_k(\sigma (·),C_b,C_W)\ge 0$ dependent only on $\sigma (·),C_b,C_W$ and degree $m\ge 0$ independent of $\sigma (·),a_1,a_2,C_b,C_W$, such that

   $$|\sigma {(x\sqrt{C_b+C_W{a}_2})}^2-\sigma {(x\sqrt{C_b+C_W{a}_1})}^2|\le P(|x|)|a_2-a_1|,\hspace{1em}\mathit{\text{for}} \mathit{\text{all}} x\in \mathbb{R}.$$(22)

2. For any $\kappa \in \mathbb{R},\hspace{0.17em}\mathbb{E}\sigma {(\kappa Z)}^4<\infty$.

Then, for any $\mathcal{\ell }=1,\dots ,L$, we have

$$\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}-{\mathcal{O}}^{(\mathcal{\ell })}{\Vert }_{L^2}\le {\displaystyle \sum _{k=1}^{\mathcal{\ell }}(4\sqrt{2}\Vert P(}|Z|){\Vert }_2{)}^{\mathcal{\ell }-k}\frac{c_k}{\sqrt{n_k}},$$

where

$$c_k=c_k(n_0,\sigma ,\mathbf{x},C_b,C_W)≔\sqrt{2\mathbb{E}\sigma {\left({\mathfrak{s}}_{k-1}Z\right)}^4}<\infty ,\hspace{1em}k=1,\dots ,L.$$(23)

The proof of the theorem is given later on in this section. We proceed stating a proposition and a remark, which clarify the generality of our assumptions on the activation function σ.

Proposition 5.2.

The following statements hold:

1. If σ is the perceptron function, that is, $\sigma (x)≔\mathbf{1}\{x\ge 0\},\hspace{0.17em}x\in \mathbb{R}$, then it satisfies Conditions (i) and (ii) of Theorem 5.1.

2. If σ is Lipschitz continuous and $C_b>0$, then σ satisfies Conditions (i) and (ii) of Theorem 5.1. In particular, Condition (i) holds with

   $$P(x)≔\frac{2|\sigma (0)|C_W\text{Lip}(\sigma )}{2\sqrt{C_b}}x+C_W\text{Lip}{(\sigma )}^2{x}^2.$$(24)

3. If σ is Lipschitz continuous and $\sigma (0)=0$, then σ satisfies Conditions (i) and (ii) of Theorem 5.1. In particular, Condition (i) holds with

   $$P(x)≔C_W\text{Lip}{(\sigma )}^2{x}^2.$$(25)

4. If σ is such that $\sigma {(·)}^2$ is Lipschitz continuous and $C_b>0$, then σ satisfies Conditions (i) and (ii) of Theorem 5.1. In particular, Condition (i) holds with

   $$P(x)=\frac{\text{Lip}({\sigma }^2)C_W}{2\sqrt{C_b}}x.$$(26)

The proof of the proposition is given later on in this section.

Remark 5.3.

As a consequence of Proposition 5.2, we have that the most common activation functions satisfy the assumptions of Theorem 5.1. Indeed, one can easily prove that the ReLU function $\sigma (x)≔x\mathbf{1}\{x\ge 0\}$, the hyperbolic tangent function $\sigma (x)≔({\text{e}}^{2x}-1)/({\text{e}}^{2x}+1)$, the $\sin$ function $\sigma (x)≔\sin (x)$, and the $\text{SWISH}$ function $\sigma (x)≔x/(1+{\text{e}}^{-x})$ are Lipschitz continuous and equal to zero in zero. Moreover, if $C_b>0$, then also the sigmoid function $\sigma (x)≔{(1+{\text{e}}^{-x})}^{-1}$ and the softplus function $\sigma (x)≔\log (1+{\text{e}}^x)$ satisfy the assumptions of Theorem 5.1, and indeed they are Lipschitz continuous. We emphasize that, if either the biases are nonnull or the activation function vanishes at 0, then the conditions on the activation function of Theorem 5.1 are more general than the one required in Basteri and Trevisan (2024), where $\sigma (·)$ is assumed Lipschitz continuous (see Theorem 3.2 and Proposition 5.2 parts (ii) and (iii)). We also emphasize that the conditions on the activation function of Theorem 5.1 are satisfied by the perceptron function that is noncontinuous and therefore non-Lipschitz (see Proposition 5.2(i). Another non-Lipschitz function that satisfies the conditions of Theorem 5.1 when the biases are nonnull I, for example, $\sigma (x)≔\sqrt{x}\mathbf{1}\{x\ge 0\}$. Indeed, ${\sigma }^2$ is the ReLU function and therefore Lipschitz continuous (see Proposition 5.2(iv)).

Proof of Theorem 5.1.

We consider separately the cases of the first hidden layer and that of the following ones.

• Case $\mathcal{\ell }=1$.

Because the random variables $z_i^{(1)},\hspace{0.17em}i=1,\dots ,n_1$, are independent and identically distributed with law $\mathcal{N}(0,{\mathfrak{s}}_0^2)$, we have

$$\begin{array}{l}\Vert {\mathcal{O}}_{n_1}^{(1)}-{\mathcal{O}}^{(1)}{\Vert }_{L^2}=\sqrt{\mathbb{E}{\left(\frac{1}{n_1}{\displaystyle \sum _{j=1}^{n_1}\sigma }{(z_j^{(1)})}^2-\mathbb{E}\sigma {\left({\mathfrak{s}}_{0}Z\right)}^2\right)}^2}\\ =\sqrt{\mathbb{E}{\left(\frac{1}{n_1}{\displaystyle \sum _{j=1}^{n_1}(\sigma {(z_j^{(1)})}^2-\mathbb{E}\sigma {(z_1^{(1)})}^2)}\right)}^2}\\ =\frac{1}{{\sqrt{n}}_1}\sqrt{\mathbb{V}\text{ar}(\sigma {(z_1^{(1)})}^2)}\\ \le \frac{1}{{\sqrt{n}}_1}\sqrt{\mathbb{E}\sigma {\left({\mathfrak{s}}_{0}Z\right)}^4}\end{array}$$(27)

$$\le \frac{c_1}{{\sqrt{n}}_1}.$$(28)

Note that this latter term is finite because of the assumption (ii).

• Case $\mathcal{\ell }=2,\dots ,L$.

Take $\mathcal{\ell }\in \{2,\dots ,L\}$. We have already noticed that, given ${\mathcal{F}}_{\mathcal{\ell }-1}$, the random variables ${\{z_i^{(\mathcal{\ell })}\}}_{i=1,\dots ,n_{\mathcal{\ell }}}$ are independent with Gaussian law with mean zero and variance

$${\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}^2≔C_b+C_W{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}.$$

Therefore, letting $Z_{\mathcal{\ell }-1}$ denote a standard Gaussian random variable, independent of ${\mathcal{F}}_{\mathcal{\ell }-1}$, we have

$$z_i^{(\mathcal{\ell })}\stackrel{d}{=}{\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}{Z}_{\mathcal{\ell }-1},\hspace{1em}i=1,\dots ,n_{\mathcal{\ell }}$$

where the symbold $\stackrel{d}{=}$ denotes the equality in law (this relation immediately follows computing, for example, the characteristic function of both random variables). Therefore, letting $p(·)$ denote the standard Gaussian density, we have

$$\mathbb{E}[\sigma {(z_i^{(\mathcal{\ell })})}^r|{\mathcal{F}}_{\mathcal{\ell }-1}]\stackrel{d}{=}{\displaystyle {\int }_{\mathbb{R}}\sigma }{\left({\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}z\right)}^{r}p(z)\text{d}z,\hspace{1em}r\in \{2,4\},\hspace{0.17em}i=1,\dots ,n_{\mathcal{\ell }}$$(29)

and

$$\mathbb{E}{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}=\mathbb{E}\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}{Z}_{\mathcal{\ell }-1}\right)}^2.$$

By assumption (i) and the fact that $Z_{\mathcal{\ell }-1}$ is independent of ${\mathcal{F}}_{\mathcal{\ell }-1}$, we have

$$\begin{array}{l}|\mathbb{E}{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}-{\mathcal{O}}^{(\mathcal{\ell })}|=|\mathbb{E}\left[\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}{Z}_{\mathcal{\ell }-1}\right)}^2\right]-\mathbb{E}\left[\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1}{Z}_{\mathcal{\ell }-1}\right)}^2\right]|\\ \le \mathbb{E}P(|Z_{\mathcal{\ell }-1}|)|{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}|\\ \le \mathbb{E}P(|Z|)\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}{\Vert }_{L^2}.\end{array}$$

Therefore,

$$\begin{array}{l}\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}-{\mathcal{O}}^{(\mathcal{\ell })}{\Vert }_{L^2}\le \Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}-\mathbb{E}{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}{\Vert }_{L^2}+|\mathbb{E}{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}-{\mathcal{O}}^{(\mathcal{\ell })}|\\ =\sqrt{\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })})}+|\mathbb{E}{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}-{\mathcal{O}}^{(\mathcal{\ell })}|\\ \le \sqrt{\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })})}+\mathbb{E}P(|Z|)\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}{\Vert }_{L^2}.\end{array}$$(30)

Note that

$$\begin{array}{l}\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })})=\frac{1}{n_{\mathcal{\ell }}^2}\left(\mathbb{E}{\left({\displaystyle \sum _{j=1}^{n_{\mathcal{\ell }}}\sigma }{(z_j^{(\mathcal{\ell })})}^2\right)}^2-n_{\mathcal{\ell }}^2{(\mathbb{E}\sigma {(z_1^{(\mathcal{\ell })})}^2)}^2\right)\\ =\frac{1}{n_{\mathcal{\ell }}^2}(n_{\mathcal{\ell }}\mathbb{E}\sigma {(z_1^{(\mathcal{\ell })})}^4+n_{\mathcal{\ell }}(n_{\mathcal{\ell }}-1)\mathbb{E}\sigma {(z_1^{(\mathcal{\ell })})}^2\sigma {(z_2^{(\mathcal{\ell })})}^2-n_{\mathcal{\ell }}^2{(\mathbb{E}\sigma {(z_1^{(\mathcal{\ell })})}^2)}^2)\\ =\frac{1}{n_{\mathcal{\ell }}^2}(n_{\mathcal{\ell }}\mathbb{E}\sigma {(z_1^{(\mathcal{\ell })})}^4-n_{\mathcal{\ell }}\mathbb{E}\sigma {(z_1^{(\mathcal{\ell })})}^2\sigma {(z_2^{(\mathcal{\ell })})}^2+n_{\mathcal{\ell }}^2\mathbb{C}\text{ov}(\sigma {(z_1^{(\mathcal{\ell })})}^2,\sigma {(z_2^{(\mathcal{\ell })})}^2))\\ =\frac{1}{n_{\mathcal{\ell }}^2}(n_{\mathcal{\ell }}\mathbb{V}\text{ar}(\sigma {(z_1^{(\mathcal{\ell })})}^2)+(n_{\mathcal{\ell }}^2-n_{\mathcal{\ell }})\mathbb{C}\text{ov}(\sigma {(z_1^{(\mathcal{\ell })})}^2,\sigma {(z_2^{(\mathcal{\ell })})}^2))\\ =\frac{1}{n_{\mathcal{\ell }}}\mathbb{V}\text{ar}(\sigma {(z_1^{(\mathcal{\ell })})}^2)+\left(1-\frac{1}{n_{\mathcal{\ell }}}\right)\mathbb{C}\text{ov}(\sigma {(z_1^{(\mathcal{\ell })})}^2,\sigma {(z_2^{(\mathcal{\ell })})}^2).\end{array}$$(31)

By (29), we have

$$\mathbb{E}[\sigma {(z_1^{(\mathcal{\ell })})}^4]=\mathbb{E}\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}{Z}_{\mathcal{\ell }-1}\right)}^4.$$

By assumption (i), we have

$$\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}{Z}_{\mathcal{\ell }-1}\right)}^2\le P(|Z_{\mathcal{\ell }-1}|)|{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}|+\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1}{Z}_{\mathcal{\ell }-1}\right)}^2,\hspace{1em}\mathbb{P}-\text{a}.\text{s}.$$

and so (using that $Z_{\mathcal{\ell }-1}$ is independent of ${\mathcal{F}}_{\mathcal{\ell }-1}$ and the inequality ${(a+b)}^2\le 2a^2+2b^2,\hspace{0.17em}a,b\in \mathbb{R}$),

$$\begin{array}{l}\mathbb{V}\text{ar}(\sigma {(z_1^{(\mathcal{\ell })})}^2)\le \mathbb{E}[\sigma {(z_1^{(\mathcal{\ell })})}^4]\\ \le 2\mathbb{E}P{(|Z|)}^2\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}{\Vert }_2^2+2\mathbb{E}\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1}Z\right)}^4\\ =A\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}{\Vert }_{L^2}^2+c_{\mathcal{\ell }}^2,\end{array}$$(32)

where $A≔2\mathbb{E}P{(|Z|)}^2<\infty$. Note that the quantities ${\mathcal{O}}^{(\mathcal{\ell })},\hspace{0.17em}\mathcal{\ell }=2,\dots ,L$, are all finite because of the assumption (ii). Then, again by assumption (ii), we have

$$\mathbb{E}\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1}Z\right)}^4<\infty ,\hspace{1em}\text{for} \text{any} \mathcal{\ell }=2,\dots ,L$$

and therefore, $c_{\mathcal{\ell }}<\infty$, for any $\mathcal{\ell }=2,\dots ,L$. By the conditional independence of the random variables $z_1^{(\mathcal{\ell })}$ and $z_2^{(\mathcal{\ell })}$, given ${\mathcal{F}}_{\mathcal{\ell }-1}$, we have

$$\begin{array}{l}\mathbb{C}\text{ov}(\sigma {(z_1^{(\mathcal{\ell })})}^2,\sigma {(z_2^{(\mathcal{\ell })})}^2)=\mathbb{C}\text{ov}(\mathbb{E}[\sigma {(z_1^{(\mathcal{\ell })})}^2\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_{\mathcal{\ell }-1}],\mathbb{E}[\sigma {(z_2^{(\mathcal{\ell })})}^2\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_{\mathcal{\ell }-1}])\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\mathbb{E}[(\mathbb{E}[\sigma {(z_1^{(\mathcal{\ell })})}^2\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_{\mathcal{\ell }-1}]-\mathbb{E}\sigma {(z_1^{(\mathcal{\ell })})}^2)(\mathbb{E}[\sigma {(z_2^{(\mathcal{\ell })})}^2\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_{\mathcal{\ell }-1}]-\mathbb{E}\sigma {(z_2^{(\mathcal{\ell })})}^2)],\end{array}$$

and so, by the Cauchy-Schwarz inequality and (29) we have

$$|\mathbb{C}\text{ov}(\sigma {(z_1^{(\mathcal{\ell })})}^2,\sigma {(z_2^{(\mathcal{\ell })})}^2)|\le \mathbb{E}[{(\mathbb{E}[\sigma {(z_1^{(\mathcal{\ell })})}^2|{\mathcal{F}}_{\mathcal{\ell }-1}]-\mathbb{E}\sigma {(z_1^{(\mathcal{\ell })})}^2)}^2].$$(33)

Letting ${\mathbb{P}}_X$ denote the law of a random variable X and again using (29), we have that the random variable $\mathbb{E}[\sigma {(z_1^{(\mathcal{\ell })})}^2|{\mathcal{F}}_{\mathcal{\ell }-1}]-\mathbb{E}\sigma {(z_1^{(\mathcal{\ell })})}^2$ has the same law as the random variable

$$\begin{array}{l}{\displaystyle {\int }_{\mathbb{R}}\left(\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}z\right)}^2-{\int }_{[0,\infty )}\sigma {\left(z\sqrt{C_b+C_{W}y}\right)}^2{\mathbb{P}}_{{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}}(\text{d}y)\right)}p(z)\text{d}z\\ \hspace{1em}={\int }_{[0,\infty )\times \mathbb{R}}(\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}z\right)}^2-\sigma {\left(z\sqrt{C_b+C_{W}y}\right)}^2){\mathbb{P}}_{{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}}(\text{d}y)p(z)\text{d}z.\end{array}$$

Therefore, by (33) and Jensen’s inequality, we have

$$\begin{array}{l}|\mathbb{C}\text{ov}(\sigma {(z_1^{(\mathcal{\ell })})}^2,\sigma {(z_2^{(\mathcal{\ell })})}^2)|\le \mathbb{E}{\int }_{[0,\infty )\times \mathbb{R}}{(\sigma {\left({\mathfrak{s}}_{\mathcal{\ell }-1,{\mathbf{n}}_{\mathcal{\ell }-1}}z\right)}^2-\sigma {\left(z\sqrt{C_b+C_{W}y}\right)}^2)}^2{\mathbb{P}}_{{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}}(\text{d}y)p(z)\text{d}z\end{array}$$

By assumption (i), it then follows that

$$|\mathbb{C}\text{ov}(\sigma {(z_1^{(\mathcal{\ell })})}^2,\sigma {(z_2^{(\mathcal{\ell })})}^2)|\le {(\mathbb{E}P(|Z|))}^2{\int }_{[0,\infty )}\mathbb{E}|{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-y{|}^2{\mathbb{P}}_{{\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}}(\text{d}y)\le A\mathbb{V}\text{ar}\left({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}\right),$$(34)

where the latter relation follows by the definition of the constant A. Combining (31), (32), and (34), we have

$$\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })})\le A(\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)})+\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}_{{\sigma }^2}^{(\mathcal{\ell }-1)}{\Vert }_{L^2}^2)+\frac{c_{\mathcal{\ell }}^2}{n_{\mathcal{\ell }}}.$$

Iterating this inequality, we have

$$\begin{array}{l}\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })})\le A\left(A(\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-2}}^{(\mathcal{\ell }-2)})+\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-2}}^{(\mathcal{\ell }-2)}-{\mathcal{O}}^{(\mathcal{\ell }-2)}{\Vert }_{L^2}^2)+\frac{c_{\mathcal{\ell }-1}^2}{n_{\mathcal{\ell }-1}}\right)+A\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}{\Vert }_{L^2}^2+\frac{c_{\mathcal{\ell }}^2}{n_{\mathcal{\ell }}}\\ =A^2\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-2}}^{(\mathcal{\ell }-2)})+A^2\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-2}}^{(\mathcal{\ell }-2)}-{\mathcal{O}}^{(\mathcal{\ell }-2)}{\Vert }_{L^2}^2+A\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}{\Vert }_{L^2}^2+A\frac{c_{\mathcal{\ell }-1}^2}{n_{\mathcal{\ell }-1}}+\frac{c_{\mathcal{\ell }}^2}{n_{\mathcal{\ell }}}\\ \le A^3\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-3}}^{(\mathcal{\ell }-3)})+A^3\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-3}}^{(\mathcal{\ell }-3)}-{\mathcal{O}}^{(\mathcal{\ell }-3)}{\Vert }_{L^2}^2+A^2\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-2}}^{(\mathcal{\ell }-2)}-{\mathcal{O}}^{(\mathcal{\ell }-2)}{\Vert }_{L^2}^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+A\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}{\Vert }_{L^2}^2+A^2\frac{c_{\mathcal{\ell }-2}^2}{n_{\mathcal{\ell }-2}}+A\frac{c_{\mathcal{\ell }-1}^2}{n_{\mathcal{\ell }-1}}+\frac{c_{\mathcal{\ell }}^2}{n_{\mathcal{\ell }}}\\ \le A^{\mathcal{\ell }-1}\mathbb{V}\text{ar}({\mathcal{O}}_{n_1}^{(1)})+{\displaystyle \sum _{k=1}^{\mathcal{\ell }-1}{A}^{\mathcal{\ell }-k}}\Vert {\mathcal{O}}_{{\mathbf{n}}_k}^{(k)}-{\mathcal{O}}^{(k)}{\Vert }_{L^2}^2+{\displaystyle \sum _{k=2}^{\mathcal{\ell }}{A}^{\mathcal{\ell }-k}}\frac{c_k^2}{n_k}\\ =2A^{\mathcal{\ell }-1}\mathbb{V}\text{ar}({\mathcal{O}}_{n_1}^{(1)})+{\displaystyle \sum _{k=2}^{\mathcal{\ell }-1}{A}^{\mathcal{\ell }-k}}\Vert {\mathcal{O}}_{{\mathbf{n}}_k}^{(k)}-{\mathcal{O}}^{(k)}{\Vert }_{L^2}^2+{\displaystyle \sum _{k=2}^{\mathcal{\ell }}{A}^{\mathcal{\ell }-k}}\frac{c_k^2}{n_k},\end{array}$$

for any $\mathcal{\ell }=2,\dots ,L$, where the latter equality follows noticing that $\mathbb{E}{\mathcal{O}}_{n_1}^{(1)}={\mathcal{O}}^{(1)}$. Here, we adopt the usual convention ${\sum }_{k=k_1}^{k_2}\cdots =0$ if $k_1>k_2$. Note that by (27), we have

$$\mathbb{V}\text{ar}({\mathcal{O}}_{n_1}^{(1)})\le \frac{1}{n_1}\mathbb{E}\sigma {\left({\mathfrak{s}}_{0}Z\right)}^4,$$

and so,

$$2\mathbb{V}\text{ar}({\mathcal{O}}_{n_1}^{(1)})\le \frac{c_1^2}{n_1}.$$

Consequently,

$$\mathbb{V}\text{ar}({\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })})\le {\displaystyle \sum _{k=2}^{\mathcal{\ell }-1}{A}^{\mathcal{\ell }-k}}\Vert {\mathcal{O}}_{{\mathbf{n}}_k}^{(k)}-{\mathcal{O}}^{(k)}{\Vert }_{L^2}^2+{\displaystyle \sum _{k=1}^{\mathcal{\ell }}{A}^{\mathcal{\ell }-k}}\frac{c_k^2}{n_k}.$$

Combining this inequality with (30) and using the elementary relation $\sqrt{a_1+a_2}\le {\sqrt{a}}_1+{\sqrt{a}}_2,\hspace{0.17em}{a}_1,a_2\ge 0$, for any $\mathcal{\ell }=2,\dots ,L$, we have

$$\begin{array}{l}\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}-{\mathcal{O}}^{(\mathcal{\ell })}{\Vert }_{L^2}\le {\displaystyle \sum _{k=2}^{\mathcal{\ell }-1}{A}^{(\mathcal{\ell }-k)/2}}\Vert {\mathcal{O}}_{{\mathbf{n}}_k}^{(k)}-{\mathcal{O}}^{(k)}{\Vert }_{L^2}+{\displaystyle \sum _{k=1}^{\mathcal{\ell }}{A}^{(\mathcal{\ell }-k)/2}}\frac{c_k}{\sqrt{n_k}}+\mathbb{E}P(|Z|)\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }-1}}^{(\mathcal{\ell }-1)}-{\mathcal{O}}^{(\mathcal{\ell }-1)}{\Vert }_{L^2}\\ \le {\displaystyle \sum _{k=2}^{\mathcal{\ell }-1}{B}^{(\mathcal{\ell }-k)/2}}\Vert {\mathcal{O}}_{{\mathbf{n}}_k}^{(k)}-{\mathcal{O}}^{(k)}{\Vert }_{L^2}+{\displaystyle \sum _{k=1}^{\mathcal{\ell }}{B}^{(\mathcal{\ell }-k)/2}}\frac{c_k}{\sqrt{n_k}},\end{array}$$(35)

where we used that $\mathbb{E}P(|Z|)\le A^{1/2}$, and we set $B≔4A$. Recalling that $\sqrt{A}=\sqrt{2}\Vert P(|Z|){\Vert }_2$ and the definition of B, one easily realizes that the claim reads as

$$\Vert {\mathcal{O}}_{{\mathbf{n}}_2}^{(\mathcal{\ell })}-{\mathcal{O}}^{(\mathcal{\ell })}{\Vert }_{L^2}\le {\displaystyle \sum _{j=1}^{\mathcal{\ell }}{(2\sqrt{B})}^{\mathcal{\ell }-j}}\frac{c_j}{\sqrt{n_j}},\hspace{1em}\mathcal{\ell }=2,\dots ,L.$$(36)

Now, we use the relation (35) to prove (36) by induction on $\mathcal{\ell }=2,\dots ,L$. Taking $\mathcal{\ell }=2$ in (35), we have

$$\Vert {\mathcal{O}}_{{\mathbf{n}}_2}^{(2)}-{\mathcal{O}}^{(2)}{\Vert }_{L^2}\le B^{1/2}\frac{c_1}{\sqrt{n_1}}+\frac{c_2}{\sqrt{n_2}}\le 2B^{1/2}\frac{c_1}{\sqrt{n_1}}+\frac{c_2}{\sqrt{n_2}},$$

for example, (36) with $\mathcal{\ell }=2$. Now, suppose that

$$\Vert {\mathcal{O}}_{{\mathbf{n}}_k}^{(k)}-{\mathcal{O}}^{(k)}{\Vert }_{L^2}\le {\displaystyle \sum _{j=1}^k{(2\sqrt{B})}^{k-j}}\frac{c_j}{\sqrt{n_j}},\hspace{1em}\text{for} \text{any} k=2,\dots ,\mathcal{\ell }-1.$$

Then, by (35), we have

$$\begin{array}{l}\Vert {\mathcal{O}}_{{\mathbf{n}}_{\mathcal{\ell }}}^{(\mathcal{\ell })}-{\mathcal{O}}^{(\mathcal{\ell })}{\Vert }_{L^2}\le {\displaystyle \sum _{k=2}^{\mathcal{\ell }-1}{B}^{(\mathcal{\ell }-k)/2}}{\displaystyle \sum _{j=1}^k{2}^{k-j}}{B}^{(k-j)/2}\frac{c_j}{\sqrt{n_j}}+{\displaystyle \sum _{k=1}^{\mathcal{\ell }}{B}^{(\mathcal{\ell }-k)/2}}\frac{c_k}{\sqrt{n_k}}\\ \le {\displaystyle \sum _{k=2}^{\mathcal{\ell }-1}{\displaystyle \sum _{j=1}^k{2}^{k-j}}}{B}^{(\mathcal{\ell }-j)/2}\frac{c_j}{\sqrt{n_j}}+{\displaystyle \sum _{k=1}^{\mathcal{\ell }}{B}^{(\mathcal{\ell }-k)/2}}\frac{c_k}{\sqrt{n_k}}\\ \le {\displaystyle \sum _{j=1}^{\mathcal{\ell }-1}{\displaystyle \sum _{k=j}^{\mathcal{\ell }-1}{2}^{k-j}}}{B}^{(\mathcal{\ell }-j)/2}\frac{c_j}{\sqrt{n_j}}+{\displaystyle \sum _{k=1}^{\mathcal{\ell }}{B}^{(\mathcal{\ell }-k)/2}}\frac{c_k}{\sqrt{n_k}}\\ ={\displaystyle \sum _{j=1}^{\mathcal{\ell }-1}\left({\displaystyle \sum _{k=j}^{\mathcal{\ell }-1}{2}^{k-j}}+1\right)}{B}^{(\mathcal{\ell }-j)/2}\frac{c_j}{\sqrt{n_j}}+\frac{c_{\mathcal{\ell }}}{\sqrt{n_{\mathcal{\ell }}}}\\ ={\displaystyle \sum _{j=1}^{\mathcal{\ell }-1}{(2\sqrt{B})}^{\mathcal{\ell }-j}}\frac{c_j}{\sqrt{n_j}}+\frac{c_{\mathcal{\ell }}}{\sqrt{n_{\mathcal{\ell }}}}={\displaystyle \sum _{j=1}^{\mathcal{\ell }}{(2\sqrt{B})}^{\mathcal{\ell }-j}}\frac{c_j}{\sqrt{n_j}},\end{array}$$

where we used that

$${\displaystyle \sum _{k=j}^{\mathcal{\ell }-1}{2}^{k-j}}+1={\displaystyle \sum _{s=0}^{\mathcal{\ell }-j-1}{2}^s}+1=2^{\mathcal{\ell }-j},\hspace{1em}\text{for} \text{any} j=1,\dots ,\mathcal{\ell }-1$$

because ${\{2^s\}}_{s\ge 0}$ is a geometric progression. The proof is completed. □

Proof of Proposition 5.2.

 

Proof of (i).

The claim immediately follows noticing that, because of the nonnegativity of the quantities $\sqrt{C_b+C_W{a}_2}$ and $\sqrt{C_b+C_W{a}_1}$, we have $|\sigma {(x\sqrt{C_b+C_W{a}_2})}^2-\sigma {(x\sqrt{C_b+C_W{a}_1})}^2|=0$, for any $x\in \mathbb{R}$.

Proof of (ii).

Because σ is Lipschitz continuous, we have

$$\begin{array}{l}|\sigma {(x\sqrt{C_b+C_W{a}_2})}^2-\sigma {(x\sqrt{C_b+C_W{a}_1})}^2|\\ =|\sigma (x\sqrt{C_b+C_W{a}_2})-\sigma (x\sqrt{C_b+C_W{a}_1})\Vert \sigma (x\sqrt{C_b+C_W{a}_2})+\sigma (x\sqrt{C_b+C_W{a}_1})|\\ \le \text{Lip}(\sigma )|x\Vert \sqrt{C_b+C_W{a}_2}-\sqrt{C_b+C_W{a}_1}|(|\sigma (x\sqrt{C_b+C_W{a}_2})|+|\sigma (x\sqrt{C_b+C_W{a}_1})|)\\ \le \text{Lip}(\sigma )|x\Vert \sqrt{C_b+C_W{a}_2}-\sqrt{C_b+C_W{a}_1}|[2|\sigma (0)|\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\text{Lip}(\sigma )|x|(\sqrt{C_b+C_W{a}_2}+\sqrt{C_b+C_W{a}_1})],\end{array}$$(37)

where the latter inequality follows noticing that (again by the Lipschitz continuity of σ) for any $x,\kappa \in \mathbb{R}$,

$$|\sigma (\kappa x)|\le |\sigma (0)|+\text{Lip}(\sigma (·))|\kappa \Vert x|.$$(38)

Multiplying and dividing the term in (37) by $\sqrt{C_b+C_W{a}_2}+\sqrt{C_b+C_W{a}_1}$, we easily have that

$$\begin{array}{l}|\sigma {(x\sqrt{C_b+C_W{a}_2})}^2-\sigma {(x\sqrt{C_b+C_W{a}_1})}^2|\\ \le C_W\text{Lip}(\sigma )|x|\left(\frac{2|\sigma (0)|}{\sqrt{C_b+C_W{a}_2}+\sqrt{C_b+C_W{a}_1}}+\text{Lip}(\sigma )|x|\right)|a_2-a_1|\\ \le P(|x|)|a_2-a_1|,\end{array}$$

where P(x) is defined by (24). As far as Condition (ii) of Theorem 5.1 is concerned, we note that by (38)

$$\sigma {(\kappa Z)}^4\le {(|\sigma (0)|+\text{Lip}(\sigma )|\kappa \Vert Z|)}^4,\hspace{1em} \text{almost} \text{surely}$$

and so the claim is an immediate consequence of the fact that the absolute moments of Z are finite.

Proof of (iii).

The proof is similar to the proof of (ii) and therefore is omitted.

Proof of (iv).

If $\sigma {(·)}^2$ is Lipschitz continuous, then

$$\begin{array}{l}|\sigma {(x\sqrt{C_b+C_W{a}_2})}^2-\sigma {(x\sqrt{C_b+C_W{a}_1})}^2|\\ \hspace{1em}\hspace{1em}\le \text{Lip}({\sigma }^2)|x\Vert \sqrt{C_b+C_W{a}_2}-\sqrt{C_b+C_W{a}_1}|\\ \hspace{1em}\hspace{1em}=\text{Lip}({\sigma }^2)|x\Vert \sqrt{C_b+C_W{a}_2}-\sqrt{C_b+C_W{a}_1}|\frac{\sqrt{C_b+C_W{a}_2}+\sqrt{C_b+C_W{a}_1}}{\sqrt{C_b+C_W{a}_2}+\sqrt{C_b+C_W{a}_1}}\\ \hspace{1em}\hspace{1em}=\text{Lip}({\sigma }^2)|x\Vert (C_b+C_W{a}_2)-(C_b+C_W{a}_1)|\frac{1}{\sqrt{C_b+C_W{a}_2}+\sqrt{C_b+C_W{a}_1}}\\ \hspace{1em}\hspace{1em}\le \frac{\text{Lip}({\sigma }^2)C_W}{2\sqrt{C_b}}|x\Vert a_2-a_1|,\end{array}$$

which shows that Condition (i) of Theorem 5.1 holds with P(x) given by (26). Furthermore, using (38) with $\sigma {(·)}^2$ in place of $\sigma (·)$, for any $\kappa \in \mathbb{R}$,

$$\sigma {(\kappa Z)}^4\le {(|\sigma (0){|}^2+\text{Lip}(\sigma {(·)}^2)|\kappa \Vert Z|)}^2,\hspace{1em}\text{almost}\hspace{0.5em}\text{surely}.$$

Therefore, Condition (ii) of Theorem 5.1 is an immediate consequence of this latter relation and the fact that the absolute moments of Z are finite. □

6. Normal Approximation of Deep Random Gaussian NNs

6.1. Normal Approximation of Deep Random Gaussian NNs in the Convex Distance

The following theorem holds.

Theorem 6.1.

Let $\text{GNN}(L,n_0,n_{L+1},{\mathbf{n}}_L,\sigma ;\mathbf{x};\mathbf{b},\mathbf{W})$ be a deep random Gaussian NN, and let the notation and assumptions of Theorem 5.1 prevail. Then,

$$\begin{array}{l}{d}_c({\mathbf{z}}^{(L+1)},\mathbf{z})\le C_1\left({\displaystyle \sum _{k=1}^L}{[4\sqrt{2}\Vert P(|Z|){\Vert }_{L^2}]}^{L-k}\frac{c_k}{\sqrt{n_k}}\right)\\ \le C_i\left({\displaystyle \sum _{k=1}^L}{[4\sqrt{2}\Vert P(|Z|){\Vert }_{L^2}]}^{L-k}\frac{c_k}{\sqrt{n_k}}\right),\hspace{1em}i=2,3\end{array}$$

where the constants c_k, $k=1,\dots ,L$, are defined by (23),

$$\begin{array}{l}{C}_1=C_1(n_0,n_{L+1},\mathbf{x},\sigma ,C_b,C_W)≔C_W\left(\frac{80}{{\mathfrak{s}}_{L+1}^3}+\frac{48}{{\mathfrak{s}}_{L+1}^2}+20\sqrt{2}\right)n_{L+1}^{59/24},\\ C_2=C_2(n_{L+1},C_b,C_W)≔C_W\left(\frac{80}{C_b^{3/2}}+\frac{48}{C_b}+20\sqrt{2}\right)n_{L+1}^{59/24}\end{array}$$(39)

and

$$C_3=C_3(n_0,n_{L+1},\mathbf{x},\sigma ,C_W)≔C_W\left(\frac{80}{{(C_W{\mathcal{O}}^{(L+1)})}^{3/2}}+\frac{48}{C_W{\mathcal{O}}^{(L+1)}}+20\sqrt{2}\right)n_{L+1}^{59/24}.$$

Here, ${\mathfrak{s}}_{L+1}≔\sqrt{C_b+C_W{\mathcal{O}}^{(L+1)}}$, and clearly the upper bound with constant C₂ holds under the additional assumption that the biases are nonnull.

Remark 6.2.

Let $\text{GNN}(L,n_0,n_{L+1},{\mathbf{n}}_L,\sigma ;\mathbf{x};\mathbf{b},\mathbf{W})$ be a deep random Gaussian NN. Theorem 3.5 in Favaro et al. (2023) shows that if

$$c_{2}n\le n_1,\dots ,n_L\le c_{1}n,\hspace{1em}\text{for} \text{some} \text{constants} c_1\ge c_2>0 \text{and} \text{some} n\ge 1$$(40)

and σ is polynomially bounded to order $r\ge 1$, then there exists a constant C₀ such that

$$d_c({\mathbf{z}}^{(L+1)},\mathbf{z})\le C_0{n}^{-1/2}.$$

We stress that the bounds in Theorem 6.1 provide (under different assumptions on σ and without any condition on the widths of the hidden layers) a detailed description of the analytical dependence of the upper estimate on the parameters of the model.

Proof.

Throughout this proof, for ease of notation, we put

$$\kappa ≔d_c({\mathbf{z}}^{(L+1)},\mathbf{z})\hspace{1em} \text{and} \hspace{1em}\gamma ≔C_W\Vert {\mathcal{O}}_{{\mathbf{n}}_L}^{(L)}-{\mathcal{O}}^{(L)}{\Vert }_{L^2}.$$

We preliminarily note that it suffices to prove that

$$\kappa \le (80\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{3/2}+48\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}+20\sqrt{2})n_{L+1}^{59/24}\gamma .$$(41)

Indeed, the claim then follows by Theorem 5.1, noticing that

$$\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}=\frac{1}{{\mathfrak{s}}_L}$$(42)

and that $C_1\le C_i$, i = 2, 3.

If $\gamma >1/\text{e}$, then the inequality (41) holds because $\kappa \le 1$ (which follows by the definition of the convex distance) and

$$20\sqrt{2}{n}_{L+1}^{59/24}\gamma >20\sqrt{2}\gamma >20\sqrt{2}/3>1.$$

From now on, we assume $\gamma \le 1/\text{e}$. Let $h(·)≔{\mathbf{1}}_C(·)$, where C is an arbitrarily fixed measurable convex set in ${\mathbb{R}}^{n_{L+1}}$, and let

$$h_t(\mathbf{y})≔\mathbb{E}h(\sqrt{t}\mathbf{z}+\sqrt{1-t}\mathbf{y}),\hspace{1em}t\in (0,1),\hspace{1em}\mathbf{y}\in {\mathbb{R}}^{n_{L+1}}.$$

For any $t\in (0,1)$, by Lemma 2.8(i),

$$f_{t,h}(\mathbf{y})≔\frac{1}{2}{\displaystyle {\int }_t^1\frac{1}{1-s}}(\mathbb{E}h(\sqrt{t}\mathbf{z}+\sqrt{1-t}\mathbf{y})-\mathbb{E}h(\mathbf{z}))\text{d}s,\hspace{1em}\mathbf{y}\in {\mathbb{R}}^{n_{L+1}}$$

solves the Stein Equation (5) with $n_{L+1}$, h_t, ${\mathrm{\Sigma }}_{n_{L+1}}$ defined by (12), and $\mathbf{z}$, in place of d, g, Σ and ${\mathbf{N}}_{\mathrm{\Sigma }}$, respectively, that is,

$$h_t(\mathbf{y})-\mathbb{E}[h_t(\mathbf{z})]={〈\mathbf{y},\nabla f_{t,h}(\mathbf{y})〉}_{n_{L+1}}-{〈{\mathrm{\Sigma }}_{n_{L+1}},\text{Hess}\hspace{0.17em}{f}_{t,h}(\mathbf{y})〉}_{H.S.},\hspace{1em}\mathbf{y}\in {\mathbb{R}}^{n_{L+1}}.$$(43)

By Lemma 2.9, it follows that

$$\kappa \le \frac{4}{3}\underset{h\in {\mathcal{I}}_{n_{L+1}}}{\sup }|\mathbb{E}{h}_t({\mathbf{z}}^{(L+1)})-\mathbb{E}{h}_t(\mathbf{z})|+\frac{20n_{L+1}}{\sqrt{2}}\frac{\sqrt{t}}{1-t},\hspace{1em}t\in (0,1).$$(44)

Without loss of generality, hereafter we assume that $\mathbf{z}$ is independent of ${\mathcal{F}}_L$. Therefore, by (43), we have

$$\begin{array}{l}\mathbb{E}[h_t({\mathbf{z}}^{(L+1)})-h_t(\mathbf{z})\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_L]={\displaystyle \sum _{i=1}^{n_{L+1}}\mathbb{E}}[z_i^{(L+1)}{\partial }_i{f}_{t,h}({\mathbf{z}}^{(L+1)})\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_L]-{\displaystyle \sum _{i,j=1}^{n_{L+1}}{\mathrm{\Sigma }}_{n_{L+1}}}(i,j)\mathbb{E}[{\partial }_{ij}^2{f}_{t,h}({\mathbf{z}}^{(L+1)})\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_L].\end{array}$$(45)

By Lemma 2.8(i), for any $t\in (0,1)$, the mapping $\mathbf{y}\mapsto {\partial }_i{f}_{t,h}(\mathbf{y})$ is in $C^1({\mathbb{R}}^{n_{L+1}})$ and has bounded first-order derivatives. Then, because ${\mathbf{z}}^{(L+1)}|{\mathcal{F}}_L$ is a centered Gaussian random vector with covariance matrix

$$\mathrm{\Sigma }{\prime }_{n_{L+1}}≔(C_b+C_W{\mathcal{O}}_{{\mathbf{n}}_L}^{(L)}){\text{Id}}_{n_{L+1}},$$(46)

by Lemma 2.10(ii), we have

$$\mathbb{E}[z_i^{(L+1)}{\partial }_i{f}_{t,h}({\mathbf{z}}^{(L+1)})|{\mathcal{F}}_L]={\displaystyle \sum _{j=1}^{n_{L+1}}\mathrm{\Sigma }}{\prime }_{n_{L+1}}(i,j)\mathbb{E}[{\partial }_{ij}^2{f}_{t,h}({\mathbf{z}}^{(L+1)})|{\mathcal{F}}_L].$$

On combining this relation with (45) and taking the expectation, we have

$$\begin{array}{l}\mathbb{E}[h_t({\mathbf{z}}^{(L+1)})-h_t(\mathbf{z})]={\displaystyle \sum _{i,j=1}^{n_{L+1}}\mathbb{E}}[(\mathrm{\Sigma }{\prime }_{n_{L+1}}(i,j)-{\mathrm{\Sigma }}_{n_{L+1}}(i,j))\mathbb{E}[{\partial }_{ij}^2{f}_{t,h}({\mathbf{z}}^{(L+1)})\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_L]]\\ ={\displaystyle \sum _{i,j=1}^{n_{L+1}}\mathbb{E}}[\mathbb{E}[(\mathrm{\Sigma }{\prime }_{n_{L+1}}(i,j)-{\mathrm{\Sigma }}_{n_{L+1}}(i,j)){\partial }_{ij}^2{f}_{t,h}({\mathbf{z}}^{(L+1)})\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_L]]\\ =\mathbb{E}{〈\mathrm{\Sigma }{\prime }_{n_{L+1}}-{\mathrm{\Sigma }}_{n_{L+1}},\text{Hess}(f_{t,h}({\mathbf{z}}^{(L+1)}))〉}_{H.S.}\end{array}$$(47)

where in the second equality we used the ${\mathcal{F}}_L$-measurability of ${\mathcal{O}}_{{\mathbf{n}}_L}^{(L)}$. Taking the modulus on the relation (47) and applying the Cauchy-Schwarz inequality, we have

$$|\mathbb{E}[h_t({\mathbf{z}}^{(L+1)})-h_t(\mathbf{z})]|\le \sqrt{\mathbb{E}\Vert \mathrm{\Sigma }{\prime }_{n_{L+1}}-{\mathrm{\Sigma }}_{n_{L+1}}{\Vert }_{H.S.}^2}\sqrt{\mathbb{E}\Vert \text{Hess}(f_{t,h}({\mathbf{z}}^{(L+1)})){\Vert }_{H.S.}^2}.$$(48)

By Lemma 2.8(ii), for any $h\in {\mathcal{I}}_{n_{L+1}}$, we have

$$\mathbb{E}\Vert \text{Hess}(f_{t,h}({\mathbf{z}}^{(L+1)})){\Vert }_{H.S.}^2\le \Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^2(n_{L+1}^2{(\log \hspace{0.17em}t)}^2\kappa +530n_{L+1}^{17/6}),\hspace{1em}t\in (0,1).$$

Moreover,

$$\mathbb{E}\Vert \mathrm{\Sigma }{\prime }_{n_{L+1}}-{\mathrm{\Sigma }}_{n_{L+1}}{\Vert }_{H.S.}^2=n_{L+1}{C}_W^2\Vert {\mathcal{O}}_{{\mathbf{n}}_L}^{(L)}-{\mathcal{O}}^{(L)}{\Vert }_{L^2}^2.$$(49)

On combining these relations and using the elementary inequality $\sqrt{a_1+a_2}\le {\sqrt{a}}_1+{\sqrt{a}}_2,\hspace{0.17em}{a}_1,a_2\ge 0$, we have

$$\begin{array}{l}\underset{C\in {\mathcal{C}}_{n_{L+1}}}{\sup }|\mathbb{P}(\sqrt{t}\mathbf{z}+\sqrt{1-t}{\mathbf{z}}^{(L+1)}\in C)-\mathbb{P}(\sqrt{t}\mathbf{z}+\sqrt{1-t}\mathbf{z}\in C)|\\ \hspace{1em}\hspace{1em}\le C_W\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}(n_{L+1}^{3/2}|\log \hspace{0.17em}t|\sqrt{\kappa }+24n_{L+1}^{23/12})\Vert {\mathcal{O}}_{{\mathbf{n}}_L}^{(L)}-{\mathcal{O}}^{(L)}{\Vert }_{L^2}.\end{array}$$

On combining this latter inequality with (44), we have

$$\kappa \le \frac{4}{3}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}(n_{L+1}^{3/2}|\log \hspace{0.17em}t|\sqrt{\kappa }+24n_{L+1}^{23/12})\gamma +\frac{20n_{L+1}}{\sqrt{2}}\frac{\sqrt{t}}{1-t},\hspace{1em}t\in (0,1).$$(50)

Because $\kappa \le 1$, by this relation, we have

$$\kappa \le \frac{4}{3}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}(n_{L+1}^{3/2}|\log \hspace{0.17em}t|+24n_{L+1}^{23/12})\gamma +\frac{20n_{L+1}}{\sqrt{2}}\frac{\sqrt{t}}{1-t},\hspace{1em}t\in (0,1).$$

Setting $t={\gamma }^2$ in this latter inequality (note that this choice of the parameter t is admissible because $\gamma \le 1/\text{e}<1$), we have

$$\begin{array}{l}\kappa \le \frac{4}{3}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}(2n_{L+1}^{3/2}|\log \hspace{0.17em}\gamma |+24n_{L+1}^{23/12})\gamma +\frac{20n_{L+1}}{\sqrt{2}}\frac{\gamma }{1-{\gamma }^2}\\ \le \frac{4}{3}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}(2n_{L+1}^{3/2}|\log \hspace{0.17em}\gamma |+24n_{L+1}^{23/12})\gamma +20\sqrt{2}{n}_{L+1}\gamma ,\end{array}$$(51)

where in the latter inequality we used the relation

$$\frac{1}{\sqrt{2}}\frac{\gamma }{1-{\gamma }^2}\le \sqrt{2}\gamma ,$$(52)

which holds because $\gamma \le 1/\text{e}<1/\sqrt{2}$. We rewrite the inequality (51) as

$$\kappa \le \frac{8}{3}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}{n}_{L+1}^{3/2}\gamma |\log \hspace{0.17em}\gamma |+(32n_{L+1}^{23/12}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}+20\sqrt{2}{n}_{L+1})\gamma .$$

Taking the square root and multiplying by $|\log \hspace{0.17em}\gamma |$, we have

$$\begin{array}{l}|\log \hspace{0.17em}\gamma |\sqrt{\kappa }\le \sqrt{\frac{8}{3}}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{1/2}{n}_{L+1}^{3/4}{\gamma }^{1/2}|\log \hspace{0.17em}\gamma {|}^{3/2}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{(32n_{L+1}^{23/12}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}+20\sqrt{2}{n}_{L+1})}^{1/2}{\gamma }^{1/2}|\log \hspace{0.17em}\gamma |.\end{array}$$

Because $\max \{{\sup }_{y\in (0,1/\text{e}]}{y}^{1/2}|\log \hspace{0.17em}y{|}^{3/2},{\sup }_{y\in (0,1/\text{e}]}{y}^{1/2}|\log \hspace{0.17em}y{|}^{1/2}\}\le 4$, we have

$$\begin{array}{l}|\log \hspace{0.17em}\gamma |\sqrt{\kappa }\le 4\sqrt{\frac{8}{3}}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{1/2}{n}_{L+1}^{3/4}+4{(32n_{L+1}^{23/12}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}+20\sqrt{2}{n}_{L+1})}^{1/2}\\ =8\frac{\sqrt{6}}{3}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{1/2}{n}_{L+1}^{3/4}+4{(32n_{L+1}^{23/12}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}+20\sqrt{2}{n}_{L+1})}^{1/2}\\ \le 8\frac{\sqrt{6}}{3}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{1/2}{n}_{L+1}^{3/4}+16\sqrt{2}{n}_{L+1}^{23/24}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{1/2}+\sqrt{20\sqrt{2}}{n}_{L+1}^{1/2}\\ \le \left[\left(8\frac{\sqrt{6}}{3}+16\sqrt{2}\right)\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{1/2}+\sqrt{20\sqrt{2}}\right]n_{L+1}^{23/24}\\ \le (30\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{1/2}+6)n_{L+1}^{23/24}.\end{array}$$(53)

By (50) with $t={\gamma }^2$, (52), and (53), we finally have (41); indeed,

$$\begin{array}{l}\kappa \le \frac{4}{3}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}(2n_{L+1}^{3/2}|\log \hspace{0.17em}\gamma |\sqrt{\kappa }+24n_{L+1}^{23/12})\gamma +20\sqrt{2}{n}_{L+1}\gamma \\ \le \frac{4}{3}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}\left[\left(60\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{1/2}+12\right)n_{L+1}^{59/24}+24n_{L+1}^{23/12}\right]\gamma +20\sqrt{2}{n}_{L+1}\gamma \\ \le (80\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}^{3/2}+48\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}+20\sqrt{2})n_{L+1}^{59/24}\gamma .\end{array}$$

The proof is completed. □

6.2. Normal Approximation of Deep Random Gaussian NNs in the 1-Wasserstein Distance

The following theorem holds.

Theorem 6.3.

Let $\text{GNN}(L,n_0,n_{L+1},{\mathbf{n}}_L,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$ be a deep random Gaussian NN, and let the notation and assumptions of Theorem 5.1 prevail. Then,

$$\begin{array}{l}{d}_{W_1}({\mathbf{z}}^{(L+1)},\mathbf{z})\le K_1\left({\displaystyle \sum _{k=1}^L}{[4\sqrt{2}\Vert P(|Z|){\Vert }_{L^2}]}^{L-k}\frac{c_k}{\sqrt{n_k}}\right)\\ \le K_i\left({\displaystyle \sum _{k=1}^L}{[4\sqrt{2}\Vert P(|Z|){\Vert }_{L^2}]}^{L-k}\frac{c_k}{\sqrt{n_k}}\right),\hspace{1em}i=2,3\end{array}$$

where the constants c_k, $k=1,\dots ,L$, are defined by (23),

$$\begin{array}{l}{K}_1=K_1(n_0,n_{L+1},\mathbf{x},\sigma ,C_b,C_W)≔\frac{n_{L+1}{C}_W}{{\mathfrak{s}}_L},\\ K_2=K_2(n_{L+1},C_b,C_W)≔\frac{n_{L+1}{C}_W}{\sqrt{C_b}}\end{array}$$

and

$$K_3=K_3(n_0,n_{L+1},\mathbf{x},\sigma ,C_W)≔\frac{n_{L+1}\sqrt{C_W}}{\sqrt{{\mathcal{O}}^{(L)}}}.$$

Here again, the upper estimate with the constant K₂ holds if $C_b>0$.

Remark 6.4.

Let $\text{GNN}(L,n_0,1,{\mathbf{n}}_L,\sigma ,\mathbf{x},\mathbf{b},\mathbf{W})$ be a deep random Gassian NN with univariate output, widths of the hidden layers satisfying (40), and a polynomially bounded to order $r\ge 1$ activation function σ. Then, by theorem 3.3 in Favaro et al. (2023), we have that there exist two constants $C,C_0>0$ such that

$$\frac{C_0}{n}\le d_{W_1}(z^{(L+1)},z)\le \frac{C}{n}.$$

Clearly, this inequality shows the optimality of the rate $1/n$. Here again, we note that the corresponding bound provided by Theorem 6.3 gives (under different assumptions on σ and without any condition on the widths of the hidden layers) a detailed description of the analytical dependence of the upper estimate on the parameters of the model.

Proof.

Let $g\in {\mathcal{L}}_{n_{L+1}}(1)$ be arbitrarily fixed. Without loss of generality, we assume that $\mathbf{z}$ is independent of ${\mathcal{F}}_L$. By Lemma 2.7, we then have

$$\begin{array}{l}\mathbb{E}[g({\mathbf{z}}^{(L+1)})-g(\mathbf{z})\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_L]={\displaystyle \sum _{i=1}^{n_{L+1}}\mathbb{E}}[z_i^{(L+1)}{\partial }_i{f}_g({\mathbf{z}}^{(L+1)})\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_L]-{\displaystyle \sum _{i,j=1}^{n_{L+1}}{\mathrm{\Sigma }}_{n_{L+1}}}(i,j)\mathbb{E}[{\partial }_{ij}^2{f}_g({\mathbf{z}}^{(L+1)})\hspace{0.17em}|\hspace{0.17em}{\mathcal{F}}_L],\end{array}$$(54)

where

$$f_g(\mathbf{y})≔{\displaystyle {\int }_0^{\infty }\mathbb{E}}[g(\mathbf{z})-g({\text{e}}^{-t}\mathbf{y}+\sqrt{1-{\text{e}}^{-2t}}\mathbf{z})]\hspace{0.17em}\text{d}t,\hspace{1em}\mathbf{y}\in {\mathbb{R}}^{n_{L+1}}.$$

Again, by Lemma 2.7, we have that the mapping $\mathbf{y}\mapsto {\partial }_i{f}_g(\mathbf{y})$ is in $C^1({\mathbb{R}}^{n_{L+1}})$ and has bounded first-order derivatives. Applying Lemma 2.10(ii) exactly as in the proof of Theorem 6.1 (see a few lines before Equation (47)), we have

$$\mathbb{E}[g({\mathbf{z}}^{(L+1)})-g(\mathbf{z})]=\mathbb{E}{〈\mathrm{\Sigma }{\prime }_{n_{L+1}}-{\mathrm{\Sigma }}_{n_{L+1}},\text{Hess}(f_g({\mathbf{z}}^{(L+1)}))〉}_{H.S.},$$

where the matrix $\mathrm{\Sigma }{\prime }_{n_{L+1}}$ is defined by (46). Therefore, applying the Cauchy-Schwarz inequality as in (48), we have

$$|\mathbb{E}[g({\mathbf{z}}^{(L+1)})-g(\mathbf{z})]|\le \sqrt{\mathbb{E}\Vert \mathrm{\Sigma }{\prime }_{n_{L+1}}-{\mathrm{\Sigma }}_{n_{L+1}}{\Vert }_{H.S.}^2}\sqrt{\mathbb{E}\Vert \text{Hess}(f_g({\mathbf{z}}^{(L+1)})){\Vert }_{H.S.}^2}$$(55)

By Lemma 2.7, we have

$$\sqrt{\mathbb{E}\Vert \text{Hess}(f_g({\mathbf{z}}^{(L+1)})){\Vert }_{H.S.}^2}\le \underset{\mathbf{y}\in {\mathbb{R}}^{n_{L+1}}}{\sup }\Vert \text{Hess}{f}_g(\mathbf{y}){\Vert }_{H.S.}\le \sqrt{n_{L+1}}\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}\Vert {\mathrm{\Sigma }}_{n_{L+1}}{\Vert }_{op}^{1/2},$$

On combining these relations with (49), we have

$$|\mathbb{E}[g({\mathbf{z}}^{(L+1)})-g(\mathbf{z})]|\le n_{L+1}{C}_W\Vert {\mathrm{\Sigma }}_{n_{L+1}}^{-1}{\Vert }_{op}\Vert {\mathrm{\Sigma }}_{n_{L+1}}{\Vert }_{op}^{1/2}\Vert {\mathcal{O}}_{{\mathbf{n}}_L}^{(L)}-{\mathcal{O}}^{(L)}{\Vert }_{L^2}.$$

The claim follows taking the supremum over g on this inequality and then using Theorem 5.1, relation (42), and the fact that $\Vert {\mathrm{\Sigma }}_{n_{L+1}}{\Vert }_{op}={\mathfrak{s}}_L^2$. □

7. Localization of the Output

In this section, we want to show the potentiality of the obtained results for practical applications. Indeed, both Theorem 4.1(ii) and Theorem 6.1 allow one to explicitly estimate the probability that the output of a random Gaussian NN evaluated at the input $\mathbf{x}$ belongs to a suitable set without resorting to computationally expensive Monte Carlo simulations. This is what we call output localization, which can suggest a suitable architecture design to estimate a target function f. Indeed, in statistical learning, given a training set

the goal is to estimate the unknown function f by a NN with a certain fixed architecture. This is performed by minimizing an empirical risk function over the space of NN’s parameters, that is, the biases and the weights. Such kinds of minimization problems are nonconvex and are usually addressed by a gradient descent procedure; the latter stabilizes toward a solution that depends on the initialization point. Because in practical applications the initialization point is given by a realization of a random Gaussian NN, it can be convenient to choose L, ${\mathbf{n}}_L$, σ, C_b, and C_W by exploiting output localization, that is, in such a way to have a good estimate of the probability $\mathbb{P}({\mathbf{z}}^{(L+1)}({\mathbf{x}}_i)\in V_i),\hspace{0.17em}i\in I$, where V_i is an appropriate neighborhood of $f({\mathbf{x}}_i)$.

Hence, in the following we want to show how to use the upper bound on the total variation distance provided in Theorem 4.1(ii), for a shallow random Gaussian NN with univariate output, and the upper bound on the convex distance provided by Theorem 6.1, for a deep random Gaussian NN, to localize the output. Indeed, given a measurable convex set $V\subset {\mathbb{R}}^{n_{L+1}}$, by the definitions of both the total variation and the convex distances, we have

where L = 1, $n_2=1$, andfor the case of a shallow NN, whereasfor the case of a deep NN, where C₁ is given by (39) and the constants c_k, $k=1,\dots ,L$, are defined by (23).

Let $V≔{\displaystyle {\prod }_{i=1}^{n_{L+1}}}[r_i,s_i]$ be a rectangle of ${\mathbb{R}}^{n_{L+1}}$. Because $\mathbf{z}$ is an $n_{L+1}$-dimensional centered Gaussian random vector with covariance matrix (12), we have

Now, we furnish numerical values of the constant C_bound in (56) and (57); in both cases, we take $\sigma (x)≔x\mathbf{1}\{x\ge 0\}$, that is, a ReLU activation function. Because the ReLU function is Lipschitz continuous with Lipschitz constant equal to 1, $\mathbb{E}{Z}^2=1$ and $\mathbb{E}{Z}^4=3$, by Proposition 5.2(ii), we have

By the expression of the constants $c_{\mathcal{\ell }}$ in Theorem 5.1, we have

By the definition of the quantities ${\mathcal{O}}^{(\mathcal{\ell })},\hspace{0.17em}\mathcal{\ell }=1,\dots ,L$, (see (10)), we have

with ${\mathcal{O}}^{(0)}$ given by (11). Therefore,

Table 2 gives the values of C_bound in (56) in the case of a shallow random Gaussian NN with architecture L = 1, $n_0=4,\hspace{0.17em}{n}_1=n$, and $n_2=1$ for different values of $n\in \{1,10,{10}^2,{10}^3,{10}^4,{10}^5\}$. Table 3 gives the values of C_bound in (57) in the case of a deep random Gaussian NN with architecture L = 3, $n_0=4,\hspace{0.17em}{n}_1=n_2=n_3=n$, and $n_4=1$, for different values of $n\in \{{10}^4,{10}^5,{10}^6,{10}^7,{10}^8,{10}^9\}$. In both tables, we consider four different inputs,

Table 2. Values of Cbound in (56) for Different Inputs $\mathbf{x}$ for a Shallow Random Gaussian NN with Architecture L = 1, $n_0=4,\hspace{0.17em}{n}_1=n,\hspace{0.17em}{n}_2=1$, and ReLU Activation Function

Table 3. Values of Cbound in (57) for Different Inputs $\mathbf{x}$ for a Deep Random Gaussian NN with Architecture L = 3, $n_0=4,\hspace{0.17em}{n}_1=n_2=n_3=n,\hspace{0.17em}{n}_4=1$, and ReLU Activation Function

whose Euclidean norm is, respectively, 0, strictly less than 1, 1 and strictly larger than 1, and

It is clear from Table 2 that for a shallow random Gaussian NN, the Gaussian approximation of the output is very good for any of the choices of the parameters n, C_b, and C_W and of the input $\mathbf{x}$. It is also clear from Table 3 that for a deep random Gaussian NN, such as the considered NN with three hidden layers, the Gaussian approximation of the output is very good for some choices of the parameters C_b, C_W, and n and of the input $\mathbf{x}$ (also when the number n of neurons in the hidden layers is not excessively large), but it is very poor for other choices of these quantities. In order to analyze the influence of the parameters C_b and C_W and of the input $\mathbf{x}$ on the value of C_bound in (57), in Table 4 we reported the value of the constant C₁ that appears in (57), and it is explicitly given in (39). As it can be seen from this table, the value of C_bound strongly depends on the value of C₁, which, in turn, is closely related to the choice of the parameters C_b and C_W and does not depend much on the norm of the input vector $\mathbf{x}$.

Table 4. Values of C1 in (39) for Different Inputs $\mathbf{x}$ for a Deep Random Gaussian NN with Architecture L = 3, $n_0=4,\hspace{0.17em}{n}_1=n_2=n_3=n,\hspace{0.17em}{n}_4=1$, and ReLU Activation Function