Random vectors

Introduction

Previously we saw how we could grasp this notion of uncertainty in probability functions and we saw what happened when our outcome was conditioned. Finally, a brief discussion was provided for the case when multiple random variables were involved. In these examples, only the situation where we had 2 random variables was discussed. This module will generalize this theory for multiple random variables. In this case an outcome of an experiment comprises N$N$ observed quantities. An example of such an observation is a noisy electroencephalography (EEG) measurement, which resembles the electrical activity of the brain and is measured over several channels.

Screencast video [⯈]

Multivariate joint probability distributions

Let us denote each of these measured quantities by the random variable Xn$X_n$, where n$n$ ranges from 1$1$ up to N$N$. Using this definition, the multivariate (meaning that multiple variables are involved) joint probability functions can be introduced. For notation purposes, all random variables Xn$X_n$ can be grouped in a random vector X=[X1,X2,…,XN]⊤$X=[X_1,X_2,\dots ,X_N{]}^{\top }$, where the ⋅⊤${\cdot }^{\top }$ operator denotes the transpose, turning this row vector into a column vector. The bold capital letter distinguishes the random vector containing multiple random variables from a single random variable. Similarly, a specific realization of this random vector can be written in lower-case as x=[x1,x2,…,xN]⊤$x=[x_1,x_2,\dots ,x_N{]}^{\top }$.

Multivariate joint cumulative distribution function

The multivariate joint cumulative distribution function of the random vector X$X$ containing random variables X1,X2,…,XN$X_1,X_2,\dots ,X_N$ is defined as PX(x)=PX1,…,XN(x1,…,xN)=Pr[X1≤x1,…,XN≤xN].$$\begin{array}{}{P}_X\left(x\right)=P_{X_1,\dots ,X_N}(x_1,\dots ,x_N)=Pr[X_1\le x_1,\dots ,X_N\le x_N].& \text{(1)}\end{array}$$ This definition holds for both discrete as continuous random variables.

Multivariate joint probability mass function

The multivariate joint probability mass function of the random vector X$X$ containing discrete random variables X1,X2,…,XN$X_1,X_2,\dots ,X_N$ is similarly defined as pX(x)=pX1,…,XN(x1,…,xN)=Pr[X1=x1,…,XN=xN].$$\begin{array}{}{p}_X\left(x\right)=p_{X_1,\dots ,X_N}(x_1,\dots ,x_N)=Pr[X_1=x_1,\dots ,X_N=x_N].& \text{(2)}\end{array}$$

Multivariate joint probability density function

The multivariate joint probability density function of the random vector X$X$ containing continuous random variables X1,X2,…,XN$X_1,X_2,\dots ,X_N$ is defined from the multivariate joint cumulative distribution function as pX(x)=pX1,…,XN(x1,…,xN)=∂NPX1,…,XN(x1,…,xN)∂x1…∂xN.$$\begin{array}{}{p}_X\left(x\right)=p_{X_1,\dots ,X_N}(x_1,\dots ,x_N)=\frac{{\partial }^N{P}_{X_1,\dots ,X_N}(x_1,\dots ,x_N)}{\partial x_1\dots \partial x_N}.& \text{(3)}\end{array}$$

Generalized probability axioms for multivariate joint distributions

From these definitions several multivariate joint probability axioms can be determined, which are similar to the case of two random variables as discussed in the last reader.

1. It holds that pX(x)≥0$p_X\left(x\right)\ge 0$, where X$X$ is a continuous or discrete random vector.
2. From the multivariate joint probability density function it follows that PX1,…,XN(x1,…,xN)=∫x1−∞⋯∫xN−∞pX1,…,XN(x1,…,xN) dx1…dxN$$P_{X_1,\dots ,X_N}(x_1,\dots ,x_N)={\int }_{-\infty }^{x_1}\cdots {\int }_{-\infty }^{x_N}{p}_{X_1,\dots ,X_N}(x_1,\dots ,x_N)d{x}_1\dots d{x}_N$$ holds for continuous random vectors.
3. Through the law of total probability it holds that
   1. ∑x1∈SX1⋯∑xN∈SXNpX1,…,XN(x1,…,xN)=1$$\sum _{x_1\in S_{X_1}}\cdots \sum _{x_N\in S_{X_N}}{p}_{X_1,\dots ,X_N}(x_1,\dots ,x_N)=1$$ for discrete random vectors and
   2. ∫∞−∞⋯∫∞−∞pX1,…,XN(x1,…,xN) dxN…dx1=1$${\int }_{-\infty }^{\infty }\cdots {\int }_{-\infty }^{\infty }{p}_{X_1,\dots ,X_N}(x_1,\dots ,x_N)d{x}_N\dots d{x}_1=1$$ for continuous random vectors.
4. The probability of an event A$A$ can be determined as
   1. Pr[A]=∑X∈SpX1,…,XN(x1,…,xN)$$Pr\left[A\right]=\sum _{X\in S}{p}_{X_1,\dots ,X_N}(x_1,\dots ,x_N)$$ for discrete random variables and
   2. Pr[A]=∫⋯∫ApX1,…,XN(x1,…,xN) dx1…dxN$$Pr\left[A\right]=\underset{A}{\int \cdots \int }{p}_{X_1,\dots ,X_N}(x_1,\dots ,x_N)d{x}_1\dots d{x}_N$$ for continuous random variables.

Axiom 1 simply states that a probability (density) cannot be smaller than 0, since no negative probabilities exist by definition. The second axiom is a direct consequence of integrating both sides of the multivariate joint probability density function allowing us to determine the multivariate joint cumulative distribution function from the multivariate joint probability density function. The third axiom is a direct consequence of the law of total probability, where the probability of all events together equal 1. The final axiom tells us to sum or integrate over all possible outcomes of an event A$A$ in order to calculate its probability.

Probability distributions of multiple random vectors

The notation of a random vector allows us to easily include multiple random variables in a single vector. Suppose now that our random vector Z$Z$ contains 2 different types of random variables, where for example each random variable corresponds to a different type of measurement. If we were to distinguish between these type of random variables using two generalized random variables Xi$X_i$ and Yi$Y_i$, the random vector Z$Z$ could be written as Z=[X1,X2,…,XN,Y1,Y2,…,YM]⊤$Z=[X_1,X_2,\dots ,X_N,Y_1,Y_2,\dots ,Y_M{]}^{\top }$. If we now were to define the random vectors X=[X1,X2,…,XN]⊤$X=[X_1,X_2,\dots ,X_N{]}^{\top }$ and Y=[Y1,Y2,…,YM]⊤$Y=[Y_1,Y_2,\dots ,Y_M{]}^{\top }$, it becomes evident that we could simplify the random vector Z$Z$ as Z=[X⊤,Y⊤]⊤$Z=[X^{\top },Y^{\top }{]}^{\top }$.

This shows that it is also possible for joint probability distributions to depend on multiple random vectors, which each can be regarded as a subset of all random variables. This can prove useful in some cases when there is a clear distinction between the subsets and is purely for notation purposes. A probability distribution depending on multiple random vectors can be regarded in all aspects as a probability distribution depending on a single random vector (which is a concatenation of all different random variables). All calculations can be performed by regarding the probability distribution as if it depends on a single (concatenated) random vector. A probability distribution involving multiple random vectors can be written for example as pX,Y(x,y)$p_{X,Y}(x,y)$.

Conditional probabilities

Similarly as in the previous reader, the conditional probability can be determined by normalizing the joint probability with the probability of the conditional event through

pX|B(x)={pX(x)Pr[B],when x∈B0.otherwise$$\begin{array}{}{p}_{X|B}\left(x\right)=\{\begin{array}{cc}\frac{p_X\left(x\right)}{Pr\left[B\right]},& \text{when}x\in B\\ 0.& \text{otherwise}\end{array}& \text{(4)}\end{array}$$

Marginal probabilities

Because the notation of a random vector X$X$ is just a shorter notation for the set of random variables X1,X2,…,XN$X_1,X_2,\dots ,X_N$, it is possible to calculate the marginalized probability distribution of a subset of random variables. This subset can also just consist of a single random variable. Again this operation is performed through marginalization as discussed in the previous reader. For the case that we are given the probability distribution pX(x)$p_X\left(x\right)$ and we would like to know the marginalized probability distribution pX2,X3(x2,x3)$p_{X_2,X_3}(x_2,x_3)$ this can be calculated as pX2,X3(x2,x3)=∫∞−∞⋯∫∞−∞pX(x) dx1dx4⋯dxN$$\begin{array}{}{p}_{X_2,X_3}(x_2,x_3)={\int }_{-\infty }^{\infty }\cdots {\int }_{-\infty }^{\infty }{p}_X\left(x\right)d{x}_{1}d{x}_4\cdots d{x}_N& \text{(5)}\end{array}$$ for continuous random variables and as pX2,X3=∑x1∈SX1∑x4∈SX4⋯∑xN∈SXNpX(x)$$\begin{array}{}{p}_{X_2,X_3}=\sum _{x_1\in S_{X_1}}\sum _{x_4\in S_{X_4}}\cdots \sum _{x_N\in S_{X_N}}{p}_X\left(x\right)& \text{(6)}\end{array}$$ for discrete random variables. Here we have integrated or summed over all possible values of all random variables except for the ones that we are interested in.

Independence

Independence is a term in probability theory which reflects that the probability of an event A$A$ is not changed after observing an event B$B$, meaning that Pr[A|B]=Pr[A]$Pr\left[A|B\right]=Pr\left[A\right]$. In other words, the occurrence of an event B$B$ has no influence on the probability of an event A$A$. Keep in mind that this does not mean that the physical occurrence of event A$A$ and B$B$ are unrelated, it just means that the probability of the occurrence of event A$A$ is unrelated to whether event B$B$ occurs or not.

Independent random variables

This notion of independence can be extended to probability functions. The random variables X1,X2,…,XN$X_1,X_2,\dots ,X_N$ can be regarded as independent if and only if the following factorization holds pX1,X2,…,XN(x1,x2,…,xN)=pX1(x1)pX2(x2)…pXN(xN).$$\begin{array}{}{p}_{X_1,X_2,\dots ,X_N}(x_1,x_2,\dots ,x_N)=p_{X_1}\left(x_1\right)p_{X_2}\left(x_2\right)\dots p_{X_N}\left(x_N\right).& \text{(7)}\end{array}$$ This equation represents that the total probability can be written a multiplication of the individual probabilities of the random variables. From a probability point of view (not a physical one) we can conclude that the random variables are independent, because the total probability solely depends on all the individual contributions of the random variables. Random variables that satisfy the independence equation and are distributed under the same probability density function are regarded as independent and identically distributed (IID or iid) random variables. This notion will become important later on when discussing random signals.

Independent random vectors

It is also possible to extend the definition of independence to random vectors. Two random vectors X$X$ and Y$Y$ can be regarded as independent if and only if the probability function can be written as pX,Y(x,y)=pX(x) pY(y).$$\begin{array}{}{p}_{X,Y}(x,y)=p_X\left(x\right)p_Y\left(y\right).& \text{(8)}\end{array}$$

Statistical characterization of random vectors

In the previous reader on random variables, several characteristics were discussed for probability distributions depending on a single random variable, such as the mean, the variance and the moments of a random variable. This section will extend these characterization to random vectors.

Expected value

The expected value of a random vector X$X$ is defined as the vector containing the expected values of the individual random variables X1,X2,…,XN$X_1,X_2,\dots ,X_N$ as E[X]=μX=[μ1,μ2,…,μN]⊤=[E[X1],E[X2],…,E[XN]]⊤.$$\begin{array}{}E\left[X\right]={\mu }_X=[{\mu }_1,{\mu }_2,\dots ,{\mu }_N{]}^{\top }={\left[E\right[X_1],E[X_2],\dots ,E[X_N\left]\right]}^{\top }.& \text{(9)}\end{array}$$

Expected value of a function

When we are interested in the expected value of a certain function g(X)$g\left(X\right)$, which accepts a random vector as argument and transforms it to a single value, this can be determined by multiplying the functions result with its corresponding probability and summing or integrating over all possible realizations of X$X$. For a discrete random vector X$X$ consisting of random variables X1,X2,…,XN$X_1,X_2,\dots ,X_N$ the expected value of a function g(X)$g\left(X\right)$ can be determined as E[g(X)]=∑x1∈SX1⋯∑xN∈SXNg(x)⋅pX(x)$$\begin{array}{}E\left[g\right(X\left)\right]=\sum _{x_1\in S_{X_1}}\cdots \sum _{x_N\in S_{X_N}}g\left(x\right)\cdot p_X\left(x\right)& \text{(10)}\end{array}$$ and for a continuous random vector as E[g(X)]=∫∞−∞⋯∫∞−∞g(x)⋅pX(x) dx1…dxN.$$\begin{array}{}E\left[g\right(X\left)\right]={\int }_{-\infty }^{\infty }\cdots {\int }_{-\infty }^{\infty }g\left(x\right)\cdot p_X\left(x\right)d{x}_1\dots d{x}_N.& \text{(11)}\end{array}$$

Covariance

Previously, we have introduced the second centralized moment of a univariate random variable as the variance. While the variance denotes the spread in the univariate case, it has a different meaning in the multivariate case. Have a look at Fig. 1, where two contour plots are shown for two distinct multivariate Gaussian probability density functions. The exact mathematical description of a multivariate Gaussian probability density function is introduced here.

It can be noted that the distributions in Fig. 1 are different, since the first contour plot shows clear circles and the second contour plot shows tilted ellipses. For both distributions, 10000 random realizations x=[x1,x2]⊤$x=[x_1,x_2{]}^{\top }$ are generated and the marginal distributions of both X1$X_1$ and X2$X_2$ are shown using a histogram and a fitted Gaussian distribution. It can be seen that the individual distributions of X1$X_1$ and X2$X_2$ are exactly equal for both multivariate distributions, but still the multivariate distributions are different. This difference can be explained by the covariance between the random variables X1$X_1$ and X2$X_2$ of the random vector X$X$.

The covariance is a measure of the relationship between 2 random variables. The second distribution in Fig. 1 has a negative covariance between X1$X_1$ and X2$X_2$, because if X1$X_1$ increases X2$X_2$ decreases. No such thing can be said about the first distribution in Fig. 1, where X1$X_1$ and X2$X_2$ seem to have no relationship and behave independently from each other.

The formal definition of the covariance between two random variables X1$X_1$ and X2$X_2$ is given by Cov[X1,X2]=E[(X1−μX1)(X2−μX2)],$$\begin{array}{}\text{Cov}[X_1,X_2]=E\left[\right(X_1-{\mu }_{X_1}\left)\right(X_2-{\mu }_{X_2}\left)\right],& \text{(12)}\end{array}$$ which is very similar to the definition of the variance, even to such an extent that it actually represents the variance if X1=X2$X_1=X_2$. Intuitively, one might regard the covariance as the expected value of the multiplication of X1$X_1$ and X2$X_2$ from which the means are subtracted. If both the normalized X1$X_1$ and X2$X_2$ have the same sign, then their multiplication would be positive and if X1$X_1$ and X2$X_2$ have different signs, then their multiplication would be negative. The covariance may therefore be regarded as a measure to indicate how X2$X_2$ behaves if X1$X_1$ increases or decreases.

¹

Correlation

The definition of the covariance can be rewritten as Cov[X1,X2]=E[(X1−μX1)(X2−μX2)],=E[X1X2−μX1X2−μX2X1+μX1μX2],=E[X1X2]−μX1E[X2]−μX2E[X1]+μX1μX2,=E[X1X2]−μX1μX2−μX1μX2+μX1μX2,=E[X1X2]−μX1μX2.$$\begin{array}{}\begin{array}{cc}\text{Cov}[X_1,X_2]& =E\left[\right(X_1-{\mu }_{X_1}\left)\right(X_2-{\mu }_{X_2}\left)\right],\\ & =E[X_1{X}_2-{\mu }_{X_1}{X}_2-{\mu }_{X_2}{X}_1+{\mu }_{X_1}{\mu }_{X_2}],\\ & =E\left[X_1{X}_2\right]-{\mu }_{X_1}E\left[X_2\right]-{\mu }_{X_2}E\left[X_1\right]+{\mu }_{X_1}{\mu }_{X_2},\\ & =E\left[X_1{X}_2\right]-{\mu }_{X_1}{\mu }_{X_2}-{\mu }_{X_1}{\mu }_{X_2}+{\mu }_{X_1}{\mu }_{X_2},\\ & =E\left[X_1{X}_2\right]-{\mu }_{X_1}{\mu }_{X_2}.\end{array}& \text{(13)}\end{array}$$ The term E[X1X2]$E\left[X_1{X}_2\right]$ is called the correlation rX1,X2$r_{X_1,X_2}$ of X1$X_1$ and X2$X_2$ and is defined as rX1,X2=E[X1X2].$$\begin{array}{}{r}_{X_1,X_2}=E\left[X_1{X}_2\right].& \text{(14)}\end{array}$$ This correlation can be regarded as a non-normalized version of the covariance. These two terms are related through Cov[X1,X2]=rX1,X2−μX1μX2.$$\begin{array}{}\text{Cov}[X_1,X_2]=r_{X_1,X_2}-{\mu }_{X_1}{\mu }_{X_2}.& \text{(15)}\end{array}$$ It can be noted that the correlation and covariance of two random variables are equal if the mean values of both random variables are 0$0$.

Uncorrelated random variables

Two random variables are called uncorrelated if the covariance between both random variables equals 0$0$ as Cov[X1,X2]=0.$$\begin{array}{}\text{Cov}[X_1,X_2]=0.& \text{(16)}\end{array}$$ Although the term suggests that it is related to the correlation between two random variables, it is defined as a zero covariance. The former has a different definition.

Orthogonality

Two random variables are called orthogonal if the correlation between both random variables equals 0$0$ as rX1,X2=0.$$\begin{array}{}{r}_{X_1,X_2}=0.& \text{(17)}\end{array}$$

Correlation coefficient

The value of the covariance depends significantly on the variance of both random variables and is therefore unbounded. In order to express the relationship between two random variables without having a dependence on the variances, it first has to be normalized. Therefore the correlation coefficient is introduced as ρX1,X2=Cov[X1,X2]√Var[X1]Var[X2]=Cov[X1,X2]σX1σX2.$$\begin{array}{}{\rho }_{X_1,X_2}=\frac{\text{Cov}[X_1,X_2]}{\sqrt{\text{Var}\left[X_1\right]\text{Var}\left[X_2\right]}}=\frac{\text{Cov}[X_1,X_2]}{{\sigma }_{X_1}{\sigma }_{X_2}}.& \text{(18)}\end{array}$$ Please note that this represents the normalized covariance and not the normalized correlation between two random variables, although the name suggests otherwise. Because of this normalization, the correlation coefficient has the property to be bounded between −1$-1$ and 1$1$ as −1≤ρX1,X2≤1.$$\begin{array}{}-1\le {\rho }_{X_1,X_2}\le 1.& \text{(19)}\end{array}$$ Fig. 2 shows realizations of three different probability distributions with negative, zero and positive correlation coefficients.

Cross-covariance matrix

We previously discussed how we could determine the covariance of two random variables. Let us turn now to the covariance of two random vectors X=[X1,X2,…,XN]⊤$X=[X_1,X_2,\dots ,X_N{]}^{\top }$ and Y=[Y1,Y2,…,YN]⊤$Y=[Y_1,Y_2,\dots ,Y_N{]}^{\top }$. Intuitively, one might say that this covariance cannot be described by a single number, because there are more than one combinations of random variables of which we want to calculate the covariance. As an example, we could determine the covariances of X1$X_1$ and Y1$Y_1$, X1$X_1$ and Y23$Y_{23}$ and XN$X_N$ and Y1$Y_1$. In order to facilitate for all these possible combinations, there is a need to introduce the cross-covariance matrix ΓXY${\Gamma }_{XY}$, which contains the covariance of all the possible combinations of the random variables in random vectors X$X$ and Y$Y$.

The covariance matrix is formally defined as ΓXY=E[(X−μX)(Y−μY)⊤]=[γ11γ12⋯γ1Nγ21γ22⋯γ2N⋮⋮⋱⋮γN1γN2⋯γNN],$$\begin{array}{}{\Gamma }_{XY}=E\left[\right(X-{\mu }_X\left)\right(Y-{\mu }_Y{)}^{\top }]=\left[\begin{array}{cccc}{\gamma }_{11}& {\gamma }_{12}& \cdots & {\gamma }_{1N}\\ {\gamma }_{21}& {\gamma }_{22}& \cdots & {\gamma }_{2N}\\ ⋮& ⋮& \ddots & ⋮\\ {\gamma }_{N1}& {\gamma }_{N2}& \cdots & {\gamma }_{NN}\end{array}\right],& \text{(20)}\end{array}$$ where the individual coefficients correspond to γnm=Cov[Xn,Ym]=E[(Xn−μXn)(Ym−μYm)].$$\begin{array}{}{\gamma }_{nm}=\text{Cov}[X_n,Y_m]=E\left[\right(X_n-{\mu }_{X_n}\left)\right(Y_m-{\mu }_{Y_m}\left)\right].& \text{(21)}\end{array}$$ The transpose operator in the first equation creates a matrix from the two column vectors filled with the covariances of all possible combinations of random variables. For each of these covariances, the correlation coefficient ρnm${\rho }_{nm}$ can be calculated similarly using the definition of the correlation coefficient. Two random vectors are called uncorrelated if ΓXY=0${\Gamma }_{XY}=0$.

Auto-covariance matrix of a random vector

For the special case that X=Y$X=Y$, the cross-covariance matrix is called the auto-covariance matrix, which calculates the covariances between all random variables in X$X$. The definition is the same as the definition of the covariance matrix, where ΓXX${\Gamma }_{XX}$ is often simplified as ΓX${\Gamma }_X$.

Cross-correlation matrix

Similarly to the cross-covariance matrix, the cross-correlation matrix can be defined containing the correlations of all combinations between the random variables in X and Y. The cross-correlation matrix of random vectors X and Y is denoted by RXY and is defined as RXY=E[XY⊤]=[r11r12⋯r1Nr21r22⋯r2N⋮⋮⋱⋮rN1rN2⋯rNN], where the individual coefficients correspond to the individual correlations rnm=E[XnYm]. Two random vectors are called orthogonal if RXY=0. Furthermore it can be proven that the cross-covariance matrix and the cross-covariance matrix are related through ΓXY=RXY−μXμ⊤Y.

Auto-correlation matrix of a random vector

For the special case that X=Y, the cross-correlation matrix is called the auto-correlation matrix, which calculates the correlations between all random variables in X. The definition is the same as the definition of the cross-correlation matrix, where RXX is often simplified as RX.

Linear transformations of random vectors

In the previous reader some calculation rules were determined for the mean and variance of a linearly transformed random variable. This subsection will continue with this line of thought, but now for random vectors. We will define an invertible transformation matrix A, with dimensions (N×N), which will linearly map a random vector X of length N to a random vector Y again with length N after adding an equally long column vector b through Y=g(X)=AX+b.

Probability density function

From the initial multivariate probability density function of X, pX(x), the new probability density function of Y can be determined as pY(y)=pX(g−1(y))|detA|=pX(A−1(y−b))|detA|, where |detA| is the absolute value of the determinant of A.

Mean vector

The new mean vector of the random vector Y can be determined as μY=E[Y]=E[AX+b]=AE[X]+b=AμX+b.

Cross-covariance and cross-correlation matrix

By the definition of the cross-covariance matrix, the cross-covariance matrices ΓXY and ΓYX can be determined from the original auto-covariance matrix ΓX of X through ΓXY=E[(X−μX)(Y−μY)⊤],=E[(X−μX)(AX+b−(AμX+b))⊤],=E[(X−μX)(A(X−μX))⊤],=E[(X−μX)(X−μX)⊤A⊤],=E[(X−μX)(X−μX)⊤]A⊤=ΓXA⊤ and similarly we can find the result ΓYX=AΓX. The new cross-correlation matrices RXY and RYX can be determined as RXY=E[XY⊤],=E[X(AX+b)⊤],=E[X(AX)⊤+Xb⊤],=E[XX⊤A⊤]+E[X]b⊤,=RXA⊤+μXb⊤ and similarly as RYX=ARX+bμ⊤X.

Auto-covariance and auto-correlation matrix

The auto-covariance matrix of Y can be determined through ΓY=E[(Y−μY)(Y−μY)⊤],=E[(AX+b−(AμX+b))(AX+b−(AμX+b))⊤],=E[(A(X−μX))(A(X−μX))⊤],=E[A(X−μX)(X−μX)⊤A⊤],=AE[(X−μX)(X−μX)⊤]A⊤=AΓXA⊤. In a similar fashion the new auto-correlation matrix of Y can be calculated as RY=E[YY⊤],=E[(AX+b)(AX+b)⊤],=E[(AX+b)(X⊤A⊤+b⊤)],=E[AXX⊤A⊤+AXb⊤+bX⊤A⊤+bb⊤],=AE[XX⊤]A⊤+AE[X]b⊤+bE[X⊤]A⊤+bb⊤,=ARXA⊤+AμXb⊤+bμ⊤XA⊤+bb⊤.

---

1. Proof that Var[X+Y]=Var[X]+Var[Y]+2E[(X−μX)(Y−μY)]: Var[X+Y]=E[(X+Y−(μX+μY))2]=E[((X−μX)+(Y−μY))2]=E[(X−μX)2+2(X−μX)(Y−μY)+(Y−μY))2] ↩︎