MVUE for Linear Models

Introduction

The minimum variance unbiased estimator (MVUE) has the lowest variance among all unbiased estimators. In general, there is no method to find the MVUE; however, for a special class of estimation problems, we can find these estimators. This module concerns a special class of signal models for which a minimum variance unbiased estimator (MVUE) is available. This special class consists of estimation problems with linear signal models and additive Gaussian noise.

The module begins by introducing the linear signal model and derives the MVUE for colored Gaussian noise. Next, we assume white Gaussian noise, which can be considered a special case of colored noise. When the information on noise distribution is not complete, an interesting approach is to approximate the noise as Gaussian by considering only the expected value and variance for the noise. Such estimators are called best linear unbiased estimators (BLUE), covered at the end of the module.

Linear Signal Model

A linear signal model is any model for which the signal s[n,θ] can be expressed as a linear combination of the weighted parameters, i.e., $\begin{matrix} (1) & s [n; θ] = \sum_{k = 0}^{K - 1} h_{k} [n] θ_{k} . \end{matrix}$ For multiple signal values, the model can be expressed as a matrix-vector product of the form $\begin{matrix} (2) & s (θ) = H θ, \end{matrix}$ where $s (θ)$ is the signal vector of length $N$ , $H$ is the so-called observation matrix of size $N \times K$ , and $θ$ is the parameter vector of length $K$ .

Example:

Assume our signal model is a polynomial of degree $K - 1$ with coefficients $θ_{0}, θ_{1}, \dots, θ_{K - 1}$ , i.e., $\begin{matrix} (3) & s [n; θ] = θ_{0} n^{0} + θ_{1} n^{1} + \dots + θ_{K - 1} n^{K - 1}, \end{matrix}$ and suppose observations for $n = 0, 1, \dots, N - 1$ are given. The corresponding observation matrix $H$ is $\begin{matrix} (4) & H = [\begin{matrix} 1 & 0 & 0 & \dots & 0 \\ 1 & 1 & 1 & 1 \\ 1 & 2 & 2^{2} & 2^{K - 1} \\ ⋮ & ⋱ \\ 1 & N - 1 & (N - 1)^{2} & \dots & (N - 1)^{K - 1} \end{matrix}] . \end{matrix}$

MVUE for Linear Models and Additive Gaussian Noise

To find the MVUE for linear signals in Gaussian noise, we will take advantage of the equality constraint of the CRLB. Therefore, we need to show that we can express the derivative of the log-likelihood function in the form $\begin{matrix} (5) & \frac{\partial}{\partial θ} l (x, θ) = I (θ) (g (x) - θ), \end{matrix}$ see module on the CRLB. To obtain the derivative of the log-likelihood function, we need first to determine the probability density function of the observations. For the linear signal model embedded in additive Gaussian noise, our observations are modeled as $\begin{matrix} (6) & x = H θ + w, \end{matrix}$ where $w$ is zero mean noise with covariance matrix $C$ , i.e., $w \sim N (0, C)$ . Thus, the PDF of the observation vector is $\begin{matrix} (7) & p (x, θ) = \frac{1}{| 2 π C |^{\frac{1}{2}}} \exp (- \frac{1}{2} (x - H θ)^{T} C^{- 1} (x - H θ)) . \end{matrix}$

The corresponding log-likelihood function is $\begin{matrix} (8) & l (x; θ) = - \frac{1}{2} \ln | 2 π C | - \frac{1}{2} (x - H θ)^{T} C^{- 1} (x - H θ), \end{matrix}$ which after expanding the second term becomes $\begin{matrix} (9) & l (x; θ) = - \frac{1}{2} \ln | 2 π C | - \frac{1}{2} (x^{T} C^{- 1} x - x^{T} C^{- 1} H θ - θ^{T} H^{T} C^{- 1} x + θ^{T} H^{T} C^{- 1} H θ) . \end{matrix}$

The gradient of the log-likelihood function is $\begin{matrix} (10) & \frac{\partial}{\partial θ} l (x; θ) = H^{T} C^{- 1} x - H^{T} C^{- 1} H θ . \end{matrix}$

How to compute the gradient.

The gradient of the log-likelihood function is $\frac{\partial}{\partial θ} l (x; θ) = [\begin{matrix} \frac{\partial}{\partial θ_{0}} l (x; θ) \\ \frac{\partial}{\partial θ_{1}} l (x; θ) \\ ⋮ \\ \frac{\partial}{\partial θ_{K - 1}} l (x; θ) \end{matrix}],$ i.e., it is a vector containing all the partial derivatives of the log-likelihood function. To evaluate the partial derivatives, we first use the linearity property of the differential operator. The single expressions in the log-likelihood function are of the form $a^{T} x$ , $x^{T} a$ , and $x^{T} A x$ . The partial derivatives and the gradient of these expressions can be obtained as follows:

The expression $a^{T} x$ is $a^{T} x = \sum_{n = 0}^{N - 1} a_{n} x_{n} = a_{0} x_{0} + a_{1} x_{1} + \dots + a_{N - 1} x_{N - 1} .$ Taking the derivative with respect to one specific element $x_{i}$ is $\frac{\partial a^{T} x}{\partial x_{i}} = a_{i},$ since only the addend $a_{i} x_{i}$ is a function of $x_{i}$ with derivative $a_{i}$ . All other addends of the sum are constants and vanish.
Since $x^{T} a = a^{T} x$ is true for real-valued vectors, we also have $\frac{\partial x^{T} a}{\partial x_{i}} = a_{i} .$
The expression $x^{T} A x$ can also be expressed as $x^{T} A x = \sum_{n = 0}^{N - 1} \sum_{m = 0}^{N - 1} x_{n} A_{n, m} x_{m},$ which after rearranging the summations is $\sum_{n = 0}^{N - 1} \sum_{m = 0}^{N - 1} x_{n} A_{n, m} x_{m} = A_{i, i} x_{i}^{2} + x_{i} \sum_{\underset{m \neq i}{m = 0}}^{N - 1} A_{i, m} x_{m} + x_{i} \sum_{\underset{n \neq i}{n = 0}}^{N - 1} x_{n} A_{n, i} + \sum_{\underset{n \neq i}{n = 0}}^{N - 1} \sum_{\underset{m \neq i}{m = 0}}^{N - 1} x_{n} A_{n, m} x_{m} .$ Taking the derivative with respect to $x_{i}$ , we obtain $\frac{\partial x^{T} A x}{\partial x_{i}} = 2 A_{i, i} x_{i} + \sum_{\underset{m \neq i}{m = 0}}^{N - 1} A_{i, m} x_{m} + \sum_{\underset{n \neq i}{n = 0}}^{N - 1} x_{n} A_{n, i}$ Note that the two summations miss the summation index $m = i$ and $n = i$ , respectively. However, we have additionally to the two sums the expression $2 A_{i, i} x_{i}$ which is exactly the missing addend in the sums. Thus, we can add distribute $2 A_{i, i} x_{i}$ over the two sums and get $\frac{\partial x^{T} A x}{\partial x_{i}} = \sum_{m = 0}^{N - 1} A_{i, m} x_{m} + \sum_{n = 0}^{N - 1} x_{n} A_{n, i} .$ The above expression can equivalently be expressed as an inner product between the $i$ th row of $A$ , denoted by $a_{i}$ , and $x$ and the inner product between the $i$ th column of $A$ , denoted by $a_{i}^{T}$ , and $x$ , i.e., $\frac{\partial x^{T} A x}{\partial x_{i}} = a_{i} x + a_{i}^{T} x$

Combining the above results, we obtain $\begin{matrix} (11) & \frac{a^{T} x}{\partial x} = \frac{x^{T} a}{\partial x} = a, \end{matrix}$ and $\begin{matrix} (12) & \frac{x^{T} A x}{\partial x} = (A + A^{T}) x . \end{matrix}$ For symmetric matrices, i.e., $A = A^{T}$ , the latter expression simplifies to $\begin{matrix} (13) & \frac{x^{T} A x}{\partial x} = 2 A x . \end{matrix}$

Note that the covariance matrix $C$ is symmetric and so is its inverse. Using $a = H^{T} C^{- 1} x$ , $a^{T} = x^{T} C^{- 1} H$ , and $A = A^{T} = H^{T} C^{- 1} H$ , we obtain $(10)$

If $H^{T} C^{- 1} H$ is invertible, we further have $\begin{matrix} (14) & \frac{\partial}{\partial θ} l (x; θ) = H^{T} C^{- 1} H ({(H^{T} C^{- 1} H)}^{- 1} H^{T} C^{- 1} x - θ) . \end{matrix}$ By comparing $(14)$ with $(5)$ , we recognize the Fisher information matrix $\begin{matrix} (15) & I (θ) = H^{T} C^{- 1} H \end{matrix}$ and the estimator $\begin{matrix} (16) & g (x) = {(H^{T} C^{- 1} H)}^{- 1} H^{T} C^{- 1} x . \end{matrix}$ We found an efficient estimator, and thus, an MVUE. In the module on the MLE, we also stated that if an efficient estimator exits, it is the MLE, which is easily verified by equating $(10)$ with zero and solving for $θ$ .

MVUE for Linear Models in White Additive Gaussian Noise

So far, we have assumed that our noise is colored, i.e., that our noise sample are correlated. For the case of additive white Gaussian noise we can reuse the above result. Therefore, we assume that our individual noise samples are IID with zero mean and variance $σ^{2}$ . In this case, the covariance matrix $C$ is a diagonal matrix with elements $σ^{2}$ on the main diagonal, i.e., $\begin{matrix} (17) & C = σ^{2} I \end{matrix}$ Substitung $(17)$ in $(14)$ yields $\begin{matrix} (18) & \frac{\partial}{\partial θ} l (x; θ) = \frac{H^{T} H}{σ^{2}} ({(H^{T} H)}^{- 1} H^{T} x - θ) \end{matrix}$ where we recognize the estimator $\begin{matrix} (19) & g (x) = (H^{T} H)^{- 1} H^{T} x \end{matrix}$ and the Fisher information matrix $\begin{matrix} (20) & I (θ) = \frac{H^{T} H}{σ^{2}} . \end{matrix}$

Example

Assume we have a signal composed of $P$ sinusoidal signals embedded in white Gaussian noise with zero mean and variance $σ^{2}$ , i.e., $\begin{matrix} (21) & x [n] = \sum_{p = 0}^{P - 1} c_{p} \cos (2 π \frac{k_{p}}{N} n - φ_{p}) + w [n], \end{matrix}$ where $c_{p}$ and $φ_{p}$ are the unknown amplitude and phase, respectively. The frequency parameters $k_{0}$ to $k_{(P - 1)}$ are assumed to be known, different, and between $0$ and $N - 1$ . The signal is linear in the parameter $c_{p}$ , but it is not linear in the unknown phase $φ_{p}$ . However, we can use the trigonometric relations $\begin{matrix} (22) & \cos (α) \cos (β) = \frac{1}{2} (\cos (α - β) + \cos (α + β)) \end{matrix}$ and $\begin{matrix} (23) & \sin (α) \sin (β) = \frac{1}{2} (\cos (α - β) - \cos (α + β)) \end{matrix}$ to express $(21)$ as $\begin{matrix} (24) & x [n] = \sum_{p = 0}^{P - 1} \underset{a_{p}}{\underset{⏟}{c_{p} \cos (φ)}} \cos (2 π \frac{k_{p}}{N} n) + \underset{b_{p}}{\underset{⏟}{c_{p} \sin (φ)}} \sin (2 π \frac{k_{p}}{N} n) + w [n] . \end{matrix}$ The transformed model is now linear in the unknown parameters $a_{p}$ and $b_{p}$ . Due to the invariance of the MLE, we can find the maximum likelihood estimate for the amplitudes $a_{p}$ and $b_{p}$ and obtain the maximum likelihood estimate for $c_{p}$ and $φ$ by using relations $\begin{matrix} (25) & c_{p} = \sqrt{a_{p}^{2} + b_{p}^{2}} \end{matrix}$ and $\begin{matrix} (26) & φ_{p} = \arctan 2 (b_{p}, a_{p}) . \end{matrix}$ The signal model in $(24)$ for multiple observation can be expressed as $\begin{matrix} (27) & x = H θ + w . \end{matrix}$ The observation matrix $H$ is $\begin{matrix} (28) & H = [\begin{matrix} c_{k_{0}} & \dots & c_{k_{(P - 1)}} & s_{k_{0}} & \dots & s_{k_{(P - 1)}} \end{matrix}] \end{matrix}$ with columns $\begin{matrix} (29) & c_{k_{p}} = {[\begin{matrix} \cos (2 π \frac{k_{p}}{N} 0) & \cos (2 π \frac{k_{p}}{N} 1) & \dots & \cos (2 π \frac{k_{p}}{N} (N - 1)) \end{matrix}]}^{T} \end{matrix}$ and $\begin{matrix} (30) & s_{k_{p}} = {[\begin{matrix} \sin (2 π \frac{k_{p}}{N} 0) & \sin (2 π \frac{k_{p}}{N} 1) & \dots & \sin (2 π \frac{k_{p}}{N} (N - 1)) \end{matrix}]}^{T}, \end{matrix}$ respectively. The parameter vector $θ$ for this model is $\begin{matrix} (31) & θ = [\begin{matrix} a_{0} & \dots & a_{(P - 1)} & b_{0} \dots b_{(P - 1)} \end{matrix}] \end{matrix}$

Note that the frequencies $k_{p}$ are multiple of the fundamental frequency $1 / N$ , and thus, the columns of the observation matrix are orthogonal. The matrix $H^{T} H$ is $\begin{matrix} (32) & H^{T} H = \frac{N}{2} I, \end{matrix}$ i.e., a diagonal matrix with entries $N / 2$ on the main diagonal. The maximum likelihood estimate is $\begin{matrix} (33) & {\hat{θ}}_{ML} = {(H^{T} H)}^{- 1} H x = [\begin{matrix} \frac{2}{N} c_{k_{0}}^{T} \\ ⋮ \\ \frac{2}{N} c_{k_{(P - 1)}}^{T} \\ \frac{2}{N} s_{k_{0}}^{T} \\ ⋮ \\ \frac{2}{N} s_{k_{(P - 1)}}^{T} \end{matrix}] x \end{matrix}$ The estimate of the parameter $a_{p}$ and $a_{b}$ are $\begin{matrix} (34) & {\hat{a}}_{p} = \frac{2}{N} \sum_{n = 0}^{N - 1} x [n] \cos (2 π \frac{k_{p}}{N} n) \end{matrix}$ and $\begin{matrix} (35) & {\hat{b}}_{p} = \frac{2}{N} \sum_{n = 0}^{N - 1} x [n] \sin (2 π \frac{k_{p}}{N} n) \end{matrix}$ Suppose that the frequency $k_{p}$ takes all values of from 0 to $N - 1$ , then $(34)$ and $(35)$ are the discrete Fourier transform for real signals.

Best Linear Unbiased Estimator (BLUE)

So far, we have derived the MVUE for linear signal models with AWGN. In many applications, it is not possible to find an MVUE estimator using techniques presented up to this point. A popular choice is to restrict the estimator to be linear in the observation $x$ , i.e., of the form $\begin{matrix} (36) & \hat{θ} = A x . \end{matrix}$ Here, we assume that $N$ observations are available and that we seek to estimate $K$ unknown parameter with $N > K$ . Thus, $A$ is of size $K \times N$ . Depending on the choice of the $A$ 's coefficients, different estimators can be obtained. Among all linear estimators, we seek an estimator that is unbiased and has minimum variance. The estimator which fulfills these properties is called the best linear unbiased estimator (BLUE). The BLUE, as we will see, is also applicable, if limited knowledge about the PDF of the observation is available.

To find the BLUE, we have to start from the assumptions at hand; that the estimator has zero bias and minimum variance. The zero bias condition is formulated as $\begin{matrix} (37) & E [\hat{θ}] = E [A x] = A E [x] = θ, \end{matrix}$ where $A$ is the $K \times N$ matrix that consists of the coefficients $a_{k, n}$ . The constraint of unbiasedness requires a linear relation between the parameter vector $θ$ and the expected value of the data such that $\begin{matrix} (38) & E [x] = H θ . \end{matrix}$ Substituting $(38)$ in $(37)$ yields $\begin{matrix} (39) & E [\hat{θ}] = A H θ . \end{matrix}$

Thus, for the linear estimator to be unbiased, we require that $\begin{matrix} (40) & A H = I, \end{matrix}$ where $I$ is the $K \times K$ identity matrix.

The variance of the $k$ th estimate ${\hat{θ}}_{k}$ is $\begin{matrix} (41) & V a r [{\hat{θ}}_{k}] = E [(a_{k}^{T} x - E [a_{k}^{T} x])^{2}] \end{matrix}$ where $a_{k}^{T}$ is the $k$ th row of $A$ . This can be equivalently expressed as $\begin{aligned} (42) & V a r [{\hat{θ}}_{k}] & = E [{(a_{k}^{T} (x - E [x]))}^{2}] \\ (43) & = E [(a_{k}^{T} (x - E [x]) (x - E [x])^{T} a_{k}] \\ (44) & = a_{k}^{T} C a_{k} . \end{aligned}$ Here, $C$ is the covariance matrix for the data, which is equivalent to the covariance matrix for the zero-mean noise due to the linear signal model.

The BLUE is found by minimizing the variance for all estimates with the constraint $A H = I$ . Thus, the optimization problem is $\begin{matrix} (45) & min_{a_{k}^{T}} a_{k}^{T} C a_{k}, s.t. a_{k}^{T} h_{l} = δ_{k, l}, \end{matrix}$ where $h_{l}$ is the $l$ th column of $H$ . The minimum can be found using Lagrange multipliers, which results in $\begin{matrix} (46) & \hat{θ} = (H^{T} C^{- 1} H)^{- 1} H^{T} C^{- 1} x . \end{matrix}$ The covariance matrix for the estimate $\hat{θ}$ is $\begin{matrix} (47) & C_{\hat{θ}} = (H^{T} C^{- 1} H)^{- 1} . \end{matrix}$

Note that the BLUE is the MVUE for the linear model in $(6)$ . This follows directly from the unbiased constraint $\begin{matrix} (48) & E [x] = E [H θ + w] = H θ . \end{matrix}$

Last updated on Sep 24, 2021