Visualizing conditional probability
machine learning
Conditional probability is a fundamental concept in probability theory; however, its interpretation is not always intuitive. Here, an easy-to-visualize example is used to illustrate the concept.
1 Model: non-directional radar tracking UFO
A UFO hovers above a field with average position \left( \overline{x},\overline{y} \right) and standard deviations \sigma_{X}, \sigma_{Y}. At the field’s center (0,0), a non-directional radar measures only the distance d to the UFO with standard deviation \sigma_{D}, but cannot determine its direction or exact position.
The UFO’s location is modeled by an uncorrelated bivariate Gaussian distribution of two random variables X and Y:
\begin{aligned} f_{X,Y}(x,y) & \equiv f_{X,Y}(X = x,Y = y) \\ & = \frac{1}{2\pi\sigma_{X}\sigma_{Y}}\exp\left\{ \frac{{- \left( x - \overline{x} \right)}^{2}}{2\sigma_{X}^{2}}\frac{{- \left( y - \overline{y} \right)}^{2}}{2\sigma_{Y}^{2}} \right\}. \end{aligned}
The distance d from the radar antenna to the UFO follows a conditional univariate Gaussian distribution of a single random variable D given the UFO’s position (X,Y):
\begin{aligned} f_{D}\left( d~|~x^{\ast},y^{\ast} \right) & \equiv f_{D}\left( D = d~|~X = x^{\ast},Y = y^{\ast} \right) \\ & = \frac{1}{\sqrt{2\pi}\sigma_{D}}\exp\left\{ \frac{{- \left( d - \sqrt{\left( x^{\ast} \right)^{2} + \left( y^{\ast} \right)^{2}} \right)}^{2}}{2\sigma_{D}^{2}} \right\}, \end{aligned}
where \left( x^{\ast},y^{\ast} \right) is a specific UFO position, and \sqrt{\left( x^{\ast} \right)^{2} + \left( y^{\ast} \right)^{2}} is the exact distance (unknown) from the radar to the UFO.
N.B. The unconditional distribution f_{D}(D = d) can be obtained by substituting x^{\ast} = \overline{x} and y^{\ast} = \overline{y}, it will model measurements of the distance to the UFO without knowing its exact position, only the average.
Our goal is to find and visualize the conditional pdf f_{X,Y}\left( x,y~|~d^{\ast} \right) of the UFO position given a measured distance d^{\ast} and compare it with the unconditional pdf f_{X,Y}(x,y).
2 Conditional probability density
Using Bayes theorem:
\Pr\left\lbrack A~|~B \right\rbrack = \frac{\Pr\lbrack A\rbrack \cdot \Pr\left\lbrack B~|~A \right\rbrack}{\Pr\lbrack B\rbrack},
the conditional probability density of the UFO being at (x,y) given a measured distance d^{\ast} is
\begin{aligned} f_{X,Y}\left( X = x,Y = y~|~D = d^{\ast} \right) & = \frac{f_{X,Y}(X = x,Y = y) \cdot f_{D}\left( D = d^{\ast}~|~X = x,Y = y \right)}{f_{D}\left( D = d^{\ast} \right)} \\ & = \underset{\text{const}}{\underbrace{\frac{1}{f_{D}\left( D = d^{\ast} \right)}}} \cdot f_{X,Y}(x,y) \cdot f_{D}\left( d^{\ast}~|~x,y \right), \end{aligned}
where \text{const} \equiv \text{const}(d^{\ast}) is the normalization constant for fixed d^{\ast}. By substituting the bivariate and univariate Gaussian distributions:
\begin{aligned} & = \text{const} \cdot \frac{1}{2\pi\sigma_{X}\sigma_{Y}}\exp\left\{ \frac{{- \left( x - \overline{x} \right)}^{2}}{2\sigma_{X}^{2}}\frac{{- \left( y - \overline{y} \right)}^{2}}{2\sigma_{Y}^{2}} \right\} \cdot \frac{1}{\sqrt{2\pi}\sigma_{D}}\exp\left\{ \frac{{- \left( d^{\ast} - \sqrt{x^{2} + y^{2}} \right)}^{2}}{2\sigma_{D}^{2}} \right\} \\ & = \underset{\text{const}'}{\underbrace{\frac{\text{const}}{{\sqrt{2\pi}}^{3}\sigma_{X}\sigma_{Y}\sigma_{D}}}}\exp\left\{ \frac{{- \left( x - \overline{x} \right)}^{2}}{2\sigma_{X}^{2}}\frac{{- \left( y - \overline{y} \right)}^{2}}{2\sigma_{Y}^{2}}\frac{{- \left( d^{\ast} - \sqrt{x^{2} + y^{2}} \right)}^{2}}{2\sigma_{D}^{2}} \right\}, \end{aligned}
where \text{const}' \equiv \frac{\text{const}}{{\sqrt{2\pi}}^{3}\sigma_{X}\sigma_{Y}\sigma_{D}} is a new normalization constant for given distribution parameters \sigma_{X}, \sigma_{Y}, and \sigma_{D}.
3 Marginalization
The normalization constant \text{const}(d^{\ast}) can be calculated by integrating the conditional probability density:
\begin{aligned} \frac{1}{\text{const}(d^{\ast})} & = f_{D}\left( D = d^{\ast} \right) \\ & = \int_{- \infty}^{+ \infty}\int_{- \infty}^{+ \infty}f_{X,Y,D}\left( X = x,Y = y,D = d^{\ast} \right)\mathrm{d}x\mathrm{d}y \\ & = \int_{- \infty}^{+ \infty}\int_{- \infty}^{+ \infty}f_{D}\left( d^{\ast}~|~x,y \right) \cdot f_{X,Y}(x,y)\mathrm{d}x\mathrm{d}y. \end{aligned}
This integral contains different exponential terms, making it challenging to solve analytically or compute numerically due to rapid growth and potential numerical overflow.
However, the pre-exponential constant \text{const}' can be omitted for visualization since it doesn’t affect density shape or maximum position.
4 Visualization
Now the unconditional f_{X,Y}(x,y) and conditional f_{X,Y}\left( x,y~|~d^{\ast} \right) densities can be visualized and compared.
The non-directional radar at position (0,0) measures distance d (depicted as a circle) to the UFO. The UFO is hovering around the mean position (4,4) with s.d. \sigma_{X} = \sigma_{Y} = 3. The corresponding probability density is shown by the color gradient. The measured distance d constrains the probable UFO location to a circular arc in the conditional density. The s.d. \sigma_{D} determines the width of the arc and was chosen to be \sigma_{D} = 0.5 (relatively small compared to fluctuations in the UFO position).
A priori distribution:
A posteriori distribution: