\Question{Quadratic Regression}
In this question, we will find the best quadratic estimator of $Y$ given $X$. First, some notation: let $\mu_i$ be the $i$th moment of $X$, i.e.\ $\mu_i = \E[X^i]$. Also, define $\beta_1 = \E[XY]$ and $\beta_2 = \E[X^2 Y]$. For simplicity, we will assume that $\E[X] = \E[Y] = 0$ and $\E[X^2] = \E[Y^2] = 1$. (Note that this poses no loss of generality, because we can always transform the random variables by subtracting their means and dividing by their standard deviations.) We claim that the best quadratic estimator of $Y$ given $X$ is
\[
\hat{Y} = \frac{1}{\mu_3^2 - \mu_4 + 1} (a X^2 + b X + c)
\]
where
\begin{align*}
a &= \mu_3 \beta_1 - \beta_2, \\
b &= (1 - \mu_4) \beta_1 + \mu_3 \beta_2, \\
c &= -\mu_3 \beta_1 + \beta_2.
\end{align*}
Your task is to prove the Projection Property for $\hat{Y}$.
\begin{Parts}
\Part Prove that $\E[Y - \hat{Y}] = 0$.
\Part Prove that $\E[(Y - \hat{Y})X] = 0$.
\Part Prove that $\E[(Y - \hat{Y})X^2] = 0$.
\end{Parts}
Any quadratic function of $X$ is a linear combination of $1$, $X$, and $X^2$. Hence, these equations together imply that $Y - \hat{Y}$ is orthogonal to any quadratic function of $X$, and so $\hat{Y}$ is the best quadratic estimator of $Y$.
\Question{Projection Property}
Use the Projection Property to answer the following questions.
\begin{Parts}
\Part Prove or disprove: for any function $\phi$, $\E[\E[Y \mid X] \phi(X)] = 0$.
\Part Prove or disprove: $\E[(Y - \E[Y \mid X]) L[Y \mid X]] = 0$.
% \Part Prove that the constant $c$ which minimizes $\E[(X - c)^2]$ is $c = \E[X]$. Use the fact that $\E[X - \E[X]] = 0$. (\textit{Note}: Although it is possible to directly minimize $\E[(X - c)^2]$ by differentiating, we would like you to try to emulate the proofs that the LLSE/MMSE minimize the mean-squared error.)
\Part Prove the following: $\E[X^2 \mid Y] = \E[(X - \E[X \mid Y])^2 \mid Y] + \E[X \mid Y]^2$. [\textit{Hint}: In the expression $\E[X^2 \mid Y]$, try replacing $X$ with $(X - \E[X \mid Y]) + \E[X \mid Y]$.]
% \Part Use the result above to compute $\E[X^2]$. (Use the law of iterated expectation.)
\Part We have already shown that $\E[\E[Y \mid X]] = \E[Y]$. Prove that $\E[L[Y \mid X]] = \E[Y]$.
\Part Prove the following property of conditional expectation:
\begin{align*}
\E[ \E[Z \mid X, Y] \mid X ] = \E[Z \mid X].
\end{align*}
[\textit{Hint}: Take a closer look at the method by which we prove properties of conditional expectation in Note 26.]
\end{Parts}
\Question{Balls in Bins Estimation}
We throw $n > 0$ balls into $m \geq 2$ bins. Let $X$ and $Y$ represent the number of balls that land in bin $1$ and $2$ respectively.
\begin{Parts}
\Part Calculate $\E[Y \mid X]$. [\textit{Hint}: Your intuition may be more useful than formal calculations.]
\Part What are $L[Y \mid X]$ and $Q[Y \mid X]$ (where $Q[Y \mid X]$ is the best quadratic estimator of $Y$ given $X$)? [\textit{Hint}: Your justification should be no more than two or three sentences, no calculations necessary! Think carefully about the meaning of the MMSE.]
\Part Unfortunately, your friend is not convinced by your answer to the previous part. Compute $\E[X]$ and $\E[Y]$.
\Part Compute $\var(X)$.
\Part Compute $\cov(X, Y)$.
\Part Compute $L[Y \mid X]$ using the formula. Ensure that your answer is the same as your answer to part (b).
\end{Parts}
\Question{Swimsuit Season}
In the swimsuit industry, it is well-known that there is a ``swimsuit season''. During this time, swimsuit sales skyrocket!
We will model this with a random variable $X$ which is either $\lambda_L$ or $\lambda_H$ with equal probability; $\lambda_L$ represents the mean number of customers in a day when swimsuits are not in season, and $\lambda_H$ represents the mean number of customers during swimsuit season. So, $\lambda_L$ is the ``low rate'' and $\lambda_H$ is the ``high rate''. The number of customer arrivals $Y$ on a particular day is modeled as a Poisson random variable with mean $X$.
You observe $Y$ customers on a certain day, and the task is to estimate $X$.
\begin{Parts}
\Part What is $L[X \mid Y]$?
\Part What is $\E[X \mid Y]$?
\end{Parts}
\Question{Political War}
Initially, there are $d$ Democrats and $r$ Republicans in a room. They begin to argue. On each day, a random person in the room leaves and returns with an additional member of his or her political party; that is, either a Democrat will leave and return with a Democrat friend, or a Republican will leave and return with a Republican friend. Let $D_n$ denote the number of democrats in the room at the end of the $n$th day. Let $D_0 = d$.
\begin{Parts}
\Part Find $\E[D_n \mid D_{n-1}]$.
\Part Find $\E[D_n]$ using the law of iterated expectation.
\Part What is the expected fraction of Democrats in the room at the end of day $n$?
\end{Parts}
\Question{Optimal Gambling}
In even-money gambling games, you bet a fixed amount of money. If you win the game, you are given back the money that you bet, and you receive an additional amount of money equal to your original bet. If you lose the game, you lose the amount of money you bet.
\begin{Parts}
\Part You are gambling and your probability of winning, on each round, is $1/2 < p < 1$: the game is in your favor! You use the following strategy: on each round, you will bet a fraction $q$ of the money you have at the start of the round. Let $X_n$ denote the amount of money you have on round $n$. $X_0$ represents your initial assets and is a constant value. What is $\E[X_n]$?
\Part What value of $q$ will maximize $\E[X_n]$? For this value of $q$, what is the distribution of $X_n$? Can you predict what will happen as $n \to \infty$? [\textit{Hint}: Under this betting strategy, what happens if you ever lose a round?]
\Part The problem with the previous approach is that we were too concerned about expected value, so our gambling strategy was too extreme. Let's start over: again we will use a gambling strategy in which we bet a fraction $q$ of our money at each round. Express $X_n$ in terms of $n$, $q$, $X_0$, and $W_n$, where $W_n$ is the number of rounds you have won up until round $n$. [\textit{Hint}: Does the order in which you win the games affect your profit?]
\Part By the law of large numbers, $W_n/n \to p$ as $n \to \infty$. Using this fact, what does $(\log X_n)/n$ converge to as $n \to \infty$?
\Part The rationale behind $(\log X_n)/n$ is that if $(\log X_n)/n \to c$, where $c$ is a constant, then that means for large $n$, $X_n$ is roughly $e^{cn}$. Therefore, $c$ is the asymptotic growth rate of your fortune! Find the value of $q$ that maximizes your asymptotic growth rate.
\Part Using the value of $q$ you found in the previous part, compute $\E[X_n]$.
\end{Parts}