Heteroskedasticity - Revision history

Rasmusen p1vaim at 00:57, 26 May 2021

2021-05-26T00:57:25Z

Rasmusen p1vaim at 00:56, 26 May 2021

2021-05-26T00:56:52Z

Rasmusen p1vaim: /* Miscellaneous Observations */

2021-05-21T14:09:18Z

‎Miscellaneous Observations

Rasmusen p1vaim: /* Miscellaneous Observations */

2021-05-20T11:25:48Z

‎Miscellaneous Observations

Rasmusen p1vaim: /* Miscellaneous Observations */

2021-05-20T11:23:45Z

‎Miscellaneous Observations

Rasmusen p1vaim: /* Miscellaneous Observations */

2021-05-17T12:40:36Z

‎Miscellaneous Observations

Rasmusen p1vaim: /* Shrinkage Estimators */

2021-05-16T22:18:57Z

‎Shrinkage Estimators

Rasmusen p1vaim at 13:42, 14 May 2021

2021-05-14T13:42:50Z

Rasmusen p1vaim: /* Median Estimators and Loss Function */

2021-05-10T02:44:40Z

‎Median Estimators and Loss Function

Rasmusen p1vaim: /* An Ultra-Small Sample Example */

2021-05-10T02:35:15Z

‎An Ultra-Small Sample Example

@@ Line 1: / Line 1: @@
 I also have some notes in LaTeX with equations,  in [http://rasmusen.org/papers/heteroskedasticity.tex Tex] and [http://rasmusen.org/papers/heteroskedasticity.tex Pdf].
-<math> y = x^2</math>
+For math format like this: <math> y = x^2</math>
 ==An  Ultra-Small Sample Example==

← Older revision		Revision as of 00:56, 26 May 2021
Line 1:		Line 1:
	I also have some notes in LaTeX with equations, in [http://rasmusen.org/papers/heteroskedasticity.tex Tex] and [http://rasmusen.org/papers/heteroskedasticity.tex Pdf].		I also have some notes in LaTeX with equations, in [http://rasmusen.org/papers/heteroskedasticity.tex Tex] and [http://rasmusen.org/papers/heteroskedasticity.tex Pdf].
		+
		+	<math> y = x^2</math>

	==An Ultra-Small Sample Example==		==An Ultra-Small Sample Example==

@@ Line 70: / Line 70: @@
 ==Miscellaneous Observations==
-*The unweighted mean is unbiased, and the sample variance is too. Are they consistent? Maybe not; it depends on how we define consistency. If it means to replicate the first $M$ observations an infinite numbers of times with independent draws but the same variances as in the first $M$, then yes, it is consistent. But if we were to increase the number of observations infinitely but with observations getting worse and worse in quality, with higher variances, then no, it might not be consistent.
+*The unweighted mean is unbiased, and the sample variance is too. Are they consistent? Maybe not; it depends on how we define consistency. If it means to replicate the first $M$ observations an infinite numbers of times with independent draws but the same variances as in the first $M$, then yes, it is consistent. But if we were to increase the number of observations infinitely but with observations getting worse and worse in quality, with higher variances, then no, it might not be consistent. The weighted mean estimator would be consistent even then, but probably not the unweighted mean.
 *The sample variance is unbiased because there are no X variables. If there were, and they were correlated with the population variances, then there would be bias. This is the classical bias: that if X_i being bigger means   $\sigma^2_i$  is bigger then the sample variance underestimates the population variance.
@@ Line 78: / Line 78: @@
 *The White heteroskedasticity consistent variance estimator might well be what I have in mind here. Kennedy's book is good on this.
-*This is probably empriical Bayes. I take as my prior the sample mean from the other two obsrvations. This is why it is a shrinkage estimator.
+*This is  related to empriical Bayes. I take as my  nuisance parameter estimate the sample mean from the other two obsrvations. This is why it is a shrinkage estimator. But that is still not actual empirical bayes. It is empricial bayes if I take the mean sample varaince as a prior adn then estimate each observation's variance using bayes rule and that prior. But actually, I should do "cheap bayes", a phrase I invent here. Instead of bothering with a conjugate prior adn using bayes's rule, etc., I  should just average teh prior adn the data. That is honest. It is just modelling. We don't know the actual distributions, so why not just pick somthing simple? Using actual distributions is good if you are thinking about adding more data and seeing
 *The shrinkage estimator works by guessing which observation has a big disturbance, and weighting it less. The heteroskedastic correction estimator works by figuring out which observation has a big \sigma^2_i and weighting it less. There is a subtle difference. The first-- the big disturbance-- is really our goal. We don't mind big \sigma_2_i if in the prticular realization the disturbance is small. But we don't know the population mean in advance. We make a preliminary guess.  If all observations have the same sigma^2, then we are guessing at which observation has the big epsilon by using the Y and our preliminary estimate. If we know observations ahve different sigma^2, and we know what it is for each, we will weight ignoring the sample mean; we can do better than that. We could actaully COMBINE the twokinds of estimators then. We could do a weighted estimate based on data quality, and then shrink it to get individual epsilons accounted for.
@@ Line 84: / Line 90: @@
 *I am not quite using shrinkage, because I am using the mean of two observations to weight the third. In shrinkage, we use the grand mean to weight each of them. Probably my way is more efficient. Or is it equivalent somehow?
-* I need a good name, a short one, for my estimator. Quality-weighted estimator. It really is perceived qualtyweighted estimator,but that is too long.
+* I need a good name, a short one, for my estimator. Quality-weighted estimator. It really is perceived qualtyweighted estimator,but that is too long.  Estimated Weighted Mean is best.
   *A  big question: how do I estimate the variance of an observation? Do I use its residual? Or do I take that and shrink it, or move it to the grand sample residual?
 ----

@@ Line 86: / Line 86: @@
 * I need a good name, a short one, for my estimator. Quality-weighted estimator. It really is perceived qualtyweighted estimator,but that is too long.
+  *A  big question: how do I estimate the variance of an observation? Do I use its residual? Or do I take that and shrink it, or move it to the grand sample residual?
 ----

← Older revision		Revision as of 11:23, 20 May 2021
Line 62:		Line 62:

	*[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2675681 "Understanding Shrinkage Estimators: From Zero to Oracle to James-Stein."] The standard estimator of the population mean is the sample mean, which is unbiased. Constructing an estimator by shrinking the sample mean results in a biased estimator, with an expected value less than the population mean. On the other hand, shrinkage always reduces the estimator's variance and can reduce its mean squared error. This paper tries to explain how that works. I start with estimating a single mean using the zero estimator and the oracle estimator, and continue with the grand-mean estimator. Thus prepared, it is easier to understand the James-Stein estimator, in its simple form with known homogeneous variance and in extensions. The James-Stein estimator combines the oracle estimate's coefficient shrinking with the grand mean estimator's cancelling out of overestimates and underestimates.		*[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2675681 "Understanding Shrinkage Estimators: From Zero to Oracle to James-Stein."] The standard estimator of the population mean is the sample mean, which is unbiased. Constructing an estimator by shrinking the sample mean results in a biased estimator, with an expected value less than the population mean. On the other hand, shrinkage always reduces the estimator's variance and can reduce its mean squared error. This paper tries to explain how that works. I start with estimating a single mean using the zero estimator and the oracle estimator, and continue with the grand-mean estimator. Thus prepared, it is easier to understand the James-Stein estimator, in its simple form with known homogeneous variance and in extensions. The James-Stein estimator combines the oracle estimate's coefficient shrinking with the grand mean estimator's cancelling out of overestimates and underestimates.
		+
		+	==Empirical Bayes==
		+	This is an example of empirical Bayes. A typical example of empirical Bayes estimation would be for [http://varianceexplained.org/r/empirical_bayes_baseball/ estimating the batting ability of each of a number of baseball players.] We would start by computing the grand sample mean of the batting averages. That would become our prior--- thus "empirical Bayes". Then we would use that prior and Bayes's rule to estimate the true mean for each player, using only his own success at batting. If he had only a few at-bats, the prior would make a big difference. If he had lots, his own data would be far more important and our estimate would be very close to his own sample mean.
		+
		+	Here, I am using the data to estimate the observation's variance. I am not using Bayes's Rule at all, nor a conjugate prior distribution, though. But I am using the idea of estimating an ancillary, nuisance, parameter with the data first, to use in estimating the parameter of interest.
		+

	==Miscellaneous Observations==		==Miscellaneous Observations==

← Older revision		Revision as of 12:40, 17 May 2021
Line 72:		Line 72:
	*The White heteroskedasticity consistent variance estimator might well be what I have in mind here. Kennedy's book is good on this.		*The White heteroskedasticity consistent variance estimator might well be what I have in mind here. Kennedy's book is good on this.

		+	*This is probably empriical Bayes. I take as my prior the sample mean from the other two obsrvations. This is why it is a shrinkage estimator.

		+	*The shrinkage estimator works by guessing which observation has a big disturbance, and weighting it less. The heteroskedastic correction estimator works by figuring out which observation has a big \sigma^2_i and weighting it less. There is a subtle difference. The first-- the big disturbance-- is really our goal. We don't mind big \sigma_2_i if in the prticular realization the disturbance is small. But we don't know the population mean in advance. We make a preliminary guess. If all observations have the same sigma^2, then we are guessing at which observation has the big epsilon by using the Y and our preliminary estimate. If we know observations ahve different sigma^2, and we know what it is for each, we will weight ignoring the sample mean; we can do better than that. We could actaully COMBINE the twokinds of estimators then. We could do a weighted estimate based on data quality, and then shrink it to get individual epsilons accounted for.
		+
		+	*I am not quite using shrinkage, because I am using the mean of two observations to weight the third. In shrinkage, we use the grand mean to weight each of them. Probably my way is more efficient. Or is it equivalent somehow?
		+
		+	* I need a good name, a short one, for my estimator. Quality-weighted estimator. It really is perceived qualtyweighted estimator,but that is too long.
		+
		+
	----		----

← Older revision		Revision as of 22:18, 16 May 2021
Line 60:		Line 60:

	*In James-Stein, I bet the real secret is this (and the Grand Mean estimator too). We have three variables with disturbances attached to their three different means. Suppose we have three numbers that add up to 180: 10, 20, and 150. The average is 60. We expand the little ones and shrink the big ones 50% towards the mean, to get 15, 30, and 105. In so doing we have added absolute amounts of 5 and 10 to the little numbers and subtracted 55 from the big number. There is some reason why that 55 helps us with mean squared error more than the 5 and 10 would hurt us.		*In James-Stein, I bet the real secret is this (and the Grand Mean estimator too). We have three variables with disturbances attached to their three different means. Suppose we have three numbers that add up to 180: 10, 20, and 150. The average is 60. We expand the little ones and shrink the big ones 50% towards the mean, to get 15, 30, and 105. In so doing we have added absolute amounts of 5 and 10 to the little numbers and subtracted 55 from the big number. There is some reason why that 55 helps us with mean squared error more than the 5 and 10 would hurt us.
		+
		+	*[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2675681 "Understanding Shrinkage Estimators: From Zero to Oracle to James-Stein."] The standard estimator of the population mean is the sample mean, which is unbiased. Constructing an estimator by shrinking the sample mean results in a biased estimator, with an expected value less than the population mean. On the other hand, shrinkage always reduces the estimator's variance and can reduce its mean squared error. This paper tries to explain how that works. I start with estimating a single mean using the zero estimator and the oracle estimator, and continue with the grand-mean estimator. Thus prepared, it is easier to understand the James-Stein estimator, in its simple form with known homogeneous variance and in extensions. The James-Stein estimator combines the oracle estimate's coefficient shrinking with the grand mean estimator's cancelling out of overestimates and underestimates.
		+
		+	==Miscellaneous Observations==
		+	*The unweighted mean is unbiased, and the sample variance is too. Are they consistent? Maybe not; it depends on how we define consistency. If it means to replicate the first $M$ observations an infinite numbers of times with independent draws but the same variances as in the first $M$, then yes, it is consistent. But if we were to increase the number of observations infinitely but with observations getting worse and worse in quality, with higher variances, then no, it might not be consistent.
		+
		+	*The sample variance is unbiased because there are no X variables. If there were, and they were correlated with the population variances, then there would be bias. This is the classical bias: that if X_i being bigger means $\sigma^2_i$ is bigger then the sample variance underestimates the population variance.
		+
		+	*The White statistic doesn't work with 0 degrees of freedom-- that is, with just a constant and no X variable.
		+
		+	*The White heteroskedasticity consistent variance estimator might well be what I have in mind here. Kennedy's book is good on this.


−	*[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2675681 "Understanding Shrinkage Estimators: From Zero to Oracle to James-Stein."] The standard estimator of the population mean is the sample mean, which is unbiased. Constructing an estimator by shrinking the sample mean results in a biased estimator, with an expected value less than the population mean. On the other hand, shrinkage always reduces the estimator's variance and can reduce its mean squared error. This paper tries to explain how that works. I start with estimating a single mean using the zero estimator and the oracle estimator, and continue with the grand-mean estimator. Thus prepared, it is easier to understand the James-Stein estimator, in its simple form with known homogeneous variance and in extensions. The James-Stein estimator combines the oracle estimate's coefficient shrinking with the grand mean estimator's cancelling out of overestimates and underestimates. .
	----		----

← Older revision		Revision as of 13:42, 14 May 2021
Line 1:		Line 1:
		+	I also have some notes in LaTeX with equations, in [http://rasmusen.org/papers/heteroskedasticity.tex Tex] and [http://rasmusen.org/papers/heteroskedasticity.tex Pdf].
		+
	==An Ultra-Small Sample Example==		==An Ultra-Small Sample Example==
	As I was suffering from food poisoning in Florida last week, I had many hours in between vomiting bouts to while away woozily. So I thought about heteroskedasticity. Suppose you are estimating the temperature of lake water using various thermometers, and the observations are 50, 52, and 64. We will assume the thermometers are all unbiased, so if x_i is observation i and T is the true temperature, E(x_i) = T. The obvious estimator is the sample mean, (50+52+64)/3 = 55.3 (about). This is also the best estimator if the thermometers are all equally good. But what if they are not? That is, what if the disturbances are heteroskedastic?		As I was suffering from food poisoning in Florida last week, I had many hours in between vomiting bouts to while away woozily. So I thought about heteroskedasticity. Suppose you are estimating the temperature of lake water using various thermometers, and the observations are 50, 52, and 64. We will assume the thermometers are all unbiased, so if x_i is observation i and T is the true temperature, E(x_i) = T. The obvious estimator is the sample mean, (50+52+64)/3 = 55.3 (about). This is also the best estimator if the thermometers are all equally good. But what if they are not? That is, what if the disturbances are heteroskedastic?

@@ Line 48: / Line 48: @@
 ==Median Estimators and Loss Function==
-  Think hard about this. Do the math. One objeciton yu might make is that even without heteroskedasticity, the outlier of the three observations is most likely to be a big error. So why not drop it? The reason is that it is also th emost importnat IF it is correct.  And that is more important with squared error loss. In fact, I bet it is for any convex loss function. For Absolute deviation loss, I bet it evens out, and it doens' matter if you drop it. That's why that give su a MEDIAN estimator. For Concave loss functions, you DO want to drop it.
+Think hard about this. Do the math. One objeciton yu might make is that even without heteroskedasticity, the outlier of the three observations is most likely to be a big error. So why not drop it? The reason is that it is also th emost importnat IF it is correct.  And that is more important with squared error loss. In fact, I bet it is for any convex loss function. For Absolute deviation loss, I bet it evens out, and it doens' matter if you drop it. That's why that give su a MEDIAN estimator. For Concave loss functions, you DO want to drop it.
 But heteroskedasticity puts a thumb o th eblance,  a thumb on the side of dropping the observation, or at least weighting it less
 ==Shrinkage Estimators==

← Older revision		Revision as of 02:35, 10 May 2021
Line 17:		Line 17:

	The only way to resolve that problem, I think, is to go Bayesian. If we have a distribution of possible variances (and, more important, of variances of variances), we can find out whether the estimator does well in expectation against that distribution. For example, we might assume that the variance of each observation is drawn randomly and independently from 0 to 100. Better, probably, would be to see how well the estimator does for a variety of true states of the world and then to eyeball it and decide whether it's worth using. I expect what I will do is use python to find two numerical examples for Monte Carlo studies, one where the heteroskedasticty correction helps, and one where it hurts.		The only way to resolve that problem, I think, is to go Bayesian. If we have a distribution of possible variances (and, more important, of variances of variances), we can find out whether the estimator does well in expectation against that distribution. For example, we might assume that the variance of each observation is drawn randomly and independently from 0 to 100. Better, probably, would be to see how well the estimator does for a variety of true states of the world and then to eyeball it and decide whether it's worth using. I expect what I will do is use python to find two numerical examples for Monte Carlo studies, one where the heteroskedasticty correction helps, and one where it hurts.
		+
		+	==Ultra-Small Samples as Low-Fat Modelling==
		+	The ultra-small sample I just used is an example of Low-Fat Modelling (or "No-Fat", or "MIT-Style", or, confusingly, "NewJersey Style modelling in computer science, since they use "MIT-Style" to mean the exact opposite). Low-Fat Modelling is the style of economic theory made popular by Joseph Stiglitz and perfected by Jean Tirole. It models an economic idea using the simplest, most specific, model possible, as opposed to the older style of trying to use the fanciest, most general model possible. Thus, instead of assuming there are $N$ goods with a continuum of possible prices, you assume there are 2 goods with 2 prices. This will remind you of international trade's 2x2x2 models (2 countries, 2 goods, 2 inputs), since they have long used Low-Fat Modelling. Sometimes you need at least 3 countries to demonstrate an effect (e.g. the Transfer Paradox), so you do that, but you don't go to N countries.
		+
		+	Thus, for a theorist like me it is natural when thinking about econometric theory to try to estimate a mean rather than a regression coefficient, and to try it with 3 observations rather than N. If we can understand it for a mean with three observations, we can understand it for a regression with M coefficients for a sample with N observations. I would have liked to reduce it to 2 observations here, but then you can't estimate the variance of an observation using the other observations' data.

	==A Large-Sample Example==		==A Large-Sample Example==