Interpreting Chi-Squared Test Results

Date: 10/26/2004 at 07:21:21
From: Joseph
Subject: Chi-Squared Question

The critical values for chi-squared increase as the confidence level 
increases.  My question is why does a distribution "pass" this test if 
the calculated value is less than the critical value and not vice- versa?

For example, let's say for 9 degrees of freedom, you calculate a value 
of 20.  Using a 97.5% confidence test, since 20 > 16.919, there is a 
significant difference from your expectation (horrible match).  Using 
a 99% confidence test, since 20 < 99%, the probability distribution 
cannot be rejected with 99% confidence (excellent match).

Isn't this a contradiction?

I've been trying to understand how chi-squared tests work.  I've been 
taught different things and am quite confused.  From how I understand 
the test, if a test "passes" for "x" confidence then it will 
automatically "pass" for all confidence levels greater than x and less 
than 100% (since the critical values increase as the confidence 
increases).  This would also indicate that a test could "pass" for x
confidence and "fail" for a confidence less than x.

Based on my understanding it would be more logical if this test were
"passed" when the calculated value is greater than (not less than) the
critical value.

I know there's something that I'm not understanding about all of this, 
but I can't seem to find what it is.  Any help would be greatly 
appreciated.  Thanks.

Date: 10/26/2004 at 09:38:27
From: Doctor George
Subject: Re: Chi-Squared Question

Hi Joseph,

Thanks for writing to Doctor Math.

It looks like you may have misread the chi-squared table, but this is
a minor point. For 9 degrees of freedom we have

              Area (1-p)        critical value
             -------------      --------------
                 0.95              16.919
                 0.99              21.666

I am here assuming that your test only rejects high values.  The area
remaining in the tail is often called p or alpha.

It will also help to clarify some terminology.  When constructing a
confidence interval for some statistic we usually speak of a large
"level of confidence" such as 95%.  When performing a hypothesis test
we usually speak of a "level of significance" such as 0.05.  This term
can be confusing.  It is a low number because we do not want a large
percentage of errors in the test.

The main issue seems to be in understanding what is being tested in
relationship to the null hypothesis.

The critical values mark the worst condition of the data.  For good
data the test statistic will be very small compared to the critical value.

The hypothesis test asks the question, "Assuming that the null
hypothesis is true, what is the interval within which 95% of the test
statistic samples will fall?"  Looking at it this way you can see that
an interval that contains 99% of the samples would have to be larger
that the interval that contains only 95% of the samples.

Now let's turn things around.  The test also means that there is only 
a 5% chance that the test statistic will be greater than 16.919 if the
null hypothesis is true, and only a 1% chance that it will be greater
than 21.666.

The test does not prove that the null hypothesis is true or false.  It
only indicates whether or not the data is consistent with it, at some
level of signifiance.

As the critical value increases we are more confident that the test
statistic will be in the interval, but there is less significance that
can be inferred from that fact.  At the extreme, virtually any set of
data will have a test statistic less than the critical value at
99.99999% of the area, but it no longer means much of anything.

There are two kinds of errors that occur.  A type I error occurs when
the null hypothesis is true and the test fails.  A type II error 
occurs when the null hypothesis is false and the test passes.  As the 
level of significance decreases, the probablility of a type I error 
decreases, but the probablility of a type II error increases.  The 
probability of a type I error is the level of significance. 

Does that make sense?  Write again if you need more help.

- Doctor George, The Math Forum
  http://mathforum.org/dr.math/