Hi:alpha1 = alpha0/N; where N is the number of tests.
Suppose we have 3 new methods of medical screening and we want to know whether: 1) any of them perform better than the existing standard method, and 2) the order of their performances, i.e., the best, the 2nd, and the 3rd.
We test them by using the same set of samples and we use the following metrics for evaluating their performances: accuracy (AC), sensitivity (SE), and specificity (SP).
Now we have many comparisons to do, and it seems that this would raise an issue of false positive.
One way to solve this issue is to divide the alpha value by the number of tests to impose more stringent criteria on each of the tests regarding the false positive. That is, instead of using the original alpha (e.g., 5%), we use the corrected one:
Then we conduct the student t-test to see if any of the tests would be statistically significant.
But now we have some questions:
1) what is the value of N?
2) by reducing the alpha from alpha0 to alpha1, we have made each of the test more difficult to be significant, wouldn't this increase the rate of false-negative? If so, how do we resolve this issue?
On Thursday, August 27, 2020 at 9:59:03 PM UTC-4, Cosine wrote:
Hi:
Suppose we have 3 new methods of medical screening and we want to
know whether: 1) any of them perform better than the existing
standard method, and 2) the order of their performances, i.e., the
best, the 2nd, and the 3rd.
We test them by using the same set of samples and we use the
following metrics for evaluating their performances: accuracy (AC), sensitivity (SE), and specificity (SP).
Now we have many comparisons to do, and it seems that this would
raise an issue of false positive.
One way to solve this issue is to divide the alpha value by the
number of tests to impose more stringent criteria on each of the
tests regarding the false positive. That is, instead of using the
original alpha (e.g., 5%), we use the corrected one: alpha1 =
alpha0/N; where N is the number of tests.
Then we conduct the student t-test to see if any of the tests
would be statistically significant.
But now we have some questions:
1) what is the value of N?
2) by reducing the alpha from alpha0 to alpha1, we have made each
of the test more difficult to be significant, wouldn't this
increase the rate of false-negative? If so, how do we resolve this
issue?
Before you proceed with t-tests, I suggest that you take a look at
the book by Robert G. Newcombe:
Notice that there is an entire chapter on screening and diagnostic
tests that includes these sections:
Background
Sensitivity and Specificity
Positive and Negative Predictive Values
Trade-Off between Sensitivity and Specificity: The ROC Curve
Simultaneous Comparison of Sensitivity and Specificity between Two
Tests
And the Support Material tab on that web-page has a link to a zip
file. It contains an Excel workbook in which Newcombe has
implemented most of the methods he describes.
As to your concern about maintaining control over the error rate, a
false discovery rate (FDR) approach might make more sense than a
Bonferroni correction.
HTH.
Hi:
Suppose we have 3 new methods of medical screening and we want to know whether: 1) any of them perform better than the existing standard method, and 2) the order of their performances, i.e., the best, the 2nd, and the 3rd.
We test them by using the same set of samples and we use the following metrics for evaluating their performances: accuracy (AC), sensitivity (SE), and specificity (SP).
Now we have many comparisons to do, and it seems that this would raise an issue of false positive.alpha1 = alpha0/N; where N is the number of tests.
One way to solve this issue is to divide the alpha value by the number of tests to impose more stringent criteria on each of the tests regarding the false positive. That is, instead of using the original alpha (e.g., 5%), we use the corrected one:
Then we conduct the student t-test to see if any of the tests would be statistically significant.
But now we have some questions:
1) what is the value of N?
2) by reducing the alpha from alpha0 to alpha1, we have made each of the test more difficult to be significant, wouldn't this increase the rate of false-negative? If so, how do we resolve this issue?
Hi:
Let's make sure about finding if a new method of screening is better than the others.
Suppose we have 3 new methods: S1, S2, and S3, and we use only the sensitivity (SE) for comparing the performance.
3. Could we conduct a statistical test to show that:
S1 performs better than S2, S1 performs better than S3, and the level
of the superiority of S1 against S2 is higher than that of S1 against S3.
3. Could we conduct a statistical test to show that:
S1 performs better than S2, S1 performs better than S3, and the level of the superiority of S1 against S2 is higher than that of S1 against S3.
Sysop: | Keyop |
---|---|
Location: | Huddersfield, West Yorkshire, UK |
Users: | 113 |
Nodes: | 8 (1 / 7) |
Uptime: | 126:40:20 |
Calls: | 2,501 |
Files: | 8,692 |
Messages: | 1,924,476 |