Table of Contents
In the class we talked about surveys having high external validity, but weak in internal validity. Does external validity take precedence (over internal validity) in terms of importance, or vice versa?
I would say that in general, it is more important to establish internal validity than external validity. If we can ensure internal validity, at the very least, we can claim to have gained some localized knowledge ($X$ causes $Y$ in the sample we have studied), even if this knowledge might not hold in another context.
However, if we cannot be sure that the findings in our current study is internally valid (i.e. if we are unable to establish a credible claim that it is indeed $X$ that caused a change in $Y$, rather than other confounding factors), then what’s the point of generalizing this invalid claim? Only when we have confidence in the internal validity of a study (more localized knowledge), then having external validity will be useful (allow us to expand on this knowledge). Otherwise, generalizing a wrong-headed conclusion only compounds the initial error, like adding more heights to a building with a faulty foundation.
Is random sampling and randomization the same thing?
Random sample refers to a sample (i.e. the subset of population that we include in the study) where each unit is chosen randomly. This concerns the cases or subjects in the study.
Randomization (a.k.a random assignment) refers the process of randomly assigning each unit in our study to receive the treatment or not. This concerns whether the units/subjects (who are already included in the study) is receiving the treatment, or will they in the control group.
How can we account for coverage error in experimental studies?
Depending on how the subjects are recruited into the experiment, coverage errors in experimental studies can be difficult to avoid. Recall that many experiments, especially lab experiments, rely on convenience sample, which usually leads to part of the population not being covered in the sampling process. If subjects are recruited among the college undergraduates, then anyone who is not a undergraduate from that university is excluded from the sample.
This problem can be difficult to “account for” if we are using convenience sample, since it is built-in to the sampling process. However, other types of non-laboratory based experiments (e.g. survey experiments or field experiments) often have better coverage, which mitigates (though does not 100% eliminate) the problems of non-representative sample that comes with coverage errors.