Table of Contents
Can you explain more about the connection between Mill’s method of difference and experiments?
The two have very similar causal logic. They both try to establish a causal claim (difference in $Y$ can be attributed to changes in $X$) by leveraging on the fact that the treatment group (X = 1) and the control group (X = 0) are similar on other confounding variables ($Z$), except the treatment variable ($Y$) — since the two groups are similar in other aspects except with regard to $X$, any observed difference in $Y$ must be caused by the difference in $X$.
Method of difference try to approximate a comparable treatment and control group by selecting cases with similar attributes except $X$ (mostly based on theory and domain knowledge about what factors could be potentially confounding variables). Experiments try to achieve this by randomly assigning the treatment.
Is the concern for external validity problems only apply to method of difference, or method of agreement as well?
It is a problem in both types of designs. External validity issue is present in all studies where we only have a small number of non-randomly selected cases.
Is selecting on dependent variable only a problem for method of agreement?
Yes it is only a problem for comparative designs that select cases using methods of agreement.
We say a study is “selecting on dependent variable” when the decision criterion to include certain units into (or exclude from) the study sample is correlated with the value of the dependent variable.
For method of agreement, we are comparing cases with the same outcome but differs in the value of independent variable. In another word, the reason we are including these cases in the comparison is because that they share the same outcome, and other cases are excluded because they have a different outcome — the decision criterion for sample selection is directly related to the status of dependent variable.
Designs using methods of difference for case selection are not selecting on dependent variable. In this method, the criterion to select cases to be included in the sample is not related to what the outcomes are. Instead, we are selecting cases based on the independent variables — we are comparing the cases that are similar in all the independent variables, except one crucial explanatory factor that we are interested in.
The lecture mentioned that method of difference has trouble estimating “multiple causes”. What are some examples of “multiple causes” cases?
An event or outcome has multiple causes when there are more than one factors that could have lead to the outcome. For example, why the U.S has low voter turnout? There could be multiple factors for this: no compulsory voting; low interests in politics; election day is not a national holiday; two party system; winner-take-all system etc.
Most of the phenomenon we are interested are quite complex, so we should expect there to be multiple-causes most of the time.