To illustrate the role of the significance level α and population size n in an Hypothesis test.  
(thanks to Brandon, Spring 2019):

Andy Long, Spring 2020

Imagine that we’re proposing an alternative hypothesis that population proportion p has gone down. Then we’ll reject the null hypothesis in favor of our alternative if we get a sample proportion (OverHat[p]) that’s much less than the hypothesized p.  

But how to measure “much less”? We set a bar, α, so that if (OverHat[p]) falls in the most extreme α-area to the left of the normal curve, then we’ll suppose that it probably wasn’t by chance -- that it’s probably because p isn’t really what the null suggested.  So we reject the null in favor of the alternative.

OverHat[p]

Manipulate[
Module[{p1,p2,p3,siga,zcrit},
siga=Sqrt[p(1-p)/n];
zcrit=InverseCDF[NormalDistribution[],alpha];
p1=Plot[{PDF[NormalDistribution[p,siga],x]},{x,p-3siga,p+3siga},PlotRange->All,PlotLabel->StringJoin[{"p:",TextString[p],"; α:",TextString[alpha]}]];
p2=Plot[{PDF[NormalDistribution[p,siga],x]},{x,p-3siga,p+zcrit*siga},PlotRange->All,Filling->Axis,PlotLabels->StringJoin[{"(OverHat[p])<",TextString[p+zcrit*siga]}]];
Show[p1,p2]
],
{p,0.1,.95},
{alpha,0.01,.10},
{n,20,300}
]