Epilogue

An epilogue is what you write when a book is finished, and this book really isn’t finished. The content will change along with feedback from students and other readers, and it must be revised anyway once more and more features are added to CogStat. This book is primarily online for students to use.

The undiscovered statistics

One thing that students often fail to realise is that their introductory statistics classes are just that: an introduction. If you want to go out into the wider world and do real data analysis, you have to learn a whole lot of new tools that extend the content of your undergraduate lectures in all sorts of different ways. Don’t assume that something can’t be done just because it wasn’t covered in undergrad. Don’t assume that something is the right thing to do just because it it was covered in an undergrad class. To stop you from falling victim to that trap, it’s useful to give an overview of some of the other ideas out there.

Omissions within the topics covered

Even within the topics covered in the book, there are a lot of omissions that must appear in future versions of the book. Just sticking to things that are purely about statistics (rather than things associated with CogStat), the following is a representative but not an exhaustive list of topics that we must expand on in later versions:

• Contrasts Since contrasts are not featured in CogStat yet, it is a topic omitted from this book version; however, it is an integral part of ANOVA if you want to understand more of the mechanics.
• Other types of correlations We talked about two types of correlation: Pearson and Spearman. Both of these methods of assessing correlation are applicable to the case where you have two continuous variables and want to assess the relationship between them. What about the case where your variables are both nominal scale? Or when one is nominal scale, and the other is continuous? There are actually methods for computing correlations in such cases (e.g., polychoric correlation).
• More detail on effect sizes In general, the treatment of effect sizes throughout the book is a little more cursory than it should be. However, for almost all tests and models, there are multiple ways of thinking about effect size, and we might go into more detail in the future.
• Dealing with violated assumptions In a number of places in the book, we’ve talked about some things you can do (or what CogStat automatically does) when you find that the assumptions of your test (or model) are violated. It would have been nice to talk in more detail about how you can transform variables to fix problems.
• Interaction terms for regression In Chapter 13, we talked about the fact that you can have interaction terms in an ANOVA, and we hinted that ANOVA could be interpreted as a kind of linear regression model. Yet, when talking about regression in Chapter 14, we made no mention of interactions at all. However, there’s nothing stopping you from including interaction terms in a regression model. It’s just a little more complicated to figure out what an “interaction” actually means when you’re talking about the interaction between two continuous predictors, and it can be done in more than one way.
• Multiple comparison methods Even within the context of talking about post hoc tests and multiple comparisons, it would have been lovely to talk about the methods in more detail, and about other methods that exist.

Statistical models missing from the book

Statistics is a huge field. The core tools in this book (chi-square tests, $t$-tests, ANOVA and regression) are basic tools that are widely used in everyday data analysis, and they form the core of most introductory stats books. However, there are plenty of other tools out there. To give you a sense of just how much more there is, here is a much shorter list.

• Analysis of covariance In Chapter 13. “Analysis of covariance” (ANCOVA) refers to the situation where some of your predictors are continuous (like in a regression model), and others are categorical (like in an ANOVA).

• Non-linear regression When discussing regression in Chapter 14, we saw that regressions assume that the relationship between predictors and outcomes is linear. On the other hand, when we talked about the simpler correlation problem, we saw that tools (e.g., Spearman correlations) could assess non-linear relationships between variables. Some non-linear regression models assume that the relationship between predictors and outcomes is monotonic (e.g. isotonic regression), while others assume that it is smooth but not necessarily monotonic (e.g. Lowess regression), while others assume that the relationship is of a known form that happens to be non-linear (e.g. polynomial regression).

• Logistic regression Yet another variation in regression occurs when the outcome variable is binary valued, but the predictors are continuous. For instance, suppose you’re investigating social media, and you want to know if it’s possible to predict whether or not someone is on Twitter as a function of their income, their age, and a range of other variables. This is basically a regression model, but you can’t use regular linear regression because the outcome variable is binary (you’re either on Twitter or you’re not): because the outcome variable is binary, there’s no way that the residuals could possibly be normally distributed. There are a number of tools that statisticians can apply to this situation, the most prominent of which is logistic regression.

• The General Linear Model (GLM) The GLM is actually a family of models that include logistic regression, linear regression, (some) non-linear regression, ANOVA and many others. The basic idea in the GLM is essentially the same idea that underpins linear models, but it allows for the idea that your data might not be normally distributed and allows for non-linear relationships between predictors and outcomes. There are a lot of very handy analyses that you can run that fall within the GLM, so it’s a handy thing to know about.

• Survival analysis In Chapter 2, we talked about “differential attrition”, the tendency for people to leave the study in a non-random fashion. Suppose you’re interested in determining how long people play different computer games in a single session. Do people play RTS (real-time strategy) games for lengthier stretches than FPS (first-person shooter) games? You might design your study like this. People come into the lab, and they can play for as long or as little as they like. Once they’re finished, you record the time they spent playing. However, due to ethical restrictions, let’s suppose that you cannot allow them to keep playing longer than two hours. A lot of people will stop playing before the two-hour limit, so you know exactly how long they played. But some people will run into the two-hour limit, and so you don’t know how long they would have kept playing if you’d been able to continue the study. As a consequence, your data are systematically censored: you’re missing all of the very long times. How do you analyse this data sensibly? This is the problem that survival analysis solves. It is specifically designed to handle this situation, where you’re systematically missing one “side” of the data because the study ended. It’s very widely used in health research, and in that context, it is often literally used to analyse survival. For instance, you may be tracking people with a particular type of cancer, some who have received treatment A and others who have received treatment B, but you only have funding to track them for five years. At the end of the study period, some people are alive, and others are not. In this context, survival analysis helps determine which treatment is more effective and tells you about the risk of death that people face over time.

• Repeated measures ANOVA The basic idea behind RM-ANOVA is to consider the fact that participants can have different overall performance levels. In other words, a repeated measures design means that we can attribute some of the variations in our measurement to individual differences, which allows us to draw stronger conclusions.

• Reliability analysis Back in Chapter 2, we talked about reliability as one of the desirable characteristics of measurement. For example, when designing a survey to measure some aspect of someone’s personality (e.g., extraversion), one generally attempts to include several different questions that all ask the same basic question in many different ways. When you do this, you tend to expect that all of these questions will tend to be correlated with one another, because they’re all measuring the same latent construct. There are a number of tools (e.g. Cronbach’s $\alpha$, or rather McDonald’s $\omega$) that you can use to check whether this is actually true for your study.

• Factor analysis One big shortcoming with reliability measures like Cronbach’s $\alpha$ is that they assume that your observed variables are all measuring a single latent construct. But that’s not true in general. If you look at most personality questionnaires, IQ tests, or almost anything where you’re taking lots of measurements, it’s probably the case that you’re actually measuring several things at once. For example, all the different tests used when measuring IQ do tend to correlate with one another, but the pattern of correlations that you see across tests suggests that there are multiple different “things” going on in the data. Factor analysis (and related tools like principal components analysis and independent components analysis) is a tool that you can use to help you figure out what these things are. Broadly speaking, what you do with these tools is take a big correlation matrix that describes all pairwise correlations between your variables and attempt to express this pattern of correlations using only a small number of latent variables. Factor analysis is an excellent way to see how your variables are related to one another – but it can be tricky to use well. A lot of people make the mistake of thinking that when factor analysis uncovers a latent variable (e.g. extraversion pops out as a latent variable when you factor analyse most personality questionnaires), it must actually correspond to a real “thing”. That’s not necessarily true. Even so, factor analysis is a very useful thing to know about, especially for psychologists.

Of course, even this list if far from complete.

Last words

In short, the big payoff for learning statistics the way of this book is extensibility. There are many other things that it pushes you to learn besides the specific analyses the book covers. You want to make sure that you learn to use the same tools that real data analysts use so that you can learn to do what they do.

And so yeah, okay, you’re a beginner right now (or you were when you started this book), but that doesn’t mean you should be given a dumbed-down story, a story where we omit the nightmare that is factorial ANOVA with unbalanced designs, or all the underlying formula and notation that we’ve probably bored some of you to the point of closing the browser. What you need is not to have the complexities of real-world data analysis hidden from you. What you need are the skills and tools that will let you handle those complexities when they inevitably ambush you in the real world.

And what we hope is that this book, with all its future changes and expansions, can help you with that.