Survivorship bias doesn't have anything to do with the criteria being used in an evaluation - it has to do with the "subset" of data points included in the analysis. See the small section about "missing bullet holes" in the wiki: https://en.wikipedia.org/wiki/Survivorship_bias. It's an interesting and counter-intuitive discussion about how our intuition works and how easily our "reasoning" can be led astray by invisible and incorrect assumptions.
In that situation, the mistake the military made was to only look at bombers that returned from combat - not bombers that didn't make it back (i.e. the ones that were shot down). That led them to draw wildly wrong conclusions about where to armor up the bomber fleet. By way of analogy, this study includes UPT graduates (bombers that "make it back") and UPT washouts (bombers that "don't make it back") - it doesn't include intel school washouts and/or AFIT graduates because that isn't going to tell you anything about graduating from UPT. It didn't make sense to include data where P-38s were or weren't getting shot up because it was a study focused on bombers.
It's not survivorship bias, you're advocating for using more dimensions of data - which is fine.
A few things. First, any prediction that is going to be made, will by definition, be "backwards looking" since there's no such thing as future data. And while there definitely may potentially be better predictor variables out there, the difficulty will be to capture them in a consistent and reliable way across a large population which is distributed across multiple communities and multiple time spans - not an easy challenge. Maybe if we could somehow capture those students who used to "bullseye womprats back on Tatooine" we could enhance our process...it's challenging to get to that level of fidelity though.
Already, the fact that > 85% of UPT candidates make it through provides a high level of confidence that UPT selection criteria are pretty good - squeezing out the last few percent becomes increasingly hard in any endeavor. Any average high school varsity basketball player is in the top 1% of all basketball players on earth. Though we all know there is an enormous difference between that kid and Michael Jordan...
And finally, this is not like saying women can't be pilots. No scientific researcher looking at that data and looking at how people were selected for pilot training back in the 80s would ever draw that conclusion. I get your point about the insight gained being limited by the data, but then so is everything else because we don't have perfect measurement for anything. In any case, all the data used in this study included women.
Correct. Though I would say the model "includes" the unsuccessful events in order to learn from them. Not emphasizes.
So is your suggestion to include people not selected for UPT and then measure how the do in UPT? Or is it to just lump random people into the study who didn't go? I'd pay to see the first executed. If you're suggesting the second, then I think all that study will conclude is that being selected for UPT is the most important data point in determining who graduates from UPT - not exactly a ground-breaking research.
The point is that a study like this is not the same as a vaccine trial. You are already selecting from a group that self-selected and there is nothing you can do as the researcher to affect the outcome you want to examine (UPT graduation) from a group of people that doesn't want to be military pilots.