About 500,000 years ago, neanderthals and modern humans diverged from a modern ancestor. Neanderthals lived throughout much of Eurasia while modern humans lived in Africa. Modern humans developed the features we have today while Neanderthals developed characteristics for living in much rougher conditions. About 50,000 years go humans and Neanderthals encountered each other and interbred. Today, about 4% of modern human genes is Neanderthal. The question is, "why did Neanderthal-human hybrid not prevail?"
A recent study was done and it suggests that because Neanderthals were less in population, natural selection was less effective. They have always lived in smaller populations because of the extreme conditions they lived in. Because of the small population size, inbreeding was frequent. Because of this inbreeding mutations were more adverse, such as greater risks for disease, but it did not prevent reproduction, therefore massing these adverse mutations to the next generation.
In 2014, a study lead by David Reich, a genetics professor at Harvard University, found that Neanderthal DNA tended to be located far away from important genes in the human genome. This was one of the first clues that natural selection was working against Neanderthal DNA. It is possible this could be due to infertility, but it could not fully explain the pattern. It was found that the most likely reason was population size. It was found that weak natural selection due to population size differences between Neanderthals and humans could explain the distance between Neanderthal DNA and genes in the human genome today.