Just imagine how crowded the place would have been in the beginning if the Christian creation myth were true and a magic man in the sky really did magic all the species into existence one day, with a hundred times as many species then as there are now!
One way evolution can drive a species to extinction is by specialisation. If the conditions are right a species can become more and more specialised to cope with them simply because each improvement produces more descendents in those conditions. The species has no choice in the matter.
But, when the conditions change, as they can do very quickly with finely balanced ecosystems and a dynamically changing climate subject to many changing factors some of them astronomical, a highly specialised species can find itself specialised for an environment which no longer exists. Not a good place to be.
I've just been reading Becoming Human: Our Past, Present and Future by Scientific American editors and it seems Neanderthals illustrate this very well. Neanderthals had lived in Europe and Western Asia for some 250,000 years - far longer than modern humans so far during which the climate became increasingly arctic as the last Ice Age expanded the ice sheets further south and turned what had been temperate woodland into Arctic tundra with the major mammal species becoming increasingly bulky to cope with the cold. Neanderthals had gradually adapted to these conditions to become, in effect, specialised Euro-Asian varieties of the Homo genus adapted to live in arctic conditions. Ice Age variant, in fact. Meanwhile our ancestors further south had had a wider range of environments to adapt to, so they had had tended to be much more generalised.
Neanderthals, for example, were short and thickset, with shorter tibia (the main lower leg bones) than modern humans. This is assumed to have been an adaptation to cold so they conserved body-heat more efficiently, having a lower surface area to mass ratio then the taller, more slender Homo sapiens. The trade off was that they were less efficient at running, taking more energy, and therefore more food, to cover the same distance as modern humans with their longer limbs and slimmer bodies. We had been evolving in Africa where it may have been an advantage to lose excess body heat. With no competition this was not a major problem for H. neanderthalensis as hunting was by ambush rather than by chase. This meant that, with a sparse population and smaller groups, the women and even the children may have had to take part in hunting the large animals, maybe to act as beaters driving the animals towards the ambush.
Modern humans, by contrast, were able to specialise with men doing the hunting and the women and children doing the gathering. Division of labour and task specialisation is always more efficient than generalisation, so, when Neanderthals came into contact and thus into competition with, modern humans, they inevitably lost out.
A number of studies have concluded that Neanderthals required more calories - some 100-350 more per day - than modern humans. Karen Steudel-Numbers of the University of Wisconsin-Madison has calculated that Neanderthals required 32 percent more energy for the same distance travelled than modern humans, for example. So, assuming competition played a part on Neanderthal extinction, the newcomers from Africa would have out-competed them purely by virtue of a more efficient metabolism with Neanderthals handicapped by virtue of their earlier adaptation and specialisation to arctic tundra. Of course we don't know for sure that modern humans competed with Neanderthals for resources but it seems very likely.
Another way in which over-specialisation would have handicapped Neanderthals would have been a reduced ability to adapt to the rapidly fluctuating climatic changes that Euroasia may have experienced towards the peak of the last Ice Age. By examining Greenland ice cores, scientists have worked out that the climate fluctuated rapidly between relatively mild, allowing woodlands to re-establish, and back to arctic tundra within as little as a generation or two.
...the [oxygen] isotope data reveal that far from progressing steadily from mild to frigid, the climate became increasingly unstable heading into the last glacial maximum, swinging severely and abruptly. With that flux came profound ecological change: forests gave way to treeless grassland; reindeer replaced certain kinds of rhinoceroses. So rapid were these oscillations that over the course of an individual’s lifetime, all the plants and animals that a person had grown up with could vanish and be replaced with unfamiliar flora and fauna. And then, just as quickly, the environment could change back again.
Scientific American Editors (2013-09-23). Becoming Human: Our Past, Present and Future
(Kindle Locations 1865-1869). Scientific American. Kindle Edition.
A species and a culture unable to adapt that quickly to these widely different environments with their different flora and fauna. Modern humans on the other hand, being less specialised, would have been more adaptable, and division of labour would have given their cultures more flexibility.
So the evidence is beginning to show that it was the process of evolution itself which was the primary cause of Neanderthal extinction. Natural selection mindlessly pushed them into an evolutionary dead end from which they were unable to reverse because evolution has no reverse gear, as they adapted to an increasingly cold Euro-asian climate. The descendants of the Neanderthal's ancestors who remained in Africa meanwhile had remained less specialised and more able to exploit the new conditions to the north as the old climate disappeared, as did the species like the mammoth and wooly rhinoceros, and H. neanderthalensis, which had adapted to it.