Questions about a hypothetical hybrid

[Zuul]

I'm working on a story that involves two species of the same genus that can interbreed, but it is rare and difficult. The Panthera hybrids are similar to what I'm thinking about, particularly the way in which imprinted genes lead to ligers being huge and tiglons being of a more modest size. Let's also say this genus is mammalian and Haldane's rule applies so that male hybrids are sterile.

So, my questions:

Does anyone have good books to recommend on hybridization like this?

Assuming a recent overlap in territory, what factors could contribute to these two species being hypothetically interfertile (at least as far as their female offspring go) and yet finding it prohibitively difficult to successfully interbreed? For the purposes of this story matings occur, but live birth is rare.

[Exy]

Assuming a recent overlap in territory, what factors could contribute to these two species being hypothetically interfertile (at least as far as their female offspring go) and yet finding it prohibitively difficult to successfully interbreed? For the purposes of this story matings occur, but live birth is rare.

Well, one factor is if they have different sexual signals (say, different coloration) or different mating practices (different mating dances or cries, or mating at different times of the year.) A lot of recognized species are interfertile but are separate different species because they rarely mate due to having some kind of incompatible mating habits rather than any incompatibility due to gametes.

[OtakuLoki]

Well, the first barrier to interbreeding, once you get them in the same territory, is going to be social. It's why the coyote and timber wolf remain separate species, even though they often share territory: While they are completely interfertile, the social structures of the two species are hugely different, and act as a barrier to prevent hybridization.

Having said that, there was a small earthquake a couple of years ago when DNA analysis of the red wolf found that that species was actually a hybridization of the timber wolf and coyote. So, the social barrier, once broken, can become a lot less of a factor as time goes by.

However, that's also with two species that are completely interfertile, not the exact situation you've mentioned.