So what's to stop the forks from collaborating so as to separate some of that duplicate code into a separate library that both the forks use?

you're basically describing a modding API/the game being factored out into an engine.

What benefit is there to playing with two different mods if they don't work well with each other?

mods don't have to be explicitly programmed to work together to so from a gameplay perspective, especially if they pertain to unrelated parts of the game. even if they don't entirely work with each other from a gameplay perspective, it can still be very fun to play. I have had lots of fun with mods that destroy the balance/progression of games. just because something is not perfect does not mean it's not worth playing.