The pale skin association to the word fair comes from two places. Fair skinned, means pale skinned and the classical association of paleness and high class. Darker skin (sun tan) means you are working class/peasants. It's implied rather than explicit in the classical use.
I'd say its fantasy usage goes even a step beyond this, given that Drow in DnD are seen as particularly evil and wicked despite that mostly being a result of a brutal society rather than any inherent evil or monsterhood (as is the case for stuff like Liches or Phase Spiders).
So, particularly in the case of fantasy, dark skin (see: orcs, goblins, etc.) tends to be associated with evil while pale skin (elves in particular) tends to get associated with good. Like, particularly with Tolkien's classics, you'll notice that pretty much all good characters are pale-skinned and pretty much all bad characters are... well, not.
Additionally, females get treated as pretty much entirely incapable in most fantasy books. They very much follow that dumb adage of, "best seen, not heard". Which also happens to relate to a lot of the direct objectification they receive in modern fantasy. In highly stratified societies like Japan's, China's or Korea's, this sort of thing is even worse in their classical fiction (despite, ironically, females often serving in historically gendered roles in those societies, such as the military or administration - though much of this changed or evaporated post-WWII, particularly the possibility or desirability of a respectful, equal relationship).