So you seem knowledgeable about all this, what has the effect of having pretty much open market of Lolicon/Shotacon have on Japanese society since it's not regulated at all?
That I'm admittedly less sure about, and I apologize if that's unsatisfactory. As I pointed out, it's still very much a niche thing that's more "tolerated" than "welcomed". I do feel like, should it become more normalized, it may influence Japanese culture in more unsavory ways, such as normalization of the sexualization of children in general and not just the sexualization of fictitious children. After all, Japan only very recently actually outlawed real child porn, and how frequently lolicon is used may be a result of that going unchecked. But for now I view it more as a problematic aspect of society that is allowed due to the current government of Japan being as it is and causing other problems as well.
This is probably the boring "closest to reality" take on the situation. The problem is why the heck is this really creepy content so damn prolific in japan???
The thing is- it's really not all that prolific. It's just the media that we get exposed to is the media that has these problematic aspects. There's a bunch of anime that localizers (whether official or fansubbers) don't touch that's devoid of this stuff (admittedly, mostly kid or family friendly shows), and J-drama and J-comedy has basically no western marketshare. It's certainly MORE prolific when you compare it to other cultures' media outputs, but in the grand scheme of Japanese media it's still niche. A problematic niche that is without question influencing things outside of its niche due to the spending power of both lolicon and moe otaku. But a niche.
That, incidentally, may somewhat also give another answer to Blackage's query: That it'll influence other media to be more similar even if the creator didn't initially want it to be, as it'll help drive sales of merchandise. This hampers artistic integrity.
Now if you're asking
how there's a market, well, someone earlier mentioned Rei and Evangelion, and I imagine that's a big part of it. You have EVA unintentionally tap into this market of people attracted to a 14 year old who previously kept to doujinshi (when it intended to in fact satirize the subculture of otaku fetishizing women characters), so when the anime market started to falter, creators start pandering to this market with spending power to stay afloat. And then with the moe boom, basically half of the characters in shows these days look like the lolis of yesterday, even if adults. As much as I hate to say it, moe culture and lolicon culture are very much intertwined and I'm unsure if it's possible for the two to not be.
As an aside, thankfully there's a good number of shows each season that don't pander to either the moe or lolicon crowd, but they are rarely the big shows of the season among most of the English speaking anime community. You have to dig to find them, even if they are localized, because, well, they don't make money via merch sales, so companies don't want to push them, even if there are at times as many of them as there are problematic shows. Hence why they're not popular. Advertisement works folks.
And if you're asking
why there's a market at all...
One article I read featured a professor- Masahiro Morioka- who is self-admittedly a lolicon and acknowledges it's problematic and pedophilic. He believes the reason he and others enjoy lolicon is not due to pedophilia itself, but instead partially due to repressed sexuality and the inablity to express one's feminine side. He noticed the start of his lolicon desires arose sometime at the height of his puberty when he found distaste at growing facial hair, becoming more muscular, and developing masculine facial features, and truly took off when he lost notable contact with his mother who was the only real woman in his life. He says he enjoys lolicon because it allows him to re-experience youth by inserting himself into the place of these girls. And he felt that if men were able to more freely express their sexuality and how they view themselves without being shunned, they'd feel less compelled to express it in problematic ways, such as lolicon.
And I mean, it's a theory that makes SOME sense, when you consider two aspects Japanese culture: the traditional value of youth and beauty, and the imported repression of sexuality outside of a very patriarchal, heteronormative viewpoint.
Youth and beauty are both
highly prized, individually and together. One of Japan's national heroes, Minamoto no Yoshitsune, was described as a child prodigy, and only lived into his 30s, and often is emphasized in war tales as being androgynous for his whole life. And most folktales involve young or beautiful (or both) people in some way. And Japanese women often find bearded or overly muscular men unattractive - western concepts of masculinity are just not desired. This worship of youth and beauty is most likely due to Shinto beliefs, and how while it's true old people are venerable and wise... they're also really close to death, and thus really close to dying and their bodies becoming instantly impure.
Further, public displays and conversation of romance or sexuality are frowned upon. You can't even hold hands with or kiss your partner without getting dirty stares. Sure, enjoy all the depraved, kinky porn you can get in private. But shut up about sexuality or romance at all in public. This of course can be linked back to the American occupation of Japan and further back to Japan's fascist stage. Both 40s/50s Protestant America and fascism alike frown upon any mention of sexuality outside of missionary, clothes on, married-for-procreation sex. And Japan just really hasn't changed since then, due to its reticence to change because it's too much effort and causes problems. So people express their sexual urges in problematic ways in private, as over time they have become desensitized to less problematic sexuality as sexuality in general is thought of as taboo, so they associate their sexual urges with being inherently taboo.
The next two paragraphs are my own theorizing and drawing parallels here, but... You can perhaps find a precursor to the phenomenon of lolicon's appeal, as described by Morioka, as being a way in which repressed men could express their non-harmful desire in harmful ways in the early to mid Edo period through wakashudo. While not
as stunted as pre-modern and modern Japanese sexuality, the Edo bakufu encouraged a rigid idea of masculinity over all and repressed the idea of femininity or androgyny being acceptable for men and the idea of sexuality being acceptable for public consumption. This was a sharp contrast to previous Japanese customs, and an attempt to incorporate even more Confucian ideals than had already been utilized. This was reflected by previously androgynous cultural heroes now being depicted as much more masculine.
But the lolicon-precursor result of this (well, a result should Morioka's theory hold true), you saw many men engaging in aforementioned wakashudo-
pederastic relationships with prepubescent boys and young teen boys. This was considered acceptable as it wasn't men giving into their lust for women (showed weakness), nor grown men choosing to display themselves as feminine or a bottom (also showed weakness), nor women expressing sexuality at all (because it was a Confucian, patriarchal society). Wakashudo was practiced before the Edo period, but often with partners around the same age, and lasted into old age. In the Edo period, it was pretty much exclusively with men 30+ with boys under 15, with the men changing out their boy when they became adults around 15~17. Late Edo Japan began to ease up on Confucian ideals a bit, and as a result wakashudo began to die out. But then Matthew Perry's black ships came, along with a new, non-Confucian brand of patriarchy- Western-style Imperialism. While this killed off wakashudo due to homosexuality being considered bad, it didn't kill that desire for youth nor the belief that men cannot be feminine.
But even if it makes sense, don't get me wrong: I don't feel it's an excuse. Lolicon is problematic, and even it's a result of men feeling shamed into liking more and more taboo things, it shouldn't be indulged due to the visible impact it's having on other media and the potential impact it has on the safety of real children. But it's sadly something that also can't be cured by societal change unless, again, progressive politicians are elected to replace the LDP and thus enable progressive voices to speak up without fear of social reprisal. Under the current government, there can be no major societal shifts, because the market demands there be none, and to the LDP, the free market comes first, nationalism second, everything else a far third.
Of course, this doesn't explain everything perfectly- there's still a good amount of Japanese media that features adult men who are androgynous or feminine who men could look to in order to idolize, for instance, instead of little girls (of course part of that may be structural, subconscious homophobia)- but it does explain how a combination of different issues can combine into one singular, truly problematic one.
-
To summarize or tl;dr: The reason Japan's got really fucking bad cultural sexual psychosis that manifests in problematic, creepy ways is most likely due to their long-standing cultural beliefs clashing with outside beliefs on sexuality and gender roles which their conservatives follow to be "modern" and "good". Said conservatives simultaneously do not want to upset the market forces they so worship if it's not in direct opposition to their continued structural power, which lolicon isn't. Under such a system, leftists and progressives feel too throttled to try to break free, and so become complacent, and thus there's no real Japanese pushback against lolicon in any major way.
Again, I know I sound like a know it all. But if anyone more knowledgeable thinks anything I'm saying is bullshit, please, call me out on it or correct me. But this is what I think I know to be true based on what I've studied about Japanese culture in college and my free time.
Also apologies if I went on too long again. I just feel like if we want to change things, and not just complain about them, we need to understand them.