Sun and Moon definitely had a boost from Go though. I just see what happened with Let's Go as a part of the trend of the decline of Pokémon in Japan, and not just a result of Let's Go being a non traditional Pokémon game.X&Y and Sun & Moon had second and third biggest launches in the franchise history (only behind Black & White) and yet they are two the lowest selling games in the franchise. Pokemon has become more frontloaded in Japan so even if the total sales are down launch sales should still be very high.
With digital vouchers also playing a part, I just don't see Sword and Shield being greater than Smash in Japan.