It's about them legs. What major competition did Infinity War face at the cinema? Deadpool 2, an R rated comedy, and a Star Wars spin-off that people had mixed feelings towards the moment that it was announced, starring a completely unknown entity in the lead role.
Endgame is going up against Detective Pikachu, Aladdin, Godzilla, Toy Story 4 and Lion King within weeks of coming out. These movies won't make the same splash as Endgame, but they're going to eat into its box office total in a concerted way that Infinity War just didn't have to face.
Additionally, sequels that see a drop in the box office are the norm, they don't just happen because people didn't like the last movie. Spider-Man 2 had a box office that dropped down from the first one, but they were both massive crowd pleasers in 2002 and 2004 respectively. Not to mention that Empire Strikes Back had a huge drop in box office from the original Star Wars. It's not impossible for Endgame to do better than Infinity War, but to do much better than an immediate predecessor is to be the exception, not the rule.