It's called "Childhood amnesia"
This is just a false narrative surrounding the memory-loss.
Most kids under 6 don't remember their childhood friends if they lose contact and don't have photos or other memorials. Kids under 3 don't remember anything apart from learned behavior and muscle memory. Kids between 4 and 6 have difficulties retaining specific memories if they don't repeat them. Hell, you have cases of orphans, aged 4 to 6, who, 10 years later, don't remember they were adopted.
You mostly "remember" part of your early childhood because Adults talk about them.
I certainly don't remember my infamous flowerbed-digging adventure with my cousin when we were 5. The. Adults. Remember! My father had a knack talking about infamous or embarrassing things we did as little kids. He was positively giddy telling our extended family or guests about them. We cousins always questioned the severity or accuracy of these tales, but asking the more grounded and serious family members, they told us those tales were true.
https://www.scientificamerican.com/article/when-do-children-start-making-long-term-memories/
https://www.popsci.com/blog-network/kinderlab/birth-memory-why-kids-forget-what-happened-age-7