I really love Warbreaker. I think Lightsong is a great balance between Brandon's normal "funny" characters and being more of an ass. I need to do a reread, see if knowing his origin changes that at all. Plus, I found the overarching plot to be fairly clever.
Warbreaker has several pretty amazing plot twists. It also has really good character development. Looking at reviews online I think a lot of people thought it was a weak entry into the cosmere series. For this reason I left it to last. Having read it, I think it might be my favorite.
Maybe people feel the premise is too similar to Elantris.