I wasn't either, until I used a randomizer hack that turns every badge, item, Star Piece, Shine Sprite or coin into a random entity that can be any badge, item, Star Piece, Shine Sprite (these seem pretty rare though, I'm now in chapter 5 and haven't gotten a single one) or even a Crystal Star (which is useless since everything tied to them is scripted in the story). Thus, enemies that drop many coins suddenly drop many other things that include badges, by chapter 4 I hit the badge limit and had to start tossing them (thankfully, there were many Double Pains and Slow Gos and the like I could toss)
Also some of the Crystal Stars you get at the end of the chapter (except Chapter 2 for some reason) is a random badge or item, though it still counts as clearing the chapter and you get the special move. It just looks weird seeing random stuff spinning around Mario.