Mario Party Advance is clearly the one with the least amount of characters, but that's clearly the easiest one to pick.
Thus, I consider the next worst one Mario Party 2 (and to a lesser extent, Mario Party). The character variety is limited in the perspective of today, where it lacks the mainstays of Daisy and Waluigi who are basically the series standard in the same way Wario is standard in Mario Kart (not in the first game(s) but still considered a famous inclusion within the series). Mario Party 2 edged out a bit because it's still the only game with costumes, and it's a shame that the series is unlikely to have costumed characters, unless they actually did that.