Hey there, Baby Luigi here, going to comment on this issue because you know, I'm here whenever you need to ask anything them Baby Mario Bros.
Anyway, yeah, as others have earlier pointed out, Baby Mario and Baby Luigi do not spend most of their time with the Yoshis. Only a fraction of the time was disrupted by Kamek and Baby Bowser, the rest are basically behind the scenes. In the Yoshi's Island games, Baby Mario and Baby Luigi were portrayed as younger, less sentient characters who are mostly unable to speak or think for themselves, the only exception being Yoshi Touch & Go, the only Yoshi game where Baby Mario receives his proper Charles Martinet voice and therefore speaks in that one.
I have to question why they speak in Italian accents though, from a narrative standpoint and not because they're younger versions of Mario and Luigi. It either means they're bilingual in this age (unlikely scenario) or they were raised with peers who also had Italian accents (more likely scenario) as kids adopt accents of their peers rather than their parents, which begs the question, is Mushroom Kingdom have a large embedded Italian culture or what?
Even if Baby Mario and Baby Luigi don't spend time with English-speaking characters, language will come to them innately, just not under English regardless. It's just what humans are, humans are wired towards language as they're adapted to it.