I think a mix is nice
If they already have some smaller pieces in their mouths, they can't pick up the bigger ones
My gravel is pretty mixed in the end with only the bigger fish taking those slightly larger pebbles in their mouths. The small ones are usually satisfied with just picking on them or grabbing some smaller rocks
The community tank at my parents' place has very round and smooth gravel with pretty small rocks, and even the 4 inch goldfish are doing great with that!