Alot of stories? I'm pretty sure there is only one And no you shouldn't be worried. I believe that might of been a bit of a freak accident. You are using gravel meant for fish tanks and I've had gravel for years that is smaller than a pea. It is probably better to go for bigger gravel to avoid those one in a million chances. So if you are really freaked out, try to find bigger gravel. However this may be more trouble than it is worth as the gravel holds alot of the good bacteria in a tank and changing all of the gravel may destory a cycled tank.

The gravel sold at the pet shop, average pea size, is fine. goldfish have the habit to suck every single piece in creation and spit it out as soon as they notice it is not edible. IF it happens that a piece of gravel gets stuck, most times they can get rid of it by themselves. Sometimes they need help.
Therefore it is very imoprtant to know your fish well. Observe their behaviour. If they behave different to their normal behaviour you should look closely and determine the cause and act accordingly.