Goldfish are fine, with most gravel as long as it is smooth. I haven't heard of a goldfish swallowing gravel, but if it's oddly shaped it can get stuck in their mouths and they will be un able to spit it back out, but generally they'll just pick gravel up and spit out while foraging for food.
I have rather large gravel, that my goldfish searches through, it's too big for her to pick up, so usually she just digs her head in
As for gravel picking up waste? Not entirely sure what you mean, but goldfish are rather messy creatures so any poo or left over food will end up in the gravel regardless and need cleaning with a gravel cleaner.