As Nossie has said that tank is too small, as goldfish not only grow a lot, they also produce a lot of waste and so need sufficient space and adequate filtration, ideally around 10x the size of the tank. As your finding when these requirements aren't met, the fish are effectively living in their own toilet and can become ill.
As Nossie has suggested a 150 - 200 litre tank would be a good way to go and you would need filtration that does 1500 - 2000 litres per hour, preferably an external filter as these have more space for media than internal filters. Which is important for the beneficial bacteria to grow on, as the beneficial bacteria are the one's that deal with the waste in the tank
What type of goldfish do you have?
Are they single tailed like comets/shubunkins/commons or are they one of the fancy varieties?
I ask because single tailed fish need more space than fancy varieties, as they are much more active with swimming and need more room. So if you have single tailed fish, I would consider a much larger tank than suggested (if you can) or even consider a pond.
Let us know how you get on