I'll just go right ahead and assume that you never cycled your tank, or you're not providing adequate maintenance. Your current tank should not have any more fish than three goldies, since you wrote in liters, thing 40-50 liters per fish. Even if they're still small, the tank is over stocked, because they still create waste that you can't see! Just by breathing they excrete ammonia.
You should always change at least 50% every week, having such an overstocked tank. Do you have a test kit? So that you could check parameters such as pH, ammonia, nitrite and nitrate?? It's important that you do this the first few weeks/months at least until you'll know about how much waste to expect in the tank after a certain amount of time after a water change. Your best bet would be testing it weekly BEFORE the water changes, and later at least once every two weeks/month.
You could always buy sinking pellets for your goldfish, that way you wouldn't have to worry about soaking them, and they would have lots of fun digging for the leftovers in the gravel
