In Iceland it is totally safe to drink cold water from the tap.
When travelling in Iceland you may notice a “rotten egg” smells when running hot water in a sink or shower—that is simply a naturally occurring smell from the geothermal water source, and it poses no health risks.