Dirt collects on the roof and washes into the barrel when it starts to rain. I magine you already have a screen to keep moquitoes from getting to the rain barrel. I do too, and am surprised to see how much fine junk gets through and clouds the water. Algae can grow in the nutrients washed in with insect parts, bird droppings, etc. Algae may not hurt the garden but could clog the barrel outlets.
Dealing with algae and gunk: Empty the barrel within a week to prevent algae growth. Add a bit of bleach to the rain barrel to disinfect ( this may run counter to some of the perceived benefits of rain water.) Paint or cover the barrel to block sunlight which in turns reduces or stops algae growth.
Preventing algae and gunk: Filter the water as it goes in. See the section in the article, "Roof washer or first flush device keeps stored water cleaner "