I have a small Gravel garden in London, England, and I adopt a similar policy, as plants from a garden center are very expensive.
A lot of the perennials seed in drifts along the gravel borders, so I thin them out if they get to much, apart from foxgloves as they grow everywhere, so I have to be quite ruthless with them.
The annuals are much the same, you can also see drifts of seedlings growing in the gravel depending on which way the wind was blowing!
I thin them out (easy in gravel) and put them in gaps; but as there are always so many, I do my wife's Hanging baskets with them and two large planters, as my wife like summer colour, until her beloved Acer's start in the Autumn.
Great article and especially for the environmentalists amongst us, no use of Gas guzzling cars to get to the garden center, heated greenhouses etc.
I enclose a picture of some Cosmos that has seeded itself amongst a herbaceous border.
Regards from a wet England