Hi twomoons, it looks like Rosebay Willow Herb to me. It's pretty invasive here - but it originated on your side of the pond before it was brought here as a garden plant. It gets white fluffy seed heads that float everywhere when they're touched. You can see it here alongside railway lines and embankments. It does look very nice but I don't really want it in my garden - but I keep finding it there where it has self-seeded anyway. You'll find some information here http://www.dgsgardening.btinternet.co.uk/rbaywherb.htm - does this look like your plant?
I think you nailed the ID Buddleja. We call it fireweed here and a mass field of it is a beautiful sight; (but perhaps not taking up one's entire garden.) It's among the first things to emerge after a wildfire, hence the name. Pretty bloom, even sideways LOL!