Gooseneck is the nightmare of my perennial garden. After smothering a number of my favorite plants out, I finally completely dug up everything about four years ago - threw away the gooseneck, sifting through the soil to try finding every piece of root. But still, every year I am digging out plants all over the place, trying to keep it from taking over again. Every little piece of root makes a new plant!
Also artemisias - I started a medicinal herb garden a number of years ago, and decided to put in mugwort and wormwood before I'd done research about them. They are now weeds all over our property, because of their hearty growth and profuse seeding. You can't just pull them out, either, they must be dug out. Watch out for Silver King as well; beautiful silver spikes for the garden, but spreading underground and up through the middle of other plants, impossible to remove unless you dig everything up and re-plant.
Tiger Lilies, while beautiful, aren't a great idea for a mixed garden. We have some along the house where nothing else but ferns grow, and that's fine, but they seed by the thousands in the garden.
I love Lysimachia, but I only plant it either in pots, or in the patch of soil between the sidewalk and my yew hedge. A weekly pass under the hedge keeps the one side in check, and the other side is kept in bounds by the sidewalk. The beauty of it is that all the passing dogs can pee to their hearts' delight, and it doesn't matter, since the Loosestrife fills in almost by the end of the day!