I'd rather see worms than millipedes, lol. Millipedes give off a weird mint-ish smell when you pick them up. They eat decaying material, but will turn to live material if necessary. Look them up to see if you have millipedes or centipedes. Centipedes hunt other insects, so I guess they would be a good thing. Either way, they do aerate the soil, lol. I found millipedes in plants I got clearanced last fall, and have been picking them out any time I see them. If your pots are really close, they might travel, but if you pick them out (use a plastic spoon and disposable cup if you're squeamish), the numbers will decrease. I throw them out hoping that birds will enjoy them, lol.
Can you post a pic to let us know what they really are? Centipedes are carnivorous; they eat other bugs. Millipedes eat plant materials that are decaying.
Centipedes bite; millipedes do not. Also, as mentioned above, the millipedes will give off an odor when they are disturbed. Centipedes don't. Centipedes are also very fast; millipedes aren't. They curl up instead of running or trying to bite when they are disturbed.
Sounds to me like you have millipedes...they can be bad news for your plant's root systems.
You could try a hydrogen peroxide infusion, and that may help. (1 tablespoon Hydrogen peroxide to 1 quart water.) This is like chicken soup; it may not help, but it can't hurt.