I have had a worm on my brugs that comes out only at night (during their summer outside), but would eat big pieces of the leaves. I couldn't find the worm, since it apparently was hiding in the soil or debris during the day. I put a cottage cheese lid loosely on the soil, and the worm hid under the lid during the day. I found the worm curled up under the lid, and disposed of it. A bit icky, but a very natural solution for me, although I don't know if you have the same kind of worms.
If your worms are hiding in the soil, get some Systemic Granules and sprinkle them on the top of the soil and water it in. It will kill everything in the soil, plus the plant will absorb it in it's "System" amd any sucking insects on the plant will die from the "juices".
You couls also spry the soil with an insecticide and see if that works quicker.
Neither of these would be "natural". They would be Chemical.
Here is a link to a site for natural Insect control. Hope you get some ideas from this.
I live in San Diego as well...great place for gardening. My brugs have no worms...I tried an alternate approach of growing a few cabbages around them to keep them interested in those instead. Then, my bearded dragon gets to enjoy the lucious, pesticide free worms/larvae/etc.
I am also told by some gardening friends that the combination of milk ( mixed with H2O as well as straight), coffe grounds, vermiculture, and the above-mentioned deterent seem to be the key for my not having too many worms or pests.
There are also some "good bugs/flies" you can get to help keep your garden safe.
If worse comes to worse, I can bring over my 3 garden guardians---chameleon, bearded dragon, and water dragon. They'll do the trick!