If a design is too dense, it just will not work on nylon. Usually designs are specially digitized for nylon to prevent this. You could try to add spray adhesive to the backing; this helps to ensure no movement.
Thanks for asking this question. I had just bought a couple little nylon bags with intention of embroidering on them, but it didn't even occur to me that they might need special treatment. And thanks to everyone for answering ;)
Meganne is right on. I use a really heavy cut-away on nylon. I also attach it to the nylon with a temporary adhesive, so nothing moves. If you don't have a really heavy cut away, use 2-3 sheets or how ever many it takes to be able to hoop it tauntly.
Meg is right (as always) so a * for the question. Also, if I haven't said so already, WELCOME to Cute.
Iris has given you the best links.
I would suggest that your stabilser is of insufficient thickness, or number of sheets.
The rule of thumb is the thinner the material the thicker the stabiliser needs to be, even if some needs to be WSS so you can spray it away later.
I believe you should be using Cutaway, not tearaway.
You should also ensure that the material is stretched sufficiently in the hoop so that it is taut, but not Drum-tight.
Check that your tension is not too tight and your stitches are even top and bottom and that your needle is still sharp with no burrs, not overused.
If all else fails you may need to use a design with less stitches that what you are trying to stitch.
You should always opt for less dense designs on thin material, but if you must stitch a particular design you could always remove some of the underlay to reduce the density.
Hope these hints help.
hugs n roses, Meganne
Here are 2 links that can help you. Good luck with the project.