It makes sense, Because you have Tilt tied to size and brush tip direction tied to tilt. Simply deactivate Pressure-Tilt.
If you wish to use your stamp as a pattern, apply it to brush texture.
If you still wish to use your current settings but no distortion: use your current stamp as auxiliary stamp (this will mask your brush) and change size/spacing, change your normal stamp to any other stamp (square, round...).
Anyway, imho it's better to use auxiliary stamp as main stamp, buecause it allows for more freedom on brush stamps. Just practice to know how it works.
I don't Know if i'm explained it well.