No. I was wondering that too, but personal fitness was a big part of my life. You could potentially cause some damage if you rush right into heavy work-outs. Just make sure you do some basic warm-ups before you start to get your blood pumping. Always try a 10-15 minute walk first, if you're working your upper body I suggest you keep light weights around and warm up your muscles for about 5 minutes. After that stretch really good until you feel everything loosened up. If you do that your work-out will be much easier and won't take a toll on your body. Don't be afraid to push yourself by any means, just make sure you warm up really good.
Anything that elevates hormone levels even if for 20 seconds technically could cause hair loss. I actually think sexual arousal can raise T levels and hence cause hair loss. Is it significant enough long term to care about though