I don't personally mind since in my case they may just be following/watching a thread to get email notifications when there are any changes or updates done it to. I don't plan on removing them since they may come back, or just don't have time at the moment to access their accounts and post and be active.
I don't do anything to inactive members like PM them, ban, etc. I just let them be. They could be away for personal reasons or just not interested in the forum anymore, that's their prerogative. My forum does send out a monthly newsletter which helps some members trickle back in who maybe aren't regular contributors, but it's only once a month and not annoying.
If many of your forum members go inactive and/or over a short period of time you should look into content first. If it is not engaging, interesting, or of high quality that could be turning people off. Forums are centered around discussions and social interaction after all.