(By "unfollow", I mean you follow some account and then later, without asking, Twitter "unfollows" that account. So you have to go back in manually and click the Follow button again to get their tweets to show up in your timeline.)
Could that be the case in your instance?