​@fxshaw are you in any clubs by chance? Because if anyone submits a post in any of your clubs, it will appear on your feed, no matter whether you follow that person. If that's so, you can try and diplomatically tell them that they're spamming people's feeds, and that their activities are already visible on the club's feed so it's really unnecessary to yet post them to the club. 
(I’ve been a party pooper more than once, asking those peeps to stop “sharing” every single activity of theirs in club posts - luckily they responded reasonably, as the only other alternative would have been leaving the club...)

This was it! I joined a club thinking I was joining a challenge. Poof, club gone!