They will see that you are no longer following them but they won't get a message that says "boobs is no longer following you and hates your guts" So I think you should be OK.