Handles after_add and after_remove outside of the AR callback flow to keep things that aren't directly tied to the model outside of it