Enhance the current "specified address" privacy control beyond just start and end of your activity, such that any stops back at that address during an activity appear as smoothed out along the nearest road/path.
For example, when doing a long run with planned loops back to my house for refueling, if I'm not super careful and specific about when I pause and restart as I approach and leave, the current map visibility will indicate I went "off-road" and near a house right at that point. This makes me less comfortable with sharing the map of the activity, given that someone with the wrong intentions might figure out my address.
An approach might be to hide any part of the activity that enters the property boundary of the specified address, then smooth out any gaps so that it appears as though you were just running by.
