POI not maintaining selected distance

Can someone please tell me why my Mavic Zoom will out of the blue not keep the selected distance in POI mode. It randomly flies closer, sometimes as close as 3m from the object. Its not even suppose to fly closer than 5m in POI mode as an build in safety feature.