The first problem I met is that attaching an object to the hand makes it the hand's child. Because of that, the device's local rotation always stays zero, it is affected by the hand's initial rotation, and its position is entirely controlled by the hand's transform. Position constraints will not help you and moving the device to its initial position every frame will just make it jitter.
As a solution to this problem, I created two parts of the device: an "interactable" part, and a "visible" part. The first one contains the
Interactable.cs script, so you can attach it to your hand, but has no mesh renderer. The "visible" part has the same hierarchy, but in the opposite way to the "interactable" part, it has a mesh rendered and no
Interactable.cs script. While the player interacts with the fully transparent "interactable" part, its script (
InteractableRotationBall.cs) constantly updates the "visible" part's rotation without changing its position. In my case, I am linearly interpolating its position for additional smoothness.