It's a constant battle with the mind....like a another being that is intent on dragging us away from the nectar of the Holy names. It's our choice in the end, we can listen to the mind or we can listen to the mantra and we make this choice every second of our Japa.
Of course the obvious choice is to listen to the mantra and ignore the mind and that is what we should be striving for with each bead that we pass between our fingers.
The material energy is constantly testing us to see if we are sincere about our Japa and she will throw up unlimited amounts of distractions every single day....it's a simple matter of seeing how the material energy is working around us and just ignoring it and giving our time to Krsna when we are with him during Japa.
This subject reminds me of a very nice quote from Srila Prabhupada:
If you stick to these principles with determination, you will be freed from all attachment to Maya, by Krishna's grace. The example is that when the sun is in the sky, then where is the question of darkness? Similarly, when Hare Krishna mantra is vibrating on your tongue, and you are hearing attentively, then your consciousness becomes clear, or Krishna consciousness, and there is no question of Maya or hazy consciousness.
