All the time! I still have a single note riff that's not that fast and sounds quite easy but I cannot play it up to speed whatever I do!
I think this is good though: You're progressing your technique through and into your own music rather than through a bunch of exercises.
I find the following helps:
1) You must allow time for your "fingers" to know it...it's no good if you have to consciously think about all the finger movements. No good at all.
2) As already said, practice it slowly. A metronome or drum machine will expose the ragged parts of it. Without something to force you to stay in time you can kind of fool yourself that you can play it in time when perhaps you can't.
3) Break it into small sections and get those down, then join them up BUT also practice *across* the sections so you are practicing the "joins".