I saw a guy use a tiny end mill and it looked like it worked great to me. I used a 20 degree V bit, and it was so so. Very sensitive as far as depth went. The end mill demo I saw looked like it was better to me. I didn’t even know those tiny end mills existed. I think it was one thirty second of an inch? It was small.
I’ve used a small glass cutting bit before. They give good results, are inexpensive and you can buy them locally in almost any hardware store. And they’re carbide!
I do not know what bit to use, but I have advise for when you do not use it for PCB. When cutting anything else with a dremel, be carefull to keep your cut depth shallow or you will have your dremel tool’s shaft bend.
Be careful to make sure that your dremel tool’s shaft is perpendicular to the work area in both X and Y dimensions or you might have odd results.
60° bit is probably good for starting. You can play with my Calculator (http://woodworkerb.com/home/pcb-isolation-routing/tool-width-calculator/) to see that even minor changes in depth have a pretty substantial impact on the width of the cut. Leveling your board as precisely as possible is it good starting point. Depending on the complexity of your board: an auto leveling routine would probably be indicated.
In a pinch I used a bit that had a small ball end that I have gotten in every Dremel bit kit I’ve ever gotten. I used it just because I had a few of them kicking around but it worked fairly well. I only made two simple boards with it before I switched back to photo resist/etching so I can’t comment on the longevity.
I wonder if those glass cutting bits will last longer in the FR4… it’s FiberGLASS after all. Wondering why I never thought of trying that. Thanks for the suggestion.