Xylene is a very powerful, very aggressive solvent. Used for the proper purpose at the proper time it is an invaluable tool. Used incorrectly it can give you trash in your dope.
It is a solvent of choice to clean crap out of the pill mass doing pill extraction, and I use it as the solvent of choice when I precipitate pseudo HCl. I want it to dissolve everything it can during those applications, because in those application I know it will not dissolve the chemical I want to isolate. It is one of the solvents of choice for washing the reaction fluid post-reaction, where I also want a solvent that will dissolve as much trash as anything will and still leave the meth.HI in solution.
I do not extract the meth into xylene after basing because it is too agressive a solvent in my opinion for this purpose, and pulls stuff from the based reaction fluid that naptha will not. I will, after extracting meth from the based reaction fluid, cover it with xylene to scavenge any remaining meth that the naptha might have missed. I let it sit for a day or so, then wash and gas it. I expect this meth to be dirtier than the stuff I got with naptha, since xylene is a more agressive solvent.
The solvent you use to extract the meth has absolutely no effect on the meth molecule. How clean you get your meth after extracting it and before using it has more to do with your skills and cleaning techniques than the solvent you use. If you taste xylene in your meth, you obviously are not taking the time to recrystalize your product. You may not even be rinsing it with acetone. Don't blame that on the solvent. Blame yourself.
Solvents are not good or bad in themselves. They are tools with particular characteristics that can be exploited. Knowing when to use which solvent is something you can learn by experience, or you can just take the time to read the information about solvents freely available on this board and on Rhodium's site. If you UTFSE you will find this matter has already been thoroughly discussed a number of time.