USB Stick sync leaving files when removed from playlist

I noticed that if I remove tracks from a playlist or even a playlist itself that was previously synced to a USB stick. That although the tracks are removed from the playlist or the playlist is removed, the mp3 files still remain on the USB stick. Is there a setting to prevent this from happening?

If I remove tracks, I also want them removed from the USB stick but obviously not deleted from my library.

I plan on syncing intelligent playlists to my USB stick in the future. Those playlists will potentially grow or shrink depending on the criteria they are based on and I don't want a bunch of orphaned track files on the USB stick.  

Thanks for any help offered!