Folliculitis can go away on its own, and usually will, but if you're desperate, you can use acne products. And they'll help.
Avoid wearing clothes that you'll sweat in
Sweaty and damp fabrics are the best place for bacteria to breed, so try to avoid those. If you work out, change into dry clothes immediately after you finish.