You’re right, the factor doesn’t want anything to do with payroll and the taxes that come with it. When you give an invoice to your customer you’ll take that invoice to the factor who will finance you for it. They pay you directly. Then you’ll turn around and use the money to pay the temp yourself. Typically there is a payroll or two that occurs before your customer pays you, so the factor just helps you receive that revenue a bit early and takes a fee for helping with the timeframe.
You’re responsible for the taxes, which is typically factored into the invoice so your customer is really paying it when they make their payment. The factor only pays you money then they’ll receive the customer’s payment from your invoice, take out their fee, and pay you what’s left. Again, they’re only paying you (the company they factor). I hope that helps.