Do you think I should prioritise Magic or Swordfighting first of all?
Also I'm level 12 and going into chapter 2 very soon. Does that seem a bit low?
I have done all sidequests.

You're fine.

I would think the answer to your question lies on what you, as a player, find yourself using more of.

Personally, I think 3-4 points is all that's needed in swordsman (whirl), and allows you to enjoy signs or alchemy without really loosing any effectiveness at swinging your sword. Your mileage may vary.