You're flooring it while shifting?
You're supposed to let off the gas while upshifting, and blip the throttle while downshifting, but the xB doesn't have a 6th gear, so theres no reason you should be on the gas going into 5th. Giving it gas while upshifting is putting excessive wear on the syncros. Ideally, you want the engine's flywheel to be rotating at the same speed as the clutch plate when they come together. In the real world, they're not exactly the same, so engineers put brass synchronizers in the transmissions, usually in the form of cones that come together before the actual gears, and since brass is a soft metal, they (somewhat) gently match the revs up, at the same time wearing down a little.
Okay, I kind of rambled, but thats a manual transmission in a (small) nutshell.
2005 Scion xB
Headunit: Kenwood eXcelon KDC-X996
Front stage: TruTechnology S44 (200x2@4Ω Bridged) / RE Audio RE6.5C 6.5" Components
Rear stage: Massive D800.4 / Pioneer GM-5400T / Pioneer TS-D1702R 6.75" Coax / 2x RE Audio REX6.5 6.5" Coax
Sub stage: 4x Sundown SAZ-1500D (0.5Ω each) / 4x RE Audio SX 15
Electrical: Mechman 270A / 2x DieHard Platinum Group 31 (Odyssey PC2150) / DieHard Platinum Group 35 (Odyssey PC1400) / 2x Custom 150Ah