I would think that it would be tougher as well. Where aluminum would bend and stay bent (plastic deformation) engineering plastics like nylon and delrin will flex and return to their original shape.
In my experience with plastic vs CF parts in large helicopters CF is stiffer and lighter but doesn't handle crashes as well.
It never ends well when people dink their RC helis.
FPV racing drones are generally made out of carbon fibre, because it's light, stiff and durable enough if the design is right.
There's a compelling case for Woven Carbon Fibre for simple chassis designs over alloy. If a chassis is laid-up right, the combination of lightness, stiffness and durability can be honed to perfection.
In the case of the Baja 5B, the chassis being a bathtub design means it is inherently stiff, so to delaminate a Carbon chassis, you'd have to have done something ridiculous.
The 5B isn't really a jump rig or extreme Kevin Talbot basher.
Better stick to milk crates for that, or 4WD, M2C laiden machines. Plenty of videos detailing that kind of crazy stuff. Stephane Z here lands his crazy jump rigs perfectly - manages not destroy his machines constantly.
The Baja is a 5th scale, classic 2WD buggy desert dweller, designed by the same person who brought us the Kyosho Ultima. The similarities are quite apparent.
It was always HPI's vision to see these rigs race in a proper, popular series.
https://www.hpiracing.com/en/article/view/2011061702
Large Scale races up to that point had been limited to FG and Yankee, which whilst high quality offerings, were dated and simply weren't as popular as 1/10th and 1/8th scale for racing.
Now we have all the fun of the HPI Baja, with the convenience and torque that brushless systems afford.
Your rig - your rules. Only you can decide what parts you want to change and why.