Yeah, without some third party/house rules, multiclassing casters is already bad for your casting, but then also harms your BAB even more than it normally is, whereas multiclassing full martials has few downsides.