Well, you need to be a fourth level fighter in order to take the feat Weapon Specialization. As in, Fighter 4, not Swashbuckler 3/Fighter 1.
If your DM lets you count the Swashbuckler levels, Swash 3/Fighter 1 qualifies for that feat. And it's really such a minor thing, I doubt your DM will care.
And yeah, both classes are very weak, in all honesty. You can min-max them a little to do some cool stuff, but noncasters are always a little on the weak side in comparison to casters. And by a little I mean a lot. But whatever, it's not always about power.
If you're interested in being a melee and having a lot of fun, ask your DM about the Tome of Battle. It gives you abilities to use that are, well, fun as hell.