Throwing in my two scene
First determining the level of technology so far as scuttling weapons. Cannons have already been mentioned but they aren't the only high seas style of making a crews day miserable. Before the invention of the cannon, highly flammable materials were used and launched by lariat catapult from ship to ship. I was highly effective. The substance know as Greek Fire would burn on the surface of water for hours and was down right impossible to remove from the surface of the ships. As the only difference, crews on the opposing ships would use a heavy parafin wax to slow the fire and make it more manageable. This wasn't the only weapon. Windless Scudda crosbows were used to poke holes in certain areas of the ship to reduce its maneuverability. The shafts of the bolts where even fluted to assure the ship took on water. The extra weight down that end of the ship causing it to list. The whole tactic was to slow the opposing ship down for the coup de tas, THE RAM.
Later developments of cannons didn't even end the use of these practices. Now, inflammatory shot was propelled over a greater distance by mortars. Even more interesting was a device used by blue beard. A cannon shot an anchor attached to a chain into the side of the hull. A trebuchet would then launch a heavy weight over the other side causing the ship to list heavily. The idea was to make maneuverability impossible and at the rick of capsizing. Chain shot was another adaptation used to destroy rigging and mast on the opposing ship. A weighted chain was shot like a bolas, decimating ropes, rigs and anyone in the way.
If you're looking for a good mechanic, look only to the 7th Sea adaptation of ship to ship. Rogue trader has some good ones too. It's basically like this. every section of the crew represents an important function of the ship. Speed maneuverability, fire power, hull capacity, the crews backbone, the leadership and their knowledge of tactics. Each are vulnerable to specific attack and each has a specific response.
In my fantasy gem there are several fazes.
Tactics and Leadership: The commanders ability to get the crew moving functionally and identifying the kinds of incoming attacks. This also includes Moral Resolve if the crew is losing moral.
The Magic roll off: Every ship is armed with three kinds of mages, Communications mage/Assault mage/Meta Mage (defense Mage). They all roll off and the dice determine who is capable of doing what (usually by contest.) Obviously the Assault mage is trying to hit the opposing ship with a spell, the Meta Mage is trying to prevent that and the communications mage is constantly trying to feed the captain with intelligence on the other ships maneuver. Any one F*cks this up? And an attack gets through.
Maneuver Faze: The ships movement comes after all this believe it or not. This is to represent attempting to maneuver a better opportunity and is largely based off of the Leadership faze.
Ordinance faze: What ever the appropriate Cannon, Mortar, Ballista, Scudda, Trebuchet, Lariat, etc. This will also include mand fire power like guns, longbows and cross bows even spears.
If the two ships have maneuvered close enough together
Boarding Faze!: Siege devices are linked between ships including more daring feats such as swinging in. You don't roll for every person of course. You roll the tactics of the boarding commander (either the captain or a crony). Of course for this section you may want to determine the difficulty modifier of boarding the ship plus the weapons being used for the boarding procedures as well as the time spent in combat (given fatigue and moral over a certain period of time). The cutlass for instance was designed for ship boarding, the awl headed polearm became a great weapon of defending against boarding parties. These are just things to consider.
Who wins?: Remmeber your crew represents that ships ability to function. the more each station loses, the smaller that stat gets. Out side of that, the ship itself has hit points segmented around certain sections (the broad side being the weakest)