In general I would say that there is a serious mechanical difference if there is more to the world than just those groups of characters. If the allies are sped up, they can carry out multiple actions, and move around. They can't, however, set things into action which will overwhelm the enemies very easily.
If the enemies are slowed down, however, things get more powerful. You can, for instance, block the doors and flood a room. The enemies have much less time to deal with it. The fires you set do damage multiple times before they can extinguish them, for example.
It also depends on where you are. If you are in the heart of their powerbase, you want speed, so you can hit and get away before assistance arrives. If you are in a place which is friendly to you and not them, assistance can be called in, so the balance of power changes.
Overall, I'd say that speeding you up is usually less powerful if you are in a place friendly to you, but good if you are in a place friendly to them. Slowing them down is usually less powerful in a place which is friendly to them, but great in a place friendly to you.
Separate mechanics are therefore necessary.