My battle system (Help needed)

I think the issue here may lie in the fact that you are using a multiplier to determine your outcome. If I am interpreting your system correctly, then "one hit = everyone hits," correct? Or - if you have five ships, and you roll one hit, then you have five hits. Conversely, if you have 20 ships, but miss with both dice, then you have zero hits. So, even if you have a large number of ships, bad dice rolls could tilt the odds quite rapidly if you are unable to land a single hit.

The traditional solution here would be to have each ship roll independently, increasing the odds that some of your ships will hit (while decreasing the odds that all of them would hit). More rolls = more stable odds. The downside to this is that this will slow your game down, which you have stated you do not want.

Another possible solution would be to simply get rid of the multiplier altogether so that one hit is only one hit, and two hits is two hits. This would be fewer die rolls than having each ship rolling independently, and would prevent one side of five ships from being obliterated in one round. The advantage of having multiple ships would be to soak up damage. Another alternative would be to scale the number of dice that you get to roll depending on the number of ships - i.e. for 1-3 ships you roll one die, for 4-6 ships you roll two, etc. This way, a large fleet would still have an offensive advantage.

Regardless of the solution, I believe more dice rolls have to be included in your battle resolution if your goal is balance - again, more dice rolls = more stable odds. Removing the multiplier (or minimizing) would also help.

Hope this helps - I'm a huge fan of 4x games, so I'm eager to see what you come up with!

Why not have ships fight 1 vs 1? Say player 1 attacks with 5 ships and player 2 defends with 1 ship. Then the first ship of player 1 attacks the one ship of player 2 and if the prior wins that settles it. But if player 2 wins the battle continues to ship 2 of the first player. In principle the second player has a chance of winning but it's not very big.
I don't know how many ships players can get, but if players end up with 20 ships each this will be a lot of dice rolling...

Another suggestion is changing the die type based on fleet size. So for instance a fleet of 1-5 ships uses D4, 6-10 ships use D6, etc. Then a single roll determines which player wins but larger fleets have an advantage.

As many ships as you want. That is good.
And I like the sound of this particular game.

Knowing how your shield, health and damage works.
Is important to know how strong your ship actually is.
This only for balance reasons though.

As one last example, I am going to fight 2 ships in 2 different little skirmishes.
On each, I start with possible 6 hits (lasers), maximum accuracy. Thus only the shields are a factor now.

Ship A has 5 shield and only 3 health.
Ship B has 3 shield but 5 health.

On ship A, only 1 laser goes through. But since the ship has 3 health, this has to happen 3 times. The durability is 3.
On ship B, 3 lasers get through. But with 5 health, it will only take 2 rounds. The durability is 2.

Thus while health and shield together is 8, the durability is different. Ship A is 50% more durable in this case.

Since the number of hits are decided by randomness, you need to take the average durability.
Ship A has a durability of 6/health. (6/(6-5))
Ship B has a durability of 2/health. (6/(6-3))
Ship A is 3 times stronger for each health.

The true worth when facing 1 laser are:
Ship A; 18
Ship B; 10
The 6 lasers are worth only 6 in this example, if the health/damage ratio equals 1.

I understand if you still don't understand. In that case how about you simply design your ships and give me the statistics. Since that particular combat mechanic looks a lot like some of my mechanics. It is easy for me to calculate the balances. I could calculate for you how strong each ship is.
This would include weapon bonusses etc.

The factor of the shields:
Shield = 0; Cost factor = 1
Shield = 1; Cost factor = 1,2
Shield = 2; Cost factor = 1,5
Shield = 3; Cost factor = 2
Shield = 4; Cost factor = 3
Shield = 5; Cost factor = 6
Shield = 6; Cost factor = infinite

***

Another question.
How to determine which ship is targeted first?
Is it the attackers choice?
Is it the defenders choice?
With the right mechanic, you could be able to make "meat" ships and "support" ships. For this you could for example allow some ships to select a target. And other ships do not have that ability.

I did not read the whole thread, but I worked a lot on ship fighting, and I wanted to avoid the fighter padding that soak hits syndrome that was in Twilight imperium. Here are a few random ideas that passes to my mind.

- Smaller ship could have an higher initiative and attack in priority to larget ships.

- You could could target ships before rolling. To avoid tiny ship to destroy large one, I used the system where larger ship roll more dice. But you need X hit to destroy a ship. So if you need 3 hit to destroy a ship, but your ship only rolls 2 dice, it cannot possibly destroy it.

- In an axis and allies system where defender choses casualties, you could make a split between capital ships and other ships. If there are more capital ships than other ships, a capital ship must be chosen as the target by the defender.

- In my ww2 pacific game, there are various type of weaponry for each ship type: Cannon, Torpedoes, bomb, etc. Each weapon attack in a specific priority order. Some weapons like cannons allow the attacker to chose target while other weapons like torpedoes allow the defender to assign hits.

- If you need different unique ship design, one solution I used is sum up the strength of all ships and each slice of 5 points makes you roll a die which can score a hit.

You might want to try using agility AND shield.
Whereas some weapons can ignore agility and some ignore shield.

Examples:
Rapid fire is a normal weapon (both agility and shield provide a die roll for blocking a hit)
Laser ignores agility (no agility die roll)
Cannon ignores shield (no shield die roll)
Rail gun ignores agility and shield (no roll at all)

Of course;
Rapid fire, plenty of rolls
Laser, depends on strength
Cannon, mostly just 1 or 2 rolls
Rail gun, the strongest laser there is.

Sorry for the late reaction.

The sequence looks logical. But have you tested it?

The fact that battle continieus on unless you retreat is a good choice.