The first and second bounces of the mutalisk glaive wurm do 1/3 and 1/9 of the base attack damage respectively, before armour. Although the HP totals displayed on the damaged units are rounded, it seems that they are actually stored in some format capable of handling fractions: for instance, a mutalisk with +1 attack does 10 base damage, and a unit with 0 armour that suffers 9 "second bounce" attacks from that mutalisk takes 10 damage in total. (tip: to test this, use hellions on hold position; hellions have 0 armour and 90 hp).
The mutalisk's base damage dealt at each upgrade level then looks like this:
target 1 target 2 target 3 total
muta +0 9 3 1 13
muta +1 10 3.33 1.11 14.44
muta +2 11 3.66 1.22 15.88
muta +3 12 4 1.33 17.33
The target armour works normally in every way except in one instance; that is, each target takes 1 less damage than is shown above for each point of armour it has. The exception is that it seems armour cannot reduce an attack's damage dealt to less than 0.5; any attack which would deal less than 0.5 damage after armour reduction instead does 0.5 damage. Testing an unupgraded marine against a fully upgraded battle cruiser (6 damage, 6 armour) seems to support this.
What follows is that upgrading armour tends to add less damage reduction as your armour increases, since as bounces become subject to the 0.5 rule they stop being reduceable.
For instance: assuming 0 base armour and unupgraded mutas:
target 1 target 2 target 3 total reduction added
armour +0 9 3 1 13
armour +1 8 2 0.5 10.5 2.5
armour +2 7 1 0.5 8.5 2
armour +3 6 0.5 0.5 7 1.5
Now, what's important to remember is, almost all protoss ground units come with 1 base armour (everything except probes, high templar, and archons). Since air units don't benefit from ground armour upgrades, archons only benefit from it on 10 of their 360 s/hp, and probes probably don't bear considering, we can pretty much consider "protoss ground armour" to upgrade base armours of 1. Meanwhile, the plasma shields upgrade will always upgrade base armours of 0. So as a tendancy, the shield upgrades will be more effective than the armour upgrades on their respective point pools.
Here's the "reduction added" values for different upgrades vs variously upgraded mutalisk, always assuming the target unit has 1 base armour: (the numbers after the // are for when guardian shield is active)
shields +1 armour +1 armour +2
muta +0 2.5 // 1.5 2.0 // 1.0 1.5 // 1.0
muta +1 2.588 // 1.833 2.0 // 1.0 1.833 // 1.0
muta +2 2.722 // 2.0 2.0 // 1.166 2.0 // 1.0
muta +3 2.833 // 2.0 2.0 // 1.5 2.0 // 1.0
It's true that some units have less shield points than hit points, and it's also true that the shield upgrades are more expensive.
However, all ground units except the zealot, immortal, and colossus (note: nothing with an air attack) have at least as many shield points as hit points. So in an army intended for fighting mutalisk (which among other things, should have access to guardian shield much of the time) the shield upgrade probably provides more benefit than +1 and almost certainly more benefit than +2.
As to the cost issue, +1 shields has the same cost as +2 ground armour, in addition to a faster reseach time.
As additional points, the shield upgrade will benefit pylons and cannons under harrassment, in addition to any air units you build.
So basically, I'm suggesting that all the protoss players currently frustrated with mass mutalisk styles consider upgrading shields +1 before ground armour +2. Since the cost reduction it seems like even the possibility of zerg tech switching out of muta is a pretty tolerable downside to the idea (nevermind that if you're having problems with muta, a tech switch out of muta should be low on your list of concerns).