Elemental damage perks should effect all respective elemental damage done

I just got the eye of the storm gem, and wanted to make a storm bringer melee build only to notice that the perks that increase lightning damage does not effect the gem. (not even sure if it effects weapon elemental enchantments as well.)

It would be cool to see magic damage be used for other builds than just staves.