So, they look almost-but-not-quite exactly like normal inhabitants of their world, they’re theoretically immortal, and they use their vast reserves of power to control fundamental aspects of nature. Yeah, alicorns are probably eldritch abominations.
 
This is what I imagine Celestia might look like if she ever let her true divine nature shine through. Comprised of light and heat and other energies unknown to mortals, she is tied directly to the sun; if it goes supernova, so does she. If she dies, so does it. Her many eyes weep constantly, tears of liquid flame that can heal any wound, like those of a phoenix.