Again, either will do for Elk also, like with any other firearm, bullet placement and bullet design will have more of an effect on performance than just caliber. I'd probably pick a good, full-caliber bullet for large deer or elk over a round ball in either caliber. BUT, consider that every game animal at one time or another has fallen to patched round balls in .50 and .54 caliber.
Of course the .54 does make "mo' bigga hoes".
Heck, get the .50 now, and you can always use "I need it for Elk" as an excuse to buy the .54 later.
To me, the bigger the bore, the better, but .50s aren't exactly poodle shooters.