Now that you've written it out I can tell that the "-ty" of "fifty" was probably shortened from some older pronunciation more resembling the word "ten". But I guarantee you that native English speakers are not thinking this in large numbers. This is the sort of insight that comes better when it's not your native language, others of us will just internalize "fifty" as its own word.

I guess it's also no coincidence that words ending in -teen are frequently confused with and misheard for those ending in -ty.

Edit: Seems like they derive from different Germanic roots for 10.