Hope this thread isn't too old, not too used to forums in general and I'm new here, but I wanted to add my 2c.
Sounds to me like the OP most likely has synesthesia. I felt the same, I'm autistic and as a kid a lot of stuff was brushed off (obviously, my family is all neurodivergent too and didn't know), and so my response later on just became annoyance and anger. 'You're not special, everyone gets that!', yeah, like saying 'I do this quirky thing called breathing'.
I associate letters and days of the week with colours, apparently A being red is rather common, S is sandy gold, Q is yellower gold, and some are just vague 'blue and green'. Monday is dark blue, Tuesday is red, Wednesday is purple/blue, Thursday is orange, Friday is red, Saturday and Sunday are gold/yellow. I do remember in primary school having coloured letters and days of the week hung up around classes, so that may be where it's from. They're all arbitrary things to remember, and childhood associations have a long time to be strengthened.
As I was reading the wiki article, it brought up something else which I didn't know could be counted as synesthesia, but I visualise tetris when I do maths. Addition and subtraction illicit much more visual reactions. Pairs of numbers that add up to multiples of 10 just look like they fit together. 8 and 2 look like they physically fit together in my head, 7 and 3, 6 and 4, etc. (they do not look like they fit together on paper, however). I think that must've just been the roundabout association I made learning maths as a very little kid, because multiplication and divison illicit a much less visual response, but there's still non-specific feelings of correctness.
Also as I've been learning to hear music more and use my ears to recognise intervals and tones, I have been intentionally trying to remember tones, and have associated them with vowels. The only one I can remember is that the D string on a guitar is /oʊ/. I'm not forcing myself to imagine specific vowels, I just hear it and focus 'what vowel does that remind me of'. Flatness is associated in my mind with backness and roundness, and sharpness with frontness (which makes sense). It's been an intentional exercise to use those associations to help me remember and hear, so it's not necessarily natural, but not necessarily forced.
I also have aphantasia, so that might explain why some colours are vague. I know what things look like, but I just can't see them and command them in my 'minds eye'. Meanwhile I can perfectly imagine what music sounds like, I can play songs in my head that I know, I can predict the next tone in a series (e.g. hear E A D and audiate what the G should be, still practicing that though). Can't imagine intervals so well, but I use songs for reference. I tune my guitar and instead of thinking 'sharp or flat' I think 'fur elise or jaws?'. There's also abstract textures to various chords, intertwined with song associations.
(Being able to remember songs you know very well is obviously a normal thing; this video on the Levitin Effect is rather neat, and inspiring for those of us without perfect pitch:
https://youtu.be/x1CBPV1_uTI)
I'd say a lot of this is relatively normal, again being autistic means it's hard to say for certain (though I subscribe to the idea that everyone has some degree of neurodivergence), but I'd imagine if there's anything odd it's likely my brain doubling down on otherwise normal mechanisms of association or making odd connections (tetris and maths) to give some grounding to arbitrary information.
There's also strong assocations between smells for me, in particular a certain brand of cherry blossom hand soap reminds me of two awful relationships. That may be more trauma related though.
Hopefully this long post is welcome, because for as long as I've used the internet I always seemed to love writing really long posts.