You don't need to ask the merchant about the memory gem, you ask Caboonga about the map, he gives you one piece but says he can't remember where the other one is so you have to go get his memory gem for him, then he gives you the map piece.
Thanks for that information, for I was also missing that level. I didn't buy the pieces of map from the merchant though, but Caboonga gave me the map and talked about the memory gem. Actually I never noticed Caboonga before.
thanks guys! now i have all 100 normal stars and the rainbow keys!
Unfortunitly, the rainbow levels (found all 4) ARE INSANLY HARD!!! and the worst part, NO CHEATING!!! now i'm pulling my hair off just to see whats inside the east temple! only if i could cheat (and yes i cheated on half the levels in the game).