If you try in ChatGPT:

“what is the numeric value of 2 * ( 5 * 2 ) replacing ‘*’ by the addition operator?”

You will mostly get wrong answers.

A simulation of 25 questions / answers using davinci 3, gets 50% of the answers right and wrong.

Answer: ’20’ – Count: 13
Answer: ‘2 + 5 + 2 = 9′ – Count: 9
Answer: ’22’ – Count: 2
Answer: ’14’ – Count: 1

Results show how the machine always remembers the expression but in half of the answers it has forgotten the basic premise.