Generally speaking, we have come to expect computers to not make arithmetic mistakes. Can doing simple arithmetic be considered to be a part of Language? ChatGPT appears to make simple arithmetics mistakes. 

Understanding of language and meaningful text generation:

What does it mean when we say that an LLM understands language? Here are some tasks that humans are able to navigate without difficulty.

Are LLMs capable of doing these tasks?

Generally speaking, we can determine the extent to which an LLM truly understands language by strategically providing pieces of text to the LLM along with probe-like queries. 

Notice how ChatGPT tied itself into knots answering a logic/arithmetic question. 

At this point of time, it is difficult to imagine that ChatGPT 3.5 could be deployed, unattended, to answer even mildly complex questions regarding uncurated text. The examples above are specifically constructed to be simple and the LLM struggled to come up with reasonable responses. Note that it did not take very long to come up with these examples.

Leave a Reply

Your email address will not be published. Required fields are marked *