Checking for "enough memory" isn't that easy. Detecting this in a userspace program like clamav won't really work.
Well, "enough memory to run this program" is not the right question, and you are right, impossible to solve in general. That's not how things work though.
A well written program (in user space or anywhere) checks the return codes for each memory allocation (new pointer). When it gets an error, it uses exceptions or some other logic to propagate up to an error handler. Ultimately the user should see something.
Programmers often write code that has no error handlers, doesn't check return codes, NULL pointers, etc. It's not that out of memory is unsolvable. It's that programmers don't exercise error discipline in their code.
To just hang dead is very, very bad behavior. One expects that from Windows, not Linux.
Mark