What are Machine Check Exceptions (or MCE)?
A machine check exception is an error detected by your system’s processor. There are 2 major types of MCE errors, a notice or warning error, and a fatal exception. The warning will be logged by a “Machine Check Event logged” notice in your system logs, and can be later viewed via some Linux utilities. A fatal MCE will cause the machine to stop responding and the details of the MCE will be printed out to the system’s console.
What causes MCE errors?
There most common reason for MCE events to occur are:
- Memory errors or Error Correction Code (ECC) problems
- Inadequate cooling / processor over-heating
- System bus errors
- Cache errors in the processor or hardware
How do I find out what the errors mean?
If you see the message “Machine Check Events logged” on your console or in your system logs, then you can run the mcelog command to read the message from the kernel. Once you run mcelog you will not be able to re-run it to see the error, so it’s best to output the text to a file so you can further analyze it. For example:
root@localhost:/root> /usr/sbin/mcelog > mcelog.out
Some systems do this for you on a regular basis and send the output to the file /var/log/mcelog . So if you see the “Machine Check Events logged” message but mcelog does not return any data, please look /var/log/mcelog.
The output received may not always be easy to understand. If you have any questions about the decoded error message please create a support ticket and we will help analyze the problem.
What if I get a fatal machine check event that causes my machine to stop responding?
These errors are almost always caused by faulty hardware. Please capture the MCE message and you can later run it through the mcelog program once the machine is back up. Here’s an example of a message you might see:
CPU 1: Machine Check Exception: 4 Bank 4: f600200137080813 TSC b0ce27165dd3 ADDR 180ee1b40
Paste or type the error message into a file, and then run it through the mcelog for example:
root@localhost:/root> /usr/sbin/mcelog --k8 --ascii < myerror
Use the –k8 option if you are using an AMD Opteron or Athlon 64 processor, or substitute it for –p4 for a Pentium 4 or Xeon. Here is the output from the previous MCE error:
HARDWARE ERROR. This is *NOT* a software problem! Please contact your hardware vendor CPU 1 4 northbridge TSC b0ce27165dd3 Northbridge Chipkill ECC error Chipkill ECC syndrome = 3700 bit32 = err cpu0 bit45 = uncorrected ecc error bit57 = processor context corrupt bit61 = error uncorrected bit62 = error overflow (multiple errors) bus error 'local node origin, request didn't time out generic read mem transaction memory access, level generic' STATUS f600200137080813 MCGSTATUS 4
This indicates that an uncorrected ECC error occurred. This indicates that one of your memory modules has failed. For further analysis please submit a support ticket with the complete MCE error message and the output of mcelog.
Request a Consultation from our team of HPC and AI Experts
Would you like to speak to one of our HPC or AI experts? We are here to help you. Submit your details, and we'll be in touch shortly.