[Linux] abrt 產生的 core dump 不見了？
今天同事回報說有程式掛掉，但是 abrt 的目錄裡沒有看到相關的 core dump～
這真是怪事一椿，在目錄裡找了半天，還真的沒看到，
但程式確實是有 crash 過，而 abrt 目錄裡也有其他的 core dump，
所以並不是 abrtd 沒有作用…
雖然 abrtd 應該是有作用，但還是先檢查一下 abrt 相關的 daemon 有沒有在執行，
用 systemctl 看一下，看起來都很正常：
root@localhost / # systemctl | egrep abrt
abrt-ccpp.service loaded active exited Install ABRT coredump hook
abrt-oops.service loaded active running ABRT kernel log watcher
abrtd.service loaded active running ABRT Automated Bug Reporting Tool
看一下 abrtd 的狀態，正在執行中，而且跑了一個多月了，
在下面的訊息裡注意到有 Deleting problem directory 的字眼：
root@localhost /var/log/abrt # systemctl status abrtd abrtd.service - ABRT Automated Bug Reporting Tool Loaded: loaded (/usr/lib/systemd/system/abrtd.service; enabled) Active: active (running) since Thu 2016-01-28 11:45:13 CST; 1 months 22 days ago Main PID: 998 (abrtd) CGroup: /system.slice/abrtd.service └─998 /usr/sbin/abrtd -d -s Mar 21 13:49:02 localhost abrt-server[23468]: Email was sent to: root@localhost Mar 21 13:55:45 localhost abrt-server[37655]: Generating core_backtrace Mar 21 13:56:06 localhost abrt-server[37655]: Duplicate: core backtrace Mar 21 13:56:06 localhost abrt-server[37655]: DUP_OF_DIR: /var/log/abrt/ccpp-2016-03-21-06:00:57-58931 Mar 21 13:56:06 localhost abrt-server[37655]: Deleting problem directory ccpp-2016-03-21-13:55:44-37481 (dup of ccpp-2016-03-21-06:00:57-58931) Mar 21 13:56:07 localhost abrt-server[37655]: Email address of sender was not specified. Would you like to do so now? If not, 'user@localhost' is to ...ed [y/N] Mar 21 13:56:07 localhost abrt-server[37655]: Email address of receiver was not specified. Would you like to do so now? If not, 'root@localhost' is t...ed [y/N] Mar 21 13:56:07 localhost abrt-server[37655]: Sending an email... Mar 21 13:56:08 localhost abrt-server[37655]: postdrop: warning: unable to look up public/pickup: No such file or directory Mar 21 13:56:08 localhost abrt-server[37655]: Email was sent to: root@localhost Hint: Some lines were ellipsized, use -l to show in full.
基本上 abrt 不會記錄重覆的 core dump，因此看到這種 DUP_OF_DIR 的訊息應該不奇怪，
不過好奇下，在 /var/log/messages 下找了一下 abrt.*deleting 的字句：
2016-03-21T06:00:58.230478+08:00 localhost abrt-hook-ccpp: /var/log/abrt is 1382018600 bytes (more than 1279MiB), deleting 'ccpp-2016-03-21-05:28:18-5996' 2016-03-21T13:48:58.190855+08:00 localhost abrt-server: Deleting problem directory ccpp-2016-03-21-13:48:39-23254 (dup of ccpp-2016-03-21-06:00:57-58931) 2016-03-21T13:56:06.163878+08:00 localhost abrt-server: Deleting problem directory ccpp-2016-03-21-13:55:44-37481 (dup of ccpp-2016-03-21-06:00:57-58931)
上面訊息裡看到了因為超過 1279 MiB 所以砍掉了某個 core dump 目錄的訊息，
這才真相大白，原來是空間的限制，導致 abrt 將舊的 core dump 給砍掉了～
看一下 /etc/abrt/abrt.conf 的內容，裡面有個 MaxCrashReportsSize 限制大小為 1000 MB，
不過 messages 裡寫的是超過 1279 MiB，這邊的差異就不太理解了：
# Enable this if you want abrtd to auto-unpack crashdump tarballs which appear # in this directory (for example, uploaded via ftp, scp etc). # Note: you must ensure that whatever directory you specify here exists # and is writable for abrtd. abrtd will not create it automatically. # #WatchCrashdumpArchiveDir = /var/log/abrt-upload # Max size for crash storage [MiB] or 0 for unlimited # MaxCrashReportsSize = 1000 # Specify where you want to store coredumps and all files which are needed for # reporting. (default:/var/spool/abrt) # DumpLocation = /var/log/abrt
另外多學習到了可以用 abrt-cli ls 來看所有的 core dump 的大略資訊，
就不用自己去 grep cmdline 了：
root@localhost / # abrt-cli ls id 5091e5be023a8573a877324d93a0c9995476f684 reason: VBoxHeadless killed by SIGSEGV time: Mon 21 Mar 2016 06:00:57 AM CST cmdline: /usr/bin/VBoxHeadless --startvm f562c344-3826-4283-829e-fb4cdf1c1bb2 --vrde config uid: 0 (root) count: 54 Directory: /var/log/abrt/ccpp-2016-03-21-06:00:57-58931 id 974b65adf0c19c3ecfb6aaeb870c5ce228555b2b reason: worker.py:27:<module>:ImportError: No module named update time: Mon 07 Mar 2016 04:34:12 PM CST cmdline: python -u -m /opt/data/testdaemon.py uid: 0 (root) count: 6 Directory: /var/log/abrt/Python-2016-03-07-16:34:12-27719
參考資料：
redhat: Automatic Bug Reporting Tool (ABRT)
TipsAndTricks: Automatic Bug Reporting Tool
(本頁面已被瀏覽過 2,043 次)