[Linux] abrt 產生的 core dump 不見了?

[Linux] abrt 產生的 core dump 不見了?

今天同事回報說有程式掛掉,但是 abrt 的目錄裡沒有看到相關的 core dump~

這真是怪事一椿,在目錄裡找了半天,還真的沒看到,

但程式確實是有 crash 過,而 abrt 目錄裡也有其他的 core dump,

所以並不是 abrtd 沒有作用…

 

雖然 abrtd 應該是有作用,但還是先檢查一下 abrt 相關的 daemon 有沒有在執行,

用 systemctl 看一下,看起來都很正常:

root@localhost / # systemctl | egrep abrt
abrt-ccpp.service  loaded active exited    Install ABRT coredump hook
abrt-oops.service  loaded active running   ABRT kernel log watcher
abrtd.service      loaded active running   ABRT Automated Bug Reporting Tool

 

看一下 abrtd 的狀態,正在執行中,而且跑了一個多月了,

在下面的訊息裡注意到有 Deleting problem directory 的字眼:

root@localhost /var/log/abrt # systemctl status abrtd
abrtd.service - ABRT Automated Bug Reporting Tool
Loaded: loaded (/usr/lib/systemd/system/abrtd.service; enabled)
Active: active (running) since Thu 2016-01-28 11:45:13 CST; 1 months 22 days ago
Main PID: 998 (abrtd)
CGroup: /system.slice/abrtd.service
└─998 /usr/sbin/abrtd -d -s
Mar 21 13:49:02 localhost abrt-server[23468]: Email was sent to: root@localhost
Mar 21 13:55:45 localhost abrt-server[37655]: Generating core_backtrace
Mar 21 13:56:06 localhost abrt-server[37655]: Duplicate: core backtrace
Mar 21 13:56:06 localhost abrt-server[37655]: DUP_OF_DIR: /var/log/abrt/ccpp-2016-03-21-06:00:57-58931
Mar 21 13:56:06 localhost abrt-server[37655]: Deleting problem directory ccpp-2016-03-21-13:55:44-37481 (dup of ccpp-2016-03-21-06:00:57-58931)
Mar 21 13:56:07 localhost abrt-server[37655]: Email address of sender was not specified. Would you like to do so now? If not, 'user@localhost' is to ...ed [y/N]
Mar 21 13:56:07 localhost abrt-server[37655]: Email address of receiver was not specified. Would you like to do so now? If not, 'root@localhost' is t...ed [y/N]
Mar 21 13:56:07 localhost abrt-server[37655]: Sending an email...
Mar 21 13:56:08 localhost abrt-server[37655]: postdrop: warning: unable to look up public/pickup: No such file or directory
Mar 21 13:56:08 localhost abrt-server[37655]: Email was sent to: root@localhost
Hint: Some lines were ellipsized, use -l to show in full.

 

基本上 abrt 不會記錄重覆的 core dump,因此看到這種 DUP_OF_DIR 的訊息應該不奇怪,

不過好奇下,在 /var/log/messages 下找了一下 abrt.*deleting 的字句:

2016-03-21T06:00:58.230478+08:00 localhost abrt-hook-ccpp: /var/log/abrt is 1382018600 bytes (more than 1279MiB), deleting 'ccpp-2016-03-21-05:28:18-5996'
2016-03-21T13:48:58.190855+08:00 localhost abrt-server: Deleting problem directory ccpp-2016-03-21-13:48:39-23254 (dup of ccpp-2016-03-21-06:00:57-58931)
2016-03-21T13:56:06.163878+08:00 localhost abrt-server: Deleting problem directory ccpp-2016-03-21-13:55:44-37481 (dup of ccpp-2016-03-21-06:00:57-58931)

 

上面訊息裡看到了因為超過 1279 MiB 所以砍掉了某個 core dump 目錄的訊息,

這才真相大白,原來是空間的限制,導致 abrt 將舊的 core dump 給砍掉了~

 

看一下 /etc/abrt/abrt.conf 的內容,裡面有個 MaxCrashReportsSize 限制大小為 1000 MB,

不過 messages 裡寫的是超過 1279 MiB,這邊的差異就不太理解了:

# Enable this if you want abrtd to auto-unpack crashdump tarballs which appear
# in this directory (for example, uploaded via ftp, scp etc).
# Note: you must ensure that whatever directory you specify here exists
# and is writable for abrtd. abrtd will not create it automatically.
#
#WatchCrashdumpArchiveDir = /var/log/abrt-upload
# Max size for crash storage [MiB] or 0 for unlimited
#
MaxCrashReportsSize = 1000
# Specify where you want to store coredumps and all files which are needed for
# reporting. (default:/var/spool/abrt)
#
DumpLocation = /var/log/abrt

 

另外多學習到了可以用 abrt-cli ls 來看所有的 core dump 的大略資訊,

就不用自己去 grep cmdline 了:

root@localhost / # abrt-cli ls
id 5091e5be023a8573a877324d93a0c9995476f684
reason:         VBoxHeadless killed by SIGSEGV
time:           Mon 21 Mar 2016 06:00:57 AM CST
cmdline:        /usr/bin/VBoxHeadless --startvm f562c344-3826-4283-829e-fb4cdf1c1bb2 --vrde config
uid:            0 (root)
count:          54
Directory:      /var/log/abrt/ccpp-2016-03-21-06:00:57-58931
id 974b65adf0c19c3ecfb6aaeb870c5ce228555b2b
reason:         worker.py:27:<module>:ImportError: No module named update
time:           Mon 07 Mar 2016 04:34:12 PM CST
cmdline:        python -u -m /opt/data/testdaemon.py
uid:            0 (root)
count:          6
Directory:      /var/log/abrt/Python-2016-03-07-16:34:12-27719

  

參考資料:

redhat: Automatic Bug Reporting Tool (ABRT)

abrt.conf(5) – Linux man page

TipsAndTricks: Automatic Bug Reporting Tool

 

(本頁面已被瀏覽過 965 次)

發佈留言

發佈留言必須填寫的電子郵件地址不會公開。

這個網站採用 Akismet 服務減少垃圾留言。進一步瞭解 Akismet 如何處理網站訪客的留言資料