[GCP] Bitnami WordPress 機器上有太多的 php-fpm 進程，導致記憶體不足
在 GCP 上的 WordPress 機器，
偶爾會出現機器非常忙碌，網站無法連線的狀況。
用 SSH 連上機器後，通常只看得到像 php、mysql 這些 process 吃很多 CPU，
且記憶體也吃了不少，但不知道實際原因。
不知道解決方法，也只能重新開機， 通常會緩解一陣子…
但是最近這招卻無效，屢屢重開機完後，CPU/記憶體又很快爆表了…
像下圖可以看到，以往的 CPU 使用率都只在 5% 上下，
但出事的時候，就會往 20% 甚至更高衝上去：
看了一下 dmesg，這才發現發生了記憶體不足 Out-of-memory (OOM) 的錯誤，
且這錯誤還會持續不斷發生：
[ 1661.471151] Call Trace: [ 1661.473829] dump_stack+0x66/0x81 [ 1661.477693] dump_header+0x6b/0x283 [ 1661.481647] ? do_try_to_free_pages+0x2ec/0x370 [ 1661.486612] oom_kill_process.cold.30+0xb/0x1cf [ 1661.491448] ? oom_badness+0xee/0x140 [ 1661.495480] out_of_memory+0x1a5/0x450 [ 1661.499597] __alloc_pages_slowpath+0xbd8/0xcb0 [ 1661.504405] __alloc_pages_nodemask+0x28b/0x2b0 [ 1661.509565] filemap_fault+0x333/0x780 [ 1661.513520] ? __switch_to+0x115/0x440 [ 1661.517755] ? filemap_map_pages+0x1e7/0x3a0 [ 1661.522441] ext4_filemap_fault+0x2c/0x40 [ext4] [ 1661.527261] __do_fault+0x34/0x90 [ 1661.530845] __handle_mm_fault+0xdf9/0x11f0 [ 1661.535396] handle_mm_fault+0xd6/0x200 [ 1661.539437] __do_page_fault+0x249/0x4f0 [ 1661.543549] ? page_fault+0x8/0x30 [ 1661.547248] page_fault+0x1e/0x30 [ 1661.550967] RIP: 0033:0x7f7c6f0cc27a [ 1661.554747] Code: Bad RIP value. [ 1661.558250] RSP: 002b:00007f7bb88f5eb0 EFLAGS: 00010217 [ 1661.563838] RAX: 0000000000000000 RBX: 00007f7bb88f6e60 RCX: 00007f7c6e6e1f59 [ 1661.571172] RDX: 0000000000000100 RSI: 0000000000000001 RDI: 00007f7c6e5c1000 [ 1661.578782] RBP: 00007f7c6e5c1000 R08: 00007f7bb88f5f20 R09: 0000000000000050 [ 1661.586114] R10: 00007f7c6b8c7fc0 R11: 0000000000000246 R12: 0000000000000001 [ 1661.593438] R13: 0000000000000000 R14: 0000000000000100 R15: 00007f7c6b8c7fc0 [ 1661.600929] Mem-Info: [ 1661.603464] active_anon:1943746 inactive_anon:15685 isolated_anon:0 [ 1661.603464] active_file:56 inactive_file:125 isolated_file:0 [ 1661.603464] unevictable:3468 dirty:0 writeback:0 unstable:0 [ 1661.603464] slab_reclaimable:3875 slab_unreclaimable:11521 [ 1661.603464] mapped:14918 shmem:15739 pagetables:12351 bounce:0 [ 1661.603464] free:25888 free_pcp:108 free_cma:0 [ 1661.638236] Node 0 active_anon:7774984kB inactive_anon:62740kB active_file:416kB inactive_file:700kB unevictable :13872kB isolated(anon):0kB isolated(file):0kB mapped:59972kB dirty:0kB writeback:0kB shmem:62956kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 6762496kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [ 1661.667250] Node 0 DMA free:15908kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file :0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0k B pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1661.694303] lowmem_reserve[]: 0 2978 7947 7947 7947 [ 1661.699413] Node 0 DMA32 free:45108kB min:25276kB low:31592kB high:37908kB active_anon:3015228kB inactive_anon:1 40kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129332kB managed:3063764kB mlocked :0kB kernel_stack:48kB pagetables:2316kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1661.727743] lowmem_reserve[]: 0 0 4969 4969 4969 [ 1661.732578] Node 0 Normal free:42032kB min:42172kB low:52712kB high:63252kB active_anon:4759756kB inactive_anon: 62600kB active_file:284kB inactive_file:308kB unevictable:13872kB writepending:0kB present:5242880kB managed:509324 4kB mlocked:13872kB kernel_stack:27120kB pagetables:47088kB bounce:0kB free_pcp:1176kB local_pcp:284kB free_cma:0kB [ 1661.763368] lowmem_reserve[]: 0 0 0 0 0 [ 1661.767458] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U ) 1*2048kB (M) 3*4096kB (M) = 15908kB [ 1661.781520] Node 0 DMA32: 63*4kB (UME) 57*8kB (UME) 17*16kB (UE) 49*32kB (UME) 39*64kB (UME) 17*128kB (UME) 4*25 6kB (UME) 2*512kB (UE) 3*1024kB (M) 6*2048kB (UME) 5*4096kB (M) = 45108kB [ 1661.798511] Node 0 Normal: 1370*4kB (UE) 859*8kB (UME) 473*16kB (UME) 143*32kB (UME) 58*64kB (UME) 32*128kB (UME ) 12*256kB (UME) 7*512kB (UME) 3*1024kB (ME) 0*2048kB 0*4096kB = 42032kB [ 1661.815443] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 1661.824483] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 1661.833456] 16957 total pagecache pages [ 1661.837705] 0 pages in swap cache [ 1661.841271] Swap cache stats: add 0, delete 0, find 0/0 [ 1661.846754] Free swap = 0kB [ 1661.849977] Total swap = 0kB [ 1661.853118] 2097051 pages RAM [ 1661.856315] 0 pages HighMem/MovableOnly [ 1661.860387] 53822 pages reserved [ 1661.863840] Tasks state (memory values in pages): [ 1661.868863] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 1661.877845] [ 234] 0 234 7391 986 94208 0 0 systemd-journal [ 1661.887125] [ 245] 0 245 5055 685 57344 0 -1000 systemd-udevd [ 1661.896249] [ 327] 0 327 2372 754 53248 0 0 dhclient [ 1661.905186] [ 333] 0 333 2021 1189 53248 0 0 haveged [ 1661.913959] [ 375] 0 375 571 18 40960 0 0 atopacctd [ 1661.922820] [ 378] 104 378 2197 492 57344 0 -900 dbus-daemon [ 1661.931828] [ 383] 0 383 1372 433 49152 0 0 atd [ 1661.940310] [ 391] 0 391 4129 3407 73728 0 0 atop [ 1661.948821] [ 402] 0 402 30162 2283 110592 0 0 google_osconfig [ 1661.958302] [ 403] 0 403 7413 1976 94208 0 0 unattended-upgr [ 1661.967600] [ 408] 0 408 1346 397 45056 0 0 agetty [ 1661.976321] [ 409] 0 409 1403 343 45056 0 0 agetty [ 1661.985225] [ 411] 106 411 3253 583 49152 0 0 chronyd [ 1661.993839] [ 414] 106 414 1204 67 49152 0 0 chronyd [ 1662.002530] [ 475] 0 475 28574 1893 90112 0 -999 google_guest_ag [ 1662.012177] [ 651] 0 651 3963 683 65536 0 -1000 sshd [ 1662.020524] [ 658] 0 658 4848 635 77824 0 0 systemd-logind [ 1662.029991] [ 661] 0 661 2126 393 53248 0 0 cron [ 1662.038420] [ 686] 0 686 79790 3734 245760 0 0 php-fpm [ 1662.047466] [ 702] 1 702 106280 40784 536576 0 0 php-fpm [ 1662.056146] [ 703] 1 703 118112 34189 491520 0 0 php-fpm [ 1662.065006] [ 704] 1 704 118662 34954 512000 0 0 php-fpm [ 1662.073698] [ 705] 1 705 101157 35677 495616 0 0 php-fpm [ 1662.082485] [ 706] 1 706 118665 35161 507904 0 0 php-fpm [ 1662.091350] [ 707] 1 707 95528 30002 450560 0 0 php-fpm [ 1662.100044] [ 708] 1 708 124756 40968 557056 0 0 php-fpm [ 1662.108730] [ 709] 1 709 100144 34615 487424 0 0 php-fpm [ 1662.118056] [ 710] 1 710 124285 40376 544768 0 0 php-fpm [ 1662.126865] [ 711] 1 711 101178 35990 495616 0 0 php-fpm [ 1662.135759] [ 712] 1 712 105763 40224 532480 0 0 php-fpm [ 1662.144363] [ 713] 1 713 87860 22331 389120 0 0 php-fpm [ 1662.152957] [ 714] 1 714 105846 40309 536576 0 0 php-fpm [ 1662.161650] [ 715] 1 715 86707 21240 380928 0 0 php-fpm [ 1662.170284] [ 716] 1 716 115071 31175 471040 0 0 php-fpm [ 1662.178892] [ 717] 1 717 112989 29122 458752 0 0 php-fpm [ 1662.187599] [ 718] 1 718 86691 21232 380928 0 0 php-fpm [ 1662.196219] [ 719] 1 719 121777 38238 524288 0 0 php-fpm [ 1662.205168] [ 720] 1 720 111441 27527 442368 0 0 php-fpm [ 1662.214000] [ 721] 1 721 96630 31127 466944 0 0 php-fpm [ 1662.222698] [ 722] 1 722 127309 43466 565248 0 0 php-fpm [ 1662.231401] [ 723] 1 723 97060 31548 462848 0 0 php-fpm [ 1662.240281] [ 724] 1 724 104537 20687 397312 0 0 php-fpm [ 1662.248882] [ 725] 1 725 115259 31654 475136 0 0 php-fpm [ 1662.257725] [ 726] 1 726 102691 37172 507904 0 0 php-fpm [ 1662.266352] [ 727] 1 727 99644 34153 491520 0 0 php-fpm [ 1662.275132] [ 728] 1 728 108835 43342 557056 0 0 php-fpm [ 1662.283999] [ 729] 1 729 102692 37186 507904 0 0 php-fpm [ 1662.292594] [ 730] 1 730 103203 37703 516096 0 0 php-fpm [ 1662.301379] [ 731] 1 731 109348 43882 565248 0 0 php-fpm [ 1662.310237] [ 732] 1 732 121181 37329 524288 0 0 php-fpm [ 1662.319006] [ 733] 1 733 105763 40240 532480 0 0 php-fpm [ 1662.327725] [ 734] 1 734 109720 44287 569344 0 0 php-fpm [ 1662.336360] [ 735] 1 735 127841 44064 577536 0 0 php-fpm [ 1662.345062] [ 736] 1 736 93475 27919 434176 0 0 php-fpm [ 1662.353755] [ 737] 1 737 96563 31090 458752 0 0 php-fpm [ 1662.362456] [ 738] 1 738 101172 35674 495616 0 0 php-fpm [ 1662.371265] [ 739] 1 739 115229 31611 475136 0 0 php-fpm [ 1662.380093] [ 740] 1 740 123927 39564 544768 0 0 php-fpm [ 1662.388968] [ 741] 1 741 101032 35487 499712 0 0 php-fpm [ 1662.397782] [ 742] 1 742 88876 23341 401408 0 0 php-fpm [ 1662.406796] [ 743] 1 743 105181 21357 393216 0 0 php-fpm [ 1662.415617] [ 744] 1 744 120653 36816 516096 0 0 php-fpm [ 1662.424341] [ 745] 1 745 127828 44026 573440 0 0 php-fpm [ 1662.433546] [ 746] 1 746 119126 35206 507904 0 0 php-fpm [ 1662.442337] [ 747] 1 747 122211 38401 532480 0 0 php-fpm [ 1662.451135] [ 748] 1 748 106304 40750 544768 0 0 php-fpm [ 1662.460291] [ 749] 1 749 99643 34135 487424 0 0 php-fpm [ 1662.469003] [ 750] 1 750 102692 37135 512000 0 0 php-fpm [ 1662.477799] [ 751] 1 751 119134 35254 503808 0 0 php-fpm [ 1662.486491] [ 752] 1 752 118605 34719 503808 0 0 php-fpm [ 1662.495203] [ 753] 1 753 118668 34827 503808 0 0 php-fpm [ 1662.504155] [ 754] 1 754 109495 44024 565248 0 0 php-fpm [ 1662.512761] [ 755] 1 755 100659 35159 495616 0 0 php-fpm [ 1662.521633] [ 756] 1 756 94507 28990 450560 0 0 php-fpm [ 1662.530331] [ 757] 1 757 106787 40188 536576 0 0 php-fpm [ 1662.539198] [ 758] 1 758 104739 39263 524288 0 0 php-fpm [ 1662.548134] [ 759] 1 759 125780 41999 565248 0 0 php-fpm [ 1662.557276] [ 760] 1 760 100243 34798 491520 0 0 php-fpm [ 1662.566458] [ 761] 1 761 116592 32709 491520 0 0 php-fpm [ 1662.575264] [ 762] 1 762 105878 22433 409600 0 0 php-fpm [ 1662.583889] [ 763] 1 763 93987 28486 438272 0 0 php-fpm [ 1662.592585] [ 764] 1 764 96571 31036 462848 0 0 php-fpm [ 1662.601326] [ 765] 1 765 125282 41497 557056 0 0 php-fpm [ 1662.610045] [ 766] 1 766 87851 22286 401408 0 0 php-fpm [ 1662.618787] [ 767] 1 767 112461 28545 446464 0 0 php-fpm [ 1662.627582] [ 768] 1 768 101668 36199 499712 0 0 php-fpm [ 1662.636242] [ 769] 1 769 97450 31954 475136 0 0 php-fpm [ 1662.645077] [ 770] 1 770 99668 34117 487424 0 0 php-fpm [ 1662.654059] [ 771] 1 771 107053 23513 421888 0 0 php-fpm [ 1662.663203] [ 773] 1 773 90419 24901 409600 0 0 php-fpm [ 1662.672065] [ 774] 1 774 103254 19315 376832 0 0 php-fpm [ 1662.680661] [ 776] 1 776 100025 34531 491520 0 0 php-fpm [ 1662.689359] [ 777] 1 777 126829 43311 565248 0 0 php-fpm [ 1662.697965] [ 778] 1 778 113004 29552 454656 0 0 php-fpm [ 1662.706683] [ 779] 1 779 100008 34488 491520 0 0 php-fpm [ 1662.715392] [ 780] 1 780 105780 40268 532480 0 0 php-fpm [ 1662.724030] [ 781] 1 781 124749 40858 552960 0 0 php-fpm [ 1662.732623] [ 783] 1 783 100131 34621 491520 0 0 php-fpm [ 1662.741403] [ 784] 1 784 93955 27508 430080 0 0 php-fpm [ 1662.750085] [ 828] 0 828 3725 1031 69632 0 0 httpd [ 1662.758786] [ 834] 1 834 581721 2609 802816 0 0 httpd [ 1662.767228] [ 835] 1 835 581726 4090 802816 0 0 httpd [ 1662.775654] [ 836] 1 836 581726 3533 802816 0 0 httpd [ 1662.784446] [ 841] 1 841 581721 2494 802816 0 0 httpd [ 1662.792875] [ 854] 1 854 581721 5142 802816 0 0 httpd [ 1662.801477] [ 868] 1 868 581869 2645 806912 0 0 httpd [ 1662.809994] [ 883] 1 883 581784 3656 802816 0 0 httpd [ 1662.818684] [ 893] 1 893 581721 2571 802816 0 0 httpd [ 1662.827115] [ 2150] 1002 2150 1007708 137960 1646592 0 0 mysqld [ 1662.835780] [ 2210] 0 2210 362537 2649 196608 0 0 gonit [ 1662.844262] [ 2263] 1 2263 581729 4554 802816 0 0 httpd [ 1662.853149] [ 2426] 1 2426 124749 40870 552960 0 0 php-fpm [ 1662.861719] [ 2428] 1 2428 87846 22321 389120 0 0 php-fpm [ 1662.870608] [ 2430] 1 2430 107336 41783 544768 0 0 php-fpm [ 1662.879323] [ 2437] 1 2437 112993 29550 466944 0 0 php-fpm [ 1662.888099] [ 2448] 0 2448 4153 723 73728 0 0 sshd [ 1662.896790] [ 2452] 1003 2452 5255 674 77824 0 0 systemd [ 1662.906110] [ 2453] 1003 2453 5723 570 86016 0 0 (sd-pam) [ 1662.914768] [ 2466] 1003 2466 4153 523 73728 0 0 sshd [ 1662.923558] [ 2467] 1003 2467 1748 572 53248 0 0 bash [ 1662.931989] [ 2468] 1003 2468 614 392 45056 0 0 sftp-server [ 1662.941466] [ 2485] 1 2485 87220 21809 389120 0 0 php-fpm [ 1662.950512] [ 2486] 1 2486 104245 38688 524288 0 0 php-fpm [ 1662.959240] [ 2489] 1 2489 93475 27898 434176 0 0 php-fpm [ 1662.967925] [ 2653] 0 2653 56456 503 86016 0 0 rsyslogd [ 1662.976803] Out of memory: Kill process 2150 (mysqld) score 67 or sacrifice child [ 1662.985163] Killed process 2150 (mysqld) total-vm:4030832kB, anon-rss:551840kB, file-rss:0kB, shmem-rss:0kB Sep 5 12:59:54 wordpress-multisite-1-vm kernel:[ 1663.016261] oom_reaper: reaped process 2150 (mysqld), now anon-r ss:0kB, file-rss:0kB, shmem-rss:0kB [ 1661.435854] mysqld invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_s core_adj=0
從 ps 的結果來看，有非常多支的 php-fpm process 在執行，
每支都吃了 1xx MB 記憶體，將近 90 支就得要吃掉 9 GB 以上，
而我的機器只有 8 GB 記憶體，自然會一直遇到記憶體不足的情況：
testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm root 688 0.0 0.2 319160 16372 ? Ss 13:52 0:00 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf) daemon 703 0.1 1.3 448156 111048 ? S 13:52 0:06 php-fpm: pool www daemon 704 0.1 2.1 509684 174852 ? S 13:52 0:06 php-fpm: pool www daemon 705 0.2 2.1 441872 176228 ? S 13:52 0:07 php-fpm: pool www daemon 706 0.1 1.1 428336 90596 ? S 13:52 0:06 php-fpm: pool www daemon 707 0.2 1.5 460424 125848 ? S 13:52 0:06 php-fpm: pool www daemon 708 0.2 2.1 511612 176308 ? S 13:52 0:08 php-fpm: pool www daemon 709 0.3 2.1 512100 174772 ? S 13:52 0:10 php-fpm: pool www daemon 710 0.3 1.4 460352 121488 ? S 13:52 0:10 php-fpm: pool www daemon 711 0.2 1.6 474680 135536 ? S 13:52 0:08 php-fpm: pool www daemon 712 0.3 2.0 505180 166296 ? S 13:52 0:10 php-fpm: pool www daemon 713 0.1 1.6 473400 135888 ? S 13:52 0:05 php-fpm: pool www daemon 714 0.2 1.7 484912 146460 ? S 13:52 0:09 php-fpm: pool www daemon 715 0.2 1.6 400656 134948 ? S 13:52 0:07 php-fpm: pool www daemon 716 0.2 2.0 511388 171504 ? S 13:52 0:08 php-fpm: pool www ...... testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm | wc -l 87
看了一下 /opt/bitnami/php/logs/php-fpm.log ，有點奇怪，
它說目前好像很忙，應該要增加啟動的 server 數量，
可是目前休息中的 server 卻有 79 個：
[05-Sep-2021 11:38:29] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 99 total children [05-Sep-2021 11:38:30] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 100 total children [05-Sep-2021 11:38:31] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 101 total children [05-Sep-2021 11:38:32] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 102 total children [05-Sep-2021 13:29:40] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 79 idle, and 91 total children
上網查了一下，也有人在抱怨 php-fpm process 吃了很多記憶體的事…
打開 /opt/bitnami/php/etc/memory.conf 看一下：
; Bitnami memory configuration for PHP-FPM ; ; Note: This will be modified on server size changes pm.max_children=120 pm.start_servers=80 pm.min_spare_servers=80 pm.max_spare_servers=90 pm.max_requests=5000
這裡的設定中沒有 pm 設定，因此預設是 pm = dynamic ，
PHP 的 process 數量會動態增減，而且一開始就會有 80 個，最多到 120 個。
就是因為這邊的設定，導致過多的 php-fpm process 占據太多的記憶體。
(不過 php-fpm.log 說目前很忙這件事，難以理解…)
簡單的改法，是設定 pm = ondemand ，有需要才增加 php-fpm process。
且指定 pm.process_idle_timeout = 20s ，
如果 php-fpm process 在 20 秒內接不到新的要求，就可以下去休息了。
這樣的話，就不會有過多的 process 在待命：
pm = ondemand pm.process_idle_timeout = 20s pm.max_children=120 pm.start_servers=80 pm.min_spare_servers=80 pm.max_spare_servers=90 pm.max_requests=5000
修改好之後，要重新啟動相關服務：
sudo /opt/bitnami/ctlscript.sh restart
這時再看一下 php-fpm 的數量，發現只剩下 4 支了：
testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php | wc -l 4
從 GCP 的監控畫面中，可以看到記憶體從開機後一路衝到快 100%，
在修改完之後，直線掉落到約 10% 上下：
當然這種改法，有省記憶體的好處，
自然也會有壞處，就是假設有一堆人跑來瀏覽網頁的話，
也許大家就要稍微等待 php-fpm process 被叫起來，會慢一點點。
不過以我的小網站來說，應該還可以接受，
畢竟記憶體一直爆掉，整個網站不能動還是比較嚴重…
用這招還有一點要注意的，就是如果調整了 VM 的設定，
像是調整了 CPU/RAM 之類的話，
/opt/bitnami/php/etc/memory.conf 就可能會被重設，
因此得再重新修改一次囉～
參考資料：
- How To Prevent PHP-FPM From Consuming Too Much RAM in Linux
- nginx – PHP-FPM creating too many proccesses & eating all the CPU – Server Fault
- Troubleshooting Agent Installation | Cloud Monitoring | Google Cloud
(本頁面已被瀏覽過 690 次)