[GCP] Bitnami WordPress 機器上有太多的 php-fpm 進程，導致記憶體不足

2021-09-19 ephrain Comments 0 Comment

在 GCP 上的 WordPress 機器，

偶爾會出現機器非常忙碌，網站無法連線的狀況。

用 SSH 連上機器後，通常只看得到像 php、mysql 這些 process 吃很多 CPU，

且記憶體也吃了不少，但不知道實際原因。

不知道解決方法，也只能重新開機，通常會緩解一陣子…

但是最近這招卻無效，屢屢重開機完後，CPU/記憶體又很快爆表了…

像下圖可以看到，以往的 CPU 使用率都只在 5% 上下，

但出事的時候，就會往 20% 甚至更高衝上去：

看了一下 dmesg，這才發現發生了記憶體不足 Out-of-memory (OOM) 的錯誤，

且這錯誤還會持續不斷發生：

[ 1661.471151] Call Trace:

[ 1661.473829] dump_stack+0x66/0x81

[ 1661.477693] dump_header+0x6b/0x283

[ 1661.481647] ? do_try_to_free_pages+0x2ec/0x370

[ 1661.486612] oom_kill_process.cold.30+0xb/0x1cf

[ 1661.491448] ? oom_badness+0xee/0x140

[ 1661.495480] out_of_memory+0x1a5/0x450

[ 1661.499597] __alloc_pages_slowpath+0xbd8/0xcb0

[ 1661.504405] __alloc_pages_nodemask+0x28b/0x2b0

[ 1661.509565] filemap_fault+0x333/0x780

[ 1661.513520] ? __switch_to+0x115/0x440

[ 1661.517755] ? filemap_map_pages+0x1e7/0x3a0

[ 1661.522441] ext4_filemap_fault+0x2c/0x40 [ext4]

[ 1661.527261] __do_fault+0x34/0x90

[ 1661.530845] __handle_mm_fault+0xdf9/0x11f0

[ 1661.535396] handle_mm_fault+0xd6/0x200

[ 1661.539437] __do_page_fault+0x249/0x4f0

[ 1661.543549] ? page_fault+0x8/0x30

[ 1661.547248] page_fault+0x1e/0x30

[ 1661.550967] RIP: 0033:0x7f7c6f0cc27a

[ 1661.554747] Code: Bad RIP value.

[ 1661.558250] RSP: 002b:00007f7bb88f5eb0 EFLAGS: 00010217

[ 1661.563838] RAX: 0000000000000000 RBX: 00007f7bb88f6e60 RCX: 00007f7c6e6e1f59

[ 1661.571172] RDX: 0000000000000100 RSI: 0000000000000001 RDI: 00007f7c6e5c1000

[ 1661.578782] RBP: 00007f7c6e5c1000 R08: 00007f7bb88f5f20 R09: 0000000000000050

[ 1661.586114] R10: 00007f7c6b8c7fc0 R11: 0000000000000246 R12: 0000000000000001

[ 1661.593438] R13: 0000000000000000 R14: 0000000000000100 R15: 00007f7c6b8c7fc0

[ 1661.600929] Mem-Info:

[ 1661.603464] active_anon:1943746 inactive_anon:15685 isolated_anon:0

[ 1661.603464] active_file:56 inactive_file:125 isolated_file:0

[ 1661.603464] unevictable:3468 dirty:0 writeback:0 unstable:0

[ 1661.603464] slab_reclaimable:3875 slab_unreclaimable:11521

[ 1661.603464] mapped:14918 shmem:15739 pagetables:12351 bounce:0

[ 1661.603464] free:25888 free_pcp:108 free_cma:0

[ 1661.638236] Node 0 active_anon:7774984kB inactive_anon:62740kB active_file:416kB inactive_file:700kB unevictable

:13872kB isolated(anon):0kB isolated(file):0kB mapped:59972kB dirty:0kB writeback:0kB shmem:62956kB shmem_thp: 0kB

shmem_pmdmapped: 0kB anon_thp: 6762496kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no

[ 1661.667250] Node 0 DMA free:15908kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file

:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0k

B pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB

[ 1661.694303] lowmem_reserve[]: 0 2978 7947 7947 7947

[ 1661.699413] Node 0 DMA32 free:45108kB min:25276kB low:31592kB high:37908kB active_anon:3015228kB inactive_anon:1

40kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129332kB managed:3063764kB mlocked

:0kB kernel_stack:48kB pagetables:2316kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB

[ 1661.727743] lowmem_reserve[]: 0 0 4969 4969 4969

[ 1661.732578] Node 0 Normal free:42032kB min:42172kB low:52712kB high:63252kB active_anon:4759756kB inactive_anon:

62600kB active_file:284kB inactive_file:308kB unevictable:13872kB writepending:0kB present:5242880kB managed:509324

4kB mlocked:13872kB kernel_stack:27120kB pagetables:47088kB bounce:0kB free_pcp:1176kB local_pcp:284kB free_cma:0kB

[ 1661.763368] lowmem_reserve[]: 0 0 0 0 0

[ 1661.767458] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U

) 1*2048kB (M) 3*4096kB (M) = 15908kB

[ 1661.781520] Node 0 DMA32: 63*4kB (UME) 57*8kB (UME) 17*16kB (UE) 49*32kB (UME) 39*64kB (UME) 17*128kB (UME) 4*25

6kB (UME) 2*512kB (UE) 3*1024kB (M) 6*2048kB (UME) 5*4096kB (M) = 45108kB

[ 1661.798511] Node 0 Normal: 1370*4kB (UE) 859*8kB (UME) 473*16kB (UME) 143*32kB (UME) 58*64kB (UME) 32*128kB (UME

) 12*256kB (UME) 7*512kB (UME) 3*1024kB (ME) 0*2048kB 0*4096kB = 42032kB

[ 1661.815443] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB

[ 1661.824483] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB

[ 1661.833456] 16957 total pagecache pages

[ 1661.837705] 0 pages in swap cache

[ 1661.841271] Swap cache stats: add 0, delete 0, find 0/0

[ 1661.846754] Free swap = 0kB

[ 1661.849977] Total swap = 0kB

[ 1661.853118] 2097051 pages RAM

[ 1661.856315] 0 pages HighMem/MovableOnly

[ 1661.860387] 53822 pages reserved

[ 1661.863840] Tasks state (memory values in pages):

[ 1661.868863] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name

[ 1661.877845] [ 234] 0 234 7391 986 94208 0 0 systemd-journal

[ 1661.887125] [ 245] 0 245 5055 685 57344 0 -1000 systemd-udevd

[ 1661.896249] [ 327] 0 327 2372 754 53248 0 0 dhclient

[ 1661.905186] [ 333] 0 333 2021 1189 53248 0 0 haveged

[ 1661.913959] [ 375] 0 375 571 18 40960 0 0 atopacctd

[ 1661.922820] [ 378] 104 378 2197 492 57344 0 -900 dbus-daemon

[ 1661.931828] [ 383] 0 383 1372 433 49152 0 0 atd

[ 1661.940310] [ 391] 0 391 4129 3407 73728 0 0 atop

[ 1661.948821] [ 402] 0 402 30162 2283 110592 0 0 google_osconfig

[ 1661.958302] [ 403] 0 403 7413 1976 94208 0 0 unattended-upgr

[ 1661.967600] [ 408] 0 408 1346 397 45056 0 0 agetty

[ 1661.976321] [ 409] 0 409 1403 343 45056 0 0 agetty

[ 1661.985225] [ 411] 106 411 3253 583 49152 0 0 chronyd

[ 1661.993839] [ 414] 106 414 1204 67 49152 0 0 chronyd

[ 1662.002530] [ 475] 0 475 28574 1893 90112 0 -999 google_guest_ag

[ 1662.012177] [ 651] 0 651 3963 683 65536 0 -1000 sshd

[ 1662.020524] [ 658] 0 658 4848 635 77824 0 0 systemd-logind

[ 1662.029991] [ 661] 0 661 2126 393 53248 0 0 cron

[ 1662.038420] [ 686] 0 686 79790 3734 245760 0 0 php-fpm

[ 1662.047466] [ 702] 1 702 106280 40784 536576 0 0 php-fpm

[ 1662.056146] [ 703] 1 703 118112 34189 491520 0 0 php-fpm

[ 1662.065006] [ 704] 1 704 118662 34954 512000 0 0 php-fpm

[ 1662.073698] [ 705] 1 705 101157 35677 495616 0 0 php-fpm

[ 1662.082485] [ 706] 1 706 118665 35161 507904 0 0 php-fpm

[ 1662.091350] [ 707] 1 707 95528 30002 450560 0 0 php-fpm

[ 1662.100044] [ 708] 1 708 124756 40968 557056 0 0 php-fpm

[ 1662.108730] [ 709] 1 709 100144 34615 487424 0 0 php-fpm

[ 1662.118056] [ 710] 1 710 124285 40376 544768 0 0 php-fpm

[ 1662.126865] [ 711] 1 711 101178 35990 495616 0 0 php-fpm

[ 1662.135759] [ 712] 1 712 105763 40224 532480 0 0 php-fpm

[ 1662.144363] [ 713] 1 713 87860 22331 389120 0 0 php-fpm

[ 1662.152957] [ 714] 1 714 105846 40309 536576 0 0 php-fpm

[ 1662.161650] [ 715] 1 715 86707 21240 380928 0 0 php-fpm

[ 1662.170284] [ 716] 1 716 115071 31175 471040 0 0 php-fpm

[ 1662.178892] [ 717] 1 717 112989 29122 458752 0 0 php-fpm

[ 1662.187599] [ 718] 1 718 86691 21232 380928 0 0 php-fpm

[ 1662.196219] [ 719] 1 719 121777 38238 524288 0 0 php-fpm

[ 1662.205168] [ 720] 1 720 111441 27527 442368 0 0 php-fpm

[ 1662.214000] [ 721] 1 721 96630 31127 466944 0 0 php-fpm

[ 1662.222698] [ 722] 1 722 127309 43466 565248 0 0 php-fpm

[ 1662.231401] [ 723] 1 723 97060 31548 462848 0 0 php-fpm

[ 1662.240281] [ 724] 1 724 104537 20687 397312 0 0 php-fpm

[ 1662.248882] [ 725] 1 725 115259 31654 475136 0 0 php-fpm

[ 1662.257725] [ 726] 1 726 102691 37172 507904 0 0 php-fpm

[ 1662.266352] [ 727] 1 727 99644 34153 491520 0 0 php-fpm

[ 1662.275132] [ 728] 1 728 108835 43342 557056 0 0 php-fpm

[ 1662.283999] [ 729] 1 729 102692 37186 507904 0 0 php-fpm

[ 1662.292594] [ 730] 1 730 103203 37703 516096 0 0 php-fpm

[ 1662.301379] [ 731] 1 731 109348 43882 565248 0 0 php-fpm

[ 1662.310237] [ 732] 1 732 121181 37329 524288 0 0 php-fpm

[ 1662.319006] [ 733] 1 733 105763 40240 532480 0 0 php-fpm

[ 1662.327725] [ 734] 1 734 109720 44287 569344 0 0 php-fpm

[ 1662.336360] [ 735] 1 735 127841 44064 577536 0 0 php-fpm

[ 1662.345062] [ 736] 1 736 93475 27919 434176 0 0 php-fpm

[ 1662.353755] [ 737] 1 737 96563 31090 458752 0 0 php-fpm

[ 1662.362456] [ 738] 1 738 101172 35674 495616 0 0 php-fpm

[ 1662.371265] [ 739] 1 739 115229 31611 475136 0 0 php-fpm

[ 1662.380093] [ 740] 1 740 123927 39564 544768 0 0 php-fpm

[ 1662.388968] [ 741] 1 741 101032 35487 499712 0 0 php-fpm

[ 1662.397782] [ 742] 1 742 88876 23341 401408 0 0 php-fpm

[ 1662.406796] [ 743] 1 743 105181 21357 393216 0 0 php-fpm

[ 1662.415617] [ 744] 1 744 120653 36816 516096 0 0 php-fpm

[ 1662.424341] [ 745] 1 745 127828 44026 573440 0 0 php-fpm

[ 1662.433546] [ 746] 1 746 119126 35206 507904 0 0 php-fpm

[ 1662.442337] [ 747] 1 747 122211 38401 532480 0 0 php-fpm

[ 1662.451135] [ 748] 1 748 106304 40750 544768 0 0 php-fpm

[ 1662.460291] [ 749] 1 749 99643 34135 487424 0 0 php-fpm

[ 1662.469003] [ 750] 1 750 102692 37135 512000 0 0 php-fpm

[ 1662.477799] [ 751] 1 751 119134 35254 503808 0 0 php-fpm

[ 1662.486491] [ 752] 1 752 118605 34719 503808 0 0 php-fpm

[ 1662.495203] [ 753] 1 753 118668 34827 503808 0 0 php-fpm

[ 1662.504155] [ 754] 1 754 109495 44024 565248 0 0 php-fpm

[ 1662.512761] [ 755] 1 755 100659 35159 495616 0 0 php-fpm

[ 1662.521633] [ 756] 1 756 94507 28990 450560 0 0 php-fpm

[ 1662.530331] [ 757] 1 757 106787 40188 536576 0 0 php-fpm

[ 1662.539198] [ 758] 1 758 104739 39263 524288 0 0 php-fpm

[ 1662.548134] [ 759] 1 759 125780 41999 565248 0 0 php-fpm

[ 1662.557276] [ 760] 1 760 100243 34798 491520 0 0 php-fpm

[ 1662.566458] [ 761] 1 761 116592 32709 491520 0 0 php-fpm

[ 1662.575264] [ 762] 1 762 105878 22433 409600 0 0 php-fpm

[ 1662.583889] [ 763] 1 763 93987 28486 438272 0 0 php-fpm

[ 1662.592585] [ 764] 1 764 96571 31036 462848 0 0 php-fpm

[ 1662.601326] [ 765] 1 765 125282 41497 557056 0 0 php-fpm

[ 1662.610045] [ 766] 1 766 87851 22286 401408 0 0 php-fpm

[ 1662.618787] [ 767] 1 767 112461 28545 446464 0 0 php-fpm

[ 1662.627582] [ 768] 1 768 101668 36199 499712 0 0 php-fpm

[ 1662.636242] [ 769] 1 769 97450 31954 475136 0 0 php-fpm

[ 1662.645077] [ 770] 1 770 99668 34117 487424 0 0 php-fpm

[ 1662.654059] [ 771] 1 771 107053 23513 421888 0 0 php-fpm

[ 1662.663203] [ 773] 1 773 90419 24901 409600 0 0 php-fpm

[ 1662.672065] [ 774] 1 774 103254 19315 376832 0 0 php-fpm

[ 1662.680661] [ 776] 1 776 100025 34531 491520 0 0 php-fpm

[ 1662.689359] [ 777] 1 777 126829 43311 565248 0 0 php-fpm

[ 1662.697965] [ 778] 1 778 113004 29552 454656 0 0 php-fpm

[ 1662.706683] [ 779] 1 779 100008 34488 491520 0 0 php-fpm

[ 1662.715392] [ 780] 1 780 105780 40268 532480 0 0 php-fpm

[ 1662.724030] [ 781] 1 781 124749 40858 552960 0 0 php-fpm

[ 1662.732623] [ 783] 1 783 100131 34621 491520 0 0 php-fpm

[ 1662.741403] [ 784] 1 784 93955 27508 430080 0 0 php-fpm

[ 1662.750085] [ 828] 0 828 3725 1031 69632 0 0 httpd

[ 1662.758786] [ 834] 1 834 581721 2609 802816 0 0 httpd

[ 1662.767228] [ 835] 1 835 581726 4090 802816 0 0 httpd

[ 1662.775654] [ 836] 1 836 581726 3533 802816 0 0 httpd

[ 1662.784446] [ 841] 1 841 581721 2494 802816 0 0 httpd

[ 1662.792875] [ 854] 1 854 581721 5142 802816 0 0 httpd

[ 1662.801477] [ 868] 1 868 581869 2645 806912 0 0 httpd

[ 1662.809994] [ 883] 1 883 581784 3656 802816 0 0 httpd

[ 1662.818684] [ 893] 1 893 581721 2571 802816 0 0 httpd

[ 1662.827115] [ 2150] 1002 2150 1007708 137960 1646592 0 0 mysqld

[ 1662.835780] [ 2210] 0 2210 362537 2649 196608 0 0 gonit

[ 1662.844262] [ 2263] 1 2263 581729 4554 802816 0 0 httpd

[ 1662.853149] [ 2426] 1 2426 124749 40870 552960 0 0 php-fpm

[ 1662.861719] [ 2428] 1 2428 87846 22321 389120 0 0 php-fpm

[ 1662.870608] [ 2430] 1 2430 107336 41783 544768 0 0 php-fpm

[ 1662.879323] [ 2437] 1 2437 112993 29550 466944 0 0 php-fpm

[ 1662.888099] [ 2448] 0 2448 4153 723 73728 0 0 sshd

[ 1662.896790] [ 2452] 1003 2452 5255 674 77824 0 0 systemd

[ 1662.906110] [ 2453] 1003 2453 5723 570 86016 0 0 (sd-pam)

[ 1662.914768] [ 2466] 1003 2466 4153 523 73728 0 0 sshd

[ 1662.923558] [ 2467] 1003 2467 1748 572 53248 0 0 bash

[ 1662.931989] [ 2468] 1003 2468 614 392 45056 0 0 sftp-server

[ 1662.941466] [ 2485] 1 2485 87220 21809 389120 0 0 php-fpm

[ 1662.950512] [ 2486] 1 2486 104245 38688 524288 0 0 php-fpm

[ 1662.959240] [ 2489] 1 2489 93475 27898 434176 0 0 php-fpm

[ 1662.967925] [ 2653] 0 2653 56456 503 86016 0 0 rsyslogd

[ 1662.976803] Out of memory: Kill process 2150 (mysqld) score 67 or sacrifice child

[ 1662.985163] Killed process 2150 (mysqld) total-vm:4030832kB, anon-rss:551840kB, file-rss:0kB, shmem-rss:0kB

Sep 5 12:59:54 wordpress-multisite-1-vm kernel:[ 1663.016261] oom_reaper: reaped process 2150 (mysqld), now anon-r

ss:0kB, file-rss:0kB, shmem-rss:0kB

[ 1661.435854] mysqld invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_s

core_adj=0

[ 1661.471151] Call Trace: [ 1661.473829] dump_stack+0x66/0x81 [ 1661.477693] dump_header+0x6b/0x283 [ 1661.481647] ? do_try_to_free_pages+0x2ec/0x370 [ 1661.486612] oom_kill_process.cold.30+0xb/0x1cf [ 1661.491448] ? oom_badness+0xee/0x140 [ 1661.495480] out_of_memory+0x1a5/0x450 [ 1661.499597] __alloc_pages_slowpath+0xbd8/0xcb0 [ 1661.504405] __alloc_pages_nodemask+0x28b/0x2b0 [ 1661.509565] filemap_fault+0x333/0x780 [ 1661.513520] ? __switch_to+0x115/0x440 [ 1661.517755] ? filemap_map_pages+0x1e7/0x3a0 [ 1661.522441] ext4_filemap_fault+0x2c/0x40 [ext4] [ 1661.527261] __do_fault+0x34/0x90 [ 1661.530845] __handle_mm_fault+0xdf9/0x11f0 [ 1661.535396] handle_mm_fault+0xd6/0x200 [ 1661.539437] __do_page_fault+0x249/0x4f0 [ 1661.543549] ? page_fault+0x8/0x30 [ 1661.547248] page_fault+0x1e/0x30 [ 1661.550967] RIP: 0033:0x7f7c6f0cc27a [ 1661.554747] Code: Bad RIP value. [ 1661.558250] RSP: 002b:00007f7bb88f5eb0 EFLAGS: 00010217 [ 1661.563838] RAX: 0000000000000000 RBX: 00007f7bb88f6e60 RCX: 00007f7c6e6e1f59 [ 1661.571172] RDX: 0000000000000100 RSI: 0000000000000001 RDI: 00007f7c6e5c1000 [ 1661.578782] RBP: 00007f7c6e5c1000 R08: 00007f7bb88f5f20 R09: 0000000000000050 [ 1661.586114] R10: 00007f7c6b8c7fc0 R11: 0000000000000246 R12: 0000000000000001 [ 1661.593438] R13: 0000000000000000 R14: 0000000000000100 R15: 00007f7c6b8c7fc0 [ 1661.600929] Mem-Info: [ 1661.603464] active_anon:1943746 inactive_anon:15685 isolated_anon:0 [ 1661.603464] active_file:56 inactive_file:125 isolated_file:0 [ 1661.603464] unevictable:3468 dirty:0 writeback:0 unstable:0 [ 1661.603464] slab_reclaimable:3875 slab_unreclaimable:11521 [ 1661.603464] mapped:14918 shmem:15739 pagetables:12351 bounce:0 [ 1661.603464] free:25888 free_pcp:108 free_cma:0 [ 1661.638236] Node 0 active_anon:7774984kB inactive_anon:62740kB active_file:416kB inactive_file:700kB unevictable :13872kB isolated(anon):0kB isolated(file):0kB mapped:59972kB dirty:0kB writeback:0kB shmem:62956kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 6762496kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [ 1661.667250] Node 0 DMA free:15908kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file :0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0k B pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1661.694303] lowmem_reserve[]: 0 2978 7947 7947 7947 [ 1661.699413] Node 0 DMA32 free:45108kB min:25276kB low:31592kB high:37908kB active_anon:3015228kB inactive_anon:1 40kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129332kB managed:3063764kB mlocked :0kB kernel_stack:48kB pagetables:2316kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 1661.727743] lowmem_reserve[]: 0 0 4969 4969 4969 [ 1661.732578] Node 0 Normal free:42032kB min:42172kB low:52712kB high:63252kB active_anon:4759756kB inactive_anon: 62600kB active_file:284kB inactive_file:308kB unevictable:13872kB writepending:0kB present:5242880kB managed:509324 4kB mlocked:13872kB kernel_stack:27120kB pagetables:47088kB bounce:0kB free_pcp:1176kB local_pcp:284kB free_cma:0kB [ 1661.763368] lowmem_reserve[]: 0 0 0 0 0 [ 1661.767458] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U ) 1*2048kB (M) 3*4096kB (M) = 15908kB [ 1661.781520] Node 0 DMA32: 63*4kB (UME) 57*8kB (UME) 17*16kB (UE) 49*32kB (UME) 39*64kB (UME) 17*128kB (UME) 4*25 6kB (UME) 2*512kB (UE) 3*1024kB (M) 6*2048kB (UME) 5*4096kB (M) = 45108kB [ 1661.798511] Node 0 Normal: 1370*4kB (UE) 859*8kB (UME) 473*16kB (UME) 143*32kB (UME) 58*64kB (UME) 32*128kB (UME ) 12*256kB (UME) 7*512kB (UME) 3*1024kB (ME) 0*2048kB 0*4096kB = 42032kB [ 1661.815443] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [ 1661.824483] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 1661.833456] 16957 total pagecache pages [ 1661.837705] 0 pages in swap cache [ 1661.841271] Swap cache stats: add 0, delete 0, find 0/0 [ 1661.846754] Free swap = 0kB [ 1661.849977] Total swap = 0kB [ 1661.853118] 2097051 pages RAM [ 1661.856315] 0 pages HighMem/MovableOnly [ 1661.860387] 53822 pages reserved [ 1661.863840] Tasks state (memory values in pages): [ 1661.868863] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 1661.877845] [ 234] 0 234 7391 986 94208 0 0 systemd-journal [ 1661.887125] [ 245] 0 245 5055 685 57344 0 -1000 systemd-udevd [ 1661.896249] [ 327] 0 327 2372 754 53248 0 0 dhclient [ 1661.905186] [ 333] 0 333 2021 1189 53248 0 0 haveged [ 1661.913959] [ 375] 0 375 571 18 40960 0 0 atopacctd [ 1661.922820] [ 378] 104 378 2197 492 57344 0 -900 dbus-daemon [ 1661.931828] [ 383] 0 383 1372 433 49152 0 0 atd [ 1661.940310] [ 391] 0 391 4129 3407 73728 0 0 atop [ 1661.948821] [ 402] 0 402 30162 2283 110592 0 0 google_osconfig [ 1661.958302] [ 403] 0 403 7413 1976 94208 0 0 unattended-upgr [ 1661.967600] [ 408] 0 408 1346 397 45056 0 0 agetty [ 1661.976321] [ 409] 0 409 1403 343 45056 0 0 agetty [ 1661.985225] [ 411] 106 411 3253 583 49152 0 0 chronyd [ 1661.993839] [ 414] 106 414 1204 67 49152 0 0 chronyd [ 1662.002530] [ 475] 0 475 28574 1893 90112 0 -999 google_guest_ag [ 1662.012177] [ 651] 0 651 3963 683 65536 0 -1000 sshd [ 1662.020524] [ 658] 0 658 4848 635 77824 0 0 systemd-logind [ 1662.029991] [ 661] 0 661 2126 393 53248 0 0 cron [ 1662.038420] [ 686] 0 686 79790 3734 245760 0 0 php-fpm [ 1662.047466] [ 702] 1 702 106280 40784 536576 0 0 php-fpm [ 1662.056146] [ 703] 1 703 118112 34189 491520 0 0 php-fpm [ 1662.065006] [ 704] 1 704 118662 34954 512000 0 0 php-fpm [ 1662.073698] [ 705] 1 705 101157 35677 495616 0 0 php-fpm [ 1662.082485] [ 706] 1 706 118665 35161 507904 0 0 php-fpm [ 1662.091350] [ 707] 1 707 95528 30002 450560 0 0 php-fpm [ 1662.100044] [ 708] 1 708 124756 40968 557056 0 0 php-fpm [ 1662.108730] [ 709] 1 709 100144 34615 487424 0 0 php-fpm [ 1662.118056] [ 710] 1 710 124285 40376 544768 0 0 php-fpm [ 1662.126865] [ 711] 1 711 101178 35990 495616 0 0 php-fpm [ 1662.135759] [ 712] 1 712 105763 40224 532480 0 0 php-fpm [ 1662.144363] [ 713] 1 713 87860 22331 389120 0 0 php-fpm [ 1662.152957] [ 714] 1 714 105846 40309 536576 0 0 php-fpm [ 1662.161650] [ 715] 1 715 86707 21240 380928 0 0 php-fpm [ 1662.170284] [ 716] 1 716 115071 31175 471040 0 0 php-fpm [ 1662.178892] [ 717] 1 717 112989 29122 458752 0 0 php-fpm [ 1662.187599] [ 718] 1 718 86691 21232 380928 0 0 php-fpm [ 1662.196219] [ 719] 1 719 121777 38238 524288 0 0 php-fpm [ 1662.205168] [ 720] 1 720 111441 27527 442368 0 0 php-fpm [ 1662.214000] [ 721] 1 721 96630 31127 466944 0 0 php-fpm [ 1662.222698] [ 722] 1 722 127309 43466 565248 0 0 php-fpm [ 1662.231401] [ 723] 1 723 97060 31548 462848 0 0 php-fpm [ 1662.240281] [ 724] 1 724 104537 20687 397312 0 0 php-fpm [ 1662.248882] [ 725] 1 725 115259 31654 475136 0 0 php-fpm [ 1662.257725] [ 726] 1 726 102691 37172 507904 0 0 php-fpm [ 1662.266352] [ 727] 1 727 99644 34153 491520 0 0 php-fpm [ 1662.275132] [ 728] 1 728 108835 43342 557056 0 0 php-fpm [ 1662.283999] [ 729] 1 729 102692 37186 507904 0 0 php-fpm [ 1662.292594] [ 730] 1 730 103203 37703 516096 0 0 php-fpm [ 1662.301379] [ 731] 1 731 109348 43882 565248 0 0 php-fpm [ 1662.310237] [ 732] 1 732 121181 37329 524288 0 0 php-fpm [ 1662.319006] [ 733] 1 733 105763 40240 532480 0 0 php-fpm [ 1662.327725] [ 734] 1 734 109720 44287 569344 0 0 php-fpm [ 1662.336360] [ 735] 1 735 127841 44064 577536 0 0 php-fpm [ 1662.345062] [ 736] 1 736 93475 27919 434176 0 0 php-fpm [ 1662.353755] [ 737] 1 737 96563 31090 458752 0 0 php-fpm [ 1662.362456] [ 738] 1 738 101172 35674 495616 0 0 php-fpm [ 1662.371265] [ 739] 1 739 115229 31611 475136 0 0 php-fpm [ 1662.380093] [ 740] 1 740 123927 39564 544768 0 0 php-fpm [ 1662.388968] [ 741] 1 741 101032 35487 499712 0 0 php-fpm [ 1662.397782] [ 742] 1 742 88876 23341 401408 0 0 php-fpm [ 1662.406796] [ 743] 1 743 105181 21357 393216 0 0 php-fpm [ 1662.415617] [ 744] 1 744 120653 36816 516096 0 0 php-fpm [ 1662.424341] [ 745] 1 745 127828 44026 573440 0 0 php-fpm [ 1662.433546] [ 746] 1 746 119126 35206 507904 0 0 php-fpm [ 1662.442337] [ 747] 1 747 122211 38401 532480 0 0 php-fpm [ 1662.451135] [ 748] 1 748 106304 40750 544768 0 0 php-fpm [ 1662.460291] [ 749] 1 749 99643 34135 487424 0 0 php-fpm [ 1662.469003] [ 750] 1 750 102692 37135 512000 0 0 php-fpm [ 1662.477799] [ 751] 1 751 119134 35254 503808 0 0 php-fpm [ 1662.486491] [ 752] 1 752 118605 34719 503808 0 0 php-fpm [ 1662.495203] [ 753] 1 753 118668 34827 503808 0 0 php-fpm [ 1662.504155] [ 754] 1 754 109495 44024 565248 0 0 php-fpm [ 1662.512761] [ 755] 1 755 100659 35159 495616 0 0 php-fpm [ 1662.521633] [ 756] 1 756 94507 28990 450560 0 0 php-fpm [ 1662.530331] [ 757] 1 757 106787 40188 536576 0 0 php-fpm [ 1662.539198] [ 758] 1 758 104739 39263 524288 0 0 php-fpm [ 1662.548134] [ 759] 1 759 125780 41999 565248 0 0 php-fpm [ 1662.557276] [ 760] 1 760 100243 34798 491520 0 0 php-fpm [ 1662.566458] [ 761] 1 761 116592 32709 491520 0 0 php-fpm [ 1662.575264] [ 762] 1 762 105878 22433 409600 0 0 php-fpm [ 1662.583889] [ 763] 1 763 93987 28486 438272 0 0 php-fpm [ 1662.592585] [ 764] 1 764 96571 31036 462848 0 0 php-fpm [ 1662.601326] [ 765] 1 765 125282 41497 557056 0 0 php-fpm [ 1662.610045] [ 766] 1 766 87851 22286 401408 0 0 php-fpm [ 1662.618787] [ 767] 1 767 112461 28545 446464 0 0 php-fpm [ 1662.627582] [ 768] 1 768 101668 36199 499712 0 0 php-fpm [ 1662.636242] [ 769] 1 769 97450 31954 475136 0 0 php-fpm [ 1662.645077] [ 770] 1 770 99668 34117 487424 0 0 php-fpm [ 1662.654059] [ 771] 1 771 107053 23513 421888 0 0 php-fpm [ 1662.663203] [ 773] 1 773 90419 24901 409600 0 0 php-fpm [ 1662.672065] [ 774] 1 774 103254 19315 376832 0 0 php-fpm [ 1662.680661] [ 776] 1 776 100025 34531 491520 0 0 php-fpm [ 1662.689359] [ 777] 1 777 126829 43311 565248 0 0 php-fpm [ 1662.697965] [ 778] 1 778 113004 29552 454656 0 0 php-fpm [ 1662.706683] [ 779] 1 779 100008 34488 491520 0 0 php-fpm [ 1662.715392] [ 780] 1 780 105780 40268 532480 0 0 php-fpm [ 1662.724030] [ 781] 1 781 124749 40858 552960 0 0 php-fpm [ 1662.732623] [ 783] 1 783 100131 34621 491520 0 0 php-fpm [ 1662.741403] [ 784] 1 784 93955 27508 430080 0 0 php-fpm [ 1662.750085] [ 828] 0 828 3725 1031 69632 0 0 httpd [ 1662.758786] [ 834] 1 834 581721 2609 802816 0 0 httpd [ 1662.767228] [ 835] 1 835 581726 4090 802816 0 0 httpd [ 1662.775654] [ 836] 1 836 581726 3533 802816 0 0 httpd [ 1662.784446] [ 841] 1 841 581721 2494 802816 0 0 httpd [ 1662.792875] [ 854] 1 854 581721 5142 802816 0 0 httpd [ 1662.801477] [ 868] 1 868 581869 2645 806912 0 0 httpd [ 1662.809994] [ 883] 1 883 581784 3656 802816 0 0 httpd [ 1662.818684] [ 893] 1 893 581721 2571 802816 0 0 httpd [ 1662.827115] [ 2150] 1002 2150 1007708 137960 1646592 0 0 mysqld [ 1662.835780] [ 2210] 0 2210 362537 2649 196608 0 0 gonit [ 1662.844262] [ 2263] 1 2263 581729 4554 802816 0 0 httpd [ 1662.853149] [ 2426] 1 2426 124749 40870 552960 0 0 php-fpm [ 1662.861719] [ 2428] 1 2428 87846 22321 389120 0 0 php-fpm [ 1662.870608] [ 2430] 1 2430 107336 41783 544768 0 0 php-fpm [ 1662.879323] [ 2437] 1 2437 112993 29550 466944 0 0 php-fpm [ 1662.888099] [ 2448] 0 2448 4153 723 73728 0 0 sshd [ 1662.896790] [ 2452] 1003 2452 5255 674 77824 0 0 systemd [ 1662.906110] [ 2453] 1003 2453 5723 570 86016 0 0 (sd-pam) [ 1662.914768] [ 2466] 1003 2466 4153 523 73728 0 0 sshd [ 1662.923558] [ 2467] 1003 2467 1748 572 53248 0 0 bash [ 1662.931989] [ 2468] 1003 2468 614 392 45056 0 0 sftp-server [ 1662.941466] [ 2485] 1 2485 87220 21809 389120 0 0 php-fpm [ 1662.950512] [ 2486] 1 2486 104245 38688 524288 0 0 php-fpm [ 1662.959240] [ 2489] 1 2489 93475 27898 434176 0 0 php-fpm [ 1662.967925] [ 2653] 0 2653 56456 503 86016 0 0 rsyslogd [ 1662.976803] Out of memory: Kill process 2150 (mysqld) score 67 or sacrifice child [ 1662.985163] Killed process 2150 (mysqld) total-vm:4030832kB, anon-rss:551840kB, file-rss:0kB, shmem-rss:0kB Sep 5 12:59:54 wordpress-multisite-1-vm kernel:[ 1663.016261] oom_reaper: reaped process 2150 (mysqld), now anon-r ss:0kB, file-rss:0kB, shmem-rss:0kB [ 1661.435854] mysqld invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_s core_adj=0

[ 1661.471151] Call Trace:
[ 1661.473829]  dump_stack+0x66/0x81
[ 1661.477693]  dump_header+0x6b/0x283
[ 1661.481647]  ? do_try_to_free_pages+0x2ec/0x370
[ 1661.486612]  oom_kill_process.cold.30+0xb/0x1cf
[ 1661.491448]  ? oom_badness+0xee/0x140
[ 1661.495480]  out_of_memory+0x1a5/0x450
[ 1661.499597]  __alloc_pages_slowpath+0xbd8/0xcb0
[ 1661.504405]  __alloc_pages_nodemask+0x28b/0x2b0
[ 1661.509565]  filemap_fault+0x333/0x780
[ 1661.513520]  ? __switch_to+0x115/0x440
[ 1661.517755]  ? filemap_map_pages+0x1e7/0x3a0
[ 1661.522441]  ext4_filemap_fault+0x2c/0x40 [ext4]
[ 1661.527261]  __do_fault+0x34/0x90
[ 1661.530845]  __handle_mm_fault+0xdf9/0x11f0
[ 1661.535396]  handle_mm_fault+0xd6/0x200
[ 1661.539437]  __do_page_fault+0x249/0x4f0
[ 1661.543549]  ? page_fault+0x8/0x30
[ 1661.547248]  page_fault+0x1e/0x30
[ 1661.550967] RIP: 0033:0x7f7c6f0cc27a
[ 1661.554747] Code: Bad RIP value.
[ 1661.558250] RSP: 002b:00007f7bb88f5eb0 EFLAGS: 00010217
[ 1661.563838] RAX: 0000000000000000 RBX: 00007f7bb88f6e60 RCX: 00007f7c6e6e1f59
[ 1661.571172] RDX: 0000000000000100 RSI: 0000000000000001 RDI: 00007f7c6e5c1000
[ 1661.578782] RBP: 00007f7c6e5c1000 R08: 00007f7bb88f5f20 R09: 0000000000000050
[ 1661.586114] R10: 00007f7c6b8c7fc0 R11: 0000000000000246 R12: 0000000000000001
[ 1661.593438] R13: 0000000000000000 R14: 0000000000000100 R15: 00007f7c6b8c7fc0
[ 1661.600929] Mem-Info:
[ 1661.603464] active_anon:1943746 inactive_anon:15685 isolated_anon:0
[ 1661.603464]  active_file:56 inactive_file:125 isolated_file:0
[ 1661.603464]  unevictable:3468 dirty:0 writeback:0 unstable:0
[ 1661.603464]  slab_reclaimable:3875 slab_unreclaimable:11521
[ 1661.603464]  mapped:14918 shmem:15739 pagetables:12351 bounce:0
[ 1661.603464]  free:25888 free_pcp:108 free_cma:0
[ 1661.638236] Node 0 active_anon:7774984kB inactive_anon:62740kB active_file:416kB inactive_file:700kB unevictable
:13872kB isolated(anon):0kB isolated(file):0kB mapped:59972kB dirty:0kB writeback:0kB shmem:62956kB shmem_thp: 0kB 
shmem_pmdmapped: 0kB anon_thp: 6762496kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1661.667250] Node 0 DMA free:15908kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file
:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0k
B pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1661.694303] lowmem_reserve[]: 0 2978 7947 7947 7947
[ 1661.699413] Node 0 DMA32 free:45108kB min:25276kB low:31592kB high:37908kB active_anon:3015228kB inactive_anon:1
40kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129332kB managed:3063764kB mlocked
:0kB kernel_stack:48kB pagetables:2316kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1661.727743] lowmem_reserve[]: 0 0 4969 4969 4969
[ 1661.732578] Node 0 Normal free:42032kB min:42172kB low:52712kB high:63252kB active_anon:4759756kB inactive_anon:
62600kB active_file:284kB inactive_file:308kB unevictable:13872kB writepending:0kB present:5242880kB managed:509324
4kB mlocked:13872kB kernel_stack:27120kB pagetables:47088kB bounce:0kB free_pcp:1176kB local_pcp:284kB free_cma:0kB
[ 1661.763368] lowmem_reserve[]: 0 0 0 0 0
[ 1661.767458] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U
) 1*2048kB (M) 3*4096kB (M) = 15908kB
[ 1661.781520] Node 0 DMA32: 63*4kB (UME) 57*8kB (UME) 17*16kB (UE) 49*32kB (UME) 39*64kB (UME) 17*128kB (UME) 4*25
6kB (UME) 2*512kB (UE) 3*1024kB (M) 6*2048kB (UME) 5*4096kB (M) = 45108kB
[ 1661.798511] Node 0 Normal: 1370*4kB (UE) 859*8kB (UME) 473*16kB (UME) 143*32kB (UME) 58*64kB (UME) 32*128kB (UME
) 12*256kB (UME) 7*512kB (UME) 3*1024kB (ME) 0*2048kB 0*4096kB = 42032kB
[ 1661.815443] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1661.824483] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1661.833456] 16957 total pagecache pages
[ 1661.837705] 0 pages in swap cache
[ 1661.841271] Swap cache stats: add 0, delete 0, find 0/0
[ 1661.846754] Free swap  = 0kB
[ 1661.849977] Total swap = 0kB
[ 1661.853118] 2097051 pages RAM
[ 1661.856315] 0 pages HighMem/MovableOnly
[ 1661.860387] 53822 pages reserved
[ 1661.863840] Tasks state (memory values in pages):
[ 1661.868863] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 1661.877845] [    234]     0   234     7391      986    94208        0             0 systemd-journal
[ 1661.887125] [    245]     0   245     5055      685    57344        0         -1000 systemd-udevd
[ 1661.896249] [    327]     0   327     2372      754    53248        0             0 dhclient
[ 1661.905186] [    333]     0   333     2021     1189    53248        0             0 haveged
[ 1661.913959] [    375]     0   375      571       18    40960        0             0 atopacctd
[ 1661.922820] [    378]   104   378     2197      492    57344        0          -900 dbus-daemon
[ 1661.931828] [    383]     0   383     1372      433    49152        0             0 atd
[ 1661.940310] [    391]     0   391     4129     3407    73728        0             0 atop
[ 1661.948821] [    402]     0   402    30162     2283   110592        0             0 google_osconfig
[ 1661.958302] [    403]     0   403     7413     1976    94208        0             0 unattended-upgr
[ 1661.967600] [    408]     0   408     1346      397    45056        0             0 agetty
[ 1661.976321] [    409]     0   409     1403      343    45056        0             0 agetty
[ 1661.985225] [    411]   106   411     3253      583    49152        0             0 chronyd
[ 1661.993839] [    414]   106   414     1204       67    49152        0             0 chronyd
[ 1662.002530] [    475]     0   475    28574     1893    90112        0          -999 google_guest_ag
[ 1662.012177] [    651]     0   651     3963      683    65536        0         -1000 sshd
[ 1662.020524] [    658]     0   658     4848      635    77824        0             0 systemd-logind
[ 1662.029991] [    661]     0   661     2126      393    53248        0             0 cron
[ 1662.038420] [    686]     0   686    79790     3734   245760        0             0 php-fpm
[ 1662.047466] [    702]     1   702   106280    40784   536576        0             0 php-fpm
[ 1662.056146] [    703]     1   703   118112    34189   491520        0             0 php-fpm
[ 1662.065006] [    704]     1   704   118662    34954   512000        0             0 php-fpm
[ 1662.073698] [    705]     1   705   101157    35677   495616        0             0 php-fpm
[ 1662.082485] [    706]     1   706   118665    35161   507904        0             0 php-fpm
[ 1662.091350] [    707]     1   707    95528    30002   450560        0             0 php-fpm
[ 1662.100044] [    708]     1   708   124756    40968   557056        0             0 php-fpm
[ 1662.108730] [    709]     1   709   100144    34615   487424        0             0 php-fpm
[ 1662.118056] [    710]     1   710   124285    40376   544768        0             0 php-fpm
[ 1662.126865] [    711]     1   711   101178    35990   495616        0             0 php-fpm
[ 1662.135759] [    712]     1   712   105763    40224   532480        0             0 php-fpm
[ 1662.144363] [    713]     1   713    87860    22331   389120        0             0 php-fpm
[ 1662.152957] [    714]     1   714   105846    40309   536576        0             0 php-fpm
[ 1662.161650] [    715]     1   715    86707    21240   380928        0             0 php-fpm
[ 1662.170284] [    716]     1   716   115071    31175   471040        0             0 php-fpm
[ 1662.178892] [    717]     1   717   112989    29122   458752        0             0 php-fpm
[ 1662.187599] [    718]     1   718    86691    21232   380928        0             0 php-fpm
[ 1662.196219] [    719]     1   719   121777    38238   524288        0             0 php-fpm
[ 1662.205168] [    720]     1   720   111441    27527   442368        0             0 php-fpm
[ 1662.214000] [    721]     1   721    96630    31127   466944        0             0 php-fpm
[ 1662.222698] [    722]     1   722   127309    43466   565248        0             0 php-fpm
[ 1662.231401] [    723]     1   723    97060    31548   462848        0             0 php-fpm
[ 1662.240281] [    724]     1   724   104537    20687   397312        0             0 php-fpm
[ 1662.248882] [    725]     1   725   115259    31654   475136        0             0 php-fpm
[ 1662.257725] [    726]     1   726   102691    37172   507904        0             0 php-fpm
[ 1662.266352] [    727]     1   727    99644    34153   491520        0             0 php-fpm
[ 1662.275132] [    728]     1   728   108835    43342   557056        0             0 php-fpm
[ 1662.283999] [    729]     1   729   102692    37186   507904        0             0 php-fpm
[ 1662.292594] [    730]     1   730   103203    37703   516096        0             0 php-fpm
[ 1662.301379] [    731]     1   731   109348    43882   565248        0             0 php-fpm
[ 1662.310237] [    732]     1   732   121181    37329   524288        0             0 php-fpm
[ 1662.319006] [    733]     1   733   105763    40240   532480        0             0 php-fpm
[ 1662.327725] [    734]     1   734   109720    44287   569344        0             0 php-fpm
[ 1662.336360] [    735]     1   735   127841    44064   577536        0             0 php-fpm
[ 1662.345062] [    736]     1   736    93475    27919   434176        0             0 php-fpm
[ 1662.353755] [    737]     1   737    96563    31090   458752        0             0 php-fpm
[ 1662.362456] [    738]     1   738   101172    35674   495616        0             0 php-fpm
[ 1662.371265] [    739]     1   739   115229    31611   475136        0             0 php-fpm
[ 1662.380093] [    740]     1   740   123927    39564   544768        0             0 php-fpm
[ 1662.388968] [    741]     1   741   101032    35487   499712        0             0 php-fpm
[ 1662.397782] [    742]     1   742    88876    23341   401408        0             0 php-fpm
[ 1662.406796] [    743]     1   743   105181    21357   393216        0             0 php-fpm
[ 1662.415617] [    744]     1   744   120653    36816   516096        0             0 php-fpm
[ 1662.424341] [    745]     1   745   127828    44026   573440        0             0 php-fpm
[ 1662.433546] [    746]     1   746   119126    35206   507904        0             0 php-fpm
[ 1662.442337] [    747]     1   747   122211    38401   532480        0             0 php-fpm
[ 1662.451135] [    748]     1   748   106304    40750   544768        0             0 php-fpm
[ 1662.460291] [    749]     1   749    99643    34135   487424        0             0 php-fpm
[ 1662.469003] [    750]     1   750   102692    37135   512000        0             0 php-fpm
[ 1662.477799] [    751]     1   751   119134    35254   503808        0             0 php-fpm
[ 1662.486491] [    752]     1   752   118605    34719   503808        0             0 php-fpm
[ 1662.495203] [    753]     1   753   118668    34827   503808        0             0 php-fpm
[ 1662.504155] [    754]     1   754   109495    44024   565248        0             0 php-fpm
[ 1662.512761] [    755]     1   755   100659    35159   495616        0             0 php-fpm
[ 1662.521633] [    756]     1   756    94507    28990   450560        0             0 php-fpm
[ 1662.530331] [    757]     1   757   106787    40188   536576        0             0 php-fpm
[ 1662.539198] [    758]     1   758   104739    39263   524288        0             0 php-fpm
[ 1662.548134] [    759]     1   759   125780    41999   565248        0             0 php-fpm
[ 1662.557276] [    760]     1   760   100243    34798   491520        0             0 php-fpm
[ 1662.566458] [    761]     1   761   116592    32709   491520        0             0 php-fpm
[ 1662.575264] [    762]     1   762   105878    22433   409600        0             0 php-fpm
[ 1662.583889] [    763]     1   763    93987    28486   438272        0             0 php-fpm
[ 1662.592585] [    764]     1   764    96571    31036   462848        0             0 php-fpm
[ 1662.601326] [    765]     1   765   125282    41497   557056        0             0 php-fpm
[ 1662.610045] [    766]     1   766    87851    22286   401408        0             0 php-fpm
[ 1662.618787] [    767]     1   767   112461    28545   446464        0             0 php-fpm
[ 1662.627582] [    768]     1   768   101668    36199   499712        0             0 php-fpm
[ 1662.636242] [    769]     1   769    97450    31954   475136        0             0 php-fpm
[ 1662.645077] [    770]     1   770    99668    34117   487424        0             0 php-fpm
[ 1662.654059] [    771]     1   771   107053    23513   421888        0             0 php-fpm
[ 1662.663203] [    773]     1   773    90419    24901   409600        0             0 php-fpm
[ 1662.672065] [    774]     1   774   103254    19315   376832        0             0 php-fpm
[ 1662.680661] [    776]     1   776   100025    34531   491520        0             0 php-fpm
[ 1662.689359] [    777]     1   777   126829    43311   565248        0             0 php-fpm
[ 1662.697965] [    778]     1   778   113004    29552   454656        0             0 php-fpm
[ 1662.706683] [    779]     1   779   100008    34488   491520        0             0 php-fpm
[ 1662.715392] [    780]     1   780   105780    40268   532480        0             0 php-fpm
[ 1662.724030] [    781]     1   781   124749    40858   552960        0             0 php-fpm
[ 1662.732623] [    783]     1   783   100131    34621   491520        0             0 php-fpm
[ 1662.741403] [    784]     1   784    93955    27508   430080        0             0 php-fpm
[ 1662.750085] [    828]     0   828     3725     1031    69632        0             0 httpd
[ 1662.758786] [    834]     1   834   581721     2609   802816        0             0 httpd
[ 1662.767228] [    835]     1   835   581726     4090   802816        0             0 httpd
[ 1662.775654] [    836]     1   836   581726     3533   802816        0             0 httpd
[ 1662.784446] [    841]     1   841   581721     2494   802816        0             0 httpd
[ 1662.792875] [    854]     1   854   581721     5142   802816        0             0 httpd
[ 1662.801477] [    868]     1   868   581869     2645   806912        0             0 httpd
[ 1662.809994] [    883]     1   883   581784     3656   802816        0             0 httpd
[ 1662.818684] [    893]     1   893   581721     2571   802816        0             0 httpd
[ 1662.827115] [   2150]  1002  2150  1007708   137960  1646592        0             0 mysqld
[ 1662.835780] [   2210]     0  2210   362537     2649   196608        0             0 gonit
[ 1662.844262] [   2263]     1  2263   581729     4554   802816        0             0 httpd
[ 1662.853149] [   2426]     1  2426   124749    40870   552960        0             0 php-fpm
[ 1662.861719] [   2428]     1  2428    87846    22321   389120        0             0 php-fpm
[ 1662.870608] [   2430]     1  2430   107336    41783   544768        0             0 php-fpm
[ 1662.879323] [   2437]     1  2437   112993    29550   466944        0             0 php-fpm
[ 1662.888099] [   2448]     0  2448     4153      723    73728        0             0 sshd
[ 1662.896790] [   2452]  1003  2452     5255      674    77824        0             0 systemd
[ 1662.906110] [   2453]  1003  2453     5723      570    86016        0             0 (sd-pam)
[ 1662.914768] [   2466]  1003  2466     4153      523    73728        0             0 sshd
[ 1662.923558] [   2467]  1003  2467     1748      572    53248        0             0 bash
[ 1662.931989] [   2468]  1003  2468      614      392    45056        0             0 sftp-server
[ 1662.941466] [   2485]     1  2485    87220    21809   389120        0             0 php-fpm
[ 1662.950512] [   2486]     1  2486   104245    38688   524288        0             0 php-fpm
[ 1662.959240] [   2489]     1  2489    93475    27898   434176        0             0 php-fpm
[ 1662.967925] [   2653]     0  2653    56456      503    86016        0             0 rsyslogd
[ 1662.976803] Out of memory: Kill process 2150 (mysqld) score 67 or sacrifice child
[ 1662.985163] Killed process 2150 (mysqld) total-vm:4030832kB, anon-rss:551840kB, file-rss:0kB, shmem-rss:0kB
Sep  5 12:59:54 wordpress-multisite-1-vm kernel:[ 1663.016261] oom_reaper: reaped process 2150 (mysqld), now anon-r
ss:0kB, file-rss:0kB, shmem-rss:0kB
 [ 1661.435854] mysqld invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_s
core_adj=0

從 ps 的結果來看，有非常多支的 php-fpm process 在執行，

每支都吃了 1xx MB 記憶體，將近 90 支就得要吃掉 9 GB 以上，

而我的機器只有 8 GB 記憶體，自然會一直遇到記憶體不足的情況：

testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm

root 688 0.0 0.2 319160 16372 ? Ss 13:52 0:00 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)

daemon 703 0.1 1.3 448156 111048 ? S 13:52 0:06 php-fpm: pool www

daemon 704 0.1 2.1 509684 174852 ? S 13:52 0:06 php-fpm: pool www

daemon 705 0.2 2.1 441872 176228 ? S 13:52 0:07 php-fpm: pool www

daemon 706 0.1 1.1 428336 90596 ? S 13:52 0:06 php-fpm: pool www

daemon 707 0.2 1.5 460424 125848 ? S 13:52 0:06 php-fpm: pool www

daemon 708 0.2 2.1 511612 176308 ? S 13:52 0:08 php-fpm: pool www

daemon 709 0.3 2.1 512100 174772 ? S 13:52 0:10 php-fpm: pool www

daemon 710 0.3 1.4 460352 121488 ? S 13:52 0:10 php-fpm: pool www

daemon 711 0.2 1.6 474680 135536 ? S 13:52 0:08 php-fpm: pool www

daemon 712 0.3 2.0 505180 166296 ? S 13:52 0:10 php-fpm: pool www

daemon 713 0.1 1.6 473400 135888 ? S 13:52 0:05 php-fpm: pool www

daemon 714 0.2 1.7 484912 146460 ? S 13:52 0:09 php-fpm: pool www

daemon 715 0.2 1.6 400656 134948 ? S 13:52 0:07 php-fpm: pool www

daemon 716 0.2 2.0 511388 171504 ? S 13:52 0:08 php-fpm: pool www

......

testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm | wc -l

testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm root 688 0.0 0.2 319160 16372 ? Ss 13:52 0:00 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf) daemon 703 0.1 1.3 448156 111048 ? S 13:52 0:06 php-fpm: pool www daemon 704 0.1 2.1 509684 174852 ? S 13:52 0:06 php-fpm: pool www daemon 705 0.2 2.1 441872 176228 ? S 13:52 0:07 php-fpm: pool www daemon 706 0.1 1.1 428336 90596 ? S 13:52 0:06 php-fpm: pool www daemon 707 0.2 1.5 460424 125848 ? S 13:52 0:06 php-fpm: pool www daemon 708 0.2 2.1 511612 176308 ? S 13:52 0:08 php-fpm: pool www daemon 709 0.3 2.1 512100 174772 ? S 13:52 0:10 php-fpm: pool www daemon 710 0.3 1.4 460352 121488 ? S 13:52 0:10 php-fpm: pool www daemon 711 0.2 1.6 474680 135536 ? S 13:52 0:08 php-fpm: pool www daemon 712 0.3 2.0 505180 166296 ? S 13:52 0:10 php-fpm: pool www daemon 713 0.1 1.6 473400 135888 ? S 13:52 0:05 php-fpm: pool www daemon 714 0.2 1.7 484912 146460 ? S 13:52 0:09 php-fpm: pool www daemon 715 0.2 1.6 400656 134948 ? S 13:52 0:07 php-fpm: pool www daemon 716 0.2 2.0 511388 171504 ? S 13:52 0:08 php-fpm: pool www ...... testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm | wc -l 87

testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm 
root       688  0.0  0.2 319160 16372 ?        Ss   13:52   0:00 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
daemon     703  0.1  1.3 448156 111048 ?       S    13:52   0:06 php-fpm: pool www
daemon     704  0.1  2.1 509684 174852 ?       S    13:52   0:06 php-fpm: pool www
daemon     705  0.2  2.1 441872 176228 ?       S    13:52   0:07 php-fpm: pool www
daemon     706  0.1  1.1 428336 90596 ?        S    13:52   0:06 php-fpm: pool www
daemon     707  0.2  1.5 460424 125848 ?       S    13:52   0:06 php-fpm: pool www
daemon     708  0.2  2.1 511612 176308 ?       S    13:52   0:08 php-fpm: pool www
daemon     709  0.3  2.1 512100 174772 ?       S    13:52   0:10 php-fpm: pool www
daemon     710  0.3  1.4 460352 121488 ?       S    13:52   0:10 php-fpm: pool www
daemon     711  0.2  1.6 474680 135536 ?       S    13:52   0:08 php-fpm: pool www
daemon     712  0.3  2.0 505180 166296 ?       S    13:52   0:10 php-fpm: pool www
daemon     713  0.1  1.6 473400 135888 ?       S    13:52   0:05 php-fpm: pool www
daemon     714  0.2  1.7 484912 146460 ?       S    13:52   0:09 php-fpm: pool www
daemon     715  0.2  1.6 400656 134948 ?       S    13:52   0:07 php-fpm: pool www
daemon     716  0.2  2.0 511388 171504 ?       S    13:52   0:08 php-fpm: pool www
......


testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm  | wc -l
87

看了一下

/opt/bitnami/php/logs/php-fpm.log

/opt/bitnami/php/logs/php-fpm.log ，有點奇怪，

它說目前好像很忙，應該要增加啟動的 server 數量，

可是目前休息中的 server 卻有 79 個：

[05-Sep-2021 11:38:29] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 99 total children

[05-Sep-2021 11:38:30] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 100 total children

[05-Sep-2021 11:38:31] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 101 total children

[05-Sep-2021 11:38:32] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 102 total children

[05-Sep-2021 13:29:40] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 79 idle, and 91 total children

[05-Sep-2021 11:38:29] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 99 total children [05-Sep-2021 11:38:30] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 100 total children [05-Sep-2021 11:38:31] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 101 total children [05-Sep-2021 11:38:32] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 102 total children [05-Sep-2021 13:29:40] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 79 idle, and 91 total children

[05-Sep-2021 11:38:29] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 99 total children
[05-Sep-2021 11:38:30] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 100 total children
[05-Sep-2021 11:38:31] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 101 total children
[05-Sep-2021 11:38:32] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 102 total children
[05-Sep-2021 13:29:40] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 79 idle, and 91 total children

上網查了一下，也有人在抱怨 php-fpm process 吃了很多記憶體的事…

打開

/opt/bitnami/php/etc/memory.conf

/opt/bitnami/php/etc/memory.conf 看一下：

; Bitnami memory configuration for PHP-FPM

;

; Note: This will be modified on server size changes

pm.max_children=120

pm.start_servers=80

pm.min_spare_servers=80

pm.max_spare_servers=90

pm.max_requests=5000

; Bitnami memory configuration for PHP-FPM ; ; Note: This will be modified on server size changes pm.max_children=120 pm.start_servers=80 pm.min_spare_servers=80 pm.max_spare_servers=90 pm.max_requests=5000

; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes
pm.max_children=120
pm.start_servers=80
pm.min_spare_servers=80
pm.max_spare_servers=90
pm.max_requests=5000

這裡的設定中沒有 pm 設定，因此預設是

pm = dynamic

pm = dynamic ，

PHP 的 process 數量會動態增減，而且一開始就會有 80 個，最多到 120 個。

就是因為這邊的設定，導致過多的 php-fpm process 占據太多的記憶體。

(不過 php-fpm.log 說目前很忙這件事，難以理解…)

簡單的改法，是設定

pm = ondemand

pm = ondemand ，有需要才增加 php-fpm process。

且指定

pm.process_idle_timeout = 20s

pm.process_idle_timeout = 20s ，

如果 php-fpm process 在 20 秒內接不到新的要求，就可以下去休息了。

這樣的話，就不會有過多的 process 在待命：

pm = ondemand

pm.process_idle_timeout = 20s

pm.max_children=120

pm.start_servers=80

pm.min_spare_servers=80

pm.max_spare_servers=90

pm.max_requests=5000

pm = ondemand pm.process_idle_timeout = 20s pm.max_children=120 pm.start_servers=80 pm.min_spare_servers=80 pm.max_spare_servers=90 pm.max_requests=5000

pm = ondemand
pm.process_idle_timeout = 20s
pm.max_children=120
pm.start_servers=80
pm.min_spare_servers=80
pm.max_spare_servers=90
pm.max_requests=5000

修改好之後，要重新啟動相關服務：

sudo /opt/bitnami/ctlscript.sh restart

sudo /opt/bitnami/ctlscript.sh restart

這時再看一下 php-fpm 的數量，發現只剩下 4 支了：

testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php | wc -l

testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php | wc -l 4

testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php | wc -l
4

從 GCP 的監控畫面中，可以看到記憶體從開機後一路衝到快 100%，

在修改完之後，直線掉落到約 10% 上下：

當然這種改法，有省記憶體的好處，

自然也會有壞處，就是假設有一堆人跑來瀏覽網頁的話，

也許大家就要稍微等待 php-fpm process 被叫起來，會慢一點點。

不過以我的小網站來說，應該還可以接受，

畢竟記憶體一直爆掉，整個網站不能動還是比較嚴重…

用這招還有一點要注意的，就是如果調整了 VM 的設定，

像是調整了 CPU/RAM 之類的話，

/opt/bitnami/php/etc/memory.conf

/opt/bitnami/php/etc/memory.conf 就可能會被重設，

因此得再重新修改一次囉～

參考資料：

(本頁面已被瀏覽過 720 次)

EPH 的程式日記

記錄程式設計生活的點點滴滴

[GCP] Bitnami WordPress 機器上有太多的 php-fpm 進程，導致記憶體不足

2021-09-19 ephrain Comments 0 Comment

發佈留言取消回覆

發佈留言 取消回覆

發佈留言取消回覆