[GCP] Bitnami WordPress 機器上有太多的 php-fpm 進程,導致記憶體不足

[GCP] Bitnami WordPress 機器上有太多的 php-fpm 進程,導致記憶體不足

在 GCP 上的 WordPress 機器,

偶爾會出現機器非常忙碌,網站無法連線的狀況。

用 SSH 連上機器後,通常只看得到像 php、mysql 這些 process 吃很多 CPU,

且記憶體也吃了不少,但不知道實際原因。

 

不知道解決方法,也只能重新開機, 通常會緩解一陣子…

但是最近這招卻無效,屢屢重開機完後,CPU/記憶體又很快爆表了…

像下圖可以看到,以往的 CPU 使用率都只在 5% 上下,

但出事的時候,就會往 20% 甚至更高衝上去:

 

看了一下 dmesg,這才發現發生了記憶體不足 Out-of-memory (OOM) 的錯誤,

且這錯誤還會持續不斷發生:

[ 1661.471151] Call Trace:
[ 1661.473829]  dump_stack+0x66/0x81
[ 1661.477693]  dump_header+0x6b/0x283
[ 1661.481647]  ? do_try_to_free_pages+0x2ec/0x370
[ 1661.486612]  oom_kill_process.cold.30+0xb/0x1cf
[ 1661.491448]  ? oom_badness+0xee/0x140
[ 1661.495480]  out_of_memory+0x1a5/0x450
[ 1661.499597]  __alloc_pages_slowpath+0xbd8/0xcb0
[ 1661.504405]  __alloc_pages_nodemask+0x28b/0x2b0
[ 1661.509565]  filemap_fault+0x333/0x780
[ 1661.513520]  ? __switch_to+0x115/0x440
[ 1661.517755]  ? filemap_map_pages+0x1e7/0x3a0
[ 1661.522441]  ext4_filemap_fault+0x2c/0x40 [ext4]
[ 1661.527261]  __do_fault+0x34/0x90
[ 1661.530845]  __handle_mm_fault+0xdf9/0x11f0
[ 1661.535396]  handle_mm_fault+0xd6/0x200
[ 1661.539437]  __do_page_fault+0x249/0x4f0
[ 1661.543549]  ? page_fault+0x8/0x30
[ 1661.547248]  page_fault+0x1e/0x30
[ 1661.550967] RIP: 0033:0x7f7c6f0cc27a
[ 1661.554747] Code: Bad RIP value.
[ 1661.558250] RSP: 002b:00007f7bb88f5eb0 EFLAGS: 00010217
[ 1661.563838] RAX: 0000000000000000 RBX: 00007f7bb88f6e60 RCX: 00007f7c6e6e1f59
[ 1661.571172] RDX: 0000000000000100 RSI: 0000000000000001 RDI: 00007f7c6e5c1000
[ 1661.578782] RBP: 00007f7c6e5c1000 R08: 00007f7bb88f5f20 R09: 0000000000000050
[ 1661.586114] R10: 00007f7c6b8c7fc0 R11: 0000000000000246 R12: 0000000000000001
[ 1661.593438] R13: 0000000000000000 R14: 0000000000000100 R15: 00007f7c6b8c7fc0
[ 1661.600929] Mem-Info:
[ 1661.603464] active_anon:1943746 inactive_anon:15685 isolated_anon:0
[ 1661.603464]  active_file:56 inactive_file:125 isolated_file:0
[ 1661.603464]  unevictable:3468 dirty:0 writeback:0 unstable:0
[ 1661.603464]  slab_reclaimable:3875 slab_unreclaimable:11521
[ 1661.603464]  mapped:14918 shmem:15739 pagetables:12351 bounce:0
[ 1661.603464]  free:25888 free_pcp:108 free_cma:0
[ 1661.638236] Node 0 active_anon:7774984kB inactive_anon:62740kB active_file:416kB inactive_file:700kB unevictable
:13872kB isolated(anon):0kB isolated(file):0kB mapped:59972kB dirty:0kB writeback:0kB shmem:62956kB shmem_thp: 0kB 
shmem_pmdmapped: 0kB anon_thp: 6762496kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 1661.667250] Node 0 DMA free:15908kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file
:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0k
B pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1661.694303] lowmem_reserve[]: 0 2978 7947 7947 7947
[ 1661.699413] Node 0 DMA32 free:45108kB min:25276kB low:31592kB high:37908kB active_anon:3015228kB inactive_anon:1
40kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129332kB managed:3063764kB mlocked
:0kB kernel_stack:48kB pagetables:2316kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1661.727743] lowmem_reserve[]: 0 0 4969 4969 4969
[ 1661.732578] Node 0 Normal free:42032kB min:42172kB low:52712kB high:63252kB active_anon:4759756kB inactive_anon:
62600kB active_file:284kB inactive_file:308kB unevictable:13872kB writepending:0kB present:5242880kB managed:509324
4kB mlocked:13872kB kernel_stack:27120kB pagetables:47088kB bounce:0kB free_pcp:1176kB local_pcp:284kB free_cma:0kB
[ 1661.763368] lowmem_reserve[]: 0 0 0 0 0
[ 1661.767458] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U
) 1*2048kB (M) 3*4096kB (M) = 15908kB
[ 1661.781520] Node 0 DMA32: 63*4kB (UME) 57*8kB (UME) 17*16kB (UE) 49*32kB (UME) 39*64kB (UME) 17*128kB (UME) 4*25
6kB (UME) 2*512kB (UE) 3*1024kB (M) 6*2048kB (UME) 5*4096kB (M) = 45108kB
[ 1661.798511] Node 0 Normal: 1370*4kB (UE) 859*8kB (UME) 473*16kB (UME) 143*32kB (UME) 58*64kB (UME) 32*128kB (UME
) 12*256kB (UME) 7*512kB (UME) 3*1024kB (ME) 0*2048kB 0*4096kB = 42032kB
[ 1661.815443] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1661.824483] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1661.833456] 16957 total pagecache pages
[ 1661.837705] 0 pages in swap cache
[ 1661.841271] Swap cache stats: add 0, delete 0, find 0/0
[ 1661.846754] Free swap  = 0kB
[ 1661.849977] Total swap = 0kB
[ 1661.853118] 2097051 pages RAM
[ 1661.856315] 0 pages HighMem/MovableOnly
[ 1661.860387] 53822 pages reserved
[ 1661.863840] Tasks state (memory values in pages):
[ 1661.868863] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 1661.877845] [    234]     0   234     7391      986    94208        0             0 systemd-journal
[ 1661.887125] [    245]     0   245     5055      685    57344        0         -1000 systemd-udevd
[ 1661.896249] [    327]     0   327     2372      754    53248        0             0 dhclient
[ 1661.905186] [    333]     0   333     2021     1189    53248        0             0 haveged
[ 1661.913959] [    375]     0   375      571       18    40960        0             0 atopacctd
[ 1661.922820] [    378]   104   378     2197      492    57344        0          -900 dbus-daemon
[ 1661.931828] [    383]     0   383     1372      433    49152        0             0 atd
[ 1661.940310] [    391]     0   391     4129     3407    73728        0             0 atop
[ 1661.948821] [    402]     0   402    30162     2283   110592        0             0 google_osconfig
[ 1661.958302] [    403]     0   403     7413     1976    94208        0             0 unattended-upgr
[ 1661.967600] [    408]     0   408     1346      397    45056        0             0 agetty
[ 1661.976321] [    409]     0   409     1403      343    45056        0             0 agetty
[ 1661.985225] [    411]   106   411     3253      583    49152        0             0 chronyd
[ 1661.993839] [    414]   106   414     1204       67    49152        0             0 chronyd
[ 1662.002530] [    475]     0   475    28574     1893    90112        0          -999 google_guest_ag
[ 1662.012177] [    651]     0   651     3963      683    65536        0         -1000 sshd
[ 1662.020524] [    658]     0   658     4848      635    77824        0             0 systemd-logind
[ 1662.029991] [    661]     0   661     2126      393    53248        0             0 cron
[ 1662.038420] [    686]     0   686    79790     3734   245760        0             0 php-fpm
[ 1662.047466] [    702]     1   702   106280    40784   536576        0             0 php-fpm
[ 1662.056146] [    703]     1   703   118112    34189   491520        0             0 php-fpm
[ 1662.065006] [    704]     1   704   118662    34954   512000        0             0 php-fpm
[ 1662.073698] [    705]     1   705   101157    35677   495616        0             0 php-fpm
[ 1662.082485] [    706]     1   706   118665    35161   507904        0             0 php-fpm
[ 1662.091350] [    707]     1   707    95528    30002   450560        0             0 php-fpm
[ 1662.100044] [    708]     1   708   124756    40968   557056        0             0 php-fpm
[ 1662.108730] [    709]     1   709   100144    34615   487424        0             0 php-fpm
[ 1662.118056] [    710]     1   710   124285    40376   544768        0             0 php-fpm
[ 1662.126865] [    711]     1   711   101178    35990   495616        0             0 php-fpm
[ 1662.135759] [    712]     1   712   105763    40224   532480        0             0 php-fpm
[ 1662.144363] [    713]     1   713    87860    22331   389120        0             0 php-fpm
[ 1662.152957] [    714]     1   714   105846    40309   536576        0             0 php-fpm
[ 1662.161650] [    715]     1   715    86707    21240   380928        0             0 php-fpm
[ 1662.170284] [    716]     1   716   115071    31175   471040        0             0 php-fpm
[ 1662.178892] [    717]     1   717   112989    29122   458752        0             0 php-fpm
[ 1662.187599] [    718]     1   718    86691    21232   380928        0             0 php-fpm
[ 1662.196219] [    719]     1   719   121777    38238   524288        0             0 php-fpm
[ 1662.205168] [    720]     1   720   111441    27527   442368        0             0 php-fpm
[ 1662.214000] [    721]     1   721    96630    31127   466944        0             0 php-fpm
[ 1662.222698] [    722]     1   722   127309    43466   565248        0             0 php-fpm
[ 1662.231401] [    723]     1   723    97060    31548   462848        0             0 php-fpm
[ 1662.240281] [    724]     1   724   104537    20687   397312        0             0 php-fpm
[ 1662.248882] [    725]     1   725   115259    31654   475136        0             0 php-fpm
[ 1662.257725] [    726]     1   726   102691    37172   507904        0             0 php-fpm
[ 1662.266352] [    727]     1   727    99644    34153   491520        0             0 php-fpm
[ 1662.275132] [    728]     1   728   108835    43342   557056        0             0 php-fpm
[ 1662.283999] [    729]     1   729   102692    37186   507904        0             0 php-fpm
[ 1662.292594] [    730]     1   730   103203    37703   516096        0             0 php-fpm
[ 1662.301379] [    731]     1   731   109348    43882   565248        0             0 php-fpm
[ 1662.310237] [    732]     1   732   121181    37329   524288        0             0 php-fpm
[ 1662.319006] [    733]     1   733   105763    40240   532480        0             0 php-fpm
[ 1662.327725] [    734]     1   734   109720    44287   569344        0             0 php-fpm
[ 1662.336360] [    735]     1   735   127841    44064   577536        0             0 php-fpm
[ 1662.345062] [    736]     1   736    93475    27919   434176        0             0 php-fpm
[ 1662.353755] [    737]     1   737    96563    31090   458752        0             0 php-fpm
[ 1662.362456] [    738]     1   738   101172    35674   495616        0             0 php-fpm
[ 1662.371265] [    739]     1   739   115229    31611   475136        0             0 php-fpm
[ 1662.380093] [    740]     1   740   123927    39564   544768        0             0 php-fpm
[ 1662.388968] [    741]     1   741   101032    35487   499712        0             0 php-fpm
[ 1662.397782] [    742]     1   742    88876    23341   401408        0             0 php-fpm
[ 1662.406796] [    743]     1   743   105181    21357   393216        0             0 php-fpm
[ 1662.415617] [    744]     1   744   120653    36816   516096        0             0 php-fpm
[ 1662.424341] [    745]     1   745   127828    44026   573440        0             0 php-fpm
[ 1662.433546] [    746]     1   746   119126    35206   507904        0             0 php-fpm
[ 1662.442337] [    747]     1   747   122211    38401   532480        0             0 php-fpm
[ 1662.451135] [    748]     1   748   106304    40750   544768        0             0 php-fpm
[ 1662.460291] [    749]     1   749    99643    34135   487424        0             0 php-fpm
[ 1662.469003] [    750]     1   750   102692    37135   512000        0             0 php-fpm
[ 1662.477799] [    751]     1   751   119134    35254   503808        0             0 php-fpm
[ 1662.486491] [    752]     1   752   118605    34719   503808        0             0 php-fpm
[ 1662.495203] [    753]     1   753   118668    34827   503808        0             0 php-fpm
[ 1662.504155] [    754]     1   754   109495    44024   565248        0             0 php-fpm
[ 1662.512761] [    755]     1   755   100659    35159   495616        0             0 php-fpm
[ 1662.521633] [    756]     1   756    94507    28990   450560        0             0 php-fpm
[ 1662.530331] [    757]     1   757   106787    40188   536576        0             0 php-fpm
[ 1662.539198] [    758]     1   758   104739    39263   524288        0             0 php-fpm
[ 1662.548134] [    759]     1   759   125780    41999   565248        0             0 php-fpm
[ 1662.557276] [    760]     1   760   100243    34798   491520        0             0 php-fpm
[ 1662.566458] [    761]     1   761   116592    32709   491520        0             0 php-fpm
[ 1662.575264] [    762]     1   762   105878    22433   409600        0             0 php-fpm
[ 1662.583889] [    763]     1   763    93987    28486   438272        0             0 php-fpm
[ 1662.592585] [    764]     1   764    96571    31036   462848        0             0 php-fpm
[ 1662.601326] [    765]     1   765   125282    41497   557056        0             0 php-fpm
[ 1662.610045] [    766]     1   766    87851    22286   401408        0             0 php-fpm
[ 1662.618787] [    767]     1   767   112461    28545   446464        0             0 php-fpm
[ 1662.627582] [    768]     1   768   101668    36199   499712        0             0 php-fpm
[ 1662.636242] [    769]     1   769    97450    31954   475136        0             0 php-fpm
[ 1662.645077] [    770]     1   770    99668    34117   487424        0             0 php-fpm
[ 1662.654059] [    771]     1   771   107053    23513   421888        0             0 php-fpm
[ 1662.663203] [    773]     1   773    90419    24901   409600        0             0 php-fpm
[ 1662.672065] [    774]     1   774   103254    19315   376832        0             0 php-fpm
[ 1662.680661] [    776]     1   776   100025    34531   491520        0             0 php-fpm
[ 1662.689359] [    777]     1   777   126829    43311   565248        0             0 php-fpm
[ 1662.697965] [    778]     1   778   113004    29552   454656        0             0 php-fpm
[ 1662.706683] [    779]     1   779   100008    34488   491520        0             0 php-fpm
[ 1662.715392] [    780]     1   780   105780    40268   532480        0             0 php-fpm
[ 1662.724030] [    781]     1   781   124749    40858   552960        0             0 php-fpm
[ 1662.732623] [    783]     1   783   100131    34621   491520        0             0 php-fpm
[ 1662.741403] [    784]     1   784    93955    27508   430080        0             0 php-fpm
[ 1662.750085] [    828]     0   828     3725     1031    69632        0             0 httpd
[ 1662.758786] [    834]     1   834   581721     2609   802816        0             0 httpd
[ 1662.767228] [    835]     1   835   581726     4090   802816        0             0 httpd
[ 1662.775654] [    836]     1   836   581726     3533   802816        0             0 httpd
[ 1662.784446] [    841]     1   841   581721     2494   802816        0             0 httpd
[ 1662.792875] [    854]     1   854   581721     5142   802816        0             0 httpd
[ 1662.801477] [    868]     1   868   581869     2645   806912        0             0 httpd
[ 1662.809994] [    883]     1   883   581784     3656   802816        0             0 httpd
[ 1662.818684] [    893]     1   893   581721     2571   802816        0             0 httpd
[ 1662.827115] [   2150]  1002  2150  1007708   137960  1646592        0             0 mysqld
[ 1662.835780] [   2210]     0  2210   362537     2649   196608        0             0 gonit
[ 1662.844262] [   2263]     1  2263   581729     4554   802816        0             0 httpd
[ 1662.853149] [   2426]     1  2426   124749    40870   552960        0             0 php-fpm
[ 1662.861719] [   2428]     1  2428    87846    22321   389120        0             0 php-fpm
[ 1662.870608] [   2430]     1  2430   107336    41783   544768        0             0 php-fpm
[ 1662.879323] [   2437]     1  2437   112993    29550   466944        0             0 php-fpm
[ 1662.888099] [   2448]     0  2448     4153      723    73728        0             0 sshd
[ 1662.896790] [   2452]  1003  2452     5255      674    77824        0             0 systemd
[ 1662.906110] [   2453]  1003  2453     5723      570    86016        0             0 (sd-pam)
[ 1662.914768] [   2466]  1003  2466     4153      523    73728        0             0 sshd
[ 1662.923558] [   2467]  1003  2467     1748      572    53248        0             0 bash
[ 1662.931989] [   2468]  1003  2468      614      392    45056        0             0 sftp-server
[ 1662.941466] [   2485]     1  2485    87220    21809   389120        0             0 php-fpm
[ 1662.950512] [   2486]     1  2486   104245    38688   524288        0             0 php-fpm
[ 1662.959240] [   2489]     1  2489    93475    27898   434176        0             0 php-fpm
[ 1662.967925] [   2653]     0  2653    56456      503    86016        0             0 rsyslogd
[ 1662.976803] Out of memory: Kill process 2150 (mysqld) score 67 or sacrifice child
[ 1662.985163] Killed process 2150 (mysqld) total-vm:4030832kB, anon-rss:551840kB, file-rss:0kB, shmem-rss:0kB
Sep  5 12:59:54 wordpress-multisite-1-vm kernel:[ 1663.016261] oom_reaper: reaped process 2150 (mysqld), now anon-r
ss:0kB, file-rss:0kB, shmem-rss:0kB
 [ 1661.435854] mysqld invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_s
core_adj=0

 

從 ps 的結果來看,有非常多支的 php-fpm process 在執行,

每支都吃了 1xx MB 記憶體,將近 90 支就得要吃掉 9 GB 以上,

而我的機器只有 8 GB 記憶體,自然會一直遇到記憶體不足的情況:

testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm 
root       688  0.0  0.2 319160 16372 ?        Ss   13:52   0:00 php-fpm: master process (/opt/bitnami/php/etc/php-fpm.conf)
daemon     703  0.1  1.3 448156 111048 ?       S    13:52   0:06 php-fpm: pool www
daemon     704  0.1  2.1 509684 174852 ?       S    13:52   0:06 php-fpm: pool www
daemon     705  0.2  2.1 441872 176228 ?       S    13:52   0:07 php-fpm: pool www
daemon     706  0.1  1.1 428336 90596 ?        S    13:52   0:06 php-fpm: pool www
daemon     707  0.2  1.5 460424 125848 ?       S    13:52   0:06 php-fpm: pool www
daemon     708  0.2  2.1 511612 176308 ?       S    13:52   0:08 php-fpm: pool www
daemon     709  0.3  2.1 512100 174772 ?       S    13:52   0:10 php-fpm: pool www
daemon     710  0.3  1.4 460352 121488 ?       S    13:52   0:10 php-fpm: pool www
daemon     711  0.2  1.6 474680 135536 ?       S    13:52   0:08 php-fpm: pool www
daemon     712  0.3  2.0 505180 166296 ?       S    13:52   0:10 php-fpm: pool www
daemon     713  0.1  1.6 473400 135888 ?       S    13:52   0:05 php-fpm: pool www
daemon     714  0.2  1.7 484912 146460 ?       S    13:52   0:09 php-fpm: pool www
daemon     715  0.2  1.6 400656 134948 ?       S    13:52   0:07 php-fpm: pool www
daemon     716  0.2  2.0 511388 171504 ?       S    13:52   0:08 php-fpm: pool www
......


testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php-fpm  | wc -l
87

 

看了一下 /opt/bitnami/php/logs/php-fpm.log ,有點奇怪,

它說目前好像很忙,應該要增加啟動的 server 數量,

可是目前休息中的 server 卻有 79 個:

[05-Sep-2021 11:38:29] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 99 total children
[05-Sep-2021 11:38:30] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 100 total children
[05-Sep-2021 11:38:31] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 101 total children
[05-Sep-2021 11:38:32] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 79 idle, and 102 total children
[05-Sep-2021 13:29:40] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 79 idle, and 91 total children

 

上網查了一下,也有人在抱怨 php-fpm process 吃了很多記憶體的事…

打開 /opt/bitnami/php/etc/memory.conf 看一下:

; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes
pm.max_children=120
pm.start_servers=80
pm.min_spare_servers=80
pm.max_spare_servers=90
pm.max_requests=5000

 

這裡的設定中沒有 pm 設定,因此預設是 pm = dynamic ,

PHP 的 process 數量會動態增減,而且一開始就會有 80 個,最多到 120 個。

就是因為這邊的設定,導致過多的 php-fpm process 占據太多的記憶體。

(不過 php-fpm.log 說目前很忙這件事,難以理解…)

 

簡單的改法,是設定 pm = ondemand ,有需要才增加 php-fpm process。

且指定 pm.process_idle_timeout = 20s ,

如果 php-fpm process 在 20 秒內接不到新的要求,就可以下去休息了。

這樣的話,就不會有過多的 process 在待命:

pm = ondemand
pm.process_idle_timeout = 20s
pm.max_children=120
pm.start_servers=80
pm.min_spare_servers=80
pm.max_spare_servers=90
pm.max_requests=5000

 

修改好之後,要重新啟動相關服務:

sudo /opt/bitnami/ctlscript.sh restart

 

這時再看一下 php-fpm 的數量,發現只剩下 4 支了:

testuser@wordpress-multisite-1-vm:~$ ps aux | egrep php | wc -l
4

 

GCP 的監控畫面中,可以看到記憶體從開機後一路衝到快 100%,

在修改完之後,直線掉落到約 10% 上下:

 

當然這種改法,有省記憶體的好處,

自然也會有壞處,就是假設有一堆人跑來瀏覽網頁的話,

也許大家就要稍微等待 php-fpm process 被叫起來,會慢一點點。

不過以我的小網站來說,應該還可以接受,

畢竟記憶體一直爆掉,整個網站不能動還是比較嚴重…

 

參考資料:

 

(本頁面已被瀏覽過 23 次)

發佈留言

發佈留言必須填寫的電子郵件地址不會公開。

這個網站採用 Akismet 服務減少垃圾留言。進一步了解 Akismet 如何處理網站訪客的留言資料