huyuzhe 發(fā)表于 2024-3-6 08:21
終于找到問題了，我設(shè)置成7天好多了

必應(yīng)爬的最勤，還老師抓一個(gè)版塊

一牛網(wǎng) 發(fā)表于 2024-3-5 16:20
我開這個(gè)插件也試過，大量蜘蛛爬取的話數(shù)據(jù)庫(kù)不斷寫入，會(huì)導(dǎo)致網(wǎng)站504

終于找到問題了，我設(shè)置成7天好多了

huyuzhe 發(fā)表于 2024-3-5 13:59

我開這個(gè)插件也試過，大量蜘蛛爬取的話數(shù)據(jù)庫(kù)不斷寫入，會(huì)導(dǎo)致網(wǎng)站504

科站網(wǎng) 發(fā)表于 2024-3-5 15:03
蜘蛛記錄太多，導(dǎo)致縮表，所以，保留有用的近期日志就行了，不用太多

感謝指點(diǎn)

蜘蛛記錄太多，導(dǎo)致縮表，所以，保留有用的近期日志就行了，不用太多

huyuzhe 發(fā)表于 2024-3-5 14:44
9樓讓我查慢日志，看出了6樓說的插件提示了，但我是基礎(chǔ)小白，你們幫我看看是這個(gè)k_spider插件影響的嗎？
...

嚇人啊！蜘蛛記錄在數(shù)據(jù)庫(kù)占用100多M，我改成了記錄7天，以前是設(shè)置的一個(gè)月！

mysql慢日志 ：


SET timestamp=1709579063;
DELETE FROM pre_k_spider WHERE spider_time ＜ ＇1706987051＇;
# Time: 2024-03-04T19:04:38.623324Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 14274
# Query_time: 36.536060  Lock_time: 29.201423 Rows_sent: 0  Rows_examined: 506615
SET timestamp=1709579078;
DELETE FROM pre_k_spider WHERE spider_time ＜ ＇1706987041＇;
# Time: 2024-03-04T19:04:38.631825Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 14326
# Query_time: 6.842157  Lock_time: 0.000065 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709579078;
REPLACE INTO pre_common_cache SET `cachekey`=＇threadclasscount_87＇ , `cachevalue`=＇a:2:{s:6:\＂sortid\＂;a:25:{i:5;s:1:\＂0\＂;i:74;s:1:\＂0\＂;i:1;s:1:\＂0\＂;i:59;s:1:\＂0\＂;i:61;s:1:\＂0\＂;i:3;s:1:\＂0\＂;i:69;s:1:\＂0\＂;i:6;s:1:\＂0\＂;i:15;s:1:\＂0\＂;i:60;s:1:\＂0\＂;i:4;s:1:\＂0\＂;i:11;s:1:\＂0\＂;i:7;s:1:\＂0\＂;i:8;s:1:\＂0\＂;i:58;s:1:\＂0\＂;i:27;s:1:\＂0\＂;i:56;s:1:\＂0\＂;i:12;s:1:\＂0\＂;i:9;s:1:\＂0\＂;i:10;s:1:\＂0\＂;i:28;s:1:\＂0\＂;i:73;s:1:\＂0\＂;i:87;s:1:\＂0\＂;i:52;s:1:\＂0\＂;i:62;s:1:\＂0\＂;}s:6:\＂typeid\＂;a:2:{i:622;s:1:\＂0\＂;i:667;s:1:\＂0\＂;}}＇;
# Time: 2024-03-04T19:06:31.957988Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 14457
# Query_time: 49.090919  Lock_time: 45.694871 Rows_sent: 0  Rows_examined: 506803
SET timestamp=1709579191;
DELETE FROM pre_k_spider WHERE spider_time ＜ ＇1706987142＇;
# Time: 2024-03-04T19:15:38.907112Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 14930
# Query_time: 21.420862  Lock_time: 0.000053 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709579738;
DELETE FROM pre_common_process WHERE processid=＇DZ_CRON_27＇ OR expiry＜1709579717;
# Time: 2024-03-04T20:01:20.621849Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 17800
# Query_time: 4.017291  Lock_time: 0.000084 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709582480;
REPLACE INTO pre_common_cache SET `cachekey`=＇threadclasscount_130＇ , `cachevalue`=＇a:2:{s:6:\＂sortid\＂;a:5:{i:5;s:1:\＂0\＂;i:7;s:1:\＂0\＂;i:103;s:1:\＂0\＂;i:127;s:1:\＂0\＂;i:90;s:1:\＂0\＂;}s:6:\＂typeid\＂;a:2:{i:586;s:1:\＂0\＂;i:685;s:1:\＂0\＂;}}＇;
# Time: 2024-03-04T21:01:44.324211Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 18892
# Query_time: 10.667173  Lock_time: 0.000098 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709586104;
REPLACE INTO pre_common_cache SET `cachekey`=＇threadclasscount_130＇ , `cachevalue`=＇a:2:{s:6:\＂sortid\＂;a:5:{i:5;s:1:\＂0\＂;i:7;s:1:\＂0\＂;i:103;s:1:\＂0\＂;i:127;s:1:\＂0\＂;i:90;s:1:\＂0\＂;}s:6:\＂typeid\＂;a:2:{i:586;s:1:\＂0\＂;i:685;s:1:\＂0\＂;}}＇;
# Time: 2024-03-04T21:01:44.324368Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 18894
# Query_time: 3.121358  Lock_time: 0.000099 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709586104;
INSERT INTO pre_k_spider SET `spider`=＇Google＇ , `spider_time`=＇1709586100＇ , `spider_url`=＇https://www.huyuzhe.com/forum.php?mod=forumdisplay&fid=87&filter=sortid&sortid=5&searchsort=1&hyfl=3&zpfl=13&jy=3&dq_5=30&xb=1&dateline=86400＇ , `url_title`=＇分類信息-呼吁者＇ , `spider_ip`=＇115.220.11.98＇ , `agent`=＇mozilla/5.0 (linux; android 6.0.1; nexus 5x build/mmb29p) applewebkit/537.36 (khtml, like gecko) chrome/121.0.6167.139 mobile safari/537.36 (compatible; googlebot/2.1; +http://www.google.com/bot.html)＇ , `sfrom`=＇mobile＇;
# Time: 2024-03-04T21:02:26.472879Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 18907
# Query_time: 3.770580  Lock_time: 0.000078 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709586146;
REPLACE INTO pre_common_cache SET `cachekey`=＇threadclasscount_130＇ , `cachevalue`=＇a:2:{s:6:\＂sortid\＂;a:5:{i:5;s:1:\＂0\＂;i:7;s:1:\＂0\＂;i:103;s:1:\＂0\＂;i:127;s:1:\＂0\＂;i:90;s:1:\＂0\＂;}s:6:\＂typeid\＂;a:2:{i:586;s:1:\＂0\＂;i:685;s:1:\＂0\＂;}}＇;
# Time: 2024-03-05T01:01:30.184863Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 28762
# Query_time: 4.827048  Lock_time: 0.000101 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709600490;
INSERT INTO pre_k_spider SET `spider`=＇Google＇ , `spider_time`=＇1709600484＇ , `spider_url`=＇https://www.huyuzhe.com/forum.php?mod=forumdisplay&fid=87&filter=sortid&sortid=5&searchsort=1&dq_5=23&zpfl=6&hyfl=27&jy=1&xb=1&dateline=86400＇ , `url_title`=＇分類信息-呼吁者＇ , `spider_ip`=＇115.220.11.14＇ , `agent`=＇mozilla/5.0 (linux; android 6.0.1; nexus 5x build/mmb29p) applewebkit/537.36 (khtml, like gecko) chrome/121.0.6167.139 mobile safari/537.36 (compatible; googlebot/2.1; +http://www.google.com/bot.html)＇ , `sfrom`=＇mobile＇;
# Time: 2024-03-05T01:02:22.289494Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 28796
# Query_time: 3.951098  Lock_time: 0.000076 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709600542;
REPLACE INTO pre_common_cache SET `cachekey`=＇threadclasscount_130＇ , `cachevalue`=＇a:2:{s:6:\＂sortid\＂;a:5:{i:5;s:1:\＂0\＂;i:7;s:1:\＂0\＂;i:103;s:1:\＂0\＂;i:127;s:1:\＂0\＂;i:90;s:1:\＂0\＂;}s:6:\＂typeid\＂;a:2:{i:586;s:1:\＂0\＂;i:685;s:1:\＂0\＂;}}＇;
/www/server/mysql/bin/mysqld, Version: 5.7.44-log (Source distribution). started with:
Tcp port: 3306  Unix socket: /tmp/mysql.sock
Time                 Id Command    Argument
/www/server/mysql/bin/mysqld, Version: 5.7.44-log (Source distribution). started with:
Tcp port: 3306  Unix socket: /tmp/mysql.sock
Time                 Id Command    Argument
# Time: 2024-03-05T04:05:38.690401Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id:  8398
# Query_time: 3.073701  Lock_time: 0.000056 Rows_sent: 0  Rows_examined: 0
use www_huyuzhe_com;
SET timestamp=1709611538;
REPLACE INTO pre_common_cache SET `cachekey`=＇threadclasscount_135＇ , `cachevalue`=＇a:1:{s:6:\＂sortid\＂;a:1:{i:1;s:1:\＂0\＂;}}＇;
# Time: 2024-03-05T05:01:59.672035Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id:  9356
# Query_time: 9.904640  Lock_time: 0.000127 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709614919;
SELECT id from pre_common_banned WHERE expiration ＞ 1709614909 AND lowerip ＜= 0x00000000000000000000000034a790b9 AND upperip ＞= 0x00000000000000000000000034a790b9;
# Time: 2024-03-05T05:03:41.489625Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id:  9380
# Query_time: 3.085228  Lock_time: 0.000084 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709615021;
REPLACE INTO pre_common_cache SET `cachekey`=＇threadclasscount_130＇ , `cachevalue`=＇a:2:{s:6:\＂sortid\＂;a:5:{i:5;s:1:\＂0\＂;i:7;s:1:\＂0\＂;i:103;s:1:\＂0\＂;i:127;s:1:\＂0\＂;i:90;s:1:\＂0\＂;}s:6:\＂typeid\＂;a:2:{i:586;s:1:\＂0\＂;i:685;s:1:\＂0\＂;}}＇;
# Time: 2024-03-05T05:17:10.431267Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id:  9603
# Query_time: 9.124723  Lock_time: 0.000079 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709615830;
REPLACE INTO pre_common_cache SET `cachekey`=＇threadclasscount_130＇ , `cachevalue`=＇a:2:{s:6:\＂sortid\＂;a:5:{i:5;s:1:\＂0\＂;i:7;s:1:\＂0\＂;i:103;s:1:\＂0\＂;i:127;s:1:\＂0\＂;i:90;s:1:\＂0\＂;}s:6:\＂typeid\＂;a:2:{i:586;s:1:\＂0\＂;i:685;s:1:\＂0\＂;}}＇;
# Time: 2024-03-05T05:17:10.433833Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id:  9605
# Query_time: 6.599067  Lock_time: 0.000097 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709615830;
INSERT INTO pre_k_spider SET `spider`=＇Google＇ , `spider_time`=＇1709615822＇ , `spider_url`=＇https://www.huyuzhe.com/forum.php?mod=forumdisplay&fid=87&filter=sortid&sortid=5&searchsort=1&zpfl=19&jy=4&hyfl=10.4&dq_5=8&xb=1&dateline=86400＇ , `url_title`=＇分類信息-呼吁者＇ , `spider_ip`=＇115.220.11.95＇ , `agent`=＇mozilla/5.0 (linux; android 6.0.1; nexus 5x build/mmb29p) applewebkit/537.36 (khtml, like gecko) chrome/121.0.6167.139 mobile safari/537.36 (compatible; googlebot/2.1; +http://www.google.com/bot.html)＇ , `sfrom`=＇mobile＇;
# Time: 2024-03-05T06:02:06.853971Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 10399
# Query_time: 12.595231  Lock_time: 0.000124 Rows_sent: 0  Rows_examined: 511196
SET timestamp=1709618526;
DELETE FROM pre_k_spider WHERE spider_time ＜ ＇1708322514＇;
# Time: 2024-03-05T06:02:20.383669Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 10409
# Query_time: 4.436246  Lock_time: 0.000096 Rows_sent: 0  Rows_examined: 227326
SET timestamp=1709618540;
DELETE FROM pre_k_spider WHERE spider_time ＜ ＇1708322535＇;
# Time: 2024-03-05T06:02:20.383696Z
# User@Host: www_huyuzhe_com[www_huyuzhe_com] @ localhost [127.0.0.1]  Id: 10410
# Query_time: 4.237844  Lock_time: 0.000082 Rows_sent: 0  Rows_examined: 0
SET timestamp=1709618540;
REPLACE INTO pre_common_cache SET `cachekey`=＇threadclasscount_130＇ , `cachevalue`=＇a:2:{s:6:\＂sortid\＂;a:5:{i:5;s:1:\＂0\＂;i:7;s:1:\＂0\＂;i:103;s:1:\＂0\＂;i:127;s:1:\＂0\＂;i:90;s:1:\＂0\＂;}s:6:\＂typeid\＂;a:2:{i:586;s:1:\＂0\＂;i:685;s:1:\＂0\＂;}}＇;

看mysql慢日志