搜索引擎粗排名计算：BM25
2022/05/25 15:25:52
BM25绠楁硶浠嬬粛BM25(BM=best matching)鏄疶DIDF鐨勪紭鍖栫増鏈紝棣栧厛鎴戜滑鏉ョ湅鐪婽FIDF鏄庝箞璁＄畻鐨
TFIDFT F 鈭 I D F = T F 鈭 I D F = 鏌 鍗 璇 鏁 閲 鍗 璇 鎬 鏁 鈭 l o g ( 鎬 鏂 妗 鍖 鍚 鏌 鍗 璇 鐨 鏂 妗 鏁 + 1 ) TF-IDF=TF*IDF=\frac{鏌愬崟璇嶆暟閲弣{鍗曡瘝鎬绘暟}*log(\frac{鎬绘枃妗{鍖呭惈鏌愬崟璇嶇殑鏂囨。鏁+1})TF鈭扞DF=TF鈭桰DF=鍗曡瘝鎬绘暟鏌愬崟璇嶆暟閲忊嬧垪log(鍖呭惈鏌愬崟璇嶇殑鏂囨。鏁+1鎬绘枃妗ｂ)
鍏朵腑tf绉颁负璇嶉锛宨df涓洪嗘枃妗ｉ鐜囥
BM25B M 25 ( i ) = 璇 i 鐨 鏁 閲 鎬 璇 鏁 鈭 ( k + 1 ) C C + k ( 1 鈭 b + b 鈭 d 鈭 鈭 a v d l 鈭 ) 鈭 l o g ( 鎬 鏂 妗 鏁 鍖 鍚 i 鐨 鏂 妗 鏁 ) BM25(i)=\frac{璇峣鐨勬暟閲弣{鎬昏瘝鏁皚*\frac{(k+1)C}{C+k(1-b+b\frac{|d|}{|avdl|})}*log(\frac{鎬绘枃妗ｆ暟}{鍖呭惈i鐨勬枃妗ｆ暟})BM25(i)=鎬昏瘝鏁拌瘝i鐨勬暟閲忊嬧垪C+k(1鈭抌+b鈭vdl鈭ｂ垼d鈭ｂ)(k+1)C鈥嬧垪log(鍖呭惈i鐨勬枃妗ｆ暟鎬绘枃妗ｆ暟鈥)
C = t f = 璇 i 鐨 鏁 閲 鎬 璇 鏁 , k > 0 , b 鈭 [ 0 , 1 ] C=tf=\frac{璇峣鐨勬暟閲弣{鎬昏瘝鏁皚,k>0, b \in [0,1]C=tf=鎬昏瘝鏁拌瘝i鐨勬暟閲忊,k>0,b鈭圼0,1], d dd涓烘枃妗 ii鐨勯暱搴︼紝a v d l avdlavdl鏄枃妗ｇ殑骞冲潎闀垮害
BM25鍜宼fidf鐨勮绠楃粨鏋滃緢鐩镐技锛屽敮涓鐨勫尯鍒湪浜庝腑澶氫簡涓椤癸紝杩欎竴椤规槸鐢ㄦ潵瀵箃f鐨勭粨鏋滆繘琛岀殑涓绉嶅彉鎹
鎶1 鈭 b + b d a v d l 1-b+b\frac{d}{avdl}1鈭抌+bavdld鈥嬩腑鐨刡 bb鐪嬫垚0,閭ｄ箞姝ゆ椂椤圭殑缁撴灉涓( k + 1 ) t f k + t f \frac{(k+1)tf}{k+tf}k+tf(k+1)tf鈥,閫氳繃璁剧疆涓涓猭 kk锛屽氨鑳藉淇濊瘉鍏舵渶澶у间负1锛岃揪鍒伴檺鍒秚 f tftf杩囧ぇ鐨勭洰鐨勩
鍗筹細
k kk涓嶅彉鐨勬儏鍐典笅锛屼笂寮忛殢鐫tf鐨勫澶ц屽澶э紝涓婇檺涓簁 + 1 k+1k+1,浣嗘槸澧炲姞鐨勭▼搴︿細鍙樺皬锛屽涓嬪浘鎵绀恒
鍦ㄤ竴涓彞瀛愪腑锛屾煇涓瘝閲嶈绋嬪害搴旇鏄殢鐫璇嶈鐨勬暟閲忛愭笎琛板噺鐨勶紝鎵浠ヤ腑闂撮」瀵硅瘝棰戣繘琛屼簡鎯╃綒锛岄殢鐫娆℃暟鐨勫鍔狅紝褰卞搷绋嬪害鐨勫鍔犱細瓒婃潵瓒婂皬銆傞氳繃璁剧疆k kk鍊硷紝鑳藉淇濊瘉鍏舵渶澶у间负k + 1 k+1k+1锛宬 kk寰寰鍙栧1.2銆
鍏跺彉鍖栧涓嬪浘锛堟棤璁簁涓哄灏戯紝涓棿椤圭殑鍙樺寲绋嬪害浼氶殢鐫娆℃暟鐨勫鍔狅紝瓒婃潵瓒婂皬锛夛細
1 鈭 b + b d a v d l 1-b+b\frac{d}{avdl}1鈭抌+bavdld鈥嬬殑浣滅敤鏄敤鏉ュ鏂囨湰鐨勯暱搴﹁繘琛屽綊涓鍖栥
渚嬪鍦ㄨ冭檻鏁翠釜鍙ュ瓙鐨則didf鐨勬椂鍊欙紝濡傛灉鍙ュ瓙鐨勯暱搴﹀お鐭紝閭ｄ箞璁＄畻鐨勬荤殑tdidf鐨勫兼槸瑕佹瘮闀垮彞瀛愮殑tdidf鐨勫艰浣庣殑銆傛墍浠ュ彲浠ヨ冭檻瀵瑰彞瀛愮殑闀垮害杩涜褰掍竴鍖栧鐞嗐
鍙互鐪嬪埌锛屽綋鍙ュ瓙鐨勯暱搴﹁秺鐭紝1 鈭 b + b d a v d l 1-b+b\frac{d}{avdl}1鈭抌+bavdld鈥嬬殑鍊兼槸瓒婂皬锛屼綔涓哄垎姣嶇殑浣嶇疆锛屼細璁╂暣涓浜岄」瓒婂ぇ锛屼粠鑰岃揪鍒版彁楂樼煭鏂囨湰鍙ュ瓙鐨凚M25鐨勫肩殑鏁堟灉銆傚綋b鐨勫间负0锛屽彲浠ョ鐢ㄥ綊涓鍖栵紝b bb寰寰鍙栧0.75
TFIDFT F 鈭 I D F = T F 鈭 I D F = 鏌 鍗 璇 鏁 閲 鍗 璇 鎬 鏁 鈭 l o g ( 鎬 鏂 妗 鍖 鍚 鏌 鍗 璇 鐨 鏂 妗 鏁 + 1 ) TF-IDF=TF*IDF=\frac{鏌愬崟璇嶆暟閲弣{鍗曡瘝鎬绘暟}*log(\frac{鎬绘枃妗{鍖呭惈鏌愬崟璇嶇殑鏂囨。鏁+1})TF鈭扞DF=TF鈭桰DF=鍗曡瘝鎬绘暟鏌愬崟璇嶆暟閲忊嬧垪log(鍖呭惈鏌愬崟璇嶇殑鏂囨。鏁+1鎬绘枃妗ｂ)
鍏朵腑tf绉颁负璇嶉锛宨df涓洪嗘枃妗ｉ鐜囥
BM25B M 25 ( i ) = 璇 i 鐨 鏁 閲 鎬 璇 鏁 鈭 ( k + 1 ) C C + k ( 1 鈭 b + b 鈭 d 鈭 鈭 a v d l 鈭 ) 鈭 l o g ( 鎬 鏂 妗 鏁 鍖 鍚 i 鐨 鏂 妗 鏁 ) BM25(i)=\frac{璇峣鐨勬暟閲弣{鎬昏瘝鏁皚*\frac{(k+1)C}{C+k(1-b+b\frac{|d|}{|avdl|})}*log(\frac{鎬绘枃妗ｆ暟}{鍖呭惈i鐨勬枃妗ｆ暟})BM25(i)=鎬昏瘝鏁拌瘝i鐨勬暟閲忊嬧垪C+k(1鈭抌+b鈭vdl鈭ｂ垼d鈭ｂ)(k+1)C鈥嬧垪log(鍖呭惈i鐨勬枃妗ｆ暟鎬绘枃妗ｆ暟鈥)
C = t f = 璇 i 鐨 鏁 閲 鎬 璇 鏁 , k > 0 , b 鈭 [ 0 , 1 ] C=tf=\frac{璇峣鐨勬暟閲弣{鎬昏瘝鏁皚,k>0, b \in [0,1]C=tf=鎬昏瘝鏁拌瘝i鐨勬暟閲忊,k>0,b鈭圼0,1], d dd涓烘枃妗 ii鐨勯暱搴︼紝a v d l avdlavdl鏄枃妗ｇ殑骞冲潎闀垮害
BM25鍜宼fidf鐨勮绠楃粨鏋滃緢鐩镐技锛屽敮涓鐨勫尯鍒湪浜庝腑澶氫簡涓椤癸紝杩欎竴椤规槸鐢ㄦ潵瀵箃f鐨勭粨鏋滆繘琛岀殑涓绉嶅彉鎹
鎶1 鈭 b + b d a v d l 1-b+b\frac{d}{avdl}1鈭抌+bavdld鈥嬩腑鐨刡 bb鐪嬫垚0,閭ｄ箞姝ゆ椂椤圭殑缁撴灉涓( k + 1 ) t f k + t f \frac{(k+1)tf}{k+tf}k+tf(k+1)tf鈥,閫氳繃璁剧疆涓涓猭 kk锛屽氨鑳藉淇濊瘉鍏舵渶澶у间负1锛岃揪鍒伴檺鍒秚 f tftf杩囧ぇ鐨勭洰鐨勩
鍗筹細
k kk涓嶅彉鐨勬儏鍐典笅锛屼笂寮忛殢鐫tf鐨勫澶ц屽澶э紝涓婇檺涓簁 + 1 k+1k+1,浣嗘槸澧炲姞鐨勭▼搴︿細鍙樺皬锛屽涓嬪浘鎵绀恒
鍦ㄤ竴涓彞瀛愪腑锛屾煇涓瘝閲嶈绋嬪害搴旇鏄殢鐫璇嶈鐨勬暟閲忛愭笎琛板噺鐨勶紝鎵浠ヤ腑闂撮」瀵硅瘝棰戣繘琛屼簡鎯╃綒锛岄殢鐫娆℃暟鐨勫鍔狅紝褰卞搷绋嬪害鐨勫鍔犱細瓒婃潵瓒婂皬銆傞氳繃璁剧疆k kk鍊硷紝鑳藉淇濊瘉鍏舵渶澶у间负k + 1 k+1k+1锛宬 kk寰寰鍙栧1.2銆
鍏跺彉鍖栧涓嬪浘锛堟棤璁簁涓哄灏戯紝涓棿椤圭殑鍙樺寲绋嬪害浼氶殢鐫娆℃暟鐨勫鍔狅紝瓒婃潵瓒婂皬锛夛細
1 鈭 b + b d a v d l 1-b+b\frac{d}{avdl}1鈭抌+bavdld鈥嬬殑浣滅敤鏄敤鏉ュ鏂囨湰鐨勯暱搴﹁繘琛屽綊涓鍖栥
渚嬪鍦ㄨ冭檻鏁翠釜鍙ュ瓙鐨則didf鐨勬椂鍊欙紝濡傛灉鍙ュ瓙鐨勯暱搴﹀お鐭紝閭ｄ箞璁＄畻鐨勬荤殑tdidf鐨勫兼槸瑕佹瘮闀垮彞瀛愮殑tdidf鐨勫艰浣庣殑銆傛墍浠ュ彲浠ヨ冭檻瀵瑰彞瀛愮殑闀垮害杩涜褰掍竴鍖栧鐞嗐
鍙互鐪嬪埌锛屽綋鍙ュ瓙鐨勯暱搴﹁秺鐭紝1 鈭 b + b d a v d l 1-b+b\frac{d}{avdl}1鈭抌+bavdld鈥嬬殑鍊兼槸瓒婂皬锛屼綔涓哄垎姣嶇殑浣嶇疆锛屼細璁╂暣涓浜岄」瓒婂ぇ锛屼粠鑰岃揪鍒版彁楂樼煭鏂囨湰鍙ュ瓙鐨凚M25鐨勫肩殑鏁堟灉銆傚綋b鐨勫间负0锛屽彲浠ョ鐢ㄥ綊涓鍖栵紝b bb寰寰鍙栧0.75