From 796a694ca52a623a8d5329b3663ad187231c874c Mon Sep 17 00:00:00 2001 From: John Bowdre Date: Sat, 13 Apr 2024 21:22:14 -0500 Subject: [PATCH] update post --- .../cloudflare-waf-status.png | Bin 0 -> 22518 bytes content/posts/blocking-ai-crawlers/index.md | 16 ++++++++++++---- 2 files changed, 12 insertions(+), 4 deletions(-) create mode 100644 content/posts/blocking-ai-crawlers/cloudflare-waf-status.png diff --git a/content/posts/blocking-ai-crawlers/cloudflare-waf-status.png b/content/posts/blocking-ai-crawlers/cloudflare-waf-status.png new file mode 100644 index 0000000000000000000000000000000000000000..76d4615221f07f237765e6aa2a1f571c59509ec5 GIT binary patch literal 22518 zcmcG$cQ~Be{yv-ti4f6=ZbV5KM0BDJLDWR=z4sQK5m6$dMf6S(ozV$`=)L#ed++VH z?7h#~=X=in@4eoAxps!{kUX!7x=%t_%pF~EFCQ#FEaZ4W z5kV}{I}B8MLdXxY-{zH?MeqzN<>Zl-yf?`!rT;NB?7EpHpB0fhkK9mo67RM%kZA8Z zbNsERVs+h-j#B)W;I)hUctS$yh}K3_=;med@i?Y!h0q-&EU!Drly~l;3jY1lL47vG zYQziezyAE!9b7(8XYBvD`|p^a^c4`}70ppC&+QP;dVCbp9S7s*2DZNFYAeJCy&P^6fo>t==R5 zV~g7hh2D2y1Pqj^X#Z)Xe?8SJ6s6#wMoy`Na@Pl{Bq)Oa4;B`@13}li_fHpop#e+f zdVQk*N8`f?e%}A53tsrhSO}H7*oyxe)&H>>STKx#yzu|af;r5wQ--%qZckPfVtj`E zYZ*S&QqFE}Zmz6k4y@_v={fKuBqb#U21;t1RMeEbqsHmG_bx`t=)z-LSi00hsssz^ zgpo3;V`_48+>Lc-x%#sLeL8#%Ct$HweUzEABrTMNmey#wGlqtSW^HY4tEQXBGq~Jh z47NS_UCy+HhGH%yog#-mjzj;H%=j!0#P$H8V7#vK_ zZl)k1Apzc4Tx|bf;yA_s(3MYd-81BKU|>#8j&WzSZO`_D`}eoUOU+w?$uvFWI?a{n zBMt=}nm*$%haCSVj**H?Ow>|Si-y3r9w2d&W5K9*t=4$n7}rXbvaDRaf&)*^K0Cme422 zjHGA3eS)Mg8^~On%$zbmuUFe9M&ThcGGU?8X3ZE00bB6!hoFPQ;fOX|dIoPq#clZlUxh`=W#bn3B0L8&@>FaCCdFD@)T-W^!vI>&sb)&uMnZ&$A1 zz>pAYD=S(bHV%${F>FLMgDwX0IIc>KEBg&*P~EHd>x0K->x-^F&;%cZRnavC;`HKT zWN4_5(A?5;D*%_yYO3n>4*$udy=H&OUg&G8F!z%wrxJ4v3=B=;ykebtq$*u!XD;@( z&dyg(+mpvz6MNDXlFT@GMo3TtQxlW7^br<(FapsJD5!$Jx=|c>$gc8bP`TZXH}@7= zNJvOf{GT;7H92%0ZH%03O(@ID5A^rL>bi)7M1vPpTKju@ZT9Ays;YR@ys+%R8oQA{ zLPkMJ&&XJJ;Nht19U4+hd0Qy-i7M@n}2@p zS~i3*?C4Gss59xKD$A(xooMu~z1;8U5x9v*Q9M-j+^IPkpZH#LG>oCjW96q+Q0H<$ z3t0sFme*+;*a&ZEqDbFA*+nkvaIr1?@SP77iFDFe&=^^zRZd$QzPGLJ>gsB2wwnA1 z7Wvr5bvp&v@#o33+o4C&6%!sYvFr7v%X!h<&JOc0`~VSeq3?E;@z-SoTCFfyBizBp zx{b}v&S1^cjlAkto|n#!hk~f4JIcgGDx2HeOeP^Nt^Gn1lT%YodbSTxM5y}sLe_ak zNimNw?-C%eVAHM}jKf*SFoFOPyPl|^xHyuVbOwUJB|GjM)xz2NdGAkeb-b&wyu73z z_Ug}MWMq8#@@V`;shY=J0}4!BSoj_v@!3JrjnodJU=^2EDw*QsFfYOvy2&M$^0qhj zA!Hy!dLy}miHYg?^XINLiQ2TUUcCaAwi*5^lPUS;jqUsQOHrHZ9UW23#{O@>cJGw$ zMd!Jjq?7K_O7*Nay_zhOgiK*bsC9mQeZ0OXrYU43IBu_*#2O2xrlz*43IjGh-COVp z&B&mw|CISUkbq_MkID4cuauq@IXysiMn$@k_40+@|2UUe6E2bNT_;o)(4=qb*-Awa z2!g4KOYj1ZvvL@?!rWvb93TP{cx+RJ#fl9*JP3WA@)qn1Y#9H=CaY&padGk5ijb-* zQHw3ub)`;LL_|amqk&0D6fL&8y1EY@lbV>CR=(dDcH5~bw_lN|!oOH1AR?+XA7MIw zuBfaGqVdb)Q~~(4J8%wk=AXOU)jMS+C5H{qv#jf`I3SCYmG;0j3r`t>WHK*y8!=7I zNr;I#EJkPN=IU)HZ?`-#vB?ON)=w3A`9|AGN#+_6>Cp!Vmf_NI9DQFottOpsu10;uwz5T(5dk=W{gW@ee0e`)NRP!7Q3ro;HeplZlc!7XfOG+Rt zJUpB`IBmr|p3g0kg2j5YQ1@WD3m+ff)%-f)b1j597f2&sU$G+ZH?T=?U@5%gF_k^GPIq z0|qV2&6R=@`1|+_4iES1XD%-#U7mZ|O z?*dhjXn10xOccmp_V)Istf&xR*&lg%T^${1VqY#lFoJ~!zw!ac=AHd6&3?nPxVBd5 zdSvFhIX*h7qN&++BdCz2U|oB*ywnk8Ds#IP&hfi{5{1)B$o0Yq7O1`GoJm?qHVuW#RPHlt)J-<00c7$Q`*$(c!Om?Y34R=oc>je$R6c zwzszr?e%&R`AA!e9wk2hLhW-3VxfympLV#(n``)*v6h1a8_7}!xnWY+;nR4<(E&)H zzyE`&k!;1BMEJGK`NF*$<=ENTS;;7-BFBE;^>wo%6)4x-F_=DSOP+RJVq}W86DScz zb?@eUudUU}8L}g7>HcYWm2L?3-1^p5*JzQBG7_6XCCFl3yrcE0lm>0>?Rvs9s-t2z zFTh5kcyyQ-U$ct}fuD6~pE_-9Y^X9F469r3QKYuGv#_u@`i69adaI22h0DIdxsjrZ zN~?7qEag+BeHumbPVEH~FHkVIF-$_WIT~+$l)z%m~Ih}m_ z_Kp9%C^0@>iU<@n0%wc3u-3JK%rm~O_V(17XQxS&Ab5?eIb5>)gn}isCl!-TI=ejR z;SL6w_&BgN4-(&$4B9dakaUljN0*(kR?J9?W$6pc7n_UG&9bn_!B~Vl$ZTlJ-DSG3L zJX5^7Gf&#`epw~<3ZvgQGLj&6A z+7?cS?ka=p&s=x;l>z~RhD)O#27^+rjlF%^I~0fk_zePqfM~CruaQ#P7wRjM;dX(y z`t2KDgaimi1^r@c#w2lPOTRwc1COv%Qi@EzJ6xmw_O0LvIR!;QpULXW8)f42xsky^ zHp6!Crc$FWVl`Su#;&HObg-wV$@ty5$f~GNRXMCC#nHb-MrIGKs}>veJ?@{e`6HnU zfcmdiR7NKFz1!#U^UrrMRP!||8-kArnKW8$<0%bN9(*b-Ek*N%K?&*(x_J*HL>F9K z4@FU9IIR=a@)B8%AA20G4*`$LGAoS@Hm3FI<)#F2#pFYtdU>`&HV8m{Gbj9Ki{Z8N zevV*wIa2+iRbM+r70Oc@uysG|+FI>PvtRCB`CzvFUCdJ2p9VawE#9kS814rLYu2vS~0i__gXo`n3Ws@QAxKF)O+a>=NOh%ANd$w>`j z4<%*gII;5S>FJRIZ8{(Ky(U7?fbb-2gGlBR%AlNIQBm;)Z2s+&-ML0=Z0!3VKP0{u zM-^(Na*jAY2sOOEJUdK(VLLA()U&2sP;A_@FC6LPgREq6bA5FyWpSEz!?Gw6lFcpMN(H%+BycqE)w ziy^2#a&q*GA&5}lT9^GBbuF!Ftpw5*qDS`+-g%<=QbD0mcJ?8?RveF&VWEVwvry2j z@S%-Yz2Ddhp>||C1MP(A$1$@MAw`Wj#1f zUIz7>qrnqwY@rEOkiCwBYj{oWk$VdY5a>YAZ!qGqV64&vh(ZR8B3z&mxDL&!xd8MilRBw7DAgwMxq>#Bb2ptCvJPVe6}Z#mLcfoIDGo?mcA$# z*t7mn5KWfnc^x;ge4sc1-@kuf1a0Ch9Ig{kL2017M|07kC(zZ`195ff279WgPQnrG zt3*12!u=2r;4K~>Zeub_dw0-IMHfFmKPW&qjT^y@eT|dAVD+lv%Ml@@+%M_r+uduV zXw8`o0zQ3u!sl9YM$1v_M}~oTWqz_fY=%#sAM2zr!~Y&SnBX^Y2h{eyCUgW;he7iZ^Qu8)Sem&d5Jk8MdaM>b|$R>;m9 zAzMh0Mc}TaE!@1k2ilA+Ar#sOgMg3_%$&k-NMOBAvT$Wj0&h#55+xaF2`Ge#_j+fh ziAhM(<(}72AzNZ@e41?y4KkRca4Rm;y^G{X10o2>t|5z4CZm?*h{6*#9{%Keb=f6QWxGl(SUjXlCNYg|jOw@I$J*1AFs*ZR{-Sfr(;gPg23FX*o# zMAPN^kSr!KHt;l8xeRpgX`c~czz9A-w$YGJA?Pi(OG`_KuUnVzr`Guu?KOfX+ZDtv zP%{e!_Hk|;!y=ugUHrFYun_J?L&^@d6RyP|zlA3IR-gk|Z1yf%p*`PnS6qv=sECM2 zTSA;nCWtRDCo7-hp(*)cd^?@}hYU@i7@}OTzk%3R*PXM2ckrG`|sB_+?6dg=`L#Gbamy~#uT`u3={uBEMPzc5`W%Y6QMRP=ybxX7>SrKvp^9!gA*X ztqh-FQ%n#Gahq(#?s$}nr^|tW51w~J@X|eTYNr;05WrA{Wkb!QvVPQ#ZSEJcGziv)r&(h?_u*eC* z!ercO_1s<#WyH)7@&ob1?OKluu3$-V@nom5Un>BSO6BErt!x8Mx5<_iPAA)+4Uo2H zsOWF|`}-n3O4c2Zn;+&i zHZ}_H=D)O_es%sFMd~>wCDc z1yYzp^pB=cUrQA&EhbRjx$-681z}M|NYLOs4C2TK6+S={!^A<$58kisw#80AVGgoN zUNPvoihhzvQkjN|h&5{gG>L;l2})2b@Q{ba^39uvitV7ZHpPR>fKM*7o{=%u0RYiA ziS)L%w!z)m`j3}6=nwECN2vwy%XuXC2QffxwupAeexd-Zxf{_hH`k~vbJhI7Z+zrw^ zsJMAJ=0IHq;8f60fgQH_)a~MU>yKVF;LDfq+L>o*wq4JE*3`($%a5gxseAyDSXmiV zt_s@P+DZsnXrVRA!Aj2x8BI`nT{=4_C+K3q(DMCf0hSZhv6j0Inme}d$(4$rtrisY zI{RHd_`%~2Hmvr1oxx9^9MQXS2#U$>?rw|ExYy}>$O{FkieU8%e(s}rT=i<#fw!#< zvN}+sZ^1wcj*22|u?5&rqCf|enn4kfg*HI(4q={$u@B9rLH8<2AF;0#O|WKbq_3Z* zmUq0n`%9dFHUI^6QXT*#+}zxU>vo%UXuhcn1HL>iPymid(Wx5K-AUxKZE(-X?CpK) zxMURj{f9#K$qYQnGr?enM*!3}z)rrf`uw*a!l2&9l{bI*n+!x$z<(4D@QPJ;nys$3 zR0aCAXHSTk3NcJu8*DW|gGFK`ff=ysf_9jeKLU&7oN*5?3zG5poe26xovOrZ3H=vO zOrQ-r-Jo+?Eke4$@PC}Kbn{gdvkriOaTDH}3x!2RwxGB~RgLB*I7I@uwcxH-sv^imM@3`hs8bo`tZC>%y3WqIZn)uTe5+?!V8vi3i8L8 zI)Qox!0mX|4cfmw-_(!7Po@|yDk>tl-TtEy=W~r!pUF z-s}g1A=i{3d2p=xe&7n$pGiZ7^!4=-u%JLXMvL@KX@Y`+X5cq`ZZ5rRI*;z(BlFV$ z2$FG6BB*owkv3Cb6U5aYcX;OL4hSBr@_;?sWZwG`0K^9~=jG+G3v8!Ynp;_A$a_vF&Ib#ijQdk~io=sqf>f&= z8L#9)lOOBS3S-7_1`Bw+a(K3qRIIX+Sf{L#Q}-hi^P$Qk)o&6sk{hct?{su1CF|j* za}PL-585Lb#*T+-h>D9B$kw@333n3`=Eugh<+8S@tTLrb$)?{V42PUV0=BI^x=i>+aYINgVJ7V7D8F|o1jNyHg7JeFpB7Z_`K<;VioM=4LwIM&VCxiCP}ackU=#h~&kZa0z7ZIuPW3vZc^hk|9_6xl%_URF7Q z)`QNT`~CX2BrO>mI)EDF@qA)nnhJ zKz!J&xSFu8GmMHC18o(}=<8P41t?E1FE2~f@@6w1efTt+FHVj2@Zs8N6*XvH0ER1| zTb+y(SCykc-`LOq^;|hTJRCGD-3)EeE5=8AO|hQa#H5psf%Xg!EkUMbzGJ%_p!jBN zv2L}A)pbt?Z)v#X3#6AZQaM8ah6myD*bFpU0J6_hQs2w_x=^vZySun(;8<@2`h}&b z)g4ZW^p_wwQBkGVlpVz#0kj_c$Hq1|Ula-mGqpUUdU+>T#xQYjZ|~C;tvBg%JWaVe z*9mUhzMj*2IXD)e5+WlYFk+!(wf56dpY*+heEwO`I5-^)Rz)QmkWa=u>2g^h;X=FZ zR{vn*Mt`O!WOYrqkG^ZwvzBE%zb)JTWbJqeLdB`?e}sUQ;Ncm6RSRI_ckkW-s6aFg zyJdh(a{46AuJRQRPizqBn!P;wj8jVy1Q4qquIar()9d()R|)@3v#x&glFK6Hv;x7f z{l~ksFQRGxNs9quM^I2O(@ZGk0%uk`tL3sJ<02b}8-ahLzb}#lFKF;I-SI*DRL33_BSyA-ypy^ybs*mvY z{a5ODAd)wpo_)Oke%jwusp}hXky%Ke^B;)WLu#`Lm=4NUzenP`1pVM z?hjfx;G$&jD8|3d{WsD6|7F4*kU%8Pf3Oo+Vi|NBQe$p|wV8Sx*9ZSQ-aE*-On{$X z|NaB(f5&^*4XmhMNDt5XqyK%U*AOt5N&+7KznSB|@6ibX-1z36s#wx3x%gkUF02CO zUQT!_?7uGnTWr1qv_BvJ#wzPSSV)NwjOM+Z0_y)f(#wkv4JaB~jLAIw{&($ut=}^B zGM~}@X)t17LmZ#xruF>8R#MRfqZxYh?$%fS`!kQ90ke@Oy+0)SZ@XD&2yU;CN1H#= zj>9j9J2s4^_q=FH~$RG;cit-e@}jl5ggUVLi*HIO^(6RvH%)aLh4ZnG)BpndYlG^N_Fc<_qR zLYfK14SoL}$(8EKVn~P2LtHb@I*^3TOVx%EpzOH17ViRt%$O_mzdsNx`di8uBhd=# zDdVBzyp4*hTBJc*@zo{zmUNY)EDepi63J%5W^$IVb@GcA_Kz=3di$#-_G=}#Td#$zr37V`}HOB9xLgTciyyz z+QGmFA{jgD9OdBuv#okP?sj<>u_SfR@0WrP=zBp#V_|+EW@3*+8ZnXgE2raD4MVB>2KV!`7km;_?{qMnd@(k4fVYFH5j}qJFA;Jv)J3LI1k?pGBtF)?)ypq4i$2X1%j439-y`_nVgeBHzaX~VOI%zRO&eLuPcSP5SMl|FE4SGVmHV`-}-)+Y+wQ~*Ts`?o{gc>b`CHY5<_c)%7b2U*nptk-FspQ4qD#=^yY8a~DT@tuYSDZ~J< zDZ}eBV2eY(=jL{*;io2g(gGk#%r4f?3?2L=qQxCJUvAvXxDglraCZRifWteWHDTBK zvQxeXCz2Pv4#OTD`<1FDhAO!)eDiinv?T@oWCv3!II4a7&1CLi1 znlqirJSO4E_$%n+F?zZn-|(58o|ZSyom!QeD433AdMC9phNtMcxK4pu#E`o~R^V!J zbMp9oV-@du1&7;1)%p4P;R0>=QhNkQ?7^?k=hHSu3h0c=!EfH)NW)}4&@V{nHBZ4j zgu>TP-AJ=z>R+NiNOC@Z`vB7M!fEU6(f5>bZ{y4G1nR9s&DG==k1;$Qe>i4*w0lHX zyMg0i;RNq&qx#C_Otf03yPa~m9K$TY@-sWn?^1iUA-oyiTkm$h6h*$;jSFcrDeRch zdzja*Rh!9A_}RFeOvDq{+^PB~1bDgH@E&XViG}m0Pdp~q-kLRhvg|t2S^xNRbPB{p z))X!An8MnYgXmxiXiIMGbBOujS0^VDtM(3shY&MmZwLxl?uY{FBem9-(608(9r_d1Qq^JWtkgH)t4_{$|Uf-Mg?5F5}Wh}oIwG_D1flJzHJ@iP-c;}+0q_*I&Mg=mt)h}j_7c0vH*#Wsv+&=k2LqeR3Etfjsd}M=jr=nq}VVk0ArvcH* zs|~-hM!2>^On-6*Zy2>UEW2a-KxH3%e;ljPGk#+uwDIe?a^mA?!Rlj$QPOK+hqJzK zX@~D{JH7{ABHi2fbnUPBN-^+)>dzQ1c!T@NJxrul{9L=P_bKWj-QCJ0w}!QSU2d}+ z@Ko8uYGXbbSIiA0o-7T&r60I5Q5{znhU5=1F+=7W{SOMN^rx6+ipy?QCFkooKwNzO-Y8ny1PR`i&ZUJm#sK5qAPUM|SZfhcy#cH zBJ|{>OnM(q6nZ0IanQ0bGh0sjN4HMd%Z##E>#DDqQfP%Qe<_Ksqf0w6!RK+g)~89?CCki^;QE)!`wkYNcpgYI~4ZwZ?beiW6~mpbF{ zSJ=hN0nzLsEtbw~OV8rqi_bN&!&xv@N1sIA?YXuTS6O>|v?vyST=adGsLs}5LMUcs ztzAf-DTc|PG@kWEahYY;@8Xh93iUHOOhrS)BFv`yvH2sou1$kX)!D++=|!&Kgpq^M z=&j3J*Ss(NN=!QONl^^@Y1mH+GCWIpd+zpEs>!U+4km!-3A3}_H}nuAVbwW@bGPBi zbW1##>H8Rt*&h*-LHd4|Y-<+V%53FF`C5DZLXPyCNb$?>bFZtG=PtG54V2>U#D#eM zS<@7f-QB^qf5hPVGWXn{%CT&#B-=?XHSs)OyFGh)jPMxggB6-jK^LOE5*8xHD{yFi zbKEP8$wvU$cxLeCWXI?uqZL22Px(wj?sVL-pwQGd$MYw`SePJPt`e9K@Z$ioBVsk$ zC0m%E=LbzY_)Cr3NpsUwkFVre>3h4ed{`uicd%9McAu*llVxxwFUTZ<0MR_&G_5Hx+uJ>=U3)co;Lkj!3(5#EOmqz`Y+YQ z%5@?axZ0(NHg*iv{C${7Lr>(@F&1*T!xm3s8E|k*2BRaV`|dDvo|3w{P$77 z==(#EAvzPe{)i8&1xP|Y+x-M&m=s{SWgTYvR+m6=Rz9OdH3hb8o<^k|0KfS9#<~Z`iXoCfF^I84 z@Z?GB_$_*V+!{<)bGSWI2ZR!Iw-^-z1499@<8mhvfIH2YP*oM^6@`Txn*s@ig(r-z z1~-lXJlF{MvU^BKT*2XO2AQ%+wJrxnfC|rpfV>8{ae&+(BDyX^X&4#vfP|$ph81&> zXL0kjFG?H^V9&i`>K=7gzPTytoCmcRot{$heRNo4W=XM$=&raGD}((CatzM(Ii#wr z`W~J3JRu43t01lFmN#ZR?BA^PV*6=sjH_O~D2`0X`aScu@bX*!4@;*3@d*W5sL$!Pn%4cygMaGonE<9@X;=3Va3lxjkGM_;>g-&$sy?Z+h^YUo7+jW@!#Vf$&4ArN>nTaA*xXGjL-g-=!f=r-!n->{ zCF#;^F)gc4KU$XI=6T=27thXH?aVC?G5dXUTz4}P+L!44#5YLSg-wLo8I*>)%>qXK zh4b>Wc=Km-6o!$f@~mpjw}Ra#`W=pHKqZUAn1%h@e7p#y@OYIIcMYePZw6V z)gVq+Q>x}`GK7MXQpQxJbUc?WuqTj5n_E~&$;f0ZdxtI!$O_c(=?AKLjD$=8Mi>iW zJ;M(eb~IL?P&)o*OG*d0VY%lCKLF`mP-oYH5pc)rpL%On>tyi5)i0SHZIalV`Y&pyM1(Y=zN8X=HnQoy$ZS~eR}Wfc|Y3&{(>WPTBSTBi&sQ6NcsfJqKw6@cv)1qSUq zu6PZwH8cTzH}%uy0Ej5ra6z<&Umqj^Jsr*0>gDgD66rpn1+_1)V6~20YOq$IcbjnI zF==h^>nW^M8)pbPI;QOvYHi4`xIFz){ydPI`%G8(aIgzqw9-^MaoM=>xOkPjg}M>hR7RwOx|x%U%lg{m8`2t1l#Fm1hm% zeI08AQ;$^R4bYU%dPCPpTlhHE$*Pw7Y%g4b=I_&az(xc(GPa-{j>3^Moc3Q`_{ic9 zG9PM}Zc*J3(yf#gqr#cJZ~ihm4r*)}Yssn4;8VeMY%HD=^Y_PYSJ`f4jm4U%dp!kgS7-J2Rmj*d|>|*aCL?}CRHwrD;U=Y3P^ybKB)QU-xcT| zLWYKhfRYao%2;1x$G62Li4htA%LgL#xo+v4A;2=YaJ! zHE7vcU>+r60+nHMz=~(c6Dh)zO4iZumM^CS@;ep?ewU?ZPM7$iqvXlX#OxW^>_e)I zbdKNimjNU?i^(Qa1l)dwPt!^E~Pl*;JHMkwcF$XZEzN-UAX z&b|E`<7o2$UqI{`?*eo zQ)%|J{{U-NqmVKls?sX-h0@r}`|UOgrxHDZXC{P z!|!r=#d`*_(RPtH2RD!QDTX5ppOHcRo^R!E7UiIib^k60Ao0=B5g@PohKheGzas`vbSFD_To4es>(HjVDZ5JXkUXArPzHX<&cO)Vk@J$V3@E2hu z;yR*dY?k?9I+2|m@mSN=K@l_h`l;sG=?_bSq%`BpugL0qfqVxS?0!-M?a!oHg`L<9 zh@w5xx~P{{e*t@Hf21852;&7S{3|}rp~m!1uHzli4o4C zh7&y{(C*r~j5d=7+KLUiAjQ1y_SR{Z7A;Df86?eWpM!6|ZhA~#1S#Lf9I;paWh5B} zIpc1h?4zcg%o(kp&68#zkB`&Dr=_K>osmXPOg5ThBMx8K6ew7;7^Kv<03C`cE;6K} z!ggWxkU_5~`WE~L;$KRiFGTvWfk5NS!NK8q(8UR;;auFEx*L9oNWsVfU2S?uzG`u^ zd#XkmgkbY6C?%mnv&@g8&(KK%bz@ zqzq)^b2zVaNf++#;o^gjr{O#NWTDMw`sg7+o=H|mttIf4q=G>uL26F};uI8e5=-Kb zQ$wM?>;ZMVttB@)b75;pL$ztDPku9VQ0~xOHzk6`^&x+sbj>zxr8;=bl7FU%9X^wEKB}dS*NFZl2Oj2%()LOb+;U9L zl3DXr1@kTmHfksPdihET;iu>VxRKqj8yDBm=eo8ede~R74>q;TBrD|& zeEjRDOYQq~6r{`3j+S=U@qdWX`;z+WGL|UXtQ@{gAL%Yo+ZytExG$<-IwF=@#D0#I zxlD(E;>F4bTF3YGWFfipK=z4pjBLdGED^V}J4SW$LE~B7E1p_W!4})e&`+;{IQ%2+ zjUWkkr~D1*fbTEfXcP24ig#bj>RnMGEc_GqzLKMScJi^8Tf5K4_gY!`JjivO_X<($ zEt5rS`8dug@tCJ%@f%NzgB^<%R-xfvI6$|C&L@;-JqG7DZr+L9weaVRRzK>g>jBwQdl4>02Q6X+5VEBpI>NbnZ4lXbkUs; z7X&sqZ~R^Sc%S+4@;Dua%~IRG-~DX&RsmZV<@*y(X$c%|paOL!&LGNBqhXrn7*fdL6s5)?QNEdvdn7ubY)Duz^3~8dI04D6(d(7;p$kc z<_@y-<=p(X53S~u3DPOF!srEsczRjycZU2N|xeVW=j1D`;DS%koWJ| zoOuj#6iDPOjh}RUA!1c&=pP>ks*Aewv-fNSwf>2Ri5R!A9pEwmHoteV3q-dAV#hl( z6u7vJaMV9rtbb8Mg32g^+&lh}@$;DaDIL0o#){#LJuiocG)FC5lCWtqyxN|6U7>5c zHwokw$eiT&nV9guJn8OwWdLmboSvQ@$nsMHs;e)6&{0xS5|G%d_S6v)Km)2OC#Uz~ z$)a7Q6m8(=&#~b27SIE+@J|hEHUEKwqeY$hZ7+hw8D<$T=6xK@N4buiSQG@;`hg`Indc!3R+up+53-ELT|`r$NgSOFr{~ zvN`O2_8h)#%;X)j8fq1}EUwG5+tkk>fz}OFjqUKmJgvgv_9eSA3l@X6*4Af?jPLEM z&F>EY@%q=VIIfUDAl(8WPlQC%`W#oc30q`jq@Tb4fwJD1C^8#*bF({7PLzG5uD%C1 zvnnC726gb$8q)@xgEc7Nh%`sbls{)=W+w4_oVQTGFIUq-*kT{B%1ePJ4yZJsQmUX| zmX8ppzPwET@#7Pp&!36$502}@G>aas*jCQ*=bN&MrK2CA&$qU<3zk%t?I!Kc48S#Nt&oZgicn!+atsW=Wx zus+@^&-nPJUb)Og5doFFscBY7NM8J8y){WBcklr82?@NSBC3bPu2qSfxQ6Sb%)GVi zM1-rYDHxmlu{_e6Dhz}J%NM$)1$#9=YXMpoCjKopw#sg)y`cf-N!no>UtWH4xHgcJ zoed;};EXS`6G`=duP4qQgi4Q<#m-h{#2Cmy592B$f^WD7H1WeJqSKKoyo&2sJzQEa+*pc#cy=k>Th$IjN+S(Q++kg|j zU@lQnOZI}`%pvfi&~&t3i!Cd#-?GY5qgLwES4c4=58n?~@}9NEFY3uq&x?@K;qpE= z2v#fCO_NGfUKJSf8~i?w7RMvmI-K|$IFawOIKMl02-9!B7r-K~+ywJKH=Bj(FLvra zm7(lUjBHA!=d`b8&NBR%tx%&+V<`=FobJ(C_A&n)JsUq)dNad%F{tm@>=^o zUv-Z4QPa;-0^!IQR^u{4gtMD^h0PZgXd=&hGk}%&6-A>@j1w#8h3HB;Ue4()F}wjZ zFm%ef5FPy6uKX2{HzZ{S6WvnfWyh0!$AA{^T0sF2!@oyYD0X*fv7Ab)!q#|txF(|> zIxiy~1(HZ69dePoP+{_?HV||rG@$NR+Yj1>1{}=-Yw;XZL0cDE#Gg9vs=cEFAQHf$ zO&jvGHtD>G7Vu#&dp?_x%wo|d!=;%&lbGgC(`H!vp=0FOX!#{ntZT_!bD0FBytVQb z>Ubtji?(AK=NHBCqs#V#S~*yV5<1X`{|9xus(H7F&Fl=0^1UF!6}jSmNPyM>b5b8V z?*J0BzSd!^w&65nS?HeT(}TX(1!qyli-X;R!z=AR%vE^NXH_mUYhUVcMr!#`0-8c; zn3=6_!ER%EcJ|VelEDFaXVYnLQYIoYl9rYht$iVX3%&Jf@}7OroO$v5agWTYtFD}7 zQ^gLw+ln?|oB?;gaNP445c$#O-!3XA_@jbKupog!udq$-|J$BkUNz{>KC&;nhKQd? z9(acmT~KLTtaa)T91U7l^;>s>4I5i{R+^nFeHEfY{XT_GWsg@f3QXdPQVsD0VPI?w zA|M)*Y+Ou?s+!t5I5i88T*LR3lb1hZC<4NZ!}vR}$i(8;UH`{O-hwvMExHHYMTNA07KBOYT=^~3`;g1RSi6 zsWj!QT+$ON0cdaM#n*@Vw7uk zgxd46KmwarJ*H&P-J>V(p}6))+e5wYRuVVgBJ$UAPCqunAjtB473Q?>=3w8%s-yf7T!1 zy9l_7<7Qu_P4D0xqkiy&0(ljMK;I(Jm@jayFE9Z}cBnDoKy0GVtoxrkoL9HK5Emi_d)^US<37Zml$s5nRW z-ovR91Cb{7u*3W;(NS7#=E>cpw}#WIRn6=sB_K5@@_mh{Tk$SCl>}DQZIUIQi#dT# zqN6}>k6lz&RmlM85G09oCo5K?&bL_#xqQSye);96+VwvfEf{WL|0$r+>Gu%6)zhh9 zjRN=)LJffk?M92HL3}o6>a+`Mowh+2>W6j9cARu?cb=q+s=$GI zYOD6up}O-giN@pv73R%EBbUp3Fz4q)v>D7dLhPpDXMW4(gYsp*UA%N(tDIZ#eIpWB zluAVvrVnM|Lt_Ht!b#YrmjXwD%)A$V#j$y{vji_WfF(o~w44+}95Be!V}F9QqMbH! zDJkzf$V#tH9ce>KXM3PX%lNS1C)hL?>TfsVDn@BuaJUZj+{MlAoOtSZ%%)jA&NcP3 z*g|@6|K?EIdD>k)&>X3lV*pLq$P<|VNLE$1D0(Z`!O`*f_}I~D$J+SASnvxqq!0N( zoV1qxq#b}|2{?uIp|5Gfhek$lVX<*>X97nK*#b)RK!rUS5Ccjn@X4TFj_uzE0*zy_J*@qAy8uM6fgULgBqmX$(-{tHK z0>I`(`3}&<1A>)=*D*Ia2}srvh<$LP6dXUADKV9IKBZ`kMn^|4x11m#ASeT(*M*i4 z(6OSHEzL~TxH$p&dsNg&rTq#>S^A!g!ZA{1I`!V5GX%E*%zZ;G&k=lY1c2Xp>TnWd z$~~{npNa(!4h}-KYbPfr;`k&;7Phv)x3$z+&(!L-1Y1s3MU<9RR5*hH%>cfXV$>cn5Yea&-99LZM=#caioUl&}ngT$MaS-U{|N|bxM33y?Zc}rZm$Tpc- z`&+&Jvpz2!Rn`mCgqS9FY8I1m|Gu1SQQ{Nsy;4nGOff%f~?uUjo!dioL& zYk>n#pwvWz7ytwyR~j_2w-g5``||Ur%dNHb^%uZ7=rILm2WF7&AU+y3uJ7Bv(pU)B z!@%JbK*3IXoM;9wL_$?zsBfgC7NdZO796X1>N7YrF%Y=5bYGmd-=xr_Kec)#Lh##q97y8hUsT@UI7KK884iG8S^(wcA|YRtvBmk zD2S9U4fxjrfWQ2Rx^8ARt=-I*Ol#By?mO2razRr^ebN_0SUe1}*nt#lnSuuEPO^5p z#Ej2Mx>bAkE7k4s#UD#IM5UK?uc$H|>MYkcoiG?~An#9%tAo{!cK{{AldlTgBHPCEMh*Xi|m}ML0<%H-=A|`&ph+I_xsNK{GR)F z-}m(`?-*w|)Tjfuns-edND&Hpr6Hu?egn~X@{26)<2V(9S^sfyWV+d+(hQC&`$A67 zinI3KTDSf81O<%)wD*$yre3zVYr)pp56x8IEc-X!N(F4XxpAA}WF4Q37FX^r)zQ_x zkiFrKO6dFQK@<*f4FW;g-U>@a*Gg|uT>No0Ab?UqTLay$V{vgqC*YVgYi@s1Q>jgi z7+XZy>n%Lpj4E6<_6RAQ2)UX|n=t7yXy(>>%b`jJQjC2mxrmP==U zK<&O5wrHxbUaY$EE=K>6@j$WMK58&KA~3~fcO3VHa#?Cvv8^O6F*Blt{l}Y*#a=yn zx>P-C|INO)^3raT+nmOm5{J?3=Z6`;2UX}|6Ns`%3J)I9w+?Cw1j-8Q70phe+N)6c`YK zU2@>@>b>cI%tYb~$U@LKkb{%N6Z=1{alxuN6HsCN=QUAF@DP*(88u2`PufMiQzy>u zpttj0D>$lrjkXaGjI<#rUL?z0@yWq^?cwfzUH)aer*C~jAM1IqNcBjs?NQ1pnmxcU>DHtwKoJR7p0Je zt`6L$x?7ew+M5Ahnxe^4zRD-zqs5m1Qgv|)0vZQ|R&7-Q{F6_KK|(Z2UTsX2FXyBkz7H*xXBVXm&r-q;h?Jv@{3YR4O^}xmvGN$fS6#>2pYMU= zeOn5qbaQ@*LrK=t{?utPtcP1ee@%{k<$wb_AV3wv=$;|i3Cqi zC7cIxHhl-tWhfl)R+CtQ2jB~2;PiUf)AVtQIy&e3HyT!uU}%Ft^mC(*J`?Ly2q4>x za%qT~>8$Y!RPBaFeB1us2(B#@4}InA>10m`7r4=a3NPgI3`Rkv_5%U9Ww;WTpJ;aGjbE6Zm$K z$B3XH|2I&kZebp2PEvmzFabFrC?rb*L#wa<3}WVKinQkLsu4alsn9Nq?VrhH`oV4s z-cG@V3#Iein40U>i6@w4fkSJEa~Tpnf~^!BllEvd1Poa3-McDFI5j)H5^{1JS>C$( zdV1Fk5Xu#WJGp$-xd$E$I5auBy2{jbr6ms>Td_$YzU+|7HS*WurAm#n1xM-Lpcc_8 z;n}uOiZVusLl3`ysvcMywYel{cx0zZl97p!{_4ClZMByOb27(Hj*=)o`4XIwxm#D3+y=p%zSSc+%41yFF z7Z-}8W_`rO-{3}rzzak1at>Wj-YArlhc_q{`2t@LQnghBP ze-ayA6vh7Pyw=Z`IF%@G;%0BJ8}|3>E+uGyKxSBn+p7bc&K=Jd7>C#x2@EEbOm-zn zIL=xvVoK1w?VxF9@_r*770ix(@k%p%a{rbp{*4$uoWMSUXCq+SY7A5q?{^Ut#UM{D z$0R!Y(Md4nPJo;1nsYOAta>ne3Syp|(~QIUoUonFX&|!#0yPoqefTiQ(?X6M(TaQ; zF^cLstx?Nsvx!dzmh(259MXJAJ3bi(w$(sz)tvcomR7Q|?MS*ABVbNH3R+_zr>bNASi{1mL+xU2zlQ=x zn@(wy%+8Ss6;bSOf^Po#y7|i-d$KJ3>+{3moZS>#ZqG4ttQBn+w(~ zH>nwjt10Lh_FSK4w%Uh8G79H2a(p{s)-y3n8ykGo^u~-;uA>_DrjGPa)8V|?fO_~7rl>y{J7gZCy}%Fh&vzydk~zp9GrPyFk*F)TL#pmSI! zDeBkxW+!oXfS-39F!TK1DN&tt*nYz!-?aYlfX$(ejAbCJDbBq5lW6|C`6L{erv1zo zqeu{p?CclMvzR&rwvK~xGxIr(ieWVjb^`lXq~{jn1hDZZoAZ0; z!Oa<2jf24s>3J5zS9Le=*wHu7THuayd>}(o?W7uC#^CaN6BGJ20eIaiEA{8O%>sfE zu%-y%IU&Qpo{!imafqH@CrZKB7-Png_lZ0U|`P5*Pm3bNjmH|8tSCU*z#6J5C%^#~pz@UjZv~JF`+# H_qcxms@~Zt literal 0 HcmV?d00001 diff --git a/content/posts/blocking-ai-crawlers/index.md b/content/posts/blocking-ai-crawlers/index.md index 843d099..71ecd84 100644 --- a/content/posts/blocking-ai-crawlers/index.md +++ b/content/posts/blocking-ai-crawlers/index.md @@ -1,7 +1,7 @@ --- title: "Blocking AI Crawlers" date: 2024-04-12 -# lastmod: 2024-04-12 +lastmod: "2024-04-14T02:21:57Z" description: "Using Hugo to politely ask AI bots to not steal my content - and then configuring Cloudflare's WAF to actively block them, just to be sure." featured: false toc: true @@ -121,9 +121,13 @@ Disallow: / Disallow: * ``` -That's all well and good, but these files carry all the weight of a "No Soliciting" sign. Do I *really* trust these bots to honor it? +That's all well and good, but these files carry all the weight and authority of a "No Soliciting" sign. Do I *really* trust these bots to honor it? -I'm hosting this site [on Neocities](/deploy-hugo-neocities-github-actions/), but it's fronted by Cloudflare. So I added a [WAF Custom Rule](https://developers.cloudflare.com/waf/custom-rules/) to block those unwanted bots. Here's the expression I'm using: +I'm hosting this site [on Neocities](/deploy-hugo-neocities-github-actions/), and Neocities unfortunately (though perhaps wisely) doesn't give me control of the web server there. But the site is fronted by Cloudflare, and that does give me a lot of options for blocking stuff I don't want. + +So I added a [WAF Custom Rule](https://developers.cloudflare.com/waf/custom-rules/) to block those unwanted bots. (I could have used their [User Agent Blocking](https://developers.cloudflare.com/waf/tools/user-agent-blocking) to accomplish the same, but you can only set 10 of those on the free tier. I can put all the user agents together in a single WAF Custom Rule.) + +Here's the expression I'm using: ```text (http.user_agent contains "AdsBot-Google") or (http.user_agent contains "Amazonbot") or (http.user_agent contains "anthropic-ai") or (http.user_agent contains "Applebot") or (http.user_agent contains "AwarioRssBot") or (http.user_agent contains "AwarioSmartBot") or (http.user_agent contains "Bytespider") or (http.user_agent contains "CCBot") or (http.user_agent contains "ChatGPT-User") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "Claude-Web") or (http.user_agent contains "cohere-ai") or (http.user_agent contains "DataForSeoBot") or (http.user_agent contains "FacebookBot") or (http.user_agent contains "Google-Extended") or (http.user_agent contains "GoogleOther") or (http.user_agent contains "GPTBot") or (http.user_agent contains "ImagesiftBot") or (http.user_agent contains "magpie-crawler") or (http.user_agent contains "Meltwater") or (http.user_agent contains "omgili") or (http.user_agent contains "omgilibot") or (http.user_agent contains "peer39_crawler") or (http.user_agent contains "peer39_crawler/1.0") or (http.user_agent contains "PerplexityBot") or (http.user_agent contains "Seekr") or (http.user_agent contains "YouBot") @@ -131,4 +135,8 @@ I'm hosting this site [on Neocities](/deploy-hugo-neocities-github-actions/), bu ![Creating a custom WAF rule in Cloudflare's web UI](cloudflare-waf-rule.png) -I'll probably streamline this in the future to be managed with a GitHub Actions workflow but this will do for now. +And checking on that rule ~24 hours later, I can see that it's doing some good: + +![It's blocked 102 bot hits already](cloudflare-waf-status.png) + +See ya, AI bots! \ No newline at end of file