|
|
发表于 2015-6-17 11:29:57
|
显示全部楼层
我也只能帮你到这里了，ebay、amazon等一些站，采集先看robots.txt,从robots.txt中找到sitemap的索引，剩下的就想办法搞吧
, C4 K0 b- r* X- Y; ~ Q5 J0 Ohttp://www.ebay.com/robots.txt # sitemaps - SRPs/ Q9 R$ s' X ^6 H
Sitemap: http://www.ebay.com/lst/SRP_US_index.xml+ z* `( O/ v1 P8 {; O( a% U, T
Sitemap: http://www.ebay.com/lst/ng/SRP_US_index.xml; f8 u# b0 D: J; m# I5 I6 g5 C
5 w7 C B! H3 u# Guides sitemaps
. ]5 F; T1 [+ x3 s- E* KSitemap: http://www.ebay.com/lst/GUIDES-0-index.xml
4 S m ~% U# r+ t9 H
u; E0 k% @; q( n( i# SSRP sitemaps
! r9 @( n, d# U0 ~Sitemap: http://www.ebay.com/lst/SSRP-0-index.xml- y, V* D8 U1 }4 b
) T8 l1 b3 x* F# m9 X, h) f: g* w
#Stores Sitemaps5 a: O5 }) B$ g; t- @
Sitemap: http://www.ebay.com/lst/STORES-0-index.xml
( ?( h% a3 v" p' j; t& {
' ?" w! K% V1 E8 `#BHP Sitemaps9 k+ m: i, r; R, T* D [- k) r
Sitemap: http://www.ebay.com/lst/BHP-0-index.xml
0 F) U! P) f) ]8 I& y c k7 M- a+ M/ F2 H
#Collections' C: u `- i) e: ]2 J# N
Sitemap: http://www.ebay.com/lst/COLLECTIONS-0-index.xml& Q) I* A5 P$ w0 {
+ K7 P- u$ r; r( T7 B, n6 u/ v( k
#VI
3 }2 H$ Q4 d, l8 @+ xSitemap: http://www.ebay.com/lst/VI-0-index.xml
. D- g# g. s8 c b) g
& |0 B/ M0 ^5 J! ~$ z' M+ [0 d% v9 H2 ?#PRP4 S( s* v' ^! |. N# w) s- H
Sitemap: http://www.ebay.com/lst/PRP-0-index.xml
+ x; f7 h$ s( b0 z$ j
|
|
发表于 2015-6-17 08:44:18
发表于 2015-6-17 09:17:34
发表于 2015-6-17 12:29:33
楼主