Documents & code‎ > ‎

Simple Website Recursive [MTA] Url Parser

Simple Website Recursive [MTA] Url Parser   
post id85 
post length1229 
post datetime7/30/2011 8:30:09 PM 
post ip10.10.10.254 

Simple Website Recursive [MTA] Url Parser [to parse web site links] in [VB.NET]


correct`ed: 07/30/2011 
     [@] Fixed parsing of frames... 
     [@] Fixed handling of long queries in the URL... 
     [@] added: pool throttling. 

as well as: 

to avoid the robot~trap`s |robot blocker| need: 

disable scripts: 

       ... 
         WithReRequest 
             .Method = "GET" 
             ... 
             WithCType(.GetResponse(), HttpWebResponse) 
                 responseString _ 
                         = StreamReader(.GetResponseStream()).ReadToEnd() 
                 responseString = responseString.Replace("script", "tpircs") 
                 IDocument.Write(responseString) 
                 .Close() 
             End With 
         End With 
        ... 

disable redirect: 

                responseString = responseString.Replace("<meta http-equiv=", "<-none equiv->") 

& change UserAgent: 

                .UserAgent = "" 

urlhttp://www.useragentstring.com/pages/useragentstring.php 

or 

urlhttp://www.vwp-online.de/ua.php 

 


ċ
LinksPrs.zip
(17k)
DMITRY MENSHOV,
Sep 4, 2013, 9:11 AM
ċ
sl.zip
(1097k)
DMITRY MENSHOV,
Sep 4, 2013, 9:11 AM
Comments