Simple Website Recursive [MTA] Url Parser

Simple Website Recursive [MTA] Url Parser

post id: 85

post length: 1229

post datetime: 7/30/2011 8:30:09 PM

post ip:

Simple Website Recursive [MTA] Url Parser [to parse web site links] in [VB.NET]

correct`ed: 07/30/2011

[@] Fixed parsing of frames...

[@] Fixed handling of long queries in the URL...

[@] added: pool throttling.

as well as:

to avoid the robot~trap`s |robot blocker| need:

disable scripts:



.Method = "GET"


WithCType(.GetResponse(), HttpWebResponse)

responseString _

= StreamReader(.GetResponseStream()).ReadToEnd()

responseString = responseString.Replace("script", "tpircs")



End With

End With


disable redirect:

responseString = responseString.Replace("<meta http-equiv=", "<-none equiv->")

& change UserAgent:

.UserAgent = ""