Simple Website Recursive [MTA] Url Parser

Simple Website Recursive [MTA] Url Parser

post id: 85

post length: 1229

post datetime: 7/30/2011 8:30:09 PM

post ip: 10.10.10.254

Simple Website Recursive [MTA] Url Parser [to parse web site links] in [VB.NET]

correct`ed: 07/30/2011

[@] Fixed parsing of frames...

[@] Fixed handling of long queries in the URL...

[@] added: pool throttling.

as well as:

to avoid the robot~trap`s |robot blocker| need:

disable scripts:

...

WithReRequest

.Method = "GET"

...

WithCType(.GetResponse(), HttpWebResponse)

responseString _

= StreamReader(.GetResponseStream()).ReadToEnd()

responseString = responseString.Replace("script", "tpircs")

IDocument.Write(responseString)

.Close()

End With

End With

...

disable redirect:

responseString = responseString.Replace("<meta http-equiv=", "<-none equiv->")

& change UserAgent:

.UserAgent = ""