Positive thread
Out of context: Reply #65
- Started
- Last post
- 72 Responses
- oBeseLilNinja0
about two hours ago i finished my intelligent feed locater/crawler that doesnt actually do much crawling/fetching compared to traditional web crawlers, it finds the most appropirate path to crawl to find the feeds via the value of anchor tags and the chance the feeds have already been found, also has a link analysis system behind each domain by how many times it has already been asked to crawl that domain or webpage (it only downloads anything once in any session). from these valued links to other sites it makes what i called a cloud - determining even if the clouds (which are subclouds within (sub)clouds like window's folders) are not valuable enough to keep analysis on and are killed all together within the system
also it uses multitasking threads with regards to the fetching so the delay between getting webpages is minimal (i.e. less bottlenecks), im able to run the application multiple times on each machine, so if the application thread(s) are set to 3, i can have 3 applications thus 9 threads fetching at the same time
that might be a load of useless incoherent info to every1 but for me thats positive for me hehe :)
forgive my grammer and spelling ive all ready started getting wasted hehe, cant believe i wrote so much