Paper Contents
Abstract
The internet is a tremendously big and dynamic place these days. A user uses the internet to get to specific websites. Information has been added to web pages through the use of text, videos, photos, and other processes. What a search engine is interface for retrieving data from websites. It is incredibly difficult to get to. information from a large web page archive stored in a URL. A search engine might use a web crawler and other web crawling techniques to obtain relevant sheets. A web crawler is a piece of software that lets users find and It does this by using crawling techniques to recover web content. Objective. This thesis proposes a new method for web crawlers called the single and multithreaded web crawling and indexing algorithm utilizing clustering technique. URLs found on pages that have been crawled are divided into two categories: intradomain and interdomain links. As hyperlinks rely on the kind and size of web page links that are saved in the queue URL, it initializes the weight for a particular webpage. The outcome of the experiment demonstrates that the suggested algorithm outperforms current techniques in terms of execution time. Initially, a crawling technique is used to extract the connections from a certain Uniform Resource Locator (URL), allowing the user to do hierarchical scanning for individual web links
Copyright
Copyright © 2023 ZUBAIR SHAFIUDDIN. This is an open access article distributed under the Creative Commons Attribution License.
