Internal linking as a data structure

Internal linking functions as a data structure within websites. Initially, editors manually insert links, which becomes unwieldy as the site grows. To address this, internal links should be treated as a graph, where each URL is a node and 'a href' pointing to another URL on the same domain is a directed edge. This allows for computing orphan pages, link depth, and missing connections programmatically.

To build the node list, the sitemap serves as the fastest source. Parse the sitemap to obtain the full set of URLs, then crawl the HTML to extract the edges. The XMLParser module helps in parsing the sitemap XML, and the fetch function is used to retrieve the URLs from the sitemap. If the sitemap contains other sitemaps, they are recursively parsed.

The edge list is built from the HTML of each URL. Cheerio library is used to parse the HTML and extract all internal anchor tags. The URLs are then normalized and added to a set to avoid duplicates. The extracted links are stored in an adjacency table, where each URL is a key and its corresponding value is an array of URLs it links to.

Depth of a page, often used as a proxy for authority, can be computed using a breadth-first search algorithm. The computeDepth function takes the adjacency table and a root URL as input. It initializes a depth map with the root page at depth 0 and a queue with the root page. The algorithm iteratively dequeues a page, checks its adjacent pages, and enqueues them if they haven't been visited yet. The depth of each page is stored in the depth map.