There are many explanations you could possibly will need to uncover many of the URLs on a web site, but your correct purpose will ascertain what you’re searching for. For example, you may want to:

Determine each indexed URL to analyze problems like cannibalization or index bloat
Gather present and historic URLs Google has noticed, especially for website migrations
Discover all 404 URLs to recover from post-migration faults
In Each individual state of affairs, only one Resource gained’t Present you with almost everything you may need. However, Google Lookup Console isn’t exhaustive, as well as a “website:example.com” look for is restricted and tough to extract info from.

Within this post, I’ll wander you thru some tools to build your URL record and ahead of deduplicating the information using a spreadsheet or Jupyter Notebook, based upon your web site’s measurement.

Old sitemaps and crawl exports
For those who’re in search of URLs that disappeared through the Reside site a short while ago, there’s a chance somebody on the crew might have saved a sitemap file or simply a crawl export prior to the modifications were being manufactured. Should you haven’t already, look for these files; they will usually provide what you need. But, when you’re reading through this, you almost certainly did not get so Blessed.

Archive.org
Archive.org
Archive.org is an invaluable Software for Website positioning duties, funded by donations. For those who seek out a domain and select the “URLs” option, it is possible to access approximately 10,000 shown URLs.

However, There are many limitations:

URL limit: You could only retrieve around web designer kuala lumpur ten,000 URLs, that is inadequate for larger internet sites.
Quality: A lot of URLs may very well be malformed or reference resource documents (e.g., images or scripts).
No export selection: There isn’t a crafted-in method to export the listing.
To bypass The dearth of an export button, make use of a browser scraping plugin like Dataminer.io. Having said that, these limitations signify Archive.org may well not offer a complete Remedy for much larger sites. Also, Archive.org doesn’t show whether Google indexed a URL—but if Archive.org identified it, there’s a fantastic prospect Google did, much too.

Moz Professional
When you might normally make use of a url index to search out external websites linking to you personally, these applications also explore URLs on your internet site in the method.


How you can utilize it:
Export your inbound inbound links in Moz Pro to secure a speedy and simple listing of concentrate on URLs out of your website. If you’re addressing a large Web-site, think about using the Moz API to export data beyond what’s workable in Excel or Google Sheets.

It’s important to note that Moz Pro doesn’t confirm if URLs are indexed or discovered by Google. Nevertheless, because most websites apply exactly the same robots.txt guidelines to Moz’s bots as they do to Google’s, this method generally is effective perfectly to be a proxy for Googlebot’s discoverability.

Google Look for Console
Google Research Console delivers many useful sources for creating your list of URLs.

Links reports:


Comparable to Moz Professional, the Backlinks segment offers exportable lists of target URLs. Regretably, these exports are capped at one,000 URLs Every single. It is possible to use filters for specific internet pages, but due to the fact filters don’t implement for the export, you may need to rely upon browser scraping instruments—limited to five hundred filtered URLs at a time. Not suitable.

Effectiveness → Search Results:


This export gives you an index of webpages receiving look for impressions. Even though the export is limited, You may use Google Research Console API for bigger datasets. There are also absolutely free Google Sheets plugins that simplify pulling far more in depth details.

Indexing → Webpages report:


This section delivers exports filtered by concern type, however they're also confined in scope.

Google Analytics
Google Analytics
The Engagement → Web pages and Screens default report in GA4 is an excellent supply for collecting URLs, that has a generous limit of one hundred,000 URLs.


Even better, you may use filters to build distinct URL lists, properly surpassing the 100k Restrict. By way of example, if you wish to export only blog site URLs, adhere to these techniques:

Stage one: Include a phase towards the report

Action 2: Click on “Make a new segment.”


Action 3: Outline the phase having a narrower URL sample, such as URLs that contains /web site/


Notice: URLs located in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they provide important insights.

Server log files
Server or CDN log documents are Potentially the ultimate Resource at your disposal. These logs seize an exhaustive checklist of each URL route queried by customers, Googlebot, or other bots in the course of the recorded interval.

Factors:

Facts size: Log information is often massive, numerous web-sites only retain the last two weeks of information.
Complexity: Examining log files may be demanding, but many resources can be found to simplify the method.
Incorporate, and good luck
As soon as you’ve collected URLs from these sources, it’s time to combine them. If your website is small enough, use Excel or, for much larger datasets, applications like Google Sheets or Jupyter Notebook. Make sure all URLs are constantly formatted, then deduplicate the listing.

And voilà—you now have an extensive listing of present-day, old, and archived URLs. Great luck!