Can You Restrict Wget to Specific Directories?

When scraping or mirroring a website using wget, you can restrict the tool’s traversal path to specific directories by utilizing the -I (or --include-directories) option. This prevents the utility from wandering into unwanted sections of a web server, ensuring that only the content within your specified paths is downloaded. By mirroring the directory structure you explicitly define, wget provides a highly targeted approach to automated web downloading.

How to Use the Include Directories Option

The -I option accepts a comma-separated list of directory paths that you want to limit the download to. It is most effective when combined with the recursive download flag (-r or -m).

Here is the basic syntax for restricting wget to specific paths:

wget -r -I /directory1,/directory2 http://example.com/

Key Considerations for Path Restricting

To ensure the restriction works as intended, keep the following behaviors in mind:

• Relative Paths: The paths specified in the include list must be relative to the hostname root. For example, if you want to download http://example.com/files/docs/, your include path should be /files/docs.
• The Parent Directory Rule: If you kick off a download at a deep sub-directory, wget might still need to look at parent directories to find files. Using the -np (or --no-parent) flag alongside -I is a common best practice to guarantee wget never ascends to a higher directory level.
• Combining with Excludes: If you want to include a large directory but skip a specific subfolder within it, you can pair the include flag with the -X (or --exclude-directories) flag to fine-tune your traversal path.