Wget Command Tutorial With Examples For Linux

RSS
EMAIL
FACEBOOK
FACEBOOK
GOOGLE
GOOGLE
https://www.poftut.com/wget-command-tutorial-examples-linux/
PINTEREST
PINTEREST
INSTAGRAM

Linux provides different tools to download files via different type of protocols like HTTP, FTP, HTTPS etc. wget is the most popular tool used to download files via command line interface. Wget is supported by Linux, BSD, Windows, MacOSX. Wget has rich feature set some of them can be listed

  • Resume downloads
  • Multiple file download single command
  • Proxy support
  • Unattended download

Curl is alternative to wget. To read curl tutorial click this link

Help

Help information of wget can be listed like below.

Wget Help

Wget Help

Simple Download

Most common usage of wget command is without providing any option or parameter except the download URL. We will use wget only providing the download URL.

Simple Download

Simple Download

Set Different File Name

While downloading the downloaded file is named the same as provided in the download URL. In previous example the file is named as wget-1.19.tar.gz as the URL provides this name. We can change the saved file name different then URL. We will use -O option with name parameter to set the name as wget.tar.gz and so will remove the version part from the file name.

Set Different File Name

Set Different File Name

Download Multiple Files

There is an other useful feature of wget which gives us the ability to download multiple files. We will provide multiple URLs in a single command. This files will be downloaded and named as in the URL. There is no explicit option specification required. We just provide URLs in a row by separating them with spaces. There is no limit the count of URLs

Download Multiple Files

Download Multiple Files

Read Download URLs From File

In the previous example we have downloaded multiple files by specifying the URLs from command like. This may became difficult to manage if there are 100 URLs to download. Another situation may be that the URLs are provided externally in a plain text format line by line. Providing these URLs into command line is hard and error prone job.

LEARN MORE  Linux Systemctl Tutorial

Hopefully wget have the feature to read URLs from a file line by line just specifying the file name. We will provide the URLs in a plan text file named downloads.txt line by line with -i option.

downloads.txt

And we will download. It have very clear presentation

Read Download URLs From File

Read Download URLs From File

Resume Uncompleted Download

Another great feature of the wget command is resuming downloads from where it left. Especially in big files the download may be interrupted after %98 completion which is a nightmare. The -c option is provided to resume the download without starting it from scratch.

Resume Uncompleted Download

Resume Uncompleted Download

As we can see from screenshot the downloaded part of the file is presented as + in the download bar. Also there is information about renaming length and all ready downloaded length with line starts Lenght:... .There is also information like Partial Content

Do Not Overwrite

By default wget do not overwrite to the file. If it see the same file name exists with the downloaded file it appends .1 to the end of downloaded file. This .1 is incremented if it is all ready exists.

Download Files In Background

Wget starts as interactive process for the download. During download the wget process remains as foreground process. But in some situations we may need to send wget to the background. With -b option the wget command will send to the background.

Download Files In Background

Download Files In Background

Restrict Download Speed

By default the download speed of the wget will be unrestricted. So it will consume the bandwidth according to remote site upload speed. This behavior can be not suitable for some situations. We may want to not use whole bandwidth and remain bandwidth to the other critical applications. We can use --limit-rate options with the bandwidth value. In this example we set bandwidth as 50KB/s

Restrict Download Speed

Restrict Download Speed

As we can see down left corner the bandwidth is limited with 50KB/s

LEARN MORE  How To Compile Linux Kernel

Specify FTP Username and Password

Security is important issue nowadays. So ftp servers are not so secure but they try to implement some security steps like user name and password. wget can use ftp authentication with ftp user name and password. We will use --ftp-user for specify username and --ftp-password to specify ftp password.

Specify FTP Username and Password

Specify FTP Username and Password

Specify HTTP Username and Password

Like in previous example we can specify the HTTP username and password. We will use --http-user to specify http user name and --http-password for HTTP password.

Specify HTTP Username and Password

Specify HTTP Username and Password

Change User Agent Information

While connecting web pages HTTP protocol provides information about the user browser. The same information is provided while downloading files with wget. Some web sites can deny access to the non standard browser. Wget provides user agent different from standard browsers by default. This browser information can be provided with --user-agent parameter with the name of the browser. In this example we will provide the browser information as Mozilla Firefox .

Change User Agent Information

Change User Agent Information

Test Download URLs

Another useful feature of the wget is testing URLs before downloading them. This will give some hints about the URL and file status. We will use --spider options to check if a remote file exists and downloadable.

Test Download URLs

Test Download URLs

As we can see from screenshot there is a message expressing Remote file exists. in the last line. We can also get the size of the file without downloading.

Set Retry Count

In problematic networks or download servers there may be some download problems. The most known problem is can not accessing the remote file for short periods. We can set retry count which will try to download for specified times. --tries options with the retry count can be used.

LEARN MORE  Linux Find Command With Examples

Download Whole Web Site or Spider Website

There are different tools to download the whole site just providing the homepage URL. wget has this ability too. This command will spider and download all pages about this URL and sub pages. This will make the site offline accessible. We will use --mirror to download whole site and provide the location to download with -P parameter. Here we want to download the

www.poftut.com and its subpages

Download Whole Web Site or Spider Website

Download Whole Web Site or Spider Website

Download Specific File Type

While downloading multiple files or mirroring a site we may want to only download a specific file or file extension. This can be specified with -A and extension or some part of file name. In this example we only want to download the .txt files.

Do Not Download Specific File Types

While downloading multiple URLs or mirroring a site there will be a lot of files we do not want to download or too big to download. We will use --reject option by providing extensions or files names.

Log To File

By default logs created by wget is printed to the standard output which is generally the command line interface we are using. But using remotely or as a batch or background process we can not get logs directly. so writing the logs to file is best solution. We will use -o option with the log file name.

Log To File

Log To File

Set Download Size Limit

Another good option to set limit total downloaded file size is -Q . We will set download size as 2 MB in the example. This setting is not effective for single file download. It will effect for recursive or mirroring of sites.

Show Version

Version information about the wget command can be get with --version option.

RSS
EMAIL
FACEBOOK
FACEBOOK
GOOGLE
GOOGLE
https://www.poftut.com/wget-command-tutorial-examples-linux/
PINTEREST
PINTEREST
INSTAGRAM

You may also like...

7 Responses

  1. george says:

    first example has a severe typo –

    It documents “-o” (lower case “o”) as the option to name the downloaded file.

    This is incorrect. the correct option for that is “-O” (upper case “O”)

    Lower case “O’ is used to redirect the reported info that wget normally sends to stdout.

    Fortunately he gets it correct in the example.

  2. John says:

    Your date for the article shows as 7/03/2017, I believe you’re a month off

  3. Bill says:

    This is a great introduction to wget. It is well presented and easy to understand.

  4. John says:

    Thank you for pointing out my simple mistake. I know of the alternative date format but wasn’t thinking that night.

Leave a Reply

Your email address will not be published.

Enjoy this blog? Please spread the word :)