Simply curl or command line tool and library for transferring data with URLs. curl provides wide range of support to download files and folders with command line interface. curl can be used with a lot of different type of protocols. Below are the protocols currently supported by curl.
DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, HTTP/2, cookies, user+password authentication (Basic, Plain, Digest, CRAM-MD5, NTLM, Negotiate and Kerberos), file transfer resume, proxy tunneling
Keep your breath because there are a lot examples about curl where we look them one by one.
Syntax of the curl command is like below.
curl [options...] <url>
Man page of curl can be get with the following command
$ man curl
Simple fast help about parameters and options can be get with the following command.
$ curl -h
Fetch And Display with HTTP/s
One of the most used type of curl is downloading html or similar files with HTTP/s. curl will automatically detect the transfer protocol which is HTTPs in this example.
$ curl https://www.poftut.com
Download with HTTP/s and Redirect
In previous example the downloaded file is printed out the standard output which is our terminal. But this is generally not a practical usage. The downloaded file can be redirected to a file with bash redirect feature like below. In this example we can see some statistics information about the download. This information provides total size, received, average download speed, total time and times spend for download.
$ curl https://www.poftut.com > poftut.html
Download with HTTP/s
There is a parameter used to save downloaded file into a file with no extra command. This parameter is
$ curl -o poftut.html https://www.poftut.com
Limit Download Rate
Another useful feature of curl is limiting download rate. This can be very useful in situations where there is limited internet bandwidth and it should be shared with other applications. In this examples we have limited bandwidth to 1K and this will make our download take some time
$ curl --limit-rate 1k -o poftut.html https://www.poftut.com
Download Sequential Files
System administrators generally prefer using sequential file names for backups or similar operations. Also log files those resides in
/var/log are generally names with sequentially. So downloading them one by one specifying the full name of the file is drudgery work. curl can download this files like following example.
$ curl ftp://www.poftut.com/backup[1-9].tar
Set SSL Version
While using secure protocols like HTTPS, FTPS, POP3S SSL protocol will be used to create encrypted channel. SSL have different version where this may create incompatible situations. SSL version can be specified with the
--sslv3 parameter. But keep in mind that curl uses GnuTLS library for cryptographic operations and GnuTLS should have support for SSLv2.
$ curl --sslv2 -o poftut.html https://www.poftut.com
Verbose and Debug Mode
While downloading and uploading files with curl there will be a of operations that occurs background. There will be also problems those we can not know what is happening. curl can provide details about the operations with the
-v parameter like below.
$ curl --ssl -v -o poftut.html https://www.poftut.com
There is another mode where there will be no output to the terminal. This can be used for clean download.
$ curl -s -o poftut.html https://www.poftut.com
Specify User and Password
While using authentication required protocols and servers curl can provide these credentials like username and password.
$ curl -u ismail:mypassword -o data.tar ftp://poftut.com/backup.tar
As we know HTTP is stateless protocol. To preserve user session cookies are used. While using HTTP protocol existing cookies can be used with
-b parameter like below.
$ curl -b aerf34fawfeawf -o statistics.html https://www.poftut.com/statistics.html