What is cURL and How to Use it? The Essential Guide for Beginners

If you work on the command line regularly, chances are you‘ve heard about cURL. The versatile data transfer tool has become ubiquitous and is installed on millions of systems worldwide. But what exactly is cURL, and why is it so popular?

In this comprehensive guide, we‘ll cover everything you need to know about cURL from what it is, to how you can use it in your day-to-day work. You‘ll learn:

• What is cURL and a brief history of the project
• An overview of the main features and capabilities
• Popular use cases and examples of cURL commands
• Detailed examples of GET, POST, uploading files, authentication and more
• How to use advanced features like cookies, proxies, and HTTPS
• How cURL compares to alternatives like Wget
• Why developers and sysadmins find cURL so indispensable

By the end of this guide, you‘ll have a solid foundation for using cURL effectively. Let‘s get started!

A Brief History of cURL

cURL first started in 1997 as a command line tool for transferring files and data. The name originally stood for "Client URL", suited for a utility that transfers data specified in a URL.

cURL was created by Daniel Stenberg, an independent developer based in Sweden. Stenberg was looking for a way to retrieve currency exchange rates for an IRC chat bot he had created. Back then, there weren‘t many existing tools available so he wrote a simple HTTP file retrieval utility that became the basis for cURL.

The initial versions only supported HTTP, FTP, GOPHER and TELNET protocols. Over the next few years, capabilities were rapidly added for other protocols including HTTPS, FTPS, IMAP, LDAP, DICT, TFTP and more. Support for SSL and proxy servers further added to cURL‘s versatility.

cURL 1.0 was officially released in March 1998. Since then, it has been adopted globally and is bundled with many operating systems and distributions. The official website curl.se states cURL is used in over 10 billion installations worldwide!

Due to its longevity, stability, and ubiquitous deployment on servers and devices, cURL has become a de facto standard for transferring data from the command line. The adoption and praise for cURL is a testament to its flexible feature set.

What is cURL Used For?

cURL allows transferring data to and from a server using one of the supported protocols. The data itself isn‘t processed by cURL – it simply facilitates the transfer.

This makes cURL a general purpose utility that can be used in many different ways:

• Automate data transfers for backups, periodic data pulls, etc
• Test APIs by sending requests and inspecting raw responses
• Download files from scripting languages
• Web scraping by downloading web pages
• Interact with web APIs by sending/receiving JSON
• Submit web forms in automation scripts
• Upload/download files to a server
• Basic automation for tasks like periodic website checks

cURL gives you the building block of transferring data over the CLI. This data transfer can then be incorporated into shell scripts and cron jobs to build complex automation workflows.

The creator Daniel Stenberg himself says:

"Anything that lets you get or send data over a network could have uses for libcurl."

Let‘s look at some of the most common use cases for cURL.

Testing APIs

One of the most popular uses for cURL is testing REST and web APIs during development. cURL allows sending formatted requests and inspecting the raw responses quickly from the command line.

For example, to test a simple REST API GET endpoint:

curl https://jsonplaceholder.typicode.com/todos/1 

The response contains the JSON document with details of the specific todo item.

To test a POST API:

curl -X POST -d ‘title=Buy milk&completed=false‘ https://jsonplaceholder.typicode.com/todos

Here we‘re using -d to send a request body for creating a new todo.

The ability to manually send requests of different types, add headers, set parameters, and view the output is invaluable when working with web APIs. cURL helps take the pain out of manually constructing requests.

Downloading Files

cURL can download files from any remote server using FTP, HTTP or HTTPS protocols. This makes it useful for scripting and automating file transfers from the command line.

For example, to store a file locally from a URL:

curl -o example_file.html https://www.example.com/files/example_file.html

The -o option names the output file.

You can also download and save using the original remote file name:

curl -O https://www.example.com/files/report-2023.pdf

This will save the file as report-2023.pdf.

These download features are handy for periodically pulling files from remote servers in cron jobs or scripts.

Web Scraping

While cURL is not designed specifically for web scraping, it can be used to download web pages and perform simple scraping tasks.

For example:

curl https://news.ycombinator.com/ > hn_frontpage.html

This grabs the HTML of the HackerNews frontpage and saves it, allowing you to then parse and extract data from it.

Of course, dedicated scraping tools like Puppeteer, Scrapy and Beautiful Soup are more full-featured. But cURL provides a quick way to pull HTML right from the terminal.

Interacting with Web APIs

cURL is commonly used for interacting with web APIs. By sending properly formatted HTTP requests and inspecting the output, you can:

• Retrieve API data in JSON/XML format
• Submit data to API endpoints
• Pass parameters and authentication credentials
• Analyze response headers and status codes

For example, to GET a user profile from a REST API:

curl https://api.example.com/users/123 -H "Authorization: Bearer api-key"

We added an authorization header here to authenticate.

To submit data:

curl -X POST -d ‘{"name":"John Smith","email":"[email protected]"}‘ https://api.example.com/users

This submits a JSON request body to create a new user.

cURL provides a quick way to prototype and test web APIs.

cURL Command Line Options

cURL has dozens of command line options available for getting the most out of the tool. Here are some of the most common ones you‘ll use:

Option	Description
-o	Write output to file
-O	Write output to file named like the remote file
-v	Verbose output for debugging
-I	Fetch headers only
-L	Follow redirects
-d	Send POST data
-u	Authentication username/password
-x	Use proxy
-H	Custom request header
--data-binary	POST binary data
-C -	Resume download

See the curl man page for all options. Let‘s look at some examples using these common flags.

GET Requests

To download a URL:

curl https://www.example.com

To save the output HTML to a file:

curl -o example.html https://www.example.com

To follow redirects:

curl -L -o example.html https://www.example.com

POST Requests

To POST data:

curl -d ‘[email protected]‘ https://www.example.com/signup

To POST JSON:

curl -H "Content-Type: application/json" -d ‘{"name":"John", "email":"[email protected]"}‘ https://www.example.com/users

Authentication

To authenticate:

curl -u username:password https://api.example.com/users/123

Debugging

To see verbose output:

curl -v https://api.example.com/users/123 > /dev/null

This includes request and response headers. The > /dev/null hides the response body for readability.

To show headers only:

curl -I https://api.example.com/users/123

Downloading Files

To download a file:

curl -O https://www.example.com/files/report.pdf

To resume a failed download:

curl -C - -O https://www.example.com/files/report.pdf 

There are many more options available – refer to the curl man page for all usage examples.

Uploading Files with cURL

In addition to downloading, cURL provides options for uploading local files to a remote server.

The -T option lets you specify a local file to upload:

curl -T image.jpg https://www.example.com/upload

This will POST the contents of image.jpg to the server.

You can also specify an upload target filename:

curl -T image.jpg -o /uploads/profile.jpg https://www.example.com

Other features like resuming allow for robust file uploading capabilities from the command line.

cURL Authentication Methods

cURL supports many different types of authentication depending on the protocol.

For basic HTTP authentication, you can specify a username and password:

curl -u myusername:password https://www.example.com

For servers using NTLM authentication (common with Windows servers):

curl --ntlm -u myusername:password https://www.example.com

When using an FTP server:

curl --ftp-ssl -u myusername:password ftp://www.example.com

This enables SSL encrypted FTP with basic authentication.

To use client SSL certificates for authentication:

curl --cert client_cert.pem https://www.example.com

Specifying the certificate authenticates your requests.

These are some common examples – cURL supports many more advanced authentication methods as well.

Using Cookies with cURL

Cookies allow you to store session data for consecutive requests.

To save cookies from a request in a file:

curl -c cookies.txt https://example.com

This stores cookies in cookies.txt

On subsequent requests, use -b to load cookies:

curl -b cookies.txt https://example.com/user

This will send cookies when making the request.

You can also pass cookies directly on the command line instead of using a file:

curl -b "jsessionid=1234; cart_items=3" https://example.com

This sets 2 cookies in the request.

cURL HTTP/2 Support

By default, cURL uses HTTP 1.1 to make requests. However, modern websites are increasingly supporting HTTP/2 for better performance.

cURL has support for HTTP/2 when compiled with the nghttp2 library. To send HTTP/2 requests:

curl --http2 https://example.com

This will use HTTP/2 instead of 1.1.

Keep in mind the server must support HTTP/2 as well for this to work.

Using Proxies

To proxy requests through a SOCKS5 proxy:

curl --socks5 localhost:1234 https://www.example.com

For an HTTP proxy:

curl -x localhost:8080 https://www.example.com

To authenticate with the proxy:

curl -x username:[email protected]:8080 https://www.example.com

This passes the proxy authentication credentials.

By routing through a proxy, you can mimic requests coming from a different IP address. Proxies are also useful for bypassing geographic restrictions on websites.

How Does cURL Compare to Wget?

cURL and Wget are two popular command line utilities used for downloading files and data. At a glance, they may seem interchangeable but there are some key differences:

• Protocols – cURL supports way more protocols including FTP, IMAP, POP3, SMTP and more. Wget only works with HTTP/HTTPS.

• HTTP features – cURL has more HTTPS and HTTP features including cookies and authentication. Wget has basic HTTP capabilities.

• File downloading – Wget specializes in recursive downloading and mirroring of sites. cURL has basic file download features.

• Proxy support – cURL has HTTP, SOCKS5 and other proxy support. Wget only supports HTTP proxies.

• Scripting – Wget has more built-in support for scripting downloads. cURL downloads need a bit more work.

• Platforms – cURL works on more platforms including Windows, Unix, macOS and more. Wget originated on Linux.

In summary:

• For generic file downloads, especially recursive mirroring, Wget is better.
• For API testing, HTTP requests, proxies, and advanced transfers, cURL is preferable.
• cURL is more of a general data transfer tool while Wget focuses on file downloads.

So while there is overlap, the tools also have complementary strengths. For most developers and sysadmins, having both installed can be beneficial.

Why is cURL So Indispensable?

Since its creation in 1997, cURL has become ubiquitous on systems worldwide. What exactly makes this utility so essential for developers and sysadmins?

It‘s installed everywhere – cURL comes pre-installed on most modern operating systems and Linux distros. You can expect it is available out-of-the-box.

Support for protocols – The sheer variety of supported protocols (over 20!) means cURL can transfer data to and from nearly any system.

Maturity and stability – cURL has been around for 20+ years and is considered very stable and reliable. The code is tried and tested.

Active development – Despite its maturity, cURL still sees active development with new features and updates released periodically.

Portability – cURL was designed as a portable library that can be used across platforms. It runs on virtually any operating system.

Scripting capabilities – cURL can be easily used in Bash or any other scripting language for automating transfers.

Ease of use – The command line syntax is simple and intuitive even for basic usage. More advanced capabilities are available when needed.

Trust and security – cURL is trusted by major companies like Facebook, Google, Netflix and more for critical transfers. The project takes security very seriously.

For these reasons and more, cURL remains a staple tool for anyone working on the command line today. It likey will continue to be indispensable for decades more!

Conclusion

cURL is an invaluable tool for working with data transfers on the command line interface. The official website summarizes it best:

"command line scripting HTTP client. Fetching URLs, interacting with web APIs, uploading/downloading files. Do everything over the command line."

In this guide, we covered the key fundamentals:

• What is cURL – An open source command line tool supporting many protocols via libcurl. Created by Daniel Stenberg in 1997.

• Key features – Supports 20+ protocols, SSL encryption, resumed downloads, file transfers, proxies, authentication and more.

• Common usage – Testing APIs, downloading files, automating transfers, web scraping, interacting with web services, and scripting HTTP requests.

• Command line options – Over 100 different options available like -o for output, -v for verbose mode, -d for sending data and much more.

• Advanced capabilities – Uploading files, using cookies, supporting HTTP/2, connecting via proxies and SSH tunnels, custom authentication and more.

• Alternatives – Tools like Wget have overlapping functionality but more limited protocol and feature support compared to cURL.

• Popularity – Installed on billions of systems and considered an indispensable tool by developers and sysadmins.

In summary, if you work on the command line, cURL is an essential tool to learn. With capabilities for transferring data across virtually any protocol, robust features, ubiquitous support and a simple interface, it‘s no surprise cURL remains a popular choice 25 years later.