How to Use cURL in Python (with Examples)

Using cURL in Python provides developers with a powerful combination of versatility and functionality for data transfer tasks. By leveraging the PycURL library, you can seamlessly incorporate cURL into your existing Python projects.
by Josephine Loo · May 2023

Contents

    cURL (Client for URLs), a versatile command line tool for data transfer, is a powerful asset in a developer's toolkit. It is platform-independent and can be used on various operating systems including Windows, macOS, Unix/Linux, IBM i, and Android to send and receive data over protocols like HTTP, FTP, SCP, IMAP, and more. You can use it to fetch web pages, download files, upload data, interact with APIs, and perform other operations that involve data exchange between a client and a server.

    While cURL can be executed directly in the terminal or command prompt of nearly any operating system, there are situations where running a cURL command within Python code becomes advantageous. For example, it allows integration with an existing Python code base, leveraging enhanced Python functionalities, and working with other Python libraries.

    In this article, we will explore how to use cURL in Python, particularly using it to make HTTP requests, and provide examples that showcase its features and practical applications.

    What is cURL

    cURL is a free and open-source command line tool and library for transferring data with URLs. Other than the operating systems mentioned previously, it can also be used on devices like routers, printers, audio equipment, mobile phones, tablets, and media players.

    All transfer-related features of cURL are powered by libcurl,  a free and easy-to-use client-side URL transfer library. It builds and works similarly on different platforms and operating systems. To use it on your machine, run the curl command in the terminal or command prompt followed by the desired options and arguments.

    curl [options / URLs]
    

    Here’s a screenshot using cURL in the terminal to fetch a web page:

    cURL in the terminal

    The command in the screenshot above will send a GET request to the specified URL and displays the response in the terminal. The received data will be written to stdout by default but you can also save them into a local file, using the -o, –output or -O, –remote-name options.

    Sending HTTP Requests in Python: PycURL vs Requests

    Making HTTP requests in Python requires a library. You can use PycURL, a wrapper around the cURL library, or Requests, a third-party library that simplifies HTTP request making and provides a high-level API for interacting with web services in Python.

    The Requests library is widely used due to its simplicity. However, it can be slow. When multiple requests are performed and connections are reused, the speed of PycURL wins over Requests significantly (2-3x). Besides that, PycURL also offers many other features, like:

    • Support protocols like DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet, and TFTP (Requests support HTTP only)
    • I/O multiplexing
    • The ability to use several TLS backends
    • More authentication options

    Both libraries come in handy when you need to make HTTP requests in Python. The Requests library is often preferred because it provides a simple API but if you need to send data over other networks too or utilize more advanced features, PycURL is a better choice.

    Pre-requisites

    To use cURL in Python/PycURL on your machine, you need to have these installed:

    Using cURL in Python to Send HTTP Requests

    PycURL is a Python wrapper around the cURL library. To use cURL in Python, install the PycURL package by running the command below in your terminal/command prompt:

    pip install pycurl
    

    After installing the package, we can import and use it in our Python code.

    Next, we will see how to make HTTP requests using the PycURL library. To show the comparison between using PycURL and Requests to make HTTP requests, we will also use some examples from Web Scraping with Python: An Introduction and Tutorial, which the Requests library is used.

    POST

    The code below shows a simple example of making a POST request in Python using PycURL:

    import pycurl
    from urllib.parse import urlencode
    
    # Create a Curl object
    curl = pycurl.Curl()
    
    # Set the URL
    curl.setopt(pycurl.URL, 'https://www.example.com/submit')
    
    # Set request type to POST
    curl.setopt(pycurl.POST, 1)
    
    # Set request data
    data = {'key': 'value'}
    curl.setopt(pycurl.POSTFIELDS, urlencode(data))
    
    # Perform the request
    curl.perform()
    
    # Get response code
    response_code = curl.getinfo(pycurl.RESPONSE_CODE)
    print(f"Response Code: {response_code}")
    
    # Close the Curl object
    curl.close()
    

    The data must be URL-encoded beforehand. If you’re not sending any data in the request body, set the POSTFIELDSIZE to 0 to indicate there’s no request body. Otherwise, the request cannot be made.

    curl.setopt(pycurl.POSTFIELDSIZE, 0)
    

    In some cases, you might need to add headers and data to your request, like making a POST request to Browserbear's API to trigger an automation task, which requires authentication:

    import pycurl
    from urllib.parse import urlencode
    
    api_key = "your_api_key"
    task_id = "your_task_id"
    run_id = "your_run_id"
    
    # Create a Curl object
    curl = pycurl.Curl() 
    
    # Set the URL
    curl.setopt(curl.URL, f"https://api.browserbear.com/v1/tasks/{task_id}/runs")
    
    # Set the header
    curl.setopt(curl.HTTPHEADER, ["User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
                                    f"Authorization: Bearer {api_key}"])
    
    # Set request type to POST
    curl.setopt(pycurl.POST, 1)
    
    # Set request data
    data = {
        'webhook_url': 'https://webhook.site/bf8ed79b-0d35-4734-883b-3b0ea909e6d8',
    }
    curl.setopt(pycurl.POSTFIELDS, urlencode(data))
    
    # Perform the request
    curl.perform()
    
    # Get response code
    response_code = curl.getinfo(pycurl.RESPONSE_CODE)
    print(f"Response Code: {response_code}")
    
    # Close the Curl object
    curl.close()
    

    Note: If you receive error code: 1010% in the response, try adding user agent to your header like the example above.

    🐻 Bear Tip: You can compare the code above with the request made in Web Scraping with Python: An Introduction and Tutorial.

    GET

    For a GET request, the code is simpler. You don’t have to set the request type explicitly like POST:

    import pycurl
    
    # Create a Curl object
    curl = pycurl.Curl()
    
    # Set the URL to send the GET request
    curl.setopt(curl.URL, 'https://www.example.com/endpoint')
    
    # Perform the request
    curl.perform()
    
    # Retrieve the response
    response = curl.getinfo(pycurl.RESPONSE_CODE)
    print(response)
    
    # Close the Curl object
    curl.close()
    

    Similarly, you can use it to make a GET request for Step 3 in Web Scraping with Python: An Introduction and Tutorial instead of using the Requests library:

    import pycurl
    
    api_key = "your_api_key"
    task_id = "your_task_id"
    run_id = "your_run_id"
    
    # Create a Curl object
    curl = pycurl.Curl()
    
    # Set the URL
    curl.setopt(curl.URL, f"https://api.browserbear.com/v1/tasks/{task_id}/runs/{run_id}") #initializing the request URL
    
    # Set the header
    curl.setopt(curl.HTTPHEADER, ["User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
                                    f"Authorization: Bearer {api_key}"])
    
    # Perform the request
    curl.perform()
    
    # Get response code
    response_code = curl.getinfo(pycurl.RESPONSE_CODE)
    print(f"Response Code: {response_code}")
    
    # Close the Curl object
    curl.close()
    

    🐻 Bear Tip : If you're hitting errors, you can debug using curl.setopt(c.VERBOSE, True).

    The examples above show you how to make simple HTTP requests using PycURL without other advanced configurations. However, sometimes, you might need to configure PycURL or perform additional operations to suit project requirements, like:

    Utilizing a Certificate Bundle

    Most websites nowadays use HTTPS, which uses TLS (SSL) to encrypt normal HTTP requests and responses, and digitally sign those requests and responses. In this case, PycURL needs to utilize a certificate bundle.

    Some operating systems supply the certificate bundle. Otherwise, you can install and use the certifi Python package:

    import certifi
    
    curl.setopt(curl.CAINFO, certifi.where())
    

    Downloading/Writing Data to a File

    PycURL does not provide storage for the network responses but you can write them to a file using the WRITEDATA function. The response body can be written bytewise without decoding or encoding, as long as the file is opened in binary mode. You can do so by indicating wb in the open() function:

    import pycurl
    
    with open('output.html', 'wb') as f:
        curl = pycurl.Curl()
        curl.setopt(curl.URL, 'https://www.browserbear.com/')
        curl.setopt(curl.WRITEDATA, f)
        curl.perform()
        curl.close()
    

    The response is saved in a file named ouptut.html:

    output.html

    Uploading Files

    The HTTPPOST option can be used to upload a file, which replicates the file upload behaviors in an HTML form. You can upload data from a physical file using the code below:

    import pycurl
    
    curl = pycurl.Curl()
    curl.setopt(curl.URL, 'https://www.example.com/upload')
    
    curl.setopt(curl.HTTPPOST, [
        ('fileupload', (
            # upload the contents of this file
            curl.FORM_FILE, __file__ ,
        )),
    ])
    
    curl.perform()
    curl.close()
    

    If the data is in memory, instead of using curl.FORM_FILE, use curl.FORM_BUFFER:

    importpycurlc = pycurl.Curl()
    curl.setopt(c.URL, 'https://www.example.com/upload')
    
    curl.setopt(curl.HTTPPOST, [
        ('fileupload', (
            curl.FORM_BUFFER, 'output.txt',
            curl.FORM_BUFFERPTR, 'The data is in memory',
        )),
    ])
    
    curl.perform()
    curl.close()
    

    For other operations like examining response headers, following redirects, and examining responses, refer to the PycURL documentation.

    Conclusion

    cURL is a valuable tool for data transfer and offers a wide range of capabilities. Whether you prefer running cURL commands directly in the terminal/command prompt or integrating them into your Python code, the choice ultimately depends on your specific requirements.

    Throughout this article, we have explored how to use cURL in Python, with a focus on making HTTP requests. By incorporating cURL into your Python projects, you can unlock a multitude of benefits. So, don't hesitate to harness the power of cURL in Python and enhance your ability to handle data transfer tasks with ease!

    About the authorJosephine Loo
    Josephine is an automation enthusiast. She loves automating stuff and helping people to increase productivity with automation.

    Automate & Scale
    Your Web Scraping

    Browserbear helps you get the data you need to run your business, with our nocode task builder and integrations

    How to Use cURL in Python (with Examples)
    How to Use cURL in Python (with Examples)