A Complete Guide to Making HTTP Requests in Node.js

Making HTTP requests in Node is one of the most important thing you would be doing as a developer and inasmuch as it is a simple to perform, there is still a lot to understand or probably be aware of. HTTP requests is all about transferring data over a network which is built on a client-server model.

The underlying technology is the HTTP (Hypertext Transfer Protocol) which is used to structure requests and responses over the Internet. It was originally designed to allow communication between web servers and web browsers with just sending html but its protocol is now used hugely for a variety of purposes. Well, our focus in this article is HTTP requests which is the protocol where a piece of data is requested from a server.

In this article, we will discuss more on HTTP, HTTP requests, ways to make this requests in Node and how we can handle errors.

Prerequisites

• This article will require you to have Node installed on your computer and all examples here will be running on Node v14.15.4, so if you have this or above as at the time of reading this article then you are good.
• Also, a basic knowledge of JavaScript and how to run each code sample is needed to understand the content of this article.

What is HTTP requests in Node

Before we talk about what HTTP requests are in Node, let us understand what requests are in general. Requests are a way to exchange information between a client and a server. The client and server has to be present before we can call it a request. The client has to be the one to initiate a request by which the server in return, send a response in accordance to the request.

Making a HTTP requests in Node is as important as anything and there are components we should consider to make a request in Node. The key specifications to describing how to make a HTTP request is in RFC 7230. The HTTP requests contains three elements which includes:

1. The request method which I called the components earlier.
2. The target, where the request is going to. This is usually the URL.
3. The protocol name and its version.

Let us look at the components we have to consider before we make a HTTP request. The first thing to note is that we are sending a request to a URL and there are some actions that need to be sent alongside that URL.

• Method (GET, POST, PUT, DELETE, etc): For example, if we make a request to a URL on our browser like https://:www.google.com, we are using the GET method and our response will definitely be the resource(what we see on our browser)

Request URL: https://www.google.com/
Request Method: GET
Status Code: 200 
Remote Address: ******
Referrer Policy: origin

In the above case, we did not perform anything serious but just requested for data. The methods here are just conventions for what needs to be done on the server. The POST sends data to the server to be saved, PUT also sends to the server but updates the resource already on the server, and DELETE deletes a resource on the server. 

• Data: This is another important component when making a request because there is a high chance you will be sending data to the server. If you are going to be using the GET method, you might not necessarily send a data. But with other methods like POST, you must be sending a data.

POST /test HTTP/1.1
Host: foo.example
Content-Type: application/x-www-form-urlencoded
Content-Length: 27

name=Victor&email=emailaddress

• In the above case, we are sending the data(name and email) to the server for a purpose known to the client.

• Headers: Headers are what I call additional information or metadata that is sent alongside the request. They do not also work with the request but with the response as well. Inasmuch as it sounds like it is less used, it is still very much needed to provide information about the subject of the request. 

• So, the headers are mostly written by the client (you) while others are automatically done. Like I said earlier, headers might be somewhat important but this depends on the type of API you are working with. Below is a sample from MDN on how headers should look like:

GET /home.html HTTP/1.1
Host: developer.mozilla.org
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://developer.mozilla.org/testpage.html
Connection: keep-alive
Upgrade-Insecure-Requests: 1
If-Modified-Since: Mon, 18 Jul 2016 02:36:04 GMT
If-None-Match: "c561c68d0ba92bbeb8b0fff2a9199f722e3a621a"
Cache-Control: max-age=0

To find out more on the Request headers field, you can read more on RFC 7231, section 5.

• Authentication: We still have to call this a request component even if we can use it or do what it does on the Headers or as a Data. More often you will be sending a token to the server so the server can authenticate and know who you are before giving you access to the resource.

Authorization:  

This is just short overview of a HTTP requests components. With this in mind, we can move on to discuss how to make these HTTP requests in Node.

Ways to make HTTP requests in Node

Since our main focus is Node, we will take a look at 5 ways to make HTTP GET  and POST request. Our aim is to look at the various ways and the similarities or better still a more convenient way to handle requests. I also want to point out that this is not supposed to tell you which is better but rather which is convenient for you.

1. HTTP Module 

This is a built in standard library or module that is basically used to build a HTTP client and server. For example, we can create a Web server that listens to HTTP requests and we can use the same module to make HTTP requests. 

Node provides http and https which are separate modules. The latter lets you communicate over SSL, encrypting the communication using a certificate. You might want to use the https when making a request that has the https url. Let’s look at how we can make a GET and POST requests using the https module.

const https = require("https");

https
  .get(`https://reqres.in/api/users`, resp => {
    let data = "";

    // A chunk of data has been recieved.
    resp.on("data", chunk => {
      data += chunk;
    });

    // The whole response has been received. Print out the result.
    resp.on("end", () => {
      let url = JSON.parse(data).message;
      console.log(url);      
    });
  })
  .on("error", err => {
    console.log("Error: " + err.message);
  });

In the above code, we make a GET request to the Dogs API, the resp is an object and inside that is two events that we are to listen to; the on data and on end. With the on data, we are streaming the data to us in chunks(pieces by pieces) and with the on end, we listen and parse our data when it is completed. We also have the error handler event on error that listens for errors.

Let’s look at making a POST request with the https module:

const https = require("https");

const options = {
  hostname: 'yourapi.com',
  port: 443,
  path: '/todos',
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': data.length
  }
}

https
  .request(options, resp => {
    // log the data
    resp.on("data", d => {
      process.stdout.write(d);
    });
  })
  .on("error", err => {
    console.log("Error: " + err.message);
  });

You can always use the https.request because it automatically uses the GET method if you don’t specify and also calls the on end event underhood.

2. Axios 

According to Axios, it is a promised-based HTTP client for the browser and Node as well. This means that it is a third party library that can be used on any JavaScript project. It uses XMLHttpRequest underhood to make requests and also does more of it in a smooth way. 

It features include intercepting requests and response, cancellation of request, transforming requests and response data, and the most important, it automatically changes the responses to JSON data.

To install Axios with npm, you run this command:

$ npm install axios

Let’s make a simple GET request and see it response.

const axios = require('axios').default;

async function getUsers() {
  try {
    const response = await axios.get(`https://reqres.in/api/users`);
    console.log(response);
  } catch (error) {
    console.error(error);
  }
}
getUsers()

First, we require the Axios library, make a request to the url with axios.get() and then log the response. This is a smooth request with less code because Axios does all the big stuffs for us.

Take a look at the response below:

{
  status: 200,
  statusText: 'OK',
  headers: {
    date: 'Sat, 26 Mar 2022 13:51:08 GMT',
    'content-type': 'application/json; charset=utf-8',
    'content-length': '996',
    connection: 'close',
    'x-powered-by': 'Express',
    'access-control-allow-origin': '*',
    etag: 'W/"3e4-2RLXvr5wTg9YQ6aH95CkYoFNuO8"',
    via: '1.1 vegur',
    'cache-control': 'max-age=14400',
    'cf-cache-status': 'HIT',
    age: '6320',
    'accept-ranges': 'bytes',
    'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
    'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=kXKzeaxshfg8WPvQvms%2FJ6YvtrgnJh3GzGw4O62LPjVjC6n24KQo6c24Tix1NHo6qfLO9V%2FLaOoqJi%2FHt2GQceMnobhDFRDIExmnDrD3kY%2FB%2Fim6tWp1BkGBi8E%3D"}],"group":"cf-nel","max_age":604800}',
    nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}',
    server: 'cloudflare',
    'cf-ray': '6f205bffa85f0871-SEA',
    'alt-svc': 'h3=":443"; ma=86400, h3-29=":443"; ma=86400'
  },
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      ...
      ...
      ...
  data: {
    page: 1,
    per_page: 6,
    total: 12,
    total_pages: 2,
    data: [ [Object], [Object], [Object], [Object], [Object], [Object] ],
    support: {
      url: 'https://reqres.in/#support-heading',
      text: 'To keep ReqRes free, contributions towards server costs are appreciated!'
    }
  }
}

Easy to understand and work with response.

With that, let’s make a POST request to the same API. 

const axios = require('axios').default;

const data = {
  "name": "victor",
  "job": "writer"
}

async function addUser(data) {
  try {
    const response = await axios.post(`https://reqres.in/api/users`, data);
    console.log(response);
  } catch (error) {
    console.error(error);
  }
}

addUser()

Our axios.post() takes in two parameters which are the url and the data to be sent to the server. Looking at the response below, we get status code context that it was sent successfully.

Hint: hit control+c anytime to enter REPL.

{
  status: 201,
  statusText: 'Created',
  headers: {
    date: 'Sat, 26 Mar 2022 14:07:58 GMT',
    'content-type': 'application/json; charset=utf-8',
    'content-length': '51',
    connection: 'close',
    'x-powered-by': 'Express',
    'access-control-allow-origin': '*',
    etag: 'W/"33-wWfab/HlR/+j60wjrpfaMpVmAek"',
    via: '1.1 vegur',
    'cf-cache-status': 'DYNAMIC',
    'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
    'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=WCBCWafrH4GtOHMHIwcxuA7XfKR3PnH3pIuyH44ugHT1hgudiMHwBBv3x8VIsW0WwxC5N6RPKUcO3IwptR99V5kp%2Bcb%2Fp4NJ9bHVQOvbUcK22YfHElZ72AtFk0w%3D"}],"group":"cf-nel","max_age":604800}',
    nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}',
    server: 'cloudflare',
    'cf-ray': '6f2074a4fd2a27d2-SEA',
    'alt-svc': 'h3=":443"; ma=86400, h3-29=":443"; ma=86400'
  },
  ...
  ...
  ...
  },
  data: { id: '210', createdAt: '2022-03-26T14:07:58.464Z' }
}

In summary, this way of making a requests looks easy to me because a lot of the heavy lifting is taken away from you and with a very nice response.

3. Got

This is another HTTP client library used for making requests in Node applications. It works almost like Axios but there a few differences in how they work. The first is that this library can not be used on the browser side i.e on the client side. It offers a Stream and Pagination API out of the box. The Got library is a native ESM which means that you have to use the import rather than CommonJS. Oh yeah, it is also a Promise-based API too.

To install Got with npm, you run this command:

$ npm install got

Looking at a GET a request with Got:

import got from 'got';

async function getUsers() {
  try {
   const response = await got.get('https://reqres.in/api/users').json();
    console.log(response.data);
  } catch (error) {
    console.error(error);
  }
}

getUsers()

Making a POST request is almost as easy:

import got from 'got';

const data = {
  "name": "victor",
  "job": "writer"
}

async function addUser(data) {
  try {
   const response = await got.get('https://reqres.in/api/users', 
    {
      json: data
    }).json();
    console.log(response.data);
  } catch (error) {
    console.error(error);
  }
}

addUser()

Got approach is quite similar to Axios as I had said earlier, the only difference is the json() at the end of it. This specifies that the data is in JSON.

4. Node-Fetch

This is another lightweight HTTP library that works just like the browser fetch API. It is quite popular with over 37m downloads from the NPM registry. Ideally, it provides consistency when working with the window.fetch in the browser, allows the use of native promises and async functions, request cancellation, and has useful extensions for response size limit or redirect limit. 

Installing with npm:

$ npm install node-fetch

With that, let us make a GET request to our api:

import fetch from 'node-fetch';

async function getUsers() {
    const response = await fetch('https://reqres.in/api/users');
    const data = await response.json();
    console.log(data);
}

getUsers()

For a POST request:

import fetch from 'node-fetch';


const data = {
  "name": "victor",
  "job": "writer"
}

async function addUser(data) {
    const response = await fetch('https://reqres.in/api/users', {
            method: 'post',
            body: JSON.stringify(data),
    });

    const data = await response.json();
    console.log(data);
}

addUser()

Working with this library is a lot like the fetch API actually and there is a little work you have to do like .json() and .stringify() you have to do on the data. Most times, you also have to set the headers yourself. 

5. SuperAgent

THis library is a VisionMedia project with over 29 million downloads per month. It is a composable and promise-based API that retries on failure, follows redirects, handles gzip, has a JSON mode, and can also cancel requests. 

You can install SuperAgent with this command: 

$ npm install superagent

Let us make a call to our API with SuperAgent with the async function:

const superagent = require('superagent');

async function getUsers() {
    try {
      const res = await superagent.get('https://reqres.in/api/users');
      console.log(res);
  } catch (err) {
      console.error(err);
  }
}

getUsers()

This almost looks like Axios if you remember, most especially the response gotten from the API. It also provides a text field along with JSON. Since SuperAgent is a promise-based API, we have to use a try/catch to handle the error.

Sending data to the server works clean too but you have to add the .send():

const superagent = require('superagent');

const data = {
  "name": "victor",
  "job": "writer"
}

async function addUser(data) {
 try {
    const res = await superagent.post('https://reqres.in/api/users').send(data);
    console.log(res);
  } catch (err) {
    console.error(err);
  }
}

addUser()

• SuperAgent is also easily extendible with plugins such as no-cache, superagent-mocker, superagent-node-http-timing and a few others. To use any of them, you import them, and use the .use() method to call them.

const nocache = require('superagent-no-cache');
const superagent = require('superagent');
const prefix = require('superagent-prefix')('/static');

superagent
  .get('/some-url')
  .use(prefix) // Prefixes *only* this request
  .use(nocache) // Prevents caching of *only* this request
  .end((err, res) => {
    // Do something
  });

Handling errors

Handling errors in HTTP requests will vary according to the application and even the client library used but we still have to factor some things in our requests. For example, if you make a request to a url, the wrong url actually with Axios, what do you get? You get unmanageable response with a response field of undefined. In that case, you come in here by adding a try/catch block to your code. For example:

async function talkToMe(reqBody) {
  try {
    let res = await Axios({
      method: 'post',
      url: 'https://api.com/to',
      data: reqBody
    });

    let data = res.data;
    return data;
  } catch (error) {
    console.log(error.response); 
    return error.response;
  }

}

Whatever error is gotten in the try section is caught in the catch section instantly. Or if you are working with callbacks, then you use the .catch() method. Most libraries provide these option to handle errors actually.

axios.get('/user/:victor')
  .catch(function (error) {
    if (error.response) {
    ...
    ...

There are also more errors that may occur, like when the site is down and you get a 50X error. Most libraries provide these option to handle errors actually.

Conclusion

We have sampled 5 ways of making HTTP requests in Node and from observations they basically do the same thing but some of them handle the bottleneck for you, most especially Axios and SuperAgent. But this is not that article to tell you which is better instead to show you all the common ways and allow you pick the most convenient. 

The main goal of HTTP request is to communicate between computers and the method, target and protocol should be taken into consideration. So, make do with this article by picking the most suitable one for your project.