logo

Watch & Learn

Debugwar Blog

Step in or Step over, this is a problem ...

Solving the Issue of NodeJS Compression Library 'Compression' Not Working Effectively in Express

2025-03-27 @ UTC+0
en-USzh-CN

I have been made to optimize the loading speed of my blog recently, the most important part of this task is to compress content during transmission. I have searched online and found a NodeJS compression library named compression. However, after applying it, the expected result of compressing the transmitted content was not achieved. This article documents the troubleshooting process for failing to enable compression.

Usage of NodeJS Library compression

According to the official usage guideref1, we wrote the following demo:

  1. import express from 'express';  
  2. import compression from 'compression';  
  3.   
  4. const app = express();  
  5. app.use(compression());  
  6. app.get(  
  7.     '*',  
  8.     (request, response) => {  
  9.         response.end('<html><body>' + 'Hello World!'.repeat(1000) + '</body></html>');  
  10.     }  
  11. );  
  12. app.listen(  
  13.     5000,  
  14.     '0.0.0.0',  
  15.     () => console.log(`listening on port 5000 ...`)  
  16. )  

It was expected that by including anAccept-Encoding header in the request, theContent-Encoding of the response returned by Express would match one of the methods specified in theAccept-Encoding header. However, this was not the case in reality, as shown in the following request example:

  1. >> curl -v http://127.0.0.1:5000/ -H 'Accept-Encoding: gzip'  
  2. *   Trying 127.0.0.1:5000...  
  3. * Connected to 127.0.0.1 (127.0.0.1) port 5000  
  4. * using HTTP/1.x  
  5. > GET / HTTP/1.1  
  6. > Host: 127.0.0.1:5000  
  7. > User-Agent: curl/8.12.1  
  8. > Accept: */*  
  9. > Accept-Encoding: gzip  
  10. >   
  11. * Request completely sent off  
  12. < HTTP/1.1 200 OK  
  13. < X-Powered-By: Express  
  14. < Date: Thu, 27 Mar 2025 03:48:20 GMT  
  15. < Connection: keep-alive  
  16. < Keep-Alive: timeout=5  
  17. < Transfer-Encoding: chunked  
  18. <   
  19. * Connection #0 to host 127.0.0.1 left intact  
  20. <html><body>Hello World!……</body></html>  

As can be seen, the response was not compressed. The demo code comes from official document, why did this discrepancy occur?

Finding the Cause in the Source Code

Fortunately, the compression library is open source. Upon reviewing its source code, we discovered a function calledshouldCompress:


Judging from the function name, its purpose is to determine whether compression is needed, with the decision based on theContent-Type.

This involves a small piece of knowledge about Expressref2, where calls to "end" function do not automatically append certain HTTP headers in the response, this directly leading to theshouldCompress function retrieving an undefined value from theContent-Type header, thereby this function returns afalse result indicating that the content should not be compressed.

At this point, the solution became clear: add the corresponding HTTP header or use a function call that automatically adds the HTTP header (for example, thesend method will automatically add this header). Here, we opted for the approach of adding the HTTP header:

  1. app.get(
  2.     '*',
  3.     (request, response) => {
  4.         response.type('text/html');
  5.         response.end(...);
  6.     }
  7. );

Note: The length of the response content inresponse.end should be sufficient; if too short, it may also lead to the content not being compressed (because the compressed version might be larger than the original).

Now, let's look at the returned content:

  1. >> curl -v http://127.0.0.1:5000/ -H 'Accept-Encoding: gzip'  
  2. *   Trying 127.0.0.1:5000...  
  3. * Connected to 127.0.0.1 (127.0.0.1) port 5000  
  4. * using HTTP/1.x  
  5. > GET / HTTP/1.1  
  6. > Host: 127.0.0.1:5000  
  7. > User-Agent: curl/8.12.1  
  8. > Accept: */*  
  9. > Accept-Encoding: gzip  
  10. >   
  11. * Request completely sent off  
  12. < HTTP/1.1 200 OK  
  13. < X-Powered-By: Express  
  14. < Content-Type: text/html; charset=utf-8  
  15. < Vary: Accept-Encoding  
  16. < Content-Encoding: gzip  
  17. < Date: Thu, 27 Mar 2025 04:20:48 GMT  
  18. < Connection: keep-alive  
  19. < Keep-Alive: timeout=5  
  20. < Transfer-Encoding: chunked  
  21. <   
  22. Warning: Binary output can mess up your terminal. Use "--output -" to tell curl to output it to your terminal anyway, or consider "--output <FILE>" to save to a file.  
  23. * client returned ERROR on write of 10 bytes  
  24. * Failed reading the chunked-encoded stream  
  25. * closing connection #0  

The output indicates that it has become binary, and the correspondingContent-Encoding header is present in the HTTP headers.

However, as debugging continued, I found that some parts of the content were still not being compressed.

Other Cases Where Compression Does Not Occur

Firstly, let's take a look at the following code:

  1. import fs from 'fs';  
  2. import express from 'express';  
  3. import compression from 'compression';  
  4.   
  5. const app = express();  
  6. app.use(compression());  
  7. app.get(  
  8.     '*',  
  9.     (request, response) => {  
  10.         response.type('image/jpeg');  
  11.         response.end(fs.readFileSync('./example.jpeg'));  
  12.     }  
  13. );  
  14. app.listen(  
  15.     5000,  
  16.     '0.0.0.0',  
  17.     () => console.log(`listening on port 5000 ...`)  
  18. )  

It's essentially no different from the code at the beginning of this article, just changing the returned content to an image (binary data), and this time using the type function to mark the content as 'image/jpeg' type in the returned HTTP header.

According to the analysis above, this content should be compressed. However, in reality, it was not compressed:

  1. >> curl -v http://127.0.0.1:5000/ -H 'Accept-Encoding: gzip' | hexdump -C | head -3  
  2.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current  
  3.                                  Dload  Upload   Total   Spent    Left  Speed  
  4.   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:5000...  
  5. * Connected to 127.0.0.1 (127.0.0.1) port 5000  
  6. * using HTTP/1.x  
  7. > GET / HTTP/1.1  
  8. > Host: 127.0.0.1:5000  
  9. > User-Agent: curl/8.12.1  
  10. > Accept: */*  
  11. > Accept-Encoding: gzip  
  12. >   
  13. * Request completely sent off  
  14. < HTTP/1.1 200 OK  
  15. < X-Powered-By: Express  
  16. < Content-Type: image/jpeg  
  17. < Date: Thu, 27 Mar 2025 04:36:08 GMT  
  18. < Connection: keep-alive  
  19. < Keep-Alive: timeout=5  
  20. < Transfer-Encoding: chunked  
  21. <   
  22. { [32588 bytes data]  
  23. 100 67275    0 67275    0     0  43.1M      0 --:--:-- --:--:-- --:--:-- 64.1M  
  24. * Connection #0 to host 127.0.0.1 left intact  
  25. 00000000  ff d8 ff e0 00 10 4a 46  49 46 00 01 01 01 00 48  |......JFIF.....H|  
  26. 00000010  00 48 00 00 ff db 00 43  00 03 02 02 03 02 02 03  |.H.....C........|  
  27. 00000020  03 03 03 04 03 03 04 05  08 05 05 04 04 05 0a 07  |................|  

There is noContent-Type header in response, the content is directly a jpeg image.

What caused this content not to be compressed? It seems we need to continue searching for answers within the source code of compression.

Back to the Source Code

Did you notice that in theshouldCompress function above, there was another function call we just ignored:

  1. function shouldCompress (req, res) {  
  2.   var type = res.getHeader('Content-Type')  
  3.   
  4.   if (type === undefined || !compressible(type)) {  
  5.     debug('%s not compressible', type)  
  6.     return false  
  7.   }  
  8.   
  9.   return true  
  10. }  

So, what does thecompressible function do?

  1. var db = require('mime-db')  
  2.   
  3. function compressible (type) {  
  4.   if (!type || typeof type !== 'string') {  
  5.     return false  
  6.   }  
  7.   
  8.   // strip parameters  
  9.   var match = EXTRACT_TYPE_REGEXP.exec(type)  
  10.   var mime = match && match[1].toLowerCase()  
  11.   var data = db[mime]  
  12.   
  13.   // return database information  
  14.   if (data && data.compressible !== undefined) {  
  15.     return data.compressible  
  16.   }  
  17.   
  18.   // fallback to regexp or unknown  
  19.   return COMPRESSIBLE_TYPE_REGEXP.test(mime) || undefined  
  20. }  

In fact, it checks the obtained content type against themime-db library, which is a json file containing information on various MIME file types. For instance, the information for the JPEG file involved this time is as follows:

  1. {  
  2.     ....  
  3.     image/jpeg: {       
  4.       source: iana,     
  5.       compressible: false,  
  6.       extensions: [jpeg,jpg,jpe]  
  7.     },  
  8.     ....  
  9. }  

Withcompressible set tofalse - it turns out that JPEGs are not compressible.

According to wiki, the JPEG format is a lossy compression image formatref3 - meaning that this type of image has already been compressed, so compressing it again would be meaningless.

Therefore, not all files transmitted over HTTP are compressible. Just because there is no compression option in theContent-Type does not mean there is a problem with compression.

Final Effect

As shown in the figure below, it is compressed to within 10 kB while the original file is around 70kB, this significantly reducing bandwidth pressure under high website traffic conditions.


Well, another casual article written, feeling quite happy ;)

References:

Catalog
Usage of NodeJS Library compression
Finding the Cause in the Source Code
Other Cases Where Compression Does Not Occur
Back to the Source Code
Final Effect
References:

CopyRight (c) 2020 - 2025 Debugwar.com

Designed by Hacksign