logo

Watch & Learn

Debugwar Blog

Step in or Step over, this is a problem ...

Solving the Issue of NodeJS Compression Library 'Compression' Not Working Effectively in Express

2025-03-27 @ UTC+0
en-USzh-CN

I have been made to optimize the loading speed of my blog recently, the most important part of this task is to compress content during transmission. I have searched online and found a NodeJS compression library named compression. However, after applying it, the expected result of compressing the transmitted content was not achieved. This article documents the troubleshooting process for failing to enable compression.

Usage of NodeJS Library compression

According to the official usage guideref1, we wrote the following demo:

  1. import express from 'express';  
  2. import compression from 'compression';  
  3.   
  4. const app = express();  
  5. app.use(compression());  
  6. app.get(  
  7.     '*',  
  8.     (request, response) => {  
  9.         response.end('<html><body>' + 'Hello World!'.repeat(1000) + '</body></html>');  
  10.     }  
  11. );  
  12. app.listen(  
  13.     5000,  
  14.     '0.0.0.0',  
  15.     () => console.log(`listening on port 5000 ...`)  
  16. )  

It was expected that by including an Accept-Encoding header in the request, the Content-Encoding of the response returned by Express would match one of the methods specified in the Accept-Encoding header. However, this was not the case in reality, as shown in the following request example:

  1. >> curl -v http://127.0.0.1:5000/ -H 'Accept-Encoding: gzip'  
  2. *   Trying 127.0.0.1:5000...  
  3. * Connected to 127.0.0.1 (127.0.0.1) port 5000  
  4. * using HTTP/1.x  
  5. > GET / HTTP/1.1  
  6. > Host: 127.0.0.1:5000  
  7. > User-Agent: curl/8.12.1  
  8. > Accept: */*  
  9. > Accept-Encoding: gzip  
  10. >   
  11. * Request completely sent off  
  12. < HTTP/1.1 200 OK  
  13. < X-Powered-By: Express  
  14. < Date: Thu, 27 Mar 2025 03:48:20 GMT  
  15. < Connection: keep-alive  
  16. < Keep-Alive: timeout=5  
  17. < Transfer-Encoding: chunked  
  18. <   
  19. * Connection #0 to host 127.0.0.1 left intact  
  20. <html><body>Hello World!……</body></html>  

As can be seen, the response was not compressed. The demo code comes from official document, why did this discrepancy occur?

Finding the Cause in the Source Code

Fortunately, the compression library is open source. Upon reviewing its source code, we discovered a function called shouldCompress:


Judging from the function name, its purpose is to determine whether compression is needed, with the decision based on the Content-Type.

这里涉及到一个Express的小知识参考2，Express的end调用并不会在响应中自动追加一些HTTP头，这也就直接导致shouldCompress函数根据Content-Type头取到的type为undefined，进而导致该函数判断响应内容不需要压缩。

到这里，解决方案就很明确了，新增对应的HTTP头或者使用会自动增加HTTP头的函数调用（例如send方法会自动添加该头）。这里我们采用增加HTTP头的方式：

This involves a small piece of knowledge about Expressref2, where calls to "end" function do not automatically append certain HTTP headers in the response, this directly leading to the shouldCompress function retrieving an undefined value from the Content-Type header, thereby this function returns a false result indicating that the content should not be compressed.

At this point, the solution became clear: add the corresponding HTTP header or use a function call that automatically adds the HTTP header (for example, the send method will automatically add this header). Here, we opted for the approach of adding the HTTP header:

  1. app.get(
  2.     '*',
  3.     (request, response) => {
  4.         response.type('text/html');
  5.         response.end(...);
  6.     }
  7. );

Note: The length of the response content in response.end should be sufficient; if too short, it may also lead to the content not being compressed (because the compressed version might be larger than the original).

Now, let's look at the returned content:

  1. >> curl -v http://127.0.0.1:5000/ -H 'Accept-Encoding: gzip'  
  2. *   Trying 127.0.0.1:5000...  
  3. * Connected to 127.0.0.1 (127.0.0.1) port 5000  
  4. * using HTTP/1.x  
  5. > GET / HTTP/1.1  
  6. > Host: 127.0.0.1:5000  
  7. > User-Agent: curl/8.12.1  
  8. > Accept: */*  
  9. > Accept-Encoding: gzip  
  10. >   
  11. * Request completely sent off  
  12. < HTTP/1.1 200 OK  
  13. < X-Powered-By: Express  
  14. < Content-Type: text/html; charset=utf-8  
  15. < Vary: Accept-Encoding  
  16. < Content-Encoding: gzip  
  17. < Date: Thu, 27 Mar 2025 04:20:48 GMT  
  18. < Connection: keep-alive  
  19. < Keep-Alive: timeout=5  
  20. < Transfer-Encoding: chunked  
  21. <   
  22. Warning: Binary output can mess up your terminal. Use "--output -" to tell curl to output it to your terminal anyway, or consider "--output <FILE>" to save to a file.  
  23. * client returned ERROR on write of 10 bytes  
  24. * Failed reading the chunked-encoded stream  
  25. * closing connection #0  

The output indicates that it has become binary, and the corresponding Content-Encoding header is present in the HTTP headers.

However, as debugging continued, I found that some parts of the content were still not being compressed.

Other Cases Where Compression Does Not Occur

Firstly, let's take a look at the following code:

  1. import fs from 'fs';  
  2. import express from 'express';  
  3. import compression from 'compression';  
  4.   
  5. const app = express();  
  6. app.use(compression());  
  7. app.get(  
  8.     '*',  
  9.     (request, response) => {  
  10.         response.type('image/jpeg');  
  11.         response.end(fs.readFileSync('./example.jpeg'));  
  12.     }  
  13. );  
  14. app.listen(  
  15.     5000,  
  16.     '0.0.0.0',  
  17.     () => console.log(`listening on port 5000 ...`)  
  18. )  

It's essentially no different from the code at the beginning of this article, just changing the returned content to an image (binary data), and this time using the type function to mark the content as 'image/jpeg' type in the returned HTTP header.

According to the analysis above, this content should be compressed. However, in reality, it was not compressed:

  1. >> curl -v http://127.0.0.1:5000/ -H 'Accept-Encoding: gzip' | hexdump -C | head -3  
  2.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current  
  3.                                  Dload  Upload   Total   Spent    Left  Speed  
  4.   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:5000...  
  5. * Connected to 127.0.0.1 (127.0.0.1) port 5000  
  6. * using HTTP/1.x  
  7. > GET / HTTP/1.1  
  8. > Host: 127.0.0.1:5000  
  9. > User-Agent: curl/8.12.1  
  10. > Accept: */*  
  11. > Accept-Encoding: gzip  
  12. >   
  13. * Request completely sent off  
  14. < HTTP/1.1 200 OK  
  15. < X-Powered-By: Express  
  16. < Content-Type: image/jpeg  
  17. < Date: Thu, 27 Mar 2025 04:36:08 GMT  
  18. < Connection: keep-alive  
  19. < Keep-Alive: timeout=5  
  20. < Transfer-Encoding: chunked  
  21. <   
  22. { [32588 bytes data]  
  23. 100 67275    0 67275    0     0  43.1M      0 --:--:-- --:--:-- --:--:-- 64.1M  
  24. * Connection #0 to host 127.0.0.1 left intact  
  25. 00000000  ff d8 ff e0 00 10 4a 46  49 46 00 01 01 01 00 48  |......JFIF.....H|  
  26. 00000010  00 48 00 00 ff db 00 43  00 03 02 02 03 02 02 03  |.H.....C........|  
  27. 00000020  03 03 03 04 03 03 04 05  08 05 05 04 04 05 0a 07  |................|  

There is no Content-Type header in response, the content is directly a jpeg image.

What caused this content not to be compressed? It seems we need to continue searching for answers within the source code of compression.

Back to the Source Code

Did you notice that in the shouldCompress function above, there was another function call we just ignored:

  1. function shouldCompress (req, res) {  
  2.   var type = res.getHeader('Content-Type')  
  3.   
  4.   if (type === undefined || !compressible(type)) {  
  5.     debug('%s not compressible', type)  
  6.     return false  
  7.   }  
  8.   
  9.   return true  
  10. }  

So, what does the compressible function do?

  1. var db = require('mime-db')  
  2.   
  3. function compressible (type) {  
  4.   if (!type || typeof type !== 'string') {  
  5.     return false  
  6.   }  
  7.   
  8.   // strip parameters  
  9.   var match = EXTRACT_TYPE_REGEXP.exec(type)  
  10.   var mime = match && match[1].toLowerCase()  
  11.   var data = db[mime]  
  12.   
  13.   // return database information  
  14.   if (data && data.compressible !== undefined) {  
  15.     return data.compressible  
  16.   }  
  17.   
  18.   // fallback to regexp or unknown  
  19.   return COMPRESSIBLE_TYPE_REGEXP.test(mime) || undefined  
  20. }  

In fact, it checks the obtained content type against the mime-db library, which is a json file containing information on various MIME file types. For instance, the information for the JPEG file involved this time is as follows:

  1. {  
  2.     ....  
  3.     image/jpeg: {       
  4.       source: iana,     
  5.       compressible: false,  
  6.       extensions: [jpeg,jpg,jpe]  
  7.     },  
  8.     ....  
  9. }  

With compressible set to false - it turns out that JPEGs are not compressible.

According to wiki, the JPEG format is a lossy compression image formatref3 - meaning that this type of image has already been compressed, so compressing it again would be meaningless.

Therefore, not all files transmitted over HTTP are compressible. Just because there is no compression option in the Content-Type does not mean there is a problem with compression.

Final Effect

As shown in the figure below, it is compressed to within 10 kB while the original file is around 70kB, this significantly reducing bandwidth pressure under high website traffic conditions.


Well, another casual article written, feeling quite happy ;)

References:

  1. NpmJS's compression page
  2. What is the difference between res.end() and res.send()?
  3. JPEG image format
Catalog
Usage of NodeJS Library compression
Finding the Cause in the Source Code
Other Cases Where Compression Does Not Occur
Back to the Source Code
Final Effect
References:

CopyRight (c) 2020 - 2025 Debugwar.com

Designed by Hacksign