[X] My Assistant

I have been using a custom multi-threaded downloader to download original archive files in fragments. After obtaining the archive link, I first use a HEAD request to get the `Content-Length` from the header to determine the size of the archive, which allows me to calculate the data range for each thread. This process worked well for most galleries before January 14th, but after that, many galleries started experiencing issues.

For larger galleries (I’m not sure of the exact threshold, but approximately >200MB), when I send a HEAD request to the gallery link, the server takes a very long time to return the header, while a GET request to the same link usually returns and starts downloading within 1 second.

Examples:

Gallery: http://ehentaihip.com/g/3194782/31ce424f3b, Archive link: [xecemudyvf.hath.network] https://xecemudyvf.hath.network/archive/319...mac9t/2?start=1
HEAD request took 9 seconds, GET request took less than 1 second.

CODE

HEAD https://xecemudyvf.hath.network/archive/3194782/a2fe212c0974913fa1a55e8eb1cbcf3a4700ea96/6kqi1kmac9t/2?start=1

HTTP/1.1 200 OK
Date: Wed, 15 Jan 2025 12:11:12 GMT
Server: Apache
Cache-Control: no-cache, no-store, max-age=0
Content-Disposition: attachment; filename="(C105) [愉悦さる (おさる1号)] ハルナの色欲とらっぷ (ブルーアーカイブ) [中国翻訳].zip"
Content-Length: 409112170
Content-Transfer-Encoding: binary
Accept-Ranges: bytes
Connection: close
Content-Type: application/zip; charset=UTF-8

响应文件已保存。
> (C105) [愉悦さる (おさる1号)] ハルナの色欲とらっぷ (ブルーアーカイブ) [中国翻訳]-4.zip

Response code: 200 (OK); Time: 9199ms (9 s 199 ms); Content length: 0 bytes (0 B)

Gallery: http://ehentaihip.com/g/3195666/fef3617678, Archive link: [cipvxqwzvk.hath.network] https://cipvxqwzvk.hath.network/archive/319...6acac/2?start=1
HEAD request took 15 seconds, GET request took less than 1 second.|

CODE

HEAD https://cipvxqwzvk.hath.network/archive/3195666/5b705f1edcdcf5779fdc1fecd1b1ba99300dbb81/89rzr46acac/2?start=1

HTTP/1.1 200 OK
Date: Wed, 15 Jan 2025 12:12:39 GMT
Server: Apache
Cache-Control: no-cache, no-store, max-age=0
Content-Disposition: attachment; filename="[ChiHel]Patreon合集[Chinese][機翻潤色].zip"
Content-Length: 738959977
Content-Transfer-Encoding: binary
Accept-Ranges: bytes
Connection: close
Content-Type: application/zip; charset=UTF-8

响应文件已保存。
> [ChiHel]Patreon合集[Chinese][機翻潤色]-1.zip

Response code: 200 (OK); Time: 15989ms (15 s 989 ms); Content length: 0 bytes (0 B)

Gallery: http://ehentaihip.com/g/3194908/89fa825dc0, Archive link: [roxysxyymp.hath.network] https://roxysxyymp.hath.network/archive/319...cac9s/2?start=1
HEAD request took 15 seconds, GET request took less than 1 second.


For smaller galleries, HEAD requests generally return within 5 seconds, but the time taken is still significantly longer than GET requests.

Gallery: http://ehentaihip.com/g/3194169/7a2e53503f/, Archive link: [taopjgvkhp.hath.network] https://taopjgvkhp.hath.network/archive/319...aacac/2?start=1
HEAD request took 2 seconds, GET request took less than 1 second.

CODE

HEAD https://taopjgvkhp.hath.network/archive/3194169/4d5ca396e650c4f3f60ee4c760b0d4a56066c88a/rmvrckaacac/2?start=1

HTTP/1.1 200 OK
Date: Wed, 15 Jan 2025 12:13:09 GMT
Server: Apache
Cache-Control: no-cache, no-store, max-age=0
Content-Disposition: attachment; filename="[HBO (変熊)] 人格排泄 私を知る [中国翻訳] [DL版].zip"
Content-Length: 80975369
Content-Transfer-Encoding: binary
Accept-Ranges: bytes
Connection: close
Content-Type: application/zip; charset=UTF-8

响应文件已保存。
> [HBO (変熊)] 人格排泄 私を知る [中国翻訳] [DL版]-2.zip

Response code: 200 (OK); Time: 2171ms (2 s 171 ms); Content length: 0 bytes (0 B)

Before January 14th, this phenomenon also occurred with some archives, but it was very rare.

I would like to know if using HEAD requests to get the archive file size is still a viable approach for gallery archives. Is the current behavior considered normal? Is it the influence brought by /index.php?showt...p;#entry6618110?

This post has been edited by hishioten: Jan 15 2025, 14:43

I also realized that archive downloading doesn't work for large galleries in some unofficial apps.
The issue only appeared since yesterday

The archiver never had any actual support for HEAD, so it would basically have acted as if you downloaded the whole thing, and then the webserver stripped the actual content before sending just the header. But I suppose it makes sense to support it. It's an easy fix so I just added explicit support for it, which should make it much faster.

Let me know if you notice any issues with this.

The archiver never had any actual support for HEAD, so it would basically have acted as if you downloaded the whole thing, and then the webserver stripped the actual content before sending it. But I suppose it makes sense to support it. It's an easy fix so I just added explicit support for it, which should make it much faster.

Let me know if you notice any issues with this.

The results are very significant, and the HEAD request now seems to be working normally again. Thank you very much!