Image upload fails because of high picture resolution

alessandro · May 27, 2019, 12:31pm

Hi, when I try to upload an asset with a big resolution the upload fails. It is not about the dimension of the picture in terms of Mb but its resolution. I’m using Cockpit 0.9.2. How to set cockpit to accept these kind of pictures?

Here are the pictures:
High resolution: https://i.ibb.co/ryJ4Vsh/High-resolution.jpg
Low resolution: https://i.ibb.co/WKsYWrG/Low-resolution.jpg

pauloamgomes · May 27, 2019, 2:00pm

Hi Alessandro, I’m pretty sure that you have another kind of error, the image that you provided is a standard image (640x1280) and it can be uploaded without any issue:

alessandro · May 28, 2019, 2:37am

Hi, this is actually the picture that is generating the error during the upload.

pauloamgomes · May 28, 2019, 2:29pm

it’s the same on my side, file uploads as expected:

alessandro · May 28, 2019, 3:20pm

I can upload it as a file but not as an asset. Also when I upload it as a file the upload is successful but I can’t really see the image preview from cockpit. How to see a log of what’s going on?

pauloamgomes · May 28, 2019, 4:08pm

If you check the screenshot it was created as an asset… and can see the image preview, maybe something related with the combination of your PHP version and installed libraries…

raffaelj · May 30, 2019, 9:31am

What is your memory_limit in php.ini? Does it work, if you increase it a bit?

I had that issue when fiddling with auto rotation by exif data and I had to increase the limit from 128M to 256M. Otherwise thumbnail creation failed for jpeg files >4MB.

alessandro · May 31, 2019, 2:21am

Thank you for the answer. I created a php file that execute phpinfo() to see what is the value of memory_limit. The value is 128M

raffaelj · June 1, 2019, 9:17pm

I tried to upload your image on my local machine and the same issue happened. After changing memory_limit to 400M or greater, it worked, but I had to wait a while until the upload was finished. Thumbnail creation took a while, too. But this is heavy… I’m not sure, if this is an issue with the SimpleImage library or if it is cockpit’s fault.

raffaelj · June 1, 2019, 11:07pm

OK, I dug into the code and did some debugging. If the memory_limit isn’t high enough, the script stops when it tries to find the main colors of an image with the ColorThief library. It stops before an Exception is fired.

github.com

agentejo/cockpit/blob/next/modules/Cockpit/module/assets.php#L77


if ($asset['image'] && !preg_match('/\.svg$/i', $file)) {


    $info = getimagesize($file);
    $asset['width']  = $info[0];
    $asset['height'] = $info[1];
    $asset['colors'] = [];


    if ($asset['width'] && $asset['height']) {


        try {
            $asset['colors'] = \ColorThief\ColorThief::getPalette($file, 5, ceil(($asset['width'] * $asset['height']) / 10000));
        } catch (\Exception $e) {
            $asset['colors'] = [];
        }


        foreach ($asset['colors'] as &$color) {
            $color = sprintf("#%02x%02x%02x", $color[0], $color[1], $color[2]);
        }
    }
}

Interestingly, sometimes I was able to upload your test image with memory_limit=256M, when I tried to upload it twice. In the first run, the upload failed and the second run worked.

I dug a bit deeper… The script stops when calling imagecreatefromjpeg(), which is part of PHP GD.

github.com

agentejo/cockpit/blob/next/lib/vendor/ksubileau/color-thief-php/lib/ColorThief/Image/Adapter/GDImageAdapter.php#L43


public function loadFile($file)
{
    list(, , $type) = @getimagesize($file);


    switch ($type) {
        case IMAGETYPE_GIF:
            $resource = @imagecreatefromgif($file);
            break;


        case IMAGETYPE_JPEG:
            $resource = @imagecreatefromjpeg($file);
            break;


        case IMAGETYPE_PNG:
            $resource = @imagecreatefrompng($file);
            break;


        default:
            throw new \RuntimeException("Image '{$file}' is not readable or does not exists.");
            break;
    }

So you could solve the issue with increasing the memory limit or if you use a smaller image.

Maybe the problem disappears, if you enable Imagick. I didn’t test this.

In the long run, there should be a check in cockpit’s core, to skip ColorThief when the images are too big.

alessandro · June 3, 2019, 12:51am

@raffaelj Thank you so much for this deep analysis.

I’m constantly having problems with image upload, and this is a little bit frustrating.

So you could solve the issue with increasing the memory limit or if you use a smaller image.

What should be a reasonable memory limit to set in order to don’t have this problem again?

thank you,

raffaelj · June 3, 2019, 11:25am

Short answer: memory_limit=512M

Long answer:

When your image is processed, each pixel needs some memory, so the problem is not the compression, but the amount of pixels and channels.

A simple rule is width x height x 4 (rgba) = needed memory in bytes, but that rule may break in some scenarios. I read somewhere, to use a multiplier of 5 to be on the save side. Let’s stick with 4.

An example with your image:

6811 * 13622 * 4 = 371117768 = 354 MiB

Cockpit needed 2 MiB already and the image processing needs 21 MiB extra. Give it some extra space, because you never know - let’s say 128 MiB.

So, 354 + 2 + 21 + 128 = 505 MiB

Conclusion

PHP GD is not the best option to process images. It has some troubles with transparent images (but that’s another story), it has inconsistencies with bundled and non-bundled PHP versions (even another story) and it may stop the script due to high RAM usage without any error message. But it exists, a lot of libraries use it, there is no native alternative, because not all (shared) hosts provide ImageMagick and it works in most cases.

If you really need to process very large images server side, you should consider increasing the memory_limit even more or you should build a workaround to process them in the background or client side.

Disabling ColorThief or switching to ImageMagick won’t solve the problem. Every time, a new thumbnail is created, the SimpleImage library is called, which uses the original image as base, uses GD and has the same problem: imagecreatefromjpeg() needs a lot of RAM.

alessandro · June 4, 2019, 2:49am

I increased it to 512M but didn’t solve the problem. Is there an integration with Cloudinary or something like that?

raffaelj · June 4, 2019, 9:47am

Hmm… I don’t know, what could cause your issue, other than that. Maybe you have another RAM hungry process somewhere, that doesn’t cleanup correctly or your system setup is different and something uses more RAM here and there.

Can you double check, if your memory_limit change took effect?
Did you try it with more than 512? If that doesn’t work, the issue might be somewhere else.

If you want to reproduce my debugging procedure, you can add my simple debug log code to /config/bootstrap.php and than call debug(memory_get_usage()); before and after the imagecreatefromejpeg() parts to see the memory peaks.

I don’t know, how and if it works: