[KPhotoAlbum] Perf suggestion: build thumbnails as images are being read

[Robert Krawitz]

At present, kpa scans all of the images, and then builds thumbnails in
a separate pass. When reading a very large number of photos (such
they don't stay in RAM), this results in each image file having to be
read twice. That's somewhat time consuming for e. g. 22 GB of photos.

(Why 22 GB? RAW, and especially RAW+JPG, adds up fast with 20MP
cameras. I shot about 800 frames at a jazz jam this evening and am
killing time while kpa reads them in and builds thumbnails.)

(It would be even faster if kpa could read photos directly from a
source and place them in the image tree according to appropriate rules
as it's reading them in.

[Johannes Zarl-Zierl]

Hello Robert,

Post by Robert Krawitz
At present, kpa scans all of the images, and then builds thumbnails in
a separate pass. When reading a very large number of photos (such
they don't stay in RAM), this results in each image file having to be
read twice. That's somewhat time consuming for e. g. 22 GB of photos.

I think this will be tricky to pull off without major changes to KPA.

OTOH you are absolutely correct that this will probably be a relevant speedup,
both for users with RAW workflows and for users with NFS shares.
For optimal performance, though, all file access should be bundled to a single
pass: reading EXIF info and other metadata, recomputing checksums - this can
also be done more intelligently, AFAIK.

Post by Robert Krawitz
(It would be even faster if kpa could read photos directly from a
source and place them in the image tree according to appropriate rules
as it's reading them in.

This part should be far easier to implement: KPA just needs to look up the
thumbnail from the freedesktop.org thumbnail store first. Writing a shell
script that creates those thumbnails is straightforward. Bonus: dolphin and
gwenview get the same thumbnails without further processing.

Before someone asks: yes, we still need the KPhotoAlbum thumbnail database -
the performance of the freedektop.org degrades with a high number of
thumbnails (having a large number of files in a single directory is quite slow
for most filesystems).

Johannes

[Robert Krawitz]

Post by Johannes Zarl-Zierl
Hello Robert,

Post by Robert Krawitz
At present, kpa scans all of the images, and then builds thumbnails in
a separate pass. When reading a very large number of photos (such
they don't stay in RAM), this results in each image file having to be
read twice. That's somewhat time consuming for e. g. 22 GB of photos.

I think this will be tricky to pull off without major changes to KPA.
OTOH you are absolutely correct that this will probably be a relevant speedup, both for users with RAW workflows and for users with NFS shares.
For optimal performance, though, all file access should be bundled to a single pass: reading EXIF info and other metadata, recomputing checksums - this can also be done more intelligently, AFAIK.

It's something that would be worth thinking about.

Post by Johannes Zarl-Zierl

Post by Robert Krawitz
(It would be even faster if kpa could read photos directly from a
source and place them in the image tree according to appropriate rules
as it's reading them in.

This part should be far easier to implement: KPA just needs to look up the thumbnail from the freedesktop.org thumbnail store first. Writing a shell script that creates those thumbnails is straightforward. Bonus: dolphin and gwenview get the same thumbnails without further processing.

I'm not sure what you're referring to (in particular, the
freedesktop.org thumbnail store). In this case, I have a card reader
with my CF card. I use a shell script to copy the new images into my
kpa directory, and then run kpa. What I'm proposing is for kpa to be
able to do that image copy (perhaps by means of a plugin, if that's
possible in this case) and do the checksum, EXIF, and thumbnail
generation in the process of doing that copy.