[KPhotoAlbum] Rebuilding thumbnails is not an exercise for the faint of heart

[Robert Krawitz]

Certainly there should be a warning issued when someone
requests rebuilding all of the thumbnails; accidentally doing that
might be very time consuming. It doesn't render the program useless
while it's going on, but it is certainly a big inconvenience.

You're totally right! I thought we had a warning message like "This could take a lot of time"? I'll check that.
For quite some time, KPA can generate the thumbnails on request, if
it's needed and not there, it's generated. We (mostly Johannes ;-)
implemented this back then for this very reason that rebuilding the
whole cache is a big task.

What I basically wanted to do was clean up my thumbnails (I had a lot
of bogus ones from some development work I was doing). It's not
something I routinely need to do.

[Johannes Zarl-Zierl]

Post by Robert Krawitz
What I basically wanted to do was clean up my thumbnails (I had a lot
of bogus ones from some development work I was doing). It's not
something I routinely need to do.

One item on my todo-list is a utility program to interact with the thumbnail database. That
should take care of your specific problem, and make life easier for users with higher
automation/scripting needs. I envision something that can:

- compact the database by purging outdated/unreferenced thumbnails
- generate thumbnails for a list of image files or a directory
- (maybe) import or export freedesktop.org-style thumbnails

That program is not a terribly big effort to write, but I'm currently barely able to keep up with
answering my emails ;-)

Cheers,
Johannes

[Robert Krawitz]

Post by Johannes Zarl-Zierl

Post by Robert Krawitz
What I basically wanted to do was clean up my thumbnails (I had a lot
of bogus ones from some development work I was doing). It's not
something I routinely need to do.

- compact the database by purging outdated/unreferenced thumbnails

Yep.

Post by Johannes Zarl-Zierl
- generate thumbnails for a list of image files or a directory

What would this use case be?

Post by Johannes Zarl-Zierl
- (maybe) import or export freedesktop.org-style thumbnails

That would be very useful for some people.

Post by Johannes Zarl-Zierl
That program is not a terribly big effort to write, but I'm currently barely able to keep up with answering my emails ;-)

One thing that might help, but it would absolutely need back
compatibility support (or a utility to do the conversion) would be to
store the pathname along with the JPEG in the thumbnail file, so it
would be possible to read the thumbnails even if the thumbnail index
file is corrupted.

[Johannes Zarl-Zierl]

Post by Robert Krawitz

Post by Johannes Zarl-Zierl
- generate thumbnails for a list of image files or a directory

What would this use case be?

You can generate thumbnails from a cron-job or from an import script. In an NFS setting, you
could possibly do that even on the server, where running a gui program is usually not much
of an option.

Post by Robert Krawitz

Post by Johannes Zarl-Zierl
- (maybe) import or export freedesktop.org-style thumbnails

That would be very useful for some people.

Post by Johannes Zarl-Zierl
That program is not a terribly big effort to write, but I'm currently
barely able to keep up with answering my emails ;-)

One thing that might help, but it would absolutely need back
compatibility support (or a utility to do the conversion) would be to
store the pathname along with the JPEG in the thumbnail file, so it
would be possible to read the thumbnails even if the thumbnail index
file is corrupted.

I'll keep that in mind...

Cheers,
Johannes

[Robert Krawitz]

Post by Johannes Zarl-Zierl

Post by Robert Krawitz

Post by Johannes Zarl-Zierl
- generate thumbnails for a list of image files or a directory

What would this use case be?

You can generate thumbnails from a cron-job or from an import
script. In an NFS setting, you could possibly do that even on the
server, where running a gui program is usually not much of an
option.

Got it, although with the current Load-Performance branch there's
little or no penalty for buliding thumbnails at image load time.
There might be some overhead from the repeated file opens on NFS, but
I don't expect any significant bulk I/O transfer since the getattr()
shoud show no change.

As for import script, what I'd rather see in kpa is something that can
do the import and return filenames to kpa as it imports each file and
places it in the filesystem. That would allow further overlapping of
I/O while allowing the user to use an arbitrary import script.

This would require rewriting NewImageFinder::searchForNewFiles to be
an iterator rather than returning a list of files, with a minor change
in loadExtraFiles(). searchForNewFiles() would be a virtual function,
and subclasses would provide their own implementations. Most likely
there would be implementations for search filesystem (as present),
script-based import, and gphoto-based import.

It needs to be done in this way rather than import-and-batch-load to
take advantage of the I/O cache; otherwise, a large import would get
kicked out of memory just in time for the files to be needed again.

When I'm importing 5000 files from a doubleheader, optimizations like
these are important.