[KPhotoAlbum] Load-performance branch

[Robert Krawitz]

If anyone wants to merge the load-performance branch, I suggest not
going beyond change set 157f8b10a60dd656f4c7c9bd23ca8357fa6df812. The
commits beyond that point are a bit more experimental, focusing on
more corner case items (in particular, image collections that span
multiple filesystems -- something that matters to people with large
image collections who for one reason or another aren't using RAID).

I probably should have branched off from that point...

[Tobias Leupold]

Post by Robert Krawitz
I probably should have branched off from that point...

Well, simply check out the last commit you consider stable and create a new
branch ;-)

[Robert Krawitz]

Post by Tobias Leupold

Post by Robert Krawitz
I probably should have branched off from that point...

Well, simply check out the last commit you consider stable and
create a new branch ;-)

Yeah, I eventually figured that out :-)

[Robert Krawitz]

BTW, has anyone else tested the load performance branch? This offers
enough improvements when loading images (functional -- image stacking
at load now fully works -- in addition to performance), and the
possibility of fixing some threading issues, that I'd like to get it
merged.

[Angel Lopez]

Dear Robert,

I'm trying to compile your Load-performance brach, but I'm getting compiler
errors:
First, I got a :

/home/angel/src/kphotoalbum-Load-performance/MainWindow/Window.cpp:1761:6:
error: no declaration matches ‘void
MainWindow::Window::slotOptimizeThumbnails()’
void MainWindow::Window::slotOptimizeThumbnails()
^~~~~~~~~~
/home/angel/src/kphotoalbum-Load-performance/MainWindow/Window.cpp:1761:6:
nota: no functions named ‘void MainWindow::Window::slotOptimizeThumbnails()’

I fixed that including a declaration in MainWindow.h

Now I got these errors:

In file included from /usr/include/qt/QtCore/qnamespace.h:43,
from /usr/include/qt/QtCore/qobjectdefs.h:48,
from /usr/include/qt/QtCore/qobject.h:46,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:
In member function 'void RemoteControl::RemoteConnection::dataReceived()':
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:52:
error: no matching function for call to 'asString(qint32&)'
<< ": Received " << qPrintable(id);
^~~~~~~~~~
In file included from /usr/include/qt/QtCore/qobject.h:47,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/usr/include/qt/QtCore/qstring.h:1862:23: note: candidate: 'const QString&
QtPrivate::asString(const QString&)' <near match>
inline const QString &asString(const QString &s) { return s; }
^~~~~~~~
/usr/include/qt/QtCore/qstring.h:1862:23: note: conversion of argument 1
would be ill-formed:
In file included from /usr/include/qt/QtCore/qnamespace.h:43,
from /usr/include/qt/QtCore/qobjectdefs.h:48,
from /usr/include/qt/QtCore/qobject.h:46,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:63:
error: invalid user-defined conversion from 'qint32' {aka 'int'} to 'const
QString&' [-fpermissive]
<< ": Received " << qPrintable(id);
^~
In file included from /usr/include/qt/QtCore/qobject.h:47,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/usr/include/qt/QtCore/qstring.h:829:5: note: candidate is:
'QString::QString(const char*)' <near match>
QString(const char *ch);
^~~~~~~
/usr/include/qt/QtCore/qstring.h:829:5: note: conversion of argument 1
would be ill-formed:
In file included from /usr/include/qt/QtCore/qnamespace.h:43,
from /usr/include/qt/QtCore/qobjectdefs.h:48,
from /usr/include/qt/QtCore/qobject.h:46,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:63:
error: invalid conversion from 'qint32' {aka 'int'} to 'const char*'
[-fpermissive]
<< ": Received " << qPrintable(id);
^~
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:52:
error: 'QString::QString(const char*)' is private within this context
<< ": Received " << qPrintable(id);
^~~~~~~~~~
In file included from /usr/include/qt/QtCore/qobject.h:47,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/usr/include/qt/QtCore/qstring.h:829:5: note: declared private here
QString(const char *ch);
^~~~~~~
In file included from /usr/include/qt/QtCore/qnamespace.h:43,
from /usr/include/qt/QtCore/qobjectdefs.h:48,
from /usr/include/qt/QtCore/qobject.h:46,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:63:
error: invalid conversion from 'qint32' {aka 'int'} to 'const char*'
[-fpermissive]
<< ": Received " << qPrintable(id);
^~
In file included from /usr/include/qt/QtCore/qobject.h:47,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/usr/include/qt/QtCore/qstring.h:829:25: note: initializing argument 1 of
'QString::QString(const char*)'
QString(const char *ch);
~~~~~~~~~~~~^~
/usr/include/qt/QtCore/qstring.h:1863:18: note: candidate: 'QString&&
QtPrivate::asString(QString&&)' <near match>
inline QString &&asString(QString &&s) { return std::move(s);
}
^~~~~~~~
/usr/include/qt/QtCore/qstring.h:1863:18: note: conversion of argument 1
would be ill-formed:
In file included from /usr/include/qt/QtCore/qnamespace.h:43,
from /usr/include/qt/QtCore/qobjectdefs.h:48,
from /usr/include/qt/QtCore/qobject.h:46,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:63:
error: invalid user-defined conversion from 'qint32' {aka 'int'} to
'QString&&' [-fpermissive]
<< ": Received " << qPrintable(id);
^~
In file included from /usr/include/qt/QtCore/qobject.h:47,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/usr/include/qt/QtCore/qstring.h:829:5: note: candidate is:
'QString::QString(const char*)' <near match>
QString(const char *ch);
^~~~~~~
/usr/include/qt/QtCore/qstring.h:829:5: note: conversion of argument 1
would be ill-formed:
In file included from /usr/include/qt/QtCore/qnamespace.h:43,
from /usr/include/qt/QtCore/qobjectdefs.h:48,
from /usr/include/qt/QtCore/qobject.h:46,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:63:
error: invalid conversion from 'qint32' {aka 'int'} to 'const char*'
[-fpermissive]
<< ": Received " << qPrintable(id);
^~
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:52:
error: 'QString::QString(const char*)' is private within this context
<< ": Received " << qPrintable(id);
^~~~~~~~~~
In file included from /usr/include/qt/QtCore/qobject.h:47,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/usr/include/qt/QtCore/qstring.h:829:5: note: declared private here
QString(const char *ch);
^~~~~~~
In file included from /usr/include/qt/QtCore/qnamespace.h:43,
from /usr/include/qt/QtCore/qobjectdefs.h:48,
from /usr/include/qt/QtCore/qobject.h:46,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:63:
error: invalid conversion from 'qint32' {aka 'int'} to 'const char*'
[-fpermissive]
<< ": Received " << qPrintable(id);
^~
In file included from /usr/include/qt/QtCore/qobject.h:47,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/usr/include/qt/QtCore/qstring.h:829:25: note: initializing argument 1 of
'QString::QString(const char*)'
QString(const char *ch);
~~~~~~~~~~~~^~
In file included from /usr/include/qt/QtCore/qnamespace.h:43,
from /usr/include/qt/QtCore/qobjectdefs.h:48,
from /usr/include/qt/QtCore/qobject.h:46,
from /usr/include/qt/QtCore/QObject:1,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.h:22,
from
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:19:
/home/angel/src/kphotoalbum-Load-performance/RemoteControl/RemoteConnection.cpp:108:52:
error: conversion to non-const reference type 'class QString&&' from rvalue
of type 'QString' [-fpermissive]
<< ": Received " << qPrintable(id);
^~~~~~~~~~
make[2]: *** [CMakeFiles/kphotoalbum.dir/build.make:3053:
CMakeFiles/kphotoalbum.dir/RemoteControl/RemoteConnection.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:69: CMakeFiles/kphotoalbum.dir/all]
Error 2
make: *** [Makefile:141: all] Error 2




Now, I have no idea about what to do. Could be related to Qt version?
I'm on Arch Linux and Qt-5.11

Regards, and thanks for your efforts to make kphotoalbum even better !!!


______________________________
----------- Angel ------------

Post by Robert Krawitz
BTW, has anyone else tested the load performance branch? This offers
enough improvements when loading images (functional -- image stacking
at load now fully works -- in addition to performance), and the
possibility of fixing some threading issues, that I'd like to get it
merged.
--
*** MIT Engineers A Proud Tradition http://mitathletics.com ***
Member of the League for Programming Freedom -- http://ProgFree.org
Project lead for Gutenprint -- http://gimp-print.sourceforge.net
"Linux doesn't dictate how I work, I dictate how Linux works."
--Eric Crampton
_______________________________________________
KPhotoAlbum mailing list
https://mail.kdab.com/mailman/listinfo/kphotoalbum

[Robert Krawitz]

Could you try revision 157f8b10a60dd656f4c7c9bd23ca8357fa6df812?

Sorry I didn't reply yesterday or today; my wife and I have been away
for several days due to a medical procedure on my father and I had
somewhat spotty connectivity.

[Robert Krawitz]

Oh !!! First thing is that your father gets better.

Thanks! He's doing well.

Concerning the software, I discovered, that this option in preferences
"Use embedded image in raw o medium sized raw"
with that option selected speed if ultra/fast (almost instant)

That makes a big difference. It allows fast JPEG decoding of the
thumbnail.
You don't need to try that revision; the setting that you found likely
accounts for the entire problem.

Total files: 857 skipped 0
Loaded 857 images in 196.089 seconds
by the way, this computer is an old laptop Celeron 1.2GHz ...

I can't evaluate that number without more context.

Your earlier posting indicated that these are 20 MP images
(5496x3670). I have a Canon 7DmkII, which produces images of that
size. RAW files from those are typically around 25 MB. If the images
are stored on a typical SATA HDD, that works out to about 109 MB/sec,
which is close to the throughput that such disks can typically
sustain. Since the process of loading images requires that the MD5
checksum be computed, and that requires reading every byte of the
images files, that would suggest that you're I/O-limited, just as you
should be.

I also don't know exactly what processor you're using. Can you get
the PassMark number for the CPU (https://www.cpubenchmark.net/)? In
my experience, that usually gives a pretty good rough approximation of
what kind of CPU performance you might expect.

For reference, if I put RAW images on a SATA SSD, I can load about
1500 of them in 100 seconds, or 75 seconds if I increase the number of
scout threads (20 files/sec). The decoding, EXIF parsing, and MD5
computation results in about 25% total CPU consumption on my system
(which has a PassMark rating a little under 9000); if my I/O system
were fast enough, I might therefore be able to load about 80
files/sec. Can you watch (with top) how busy your system is while
loading the images?

If you want to go back to current master and see how long it takes to
load images (you'll need to delete index.xml, .thumbnails, and
exif.db), it would make an interesting comparison.

[Robert Krawitz]

CPU Intel celeron SU2300 (776 points in cpu mark)
Hard drive is SSD
Images are Raw files from camera canon 70D (25MB, 5496x3670 (20.2 Mpixel))
test folder with only 857 images
Total files: 857 skipped 0
Loaded 857 images in 170 seconds
Thumbnail load is very fast. First 32 are already there, if I scroll down
(procesor usage attached)
kphotoalbum-5.3: (I think, it is last stable version)
Loaded 857 images in 246 seconds
Thumbnail load takes 325 seconds

So this is on ntfs, not ext4 or other native Linux filesystem? That
makes it harder to judge, especially since it looks like you were
running on battery for the load-performance test and AC for 5.3
(sometimes laptops go into lower power mode when running on battery).

You had essentially 100% CU utilization (user+system) for the
load-performance test, and got 5 images/sec. My tests got (depending
upon how many scout threads I used) 15 images/sec at 18% CPU
utilization and 20 images/sec at 25% CPU utilization. Either way,
that corresponds to about 80 images/sec if I could achieve 100% CPU
utilization (which I can't, due to being I/O limited).

My CPU has a PassMark rating of 8887; that's about 11-12x as fast, and
I (hypothetically) get about 16x as many images/sec. That's a bit
slower than I'd expect, but you're using NTFS which imposes additional
overhead (it appears to be a userland filesystem driver, for one). So
all in all I'd say you're probably pretty close.

BTW, which is the way to get the last version of code? (I'm not really used
to git) If I do a "git pull" inside the folder it says it is up-to-date.
But I can update, by using this method, the stable branch., for example,
today I got : "v5.3-54-gef8f68c5-dirty" (which is not the one used in the
comparative above)

What do you get from

git rev-parse @

? That tells you the revision you're currently on.

I haven't pushed anything to the load-performance branch for a while,
so you're probably up to date.

[Robert Krawitz]

Post by Angel Lopez
Dear Robert,
I have been testing your brach.
First launch is indeed much faster (my computer is not very fast anyway).
Displaying of thumbnails takes longer (more than 10 seconds per image) . I

This is when displaying the thumbnails (the small images that are
displayed maybe 100 per screen)? Each thumbnail takes 10 seconds to display?

Post by Angel Lopez
---------------------------------------------------------
libkdcraw: LibRaw progress: Reading metadata pass 1 of 2
libkdcraw: LibRaw progress: Reading RAW data pass 0 of 2
libkdcraw: LibRaw progress: Reading RAW data pass 1 of 2
libkdcraw: Disabling LibRaw highlights adjustments
libkdcraw: LibRaw progress: Scaling colors pass 0 of 2
libkdcraw: LibRaw progress: Scaling colors pass 1 of 2
libkdcraw: LibRaw progress: Pre-interpolating pass 0 of 2
libkdcraw: LibRaw progress: Pre-interpolating pass 1 of 2
libkdcraw: LibRaw progress: Interpolating pass 0 of 3
libkdcraw: LibRaw progress: Interpolating pass 1 of 3
libkdcraw: LibRaw progress: Interpolating pass 2 of 3
libkdcraw: LibRaw progress: Converting to RGB pass 0 of 2
libkdcraw: LibRaw progress: Converting to RGB pass 1 of 2
libkdcraw: LibRaw: data info: width= 5496 height= 3670 rgbmax= 255
libkdcraw: "/datos/Fotos/2018-06-01_Tesis Maria Cristina
Momblona/2018-06-01_012.CR2"
-- RAW DECODING SETTINGS --------------------------------
-- autoBrightness: true
-- sixteenBitsImage: false
-- brightness: 1
-- RAWQuality: 0
-- inputColorSpace: 0
-- outputColorSpace: 1
-- RGBInterpolate4Colors: false
-- DontStretchPixels: false
-- unclipColors: 0
-- whiteBalance: 1
-- customWhiteBalance: 6500
-- customWhiteBalanceGreen: 1
-- halfSizeColorImage: false
-- enableBlackPoint: false
-- blackPoint: 0
-- enableWhitePoint: false
-- whitePoint: 0
-- NoiseReductionType: 0
-- NoiseReductionThreshold: 0
-- enableCACorrection: false
-- caMultiplier: 0, 0
-- medianFilterPasses: 0
-- inputProfile: ""
-- outputProfile: ""
-- deadPixelMap: ""
-- whiteBalanceArea: QRect(0,0 0x0)
-- dcbIterations: -1
-- dcbEnhanceFl: false
-- eeciRefine: false
-- esMedPasses: 0
-- NRChrominanceThreshold: 0
-- expoCorrection: false
-- expoCorrectionShift: 1
-- expoCorrectionHighlight: 0
---------------------------------------------------------
libkdcraw: LibRaw progress: Reading metadata pass 1 of 2
libkdcraw: LibRaw progress: Reading RAW data pass 0 of 2
libkdcraw: LibRaw progress: Reading RAW data pass 1 of 2
libkdcraw: Disabling LibRaw highlights adjustments
libkdcraw: LibRaw progress: Scaling colors pass 0 of 2
libkdcraw: LibRaw progress: Scaling colors pass 1 of 2
libkdcraw: LibRaw progress: Pre-interpolating pass 0 of 2
libkdcraw: LibRaw progress: Pre-interpolating pass 1 of 2
libkdcraw: LibRaw progress: Interpolating pass 0 of 3
libkdcraw: LibRaw progress: Interpolating pass 1 of 3
libkdcraw: LibRaw progress: Interpolating pass 2 of 3
libkdcraw: LibRaw progress: Converting to RGB pass 0 of 2
libkdcraw: LibRaw progress: Converting to RGB pass 1 of 2
libkdcraw: LibRaw: data info: width= 5496 height= 3670 rgbmax= 255
libkdcraw: "/datos/Fotos/2018-06-01_Tesis Maria Cristina
Momblona/2018-06-01_019.CR2"
-- RAW DECODING SETTINGS --------------------------------
-- autoBrightness: true
-- sixteenBitsImage: false
-- brightness: 1
-- RAWQuality: 0
-- inputColorSpace: 0
-- outputColorSpace: 1
-- RGBInterpolate4Colors: false
-- DontStretchPixels: false
-- unclipColors: 0
-- whiteBalance: 1
-- customWhiteBalance: 6500
-- customWhiteBalanceGreen: 1
-- halfSizeColorImage: false
-- enableBlackPoint: false
-- blackPoint: 0
-- enableWhitePoint: false
-- whitePoint: 0
-- NoiseReductionType: 0
-- NoiseReductionThreshold: 0
-- enableCACorrection: false
-- caMultiplier: 0, 0
-- medianFilterPasses: 0
-- inputProfile: ""
-- outputProfile: ""
-- deadPixelMap: ""
-- whiteBalanceArea: QRect(0,0 0x0)
-- dcbIterations: -1
-- dcbEnhanceFl: false
-- eeciRefine: false
-- esMedPasses: 0
-- NRChrominanceThreshold: 0
-- expoCorrection: false
-- expoCorrectionShift: 1
-- expoCorrectionHighlight: 0
---------------------------------------------------------
libkdcraw: LibRaw progress: Reading metadata pass 1 of 2
libkdcraw: LibRaw progress: Reading RAW data pass 0 of 2
Aborting aboutToFinish handling.
libkdcraw: LibRaw progress: Reading RAW data pass 1 of 2
libkdcraw: Disabling LibRaw highlights adjustments
libkdcraw: LibRaw progress: Scaling colors pass 0 of 2
libkdcraw: LibRaw progress: Scaling colors pass 1 of 2
libkdcraw: LibRaw progress: Pre-interpolating pass 0 of 2
libkdcraw: LibRaw progress: Pre-interpolating pass 1 of 2
libkdcraw: LibRaw progress: Interpolating pass 0 of 3
libkdcraw: LibRaw progress: Interpolating pass 1 of 3
libkdcraw: LibRaw progress: Interpolating pass 2 of 3
libkdcraw: LibRaw progress: Converting to RGB pass 0 of 2
libkdcraw: LibRaw progress: Converting to RGB pass 1 of 2
libkdcraw: LibRaw: data info: width= 5496 height= 3670 rgbmax= 255
Hope this helps.
Or could be just something wrong with my setup and not with your code, of
course !!

[Johannes Zarl-Zierl]

Hi Robert,

I've (finally) started reviewing the load-performance-stable branch.

I'm still at the beginning, but no serious problems so far ;-)
I'm wondering why you start so many transactions in ExifDB, but I haven't touched that part
for a while, so quite possibly I just need to brush up on my QSqlQuery knowledge a bit...

Thanks for all the work!

Johannes

Post by Robert Krawitz
If anyone wants to merge the load-performance branch, I suggest not
going beyond change set 157f8b10a60dd656f4c7c9bd23ca8357fa6df812. The
commits beyond that point are a bit more experimental, focusing on
more corner case items (in particular, image collections that span
multiple filesystems -- something that matters to people with large
image collections who for one reason or another aren't using RAID).
I probably should have branched off from that point...

[Robert Krawitz]

Post by Johannes Zarl-Zierl
I've (finally) started reviewing the load-performance-stable branch.
I'm still at the beginning, but no serious problems so far ;-)

That's good to hear :-)

Post by Johannes Zarl-Zierl
I'm wondering why you start so many transactions in ExifDB, but I haven't touched that part for a while, so quite possibly I just need to brush up on my QSqlQuery knowledge a bit...

I'm not actaully using more than one transaction (m_insertTransaction)
per image scan. However, in order to simplify the load logic, I need
to be able to get at the insert query started within that transaction,
hence getInsertQuery.

Post by Johannes Zarl-Zierl
Thanks for all the work!
Johannes

Post by Robert Krawitz
If anyone wants to merge the load-performance branch, I suggest not
going beyond change set 157f8b10a60dd656f4c7c9bd23ca8357fa6df812. The
commits beyond that point are a bit more experimental, focusing on
more corner case items (in particular, image collections that span
multiple filesystems -- something that matters to people with large
image collections who for one reason or another aren't using RAID).
I probably should have branched off from that point...

[Johannes Zarl-Zierl]

Hi Robert,

And another mini-reviewing session ;-)

In void ImageManager::AsyncLoader::requestExit(), why don't you simply call wakeAll()
instead of calling wakeOne() n times?

Also, it might be worthwhile to simply connect the finished() signals of our threads to
deleteLater() so that we don't have to manually delete them.

Post by Robert Krawitz

Post by Johannes Zarl-Zierl
I'm wondering why you start so many transactions in ExifDB, but I haven't
touched that part for a while, so quite possibly I just need to brush up
on my QSqlQuery knowledge a bit...

I'm not actaully using more than one transaction (m_insertTransaction)
per image scan. However, in order to simplify the load logic, I need
to be able to get at the insert query started within that transaction,
hence getInsertQuery.

I was thinking more about stuff like this:
m_db.transaction();
QSqlQuery *query = getInsertQuery();
m_db.transaction();

My thinking is that normally each m_db.transaction() call should correspond to a call to
either m_db.commit() or m_db.rollback().

Cheers,
Johannes

[Robert Krawitz]

Post by Johannes Zarl-Zierl
And another mini-reviewing session ;-)
In void ImageManager::AsyncLoader::requestExit(), why don't you
simply call wakeAll() instead of calling wakeOne() n times?

Just not familiar enough with Qt, and I copied this code from
elsewhere in kpa.

Post by Johannes Zarl-Zierl
Also, it might be worthwhile to simply connect the finished()
signals of our threads to deleteLater() so that we don't have to
manually delete them.

I'll take a look at this. However, I must admit I don't entirely
understand all of the intricacies of QThreads.

Post by Johannes Zarl-Zierl

Post by Robert Krawitz

Post by Johannes Zarl-Zierl
I'm wondering why you start so many transactions in ExifDB, but I haven't
touched that part for a while, so quite possibly I just need to brush up
on my QSqlQuery knowledge a bit...

I'm not actaully using more than one transaction (m_insertTransaction)
per image scan. However, in order to simplify the load logic, I need
to be able to get at the insert query started within that transaction,
hence getInsertQuery.

m_db.transaction();
QSqlQuery *query = getInsertQuery();
m_db.transaction();
My thinking is that normally each m_db.transaction() call should
correspond to a call to either m_db.commit() or m_db.rollback().

This particular case (openInsertQuery) can go away; it's unused. And
with that, there's exactly one m_db.transaction() (in
startInsertTransaction), one m_db.commit() (in
commitInsertTransaction) and one m_db.rollback (in
abortInsertTransaction).

In order to minimize I/O, I need to hold a transaction open while
DB::NewImageFinder::loadExtraFiles() loads files. If I load all of
the files first to get their MD5 checksum and build thumbnails, and
then do a batch EXIF load, I have to read at least part of each file
twice, which is expensive. So unless I want to hoist the EXIF
database code directly into NewImageFinder (which isn't really where
it belongs), the m_db calls are going to be in separate calls. But
those calls are all made from loadExtraFiles(), so I think it's clean
enough.