Tag: kde

Fast SFTP transfers with KDE / kio-extras / Dolphin

For a few years now, I’ve used Dolphin / Konqueror / Krusader for transferring files to or from my home server via SFTP. However when there were larger amounts to be transferred, I opened the shell and used rsync. That’s because I felt it was always somewhat slower with SFTP. I blamed it on the weak file server, on the network, or whatever.

Recently I again wanted to clean up my desktop’s home directory, where a bunch of raw or edited videos were sitting. All in all ~250 GiB. Because I wanted to sort the videos into different sub-directories on the server, I used Dolphin and SFTP once again. While I was watching the transfer, I began to doubt whether the network between my desktop and the home server was actually Gigabit, because I only saw 10-15 MiB/s. Quick check with rsync and scp: nope, 80+ MB/s is possible, just not with Dolphin.

That’s when I decided to get to the bottom of the issue. I discovered KDE bug 296526 – Dolphin is too slow when upload a file on a SSH server. It fit exactly what I was observing, even though it’s from 2012. That’s 14 years! I went on to read all the comments, and it became apparent that none of the users that had commented had spent the time to do a proper side-by-side comparison, and document it properly. So I took up that task and posted comment 40, where I wrote pretty much what I wrote here, but also the results of some tests that I had done with ‘libssh’. That’s the underlying library which kio-extras/sftp uses to interact with SSH/SFTP servers. And Dolphin / Konqueror / Krusader in turn use kio-extras/sftp to do SFTP transfers.

These tests showed very promising results: I could actually saturate my Gigabit link and transfer >750 Mbps. What I didn’t know at that time (September): I had tested a brand new version of libssh (0.11.0), released in August, that came with major changes. Namely a new async I/O API had been added. The transfers with Dolphin had still used libssh 0.10.x though.

Upon learning about these important changes, I opened a version bump request for libssh 0.11.0 in Gentoo’s bug tracker to make the Gentoo devs aware of this new libssh version. Fast forward two months, libssh 0.11.1 was available in Gentoo. I then erroneously tested with kio-extras linked against libssh 0.10.x (and thus disabled new async I/O API), even though I had 0.11.1 on my system. The reason was, that I hadn’t rebuilt kio-extras. That resulted in comment 45. A few minutes later I realized my error, rebuilt kio-extras (actually most of KDE, because an update was coming in anyway), and voilà: ~230 MiB/s or ~1840 Mbps (with peaks going >2 Gbps)! Hurray!

Side note: I had upgraded my network from Gigabit to 2.5 Gigabit in the meantime, otherwise that would obviously not be possible. But with the old libssh only around ~90 MiB/s or ~720 Mbps were possible over the same network, 2.5x slower than with the new libssh 0.11.x.

So, if you find that your SFTP transfers in KDE are slower than they should be, check which libssh version comes with your distribution. If it’s <0.11.0, you know you need to upgrade. With non-rolling binary distributions, you’ll probably have to wait a bit and then upgrade your whole distribution. For example Ubuntu will only get libssh 0.11.1 in “Plucky” aka version 25.04. For rolling distributions like Gentoo or Arch it’s already available.

This is probably the biggest single improvement to my Linux on the Desktop experience of the last years… hence this blog post 🙂

Getting FRITZ!Box upload/download rate nicely displayed in a KDE Plasma panel

It’s been a while that I’ve been setting up a private computer. For a long time I only had my old laptop, where I had not much motivation to tinker and optimize. Now that I have my new desktop computer (yay!), I’ve spent some time arranging the workspace / desktop. And of course I’m using KDE, as I have been for so many years.

The Plasma 5 system monitor is pretty cool, because it’s very configurable, and each of the graphs can be placed as Plasma widget on the desktop or in any panel. So I set up some nice graphs for SSD I/O, network throughput, CPU, memory, temperatures etc.. But then I thought: Can’t I also get some info from my Internet router, e.g. the current download rate of the whole house (as this PC is by far not the only device using the Internet connection). Or when it last reconnected, and what the external IP address is.

So I looked a bit into the TR-064 protocol and found a really nice Python module – fritzconnection by Klaus Bremer – to interface with my router (an AVM FRITZ!Box 5530 Fiber). It should also work with pretty much any other FRITZ!Box model, which is a very common Internet router throughout Germany. You can simply get it by running pip install fritzconnection.

So after a few minutes I was able to query the desired data from the router with very few lines of code, like so:

os.environ['FRITZ_USERNAME'] = 'user'
os.environ['FRITZ_PASSWORD'] = 'password'
fc = FritzStatus(address='fritz.box')
print('WAN Download: ' + fc.str_transmission_rate[1])

I found that querying it more than once every 5 seconds doesn’t make sense, because it doesn’t update the numbers more often than that. So a loop with a 5 second sleep it is.

But I also wanted to get the data into my nice panel, together with the CPU load etc… And by chance I stumbled over the Do It Yourself Bar – funny name, but it kind of fits: It’s a bar-shaped Plasma widget that you can populate via DBUS messages through any kind of script – typically Shell scripts but of course Python scripts work just as well.

So I modified my “text-only” Python script to send DBUS messages and hooked it into the DIY bar – and suddenly I have the data queried from the router in my KDE panel, with configurable colors and all!

Running a speed test – here you can see the local network adapter’s data and below the Do It Yourself Bar with the FRITZ!Box data (I highlighted it with a yellow frame in this screenshot).

Hovering over the “FRITZ!Box conn. up: …” label also shows the external IP address, and clicking on it would open the browser with the FRITZ!Box’s web interface.

Here is the current Python script and my Pull Request.

#!/usr/bin/python

# Prerequisites:
# - Have a FRITZ!Box Internet router that is reachable at the address http://fritz.box
# - Add a user + password to your FRITZ!Box (via the web interface)
# - Run `pip install fritzconnection` to get the handy Python module by Klaus Bremer
#   (see https://fritzconnection.readthedocs.io)

import os, sys, time
from fritzconnection.lib.fritzstatus import FritzStatus

os.environ['FRITZ_USERNAME'] = 'scriptuser'
os.environ['FRITZ_PASSWORD'] = 'the_password'

fc = None
while not fc:
    try:
        fc = FritzStatus(address='fritz.box')
    except:
        data = '| A | FRITZ!Box not reachable | Have you changed username and password in get-fritz-wan-stats.py? Is your network up? | |'
        os.system('/usr/bin/qdbus org.kde.plasma.doityourselfbar /id_' + sys.argv[1] + ' org.kde.plasma.doityourselfbar.pass \'' + data + '\'')
        pass
    time.sleep(5)

while fc:
    data = '| A | FRITZ!Box conn. up: ' + fc.str_uptime + ' | External IP: ' + fc.external_ip + ' | xdg-open http://fritz.box |'
    data += '| B | WAN Download: ' + fc.str_transmission_rate[1] + '/s | FRITZ!Box WAN Download | |'
    data += '| C | WAN Upload: ' + fc.str_transmission_rate[0] + '/s | FRITZ!Box WAN Upload | |'
    os.system('/usr/bin/qdbus org.kde.plasma.doityourselfbar /id_' + sys.argv[1] + ' org.kde.plasma.doityourselfbar.pass \'' + data + '\'')
    time.sleep(5)

Pretty neat. Thanks a lot to Klaus Bremer and the other fritzconnection authors and wsdfhjxc for making this possible!