KDE Frameworks 5 Tech Preview released, with updated ThreadWeaver

Today, the KDE Community released a tech preview of the upcoming KDE 5 Frameworks, the new, modularised incarnation of what was previously distributed simply as the KDE libraries. The new frameworks are drop-in extensions to Qt applications, with minimal and well-documented dependencies for easier deployment. The tech preview contains two frameworks that are marked as mature, namely KArchive and ThreadWeaver. The updated ThreadWeaver was my major piece of library coding work in 2013, and was finished just in time for the release. Even though it is a tech preview, it is stable, and no major (or even significant but minor) changes in the current API are expected until the final release. Programmers are already encouraged to use it, and provide feedback and bug reports.

2897019812_c6bddd5fb1_oThreadWeaver is a concurrent execution scheduler written in C++. Available for all target platforms of the Qt framework, including desktop, mobile and embedded environments, ThreadWeaver delivers concurrent execution of tasks, load balancing with regard to user-defined criteria, multiple independent queues, processing graph modelling, aggregate jobs and other comprehensive features. As all other KDE frameworks, ThreadWeaver is Free Software. Its only dependency is Qt, which makes it a tier 1 framework in KDE’s lingo.

A number of the new features of ThreadWeaver were announced at Akademy 2013. Jobs, the unit of concurrent execution in ThreadWeaver, are now managed by the queue using shared pointers, meaning that auto-delete behaviour is implicit and controlled by the user. Helper templates are available to queue stack or member variables, so allocation of jobs can be static or dynamic. Functors or lambda functions can be used to construct jobs. Job aggregates like collections and sequences now execute their own run() method before queueing their elements, so that aggregates can generate their own elements. Success and queueing state of jobs are now integrated into a single status. Jobs can signal the result of execution by setting a status, but also using exceptions, simplifying error reporting in more complex job classes. Jobs can be decorated, and no more inherit QObject by default. Decorators can be used to add signals, change priorities or modify just about any behaviour of jobs independently of the actual job class used. The construction of the global queue can now be customised using a queue factory. The QueueStream API greatly simplifies queueing jobs with a familiar iostream-like C++ syntax.

ThreadWeaver follows the Unix idiom of doing one thing, and doing it right. Similar to how small Unix programs can be combined to create an practically infinite space of computing solutions, ThreadWeaver offers itself to programmers as an add-on module with minimal dependencies. Including it extends an application with concurrent scheduling capability. But the same Unix idiom is also applied in a second sense. Within ThreadWeaver, a few basic concepts – jobs and their aggregates, queues and policies – are implemented that again provide simple building blocks that can be combined creatively, offering a vast space of potential solutions within the scope of the application.

The history of ThreadWeaver goes back to KDE 3. The idea of implementing a thread pool based execution scheduler that manages dependencies between jobs was implemented as a proof of concept using Qt 3. However it turned out to be difficult to implement and use because of the lack of thread-safe reference counting of the implicitly shared classes at the time. These fundamental problems have been solved with the release of Qt 4. Additionally, the introduction of cross-thread signal-slot connections further simplified the communication between jobs and the application’s user interface. The first production ready version of ThreadWeaver was released as part of KDELibs with KDE 4.0. For KDE Frameworks 5, it was almost completely re-written to simplify memory management of jobs, make use of new Qt 5 features like atomic variables, and in part to reflect new language constructs in C++11 like lambda functions. ThreadWeaver comes with an extensive set of unit tests that all pass in the tech preview (hear, hear).

In the following weeks and months, the framework will be polished and debugged based on user feedback. Also, a series of posts here on this blog will introduce individual ThreadWeaver concepts and features in depth, mostly based on example programs, including contrasting it to thread handling in Qt using QThread or Qt Concurrent. ThreadWeaver is very close to production quality, having been tested continuously in the last couple of months. There may still be smaller, source compatible changes to the framework. We ask interested programmers out there to provide feedback and bug reports to make ThreadWeaver what it should be — a worry-free, easy to use and powerful add-on to Qt that programmers enjoy using. Have fun!

[Image by Shannan Sinclair, thanks: http://www.flickr.com/photos/originalbliss/2897019812%5D


9 responses to “KDE Frameworks 5 Tech Preview released, with updated ThreadWeaver

  1. Congratulations! However, the word “framework” conjures an image of smth rather large and complex. Even though it’s not a technically strict question, how “big” and “complex” is ThreadWeaver? What does it consist of? Also, what libraries and/or projects does ThreadWeaver compare to in the OpenSource and proprieatary worlds? OpenMP, MPI, some part of Boost, etc?

    Thank you!

    • ThreadWeaver is called framework because it is part of KDE Frameworks 5, not because it is big and complex. Actually, it got smaller with the current release because it was stripped to contain only essential functionality. As for complex, ThreadWeaver tries to make concurrent execution easier to manage for programmers. For example, it helps to keep the code that specifies what should be executed and how relatively local. However, concurrency has some inherent complexity. To completely encapsulate this complexity also means to limit what programmers can do with it. I tried to strike a balance here where basic concepts can be implemented more easily, but advanced approaches are also possible (see the job decorators or the possibility to use a customised global queue). As for what ThreadWeaver contains and how it compares to other Open Source toolkits, hang on for future posts. I will leave comparing it to proprietary toolkits to the reader, mostly out of lack of opportunity.

  2. Although I am only a user of KDE and know nothing about development, I am really happy that KDE is moving forward. I am really looking forward to what changes will be made and how it will all work together. Well done.

  3. Plasma Media Center team is quite eager to try out ThreadWeaver as we have lots of jobs to fetch media, metadata and what not. We just need to clone https://projects.kde.org/projects/frameworks/threadweaver/repository and start using it?

    • Yes, that is essentially it. Check the KDE Frameworks 5 guide for how to use the CMake find functions to include it. The tests should be quite helpful to find usage examples. Feel free to send questions my way.

  4. What is “ECM” that cmake needs when I run it? http://paste.kde.org/p5c85f50e
    Ubuntu has libecm-dev which doesn’t seem to help (description is “Factor integers using the Elliptic Curve Method (library) “)

    • ah it is https://projects.kde.org/projects/kdesupport/extra-cmake-modules/repository
      Also, our codebase runs on Qt4 right now and we’re waiting for Plasma2 to be out so that we do the Qt4->Qt5 and Plasma->Plasma2 migration at once. Till then, any suggestions to use threadviewer without porting the entire app to Qt5?

      • Unfortunately, the new iteration of ThreadWeaver relies on Qt 5. While it is possible to start out with using the version of ThreadWeaver that is included in kdelibs for KDE 4, there are quite some changes both in API and in concept. I suggest to focus on trying out features and ideas for now, and then start using ThreadWeaver in earnest when porting to Qt 5. I am happy to help.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s