Integrating the RawTherapee engine

RawTherapee is one of the two major open source RAW photo processing applications, the other is Darktable.

Can I leverage RawTherapee RAW processing code for use in Niepce? Yes I can.

So let's review of I did it.

Preamble

License-wise GPL-3.0 is a match.

In term of tech stack, there are a few complexities.

1. RawTherapee is written in C++, while Niepce is being converted to Rust. Fortunately it's not really an issue, it require just a bit of work, even at the expense of writing a bit of C++.
2. It is not designed to be used as a library: it's an application. Fortunately there is a separation between the engine (rtengine) and the UI (rtgui) which will make our life easier. There are a couple of places where this separation blurs, but nothing that can't be fixed.
3. The UI toolkit is gtkmm 3.0. This is a little point of friction here as Niepce uses GTK4 with some leftovers C++ code using gtkmm 4.0. We are porting the engine, so it shouldn't matter except that the versions of neither glibmm nor cairomm match (ie they are incompatible) and the engine relies on them heavily.
4. Build system: it uses CMake. Given that RawTherapee is not meant to be built as a library, changes will be required. I will take a different approach though.

Organization

The code will not be imported in the repository and instead will be used as a git submodule. I already have cxx that way for the code generator. Given that some code needs to be changed, it will reference my own fork of RawTherapee, based on 5.9, with as much as I can upstreamed.

The Rust wrappers will live in their own crate: Niepce application code is setup as a workspace with 4 crates: npc-fwk, npc-engine, npc-craw and niepce. This would be the fifth: rtengine.

The rtengine crate will provide the API for the Rust code. No C++ will be exposed.

npc-craw (Niepce Camera Raw¹), as it is meant to implement the whole image processing pipeline, will use this crate. We'll create a trait for the pipeline and implement it for both the ncr pipeline and rtengine.

Integrating

Build system

Niepce wrap everything into a meson build. So to build rtengine we will build a static library and install the supporting file. We have to bring in a lot of explicit dependencies, which bring a certain amount bloat, but we can see later if there is a way to reduce this. It's tedious to assemble everything.

The first build didn't include everything needed. I had to fix this as I was writing the wrappers.

Dependencies

glibmm and cairomm: the version used for gtkmm-3.0 and gtkmm-4.0 differs. glibmm changed a few things like some enum are now C++ enum class (better namespacing), and Glib::RefPtr<> is now a std::shared_ptr<>. The biggest hurdle is the dependency on the concurrency features of glibmm (Glib::Mutex) that got completely removed in glibmm-2.68 (gtkmm-3.0 uses glibmm-2.4). I did a rough port to use the C++ library, and upstream has a languishing work in progress pull request. Other changes include adding explicit includes. I also need to remove gtkmm dependencies leaking into the engine.

Rust wrapper

I recommend heavily to make sure you can build your code with the address sanitizer. In the case of Niepce, I have had it for a long time, and made sure it still worked when I inverted the build order to link the main binary with Rust instead of C++.

Using cxx I created a minimum interface to the C++ code. The problem was to understand how it works. Fortunately the command line interface for RawTherapee does exactly that. This is the logic we'll follow in the Rust code.

Lets create the bridge module. We need to bridge the following types:

• InitialImage which represents the image to process.
• ProcParams which represents the parameters for processing the the image.
• PartialProfile which is used to populate the ProcParams from a processing profile.
• ProcessingJob which represents the job of processing the image.
• ImageIO which is one of the classes the processed image data inherit from, the one that implement getting the scanlines.

Ownership is a bit complicated you should pay attention how these types get cleaned up. For example a ProcessingJob ownership get transfered to the processImage() function, unless there is an error, in which case there is a destroy() function (it's a static method) to call. While PartialProfile needs deleteInstance() to be called before being destroyed, or it will leak.

Example:

let mut proc_params = ffi::proc_params_new();
let mut raw_params = unsafe {
    ffi::profile_store_load_dynamic_profile(image.pin_mut().get_meta_data())
};
ffi::partial_profile_apply_to(&raw_params, proc_params.pin_mut(), false);

We have created proc_params as a UniquePtr<ProcParams>. We obtain a raw_params as a UniquePtr<PartialProfile>. UniquePtr<> is like a Box<> but for use when coming from a C++ std::unique_ptr<>.

raw_params.pin_mut().delete_instance();

raw_params will be freed when getting out of scope, but if you don't call delete_instance() (the function is renamed in the bridge to follow Rust conventions), memory will leak. The pin_mut() is necessary to obtain a Pin<> of the pointer for a mutable pointer required as the instance.

let job = ffi::processing_job_create(
    image.pin_mut(),
    proc_params.as_ref().unwrap(),
    false,
);
let mut error = 0_i32;
// Warning: unless there is an error, process_image will consume it.
let job = job.into_raw();
let imagefloat = unsafe { ffi::process_image(job, &mut error, false) };
if imagefloat.is_null() {
    // Only in case of error.
    unsafe { ffi::processing_job_destroy(job) };
    return Err(Error::from(error));
}

This last bit, we create the job as a UniquePtr<ProcessingJob> but then we have to obtain the raw pointer to sink either with process_image(), or in case of error, sink with processing_job_destroy(). into_raw() do consume the UniquePtr<>.

image is also is a UniquePtr<InitialImage> and InitialImage has a decreaseRef() to unref the object that must be called to destroy the object. It would be called like this:

unsafe { ffi::decrease_ref(image.into_raw()) };

Most issues got detected with libasan, either as memory errors or as memory leaks. There is a lot of pointer manipulations, but let's limit this to the bridge and not expose it ; at least unlike in C++, cxx::UniquePtr<> consume the smart pointer when turning it into a raw pointer, there is no risk to use it again, at least in the Rust code.

Also, some glue code needed to be written as some function take Glib::ustring instead of std::string, constructors needs to be wrapped to return UniquePtr<>. Multiple inheritence make some direct method call not possible, and static methods are still work in progress with cxx.

One good way to test this was to write a simple command line program. As the code shown above, it's tricky to use correctly, so I wrote a safe API to use the engine, one that is more in line with Niepce "architecture".

At that point rendering an image is the following code:

use rtengine::RtEngine;

let engine = RtEngine::new();
if engine.set_file(filename, true /* is_raw */).is_err() {
    std::process::exit(3);
}

match engine.process() {
    Err(error) => {
        println!("Error, couldn't render image: {error}");
        std::process::exit(2);
    }
    Ok(image) => {
        image.save_png("image.png").expect("Couldn't save image");
    }
}

Results

I have integrated it in the app. For now switching rendering engine needs a code change, there is a bit more work to integrate rendering parameters to the app logic.

Here is how a picture from my Canon G7X MkII looked with the basic pipeline from ncr:

Here is how it looks with the RawTherapee engine:

As you can notice, lens correction is applied.