Why should you use Processing Algorithms

• Well tested and rigorous implementations
• Majority are written in C++ and are faster than alternatives
• Long processes can run the the background while you continue to use QGIS
• Many multi-threaded algorithms can take advantage of multi-core CPU on your machines and give you better performance
• Robust handling of invalid geometry
• Ability to see progress of the operation and cancel it
• Easily run on all or just selected features

Review default settings

You can control what providers are available in settings. The Processing Options menu also provides a way to fine-tune the configuration of the framework.

I strongly recommend, changing the default settings and enable the option Prefer output filename for layer names. This option ensures that when you use batch processing, the resulting layers are unique.

Exercise: Find the length of national highways in a state

The aim of this exercise is to show how a multi-step spatial analysis problem can be solved using a purely processing-based workflow. This exercise also shows the richness of available algorithms in QGIS that are able to do sophisticated operations that previously needed plugins or were more complex.

We will work with roads extracted from OpenStreetMap for the state of Karnataka in India. The admin boundary for the state and the districts come from DataMeet.

Browse to the data directory and expand karnataka.gpkg. Drag and drop the karnataka, karnataka_districts and karnataka_major_roads layers to the canvas.

The layer karnataka_major_roads contain all major roads, including national highways, state highways, major arterial roads etc. Select the layer and use the keyboard shortcut F6 to open the attribute table.

You will notice that the ref column has information about road designation. As we are interested in only national highways, we can use the information in this column to extract relevant road segments.

You can use tools such as Select by expression, export the selected features as a new layer and continue to work. But the processing toolbox providers a much seamless workflow. Search for the algorithm Extract by expression.

Enter the following expression to extract the features where the value of the ref field starts with the letters NH.

regexp_match("ref", '^NH')

You will get a new layer Matching Features in the Layers Panel. Next, we want to calculate the length of each segment. You can use the built-in algorithm Add geometry attributes

The source layer is in the Geographic CRS EPSG:4326. But for the analysis, we want the lengths to be measured in meters/kilometers. The algorithm provides us with a handy option to calculate the distances in Elliposidal math - which is ideal for layers in the geographic CRS.

A new layer with an additional field called length will be added to the Layers panel. The distances in this field are in meters. Let’s convert them to kilometers. You may reach out for the trusty QGIS field calculator to add a new field. That’s a perfectly valid way - but as mentioned earlier, there is a processing way to do things which is the preferred way. Search and open the Field Calculator processing algorithm instead and enter the following expression.

"length"/1000

A new layer with the field length_km will be added to the Layers panel. Now we are ready to find out the answer. We just need to sum of the values in the length_km field. Use the Basic Statictics for Fields algorithm.

The result is a table of different statistics on the column. As the result is not a layer, it will be displayed in the Results Viewer. The panel will contain the link to a HTML file containing the statistics. The Sum contains the total length of national highways in the state.

What do you think of the results? The resulting number may not be perfect because the OpenStreetMap database may have missing roads or are classified differently. But it is close to the number provided in the official statistics.

The final layer is called Calculated and is a temporary memory layer. Let’s save it to the disk so we can use it later. The layer contains many fields which are not relevant to us, so let’s delete some columns before saving. The classic way to do this is to toggle editing and use the Delete Column button from the Attribute Table. If you wanted to rename/reorder certain fields, that needed a plugin. But now, we have a really easy processing algorithm called Refactor Fields that can add, delete, rename, re-order and change the field types all at once. Delete fields that are not required and save the result as the layer national_highways in the source karnataka.gpkg.

The layer national_highways will be added to the Layers panel. We achieved the goal of the exercise, but we can explore the results a bit better if we can break down the results by a smaller administrative unit. Let’s try to calculate the length of national highways for each district in the state.

We have the district information in the karnataka_districts layer, but not in the national_highways layer. To add the name of the district to the roads layer, we need to perform a spatial-join. This is done using the Join attributes by Location algorithm.

Select the national_highways as the input layer and do a one-to-one join with the karnataka_districts layer. Select only the DISTRICT field to be added to the output.

The new layer Joined layer now has the intersecting district name in the DISTRICT field. We can now sum the road lengths and group them for each district. You may recall that in earlier versions of QGIS ,you needed a plugin called Group Stats to do this. But now we can do this via the built-in Statistic by Categories algorithm.

The output of the algorithm is a table containing various statistics on the length_km column for each district. The values in the Sum column is the total length of national highways in the district.

Note that we did data processing, spatial analysis and statistical analysis - all using just processing algorithms in a fast, re-producible and intuitive workflow.

The Locator Bar

To take your processing experience to the next-level, you can use the built-in Locator Bar. At the bottom-left of QGIS main window, there is a universal search bar that can do keyword-search across layers, settings, processing algorithms and more. You can open the locator bar using the keyboard shortcut Ctrl+K.

I find that rather than clicking-around the processing toolbox, you can just use locator bar to search and open the algorithms. Type Ctrl+K, followed by a (to restrict search to algorithms), followed by a space and a few characters. Use the arrow keys to select and press Enter to open the algorithm.

In-place Editing

Processing algorithms are designed to take inputs and produce outputs. The default behavior is to create a new layer after each operation. This is useful for many workflows, especially in an enterprise setting, where you may not have the ability to edit the source data. If your algorithms are altering the source data, that also means that the workflows cannot be reproduced easily. So you would want a setup where the algorithms read from a source data and create modified outputs.

An exception to this workflow is when you are doing data editing. When your workflow involves creating new features or editing them - creating a new layer for every edit is undesirable. A recent QGIS crowd-funding campaign added the ability for processing algorithms to modify the features in-place and this functionality is available out-of-the-box in QGIS now.

Load the basic_network_analysis project from the data package. This project contains a street network for Washington DC from DCGISopendata where the arrows display the digitizing direction of the line segments. You can click the Edit Features In-Place button in the processing toolbox to use algorithms that support this functionality. Once this mode is activated, processing algorithms will modify the selected features in the chosen layer instead of creating a new layer.

Select a line segment and run the Reverse line direction algorithm. The algorithm will enable editing on the layer, perform the operation and overwrite the existing feature with a segment in the reverse direction. You will see that the arrow rendering is now in the opposite direction.

Exercise: Clip multiple layers to a polygon

We will take multiple country-level data layers and use the batch processing operation to clip them to a state polygon in a single operation.

Open the batch_processing project from the data package.Select the India-States layer and use the Select Features tool to select a state by clicking it.

Next, use the Extract selected features processing algorithm to create a new layer from the selected feature. This will create a new layer called Selected features

Search for the Vector Overlay → Clip algorithm and right-click on it. Select Execute as Batch Process.

In the batch processing dialog, click the … button on the first row of the Input layer column and choose Select from Open Layers... Select all data layers that you want to clip.

Similarly, select the Selected features layer as the Overlay layer.

As all input layers need to be clipped with the same overlay layer, you can click Autofill.. and select Fill Down to autofill all the remaining rows with the same value.

In the last Clipped column, click the … button and name the output clipped_. When prompted, choose Fill with parameter values as the autofill mode, and Input layer as the Parameter to use.

Make sure the Load layers on completion box is checked and click Run.

Resulting clipped layers will be added to the Layers panel. We can combine all the clipped layers into a single geopackage file for ease of sharing. Run the Package Layers processing algorithms.

Choose all the clipped layers and save the output as clipped.gpkg.

Exercise: Find the hotspots of piracy indidents

National Geospatial-Intelligence Agency’s Maritime Safety Information portal provides a shapefile of all incidents of maritime piracy in the form on Anti-shipping Activity Messages. We can create a density map by aggregating the incident pints over a global hexagonal grid.

The steps needed to create a hex-bin layer suitable for visualization is as follows

• Reproject the input to an equal-area projection
• Create a global hexagonal grid layer
• Select all grids that intersect with at least 1 point
• Count points within each grid

We will now learn how to build a model that runs the above processing steps in a single workflow.

Open the maritime_piracy project.

Launch the modeler from Processing → Graphical Modeler

In the Processing Modeler dialog, locate the Model Properties panel. Enter piracy_hexbin as the Name of the model and projects as the Groups. Click the Save button.

Save the model as piracy_hexbin.

Now we can start building a graphical model of our processing pipeline. The Processing modeler dialog contains a left-hand panel and a main canvas. On he left-hand panel, locate the Inputs panel listing various types of input data types. Scroll down and select the + Vector Layer input. Drag it to the canvas.

Enter Input Points as the Parameter name and Point as the Geometry type. This input represents the piracy incidents point layer.

Next, drag another + Vector Layer input to the canvas. Enter Base Layer as the Parameter name and Polygon as the Geometry type. This input represents the natural earth global land layer which we will use as the extent of the grid layer.

As we are generating a global hexagonal grid, we can ask the user to supply us the grid size as an input instead of hard-coding it as part of our model. This way, the user can quickly experiment with different grid sizes without changing the model at all. select a + Number input and drag it to the canvas. Enter Grid Size as the Parameter name and click OK.

Now that we have our user inputs defined, we are ready to add processing steps. All of the processing algorithms are available to you under the Algorithms tab. The first step in our pipeline will be to reproject the base layer to the Project CRS. Search for Reproject layer algorithm and drag it to the canvas.

In the Reproject layer dialog, select Base Layer as the Input layer. Check the Use project CRS as the Target CRS. Click OK.

In the Processing Modeler canvas, you will notice a connection appear between the + Base Layer input and the Reproject layer algorithm. This connection indicates the flow of our processing pipeline. Next step is to create a hexagonal grid. Search for the Create grid algorithm and drag it to the canvas.

In the Generate grid dialog, choose Hexagon (polygon) as the Grid type. Select Extent of ‘Reprojected’ from algorithm ‘Reproject Layer’ as the Grid extent. Click the 123 button under the Horizontal spacing label and choose Model input.

Select Grid Size input for Using model input. Repeat the same process for Vertical Spacing. Click OK.

At this point, we have a global hexagonal grid. The grid spans the full extent of the base layer, including land areas and places where there are no points. Let’s filter out those grid polygons where there are no input points. Search for Extract by location algorithm and drag it to the canvas.

For Extract features from, select ‘Grid’ from algorithm ‘Generate Grid’, Where the features (geometric predicate) as Intersect and By comparing to the features from as Input points. Click OK.

Now we have only those grid polygons that contain some input points. To aggregate these points, we will use Count points in polygon algorithm. Search and drag it to the canvas.

Select ‘Extracted (location)’ from algorithm ‘Extract by location’ as the value for Polygons. The Points layer would be Input Points. At the bottom, name the Count output layer as counts. Click OK.

The model is now complete. Click the Save button.

Switch to the main QGIS window. You can find your newly created model in the Processing Toolbox under ** Models → projects → piracy_hexbin**. Double-click to run the model.

Our Base Layer is the ne_10m_land and the Input Points layer is ASAM_events. The Grid Size needs to be specified in the units of the selected CRS. Enter 100000 (100 Kms) as the Grid Size. Click Run to start the processing pipeline. Once the process finishes, click Close.

You will see a new layer Counts loaded as the result of the model. As you explore, you will notice the layer contains an attribute called NUMPOINTS containing the number of piracy incidents points contained within that grid feature. Let’s style this layer to display this information better. Click the Open Layer Styling Panel button from the Layers panel. Select Graduated symbology and NUMPOINTS as the Column. Click the dropdown next to Color ramp and select the Viridis ramp. Click the dropdown again and select Invert Color Ramp to reverse the order of color. The Graduated symbology will divide the values in the selected column into distinct classes and assign a different color to each of the classes. Select Natural Breaks (Jenks) as the Mode and click Classify.

We will use a clever trick to make the visualization better. We can use QGIS variables to set the edges of the hexagon a slightly darker shade of the fill color. Click on the Symbol. Click Simple Fill. Click the Data defined override button next to Stroke color and select Edit>

Enter the following expression to set the stroke color to 30% darker than the fill color and click OK. The map rendering will change to a much smoother visualization.

darker( @symbol_color , 130)

Challenge: Improve the model

Can you change the model so that instead of entering the grid size in meters, the user can enter the size in kilometers?

Hint: The Create Grid algorithm expects the size in meters, so you will have to convert the input to meters.

Enabling Reproducible Workflows

To ensure that you are always able to reproduce your results, it is recommended to bundle the model in your project. The modeler interface has a button save model in project that will embed the model in your QGIS project file.

Once the model is embedded, Whenever you open the project, the model will be available to the user under the Project models in the Processing Toolbox.