Blog d’Anita Graser

https://anitagraser.com

  • Scalable spatial vector data processing 18 mai 2018
    Working with movement data analysis, I’ve banged my head against performance issues every once in a while. For example, PostgreSQL – and therefore PostGIS – run queries in a single thread of execution. This is now changing, with more and more functionality being parallelized. PostgreSQL version 9.6 (released on 2016-09-29) included important steps towards parallelization, including parallel execution of sequential scans, joins and aggregates. Still, there is no parallel processing in PostGIS so far (but it is under development as described by Paul Ramsey in his posts “Parallel PostGIS II” and “PostGIS Scaling” from late 2017). At the FOSS4G2016 in Bonn, I had the pleasure to chat with Shoaib Burq who ran the “An intro to Apache PySpark for Big Data GeoAnalysis” workshop. Back home, I downloaded the workshop material and gave it a try but since I wanted a scalable system for storing, analyzing, and visualizing spatial data, it didn’t really seem to fit the bill. Around one year ago, my search grew more serious since we needed a solution that would support our research group’s new projects where we expected to work with billions of location records (timestamped points and associated …
  • Movement data in GIS #13: Timestamp labels for trajectories 9 mai 2018
    In Movement data in GIS #2: visualization I mentioned that it should be possible to label trajectory segments without having to break the original trajectory feature. While it’s not a straightforward process, it is indeed possible to create timestamp labels at desired intervals: The main point here is that we cannot use regular labels because there would be only one label for the whole trajectory feature. Instead, we are using a marker line with a font marker: By default, font markers only display one character from a given font but by using expressions we can make it display longer text, including datetime strings: If you want to have a label at every node of the trajectory, the expression looks like this: format_date( to_datetime(‘1970-01-01T00:00:00Z’)+to_interval( m(start_point(geometry_n( segments_to_lines( $geometry ), @geometry_part_num) ))||’ seconds’ ), ‘HH:mm:ss’ ) You probably remember those parts of the expression that extract the m value from previous posts. Note that – compared to 2016 – it is now necessary to add the segments_to_lines() function. The m value (which stores time as seconds since Unix epoch) is then converted to datetime and finally formatted to only sh …
  • Movement data in GIS #12: Why you should be using PostGIS trajectories 16 avril 2018
    In short: both writing trajectory queries as well as executing them is considerably faster using PostGIS trajectories (as LinestringM) rather than the commonly used point-based approach. Here are a couple of examples to give you an impression of the differences. Spoiler alert! Trajectory queries are up to 500 times faster than comparable point-based queries. A quick look at indexing In both cases, we have indexed the tracker id, geometry, and time columns to speed up query processing. The trajectory table has 3 indexes gist (time_range) gist (track gist_geometry_ops_nd) btree (tracker) The point-based table has 4 indexes gist (pt) btree (trajectory_id) btree (tracker) btree (t) Length First, let’s see how to determine trajectory length for all observed moving objects (identified by a tracker id). Using the point-based approach, we first need to ensure that the points are in the correct temporal order, create the lines, and finally sum up their length: WITH ordered AS ( SELECT trajectory_id, tracker, t, pt FROM geolife.trajectory_pt ORDER BY t ), tmp AS ( SELECT trajectory_id, tracker, st_makeline(pt) traj FROM ordered GROUP BY trajectory_id, tracker ) SELECT tracker, round(sum(ST_L …
  • Optional parameters in QGIS Processing scripts & models 1 avril 2018
    Remember the good old times when all parameters in Processing were mandatory? Inputs and outputs are fixed, and optional parameters or outputs are not supported. [Graser & Olaya, 2015] Since QGIS 2.14, this is no longer the case. Scripts, as well as models, can now have optional parameters. Here is how for QGIS 3: When defining a Processing script parameter, the parameter’s constructor takes a boolean flag indicating whether the parameter should be optional. It’s false by default: class qgis.core.QgsProcessingParameterNumber( name: str, description: str =  »,  type: QgsProcessingParameterNumber.Type = QgsProcessingParameterNumber.Integer,  defaultValue: Any = None,  optional: bool = False,  minValue: float = -DBL_MAX+1, maxValue: float = DBL_MAX) (Source: http://python.qgis.org/api/core/Processing/QgsProcessingParameterNumber.html) One standard tool that uses optional parameters is Add autoincremental field: From Python, this algorithm can be called with or without the optional parameters: When building a model, an optional input can be assigned to the optional parameter. To create an optional input, make sure to deactivate the mandatory checkbox at the bottom of the input parameter …
  • Processing script template for QGIS3 25 mars 2018
    Processing has been overhauled significantly for QGIS 3.0. Besides speed-ups, one of the most obvious changes is the way to write Processing scripts. Instead of the old Processing-specific syntax, Processing scripts for QGIS3 are purely pythonic implementations of QgsProcessingAlgorithm. Here’s a template that you can use to develop your own algorithms: from qgis.PyQt.QtCore import QCoreApplication, QVariant from qgis.core import (QgsField, QgsFeature, QgsFeatureSink, QgsFeatureRequest, QgsProcessing, QgsProcessingAlgorithm, QgsProcessingParameterFeatureSource, QgsProcessingParameterFeatureSink) class ExAlgo(QgsProcessingAlgorithm): INPUT = ‘INPUT’ OUTPUT = ‘OUTPUT’ def __init__(self): super().__init__() def name(self): return « exalgo » def tr(self, text): return QCoreApplication.translate(« exalgo », text) def displayName(self): return self.tr(« Example script ») def group(self): return self.tr(« Examples ») def groupId(self): return « examples » def shortHelpString(self): return self.tr(« Example script without logic ») def helpUrl(self): return « https://qgis.org » def createInstance(self): return type(self)() def initAlgorithm(self, config=None): self.addParameter(QgsProcessingParameterFeat …
  • Revisiting point & polygon joins 24 mars 2018
    Joining polygon attributes to points based on their location is a very common GIS task. In QGIS 2, QGIS’ own implementation of “Join attributes by location” was much slower than SAGA’s “Add polygon attributes to points”. Thus, installations without SAGA were out of good options. Luckily this issue (and many more) has been fixed by the rewrite of many geoprocessing algorithms for QGIS 3! Let’s revisit the comparison: I’m using publicly available datasets from Naturalearth: The small scale populated places (243 points) and the large scale countries (255 polygons with many nodes). Turns out that QGIS 3’s built-in tool takes a little less than two seconds while the SAGA Processing tool requires a litte less than six seconds: Like in the previous comparison, times were measured using the Python Console: In both tools, only the countries’ SOVEREIGNT attribute is joined to the point attribute table: import processing t0 = datetime.datetime.now() print(« QGIS Join attributes by location … ») processing.runAndLoadResults( « qgis:joinattributesbylocation », {‘INPUT’:’E:/Geodata/NaturalEarth/vector_v4/natural_earth_vector/110m_cultural/ne_110m_populated_places.shp’, ‘JOIN’:’E:/Geodata/NaturalEa …
  • Resources for QGIS3 21 février 2018
    The release of 3.0 is really close now. If you want to know what’s new or are just looking for interesting ways to pass the time until the packages land, check out the following QGIS3 resources. For users “24 Days of QGIS 3.0 Features” by Nyall Dawson “Exploring Reports in QGIS 3.0 – the Ultimate Guide!” by Nyall Dawson “Data exploration with Data Plotly for QGIS3” by your’s truly “Intro to QGIS3 3D view with Viennese building data” by your’s truly For more recordings from the developer meeting in Madeira check my Youtube playlist. For developers The official “Plugin migration to QGIS 3” wiki page “The PyQGIS Programmer’s Guide – Extending QGIS 3 with Python 3” by Gary Sherman “Porting Processing scripts to QGIS3” by your’s truly There’s still time to get the PyQGIS 3 Programmer’s Guide at discount before QGIS 3.0 is released! See: https://t.co/xJ7LLaX5UU #qgis #qgis3 #pyqgis #python pic.twitter.com/9k3FUmNFkH — Locate Press (@locatepress) February 13, 2018 Migrating your plugin to #qgis 3, you can use First Aid plugin to help you debugging issues. To learn more about First Aid:https://t.co/5k7RXUDfGD — Lutra Consulting (@lutraconsulting) February 1, 2018 If you have further readin …
  • TimeManager 2.5 published 12 février 2018
    TimeManager 2.5 is quite likely going to be the final TimeManager release for the QGIS 2 series. It comes with a couple of bug fixes and enhancements: Fixed #245: updated help.htm Fixed #240: now hiding unmanageable WFS layers Fixed #220: fixed issues with label size Fixed #194: now exposing additional functions: animation_time_frame_size, animation_time_frame_type, animation_start_datetime, animation_end_datetime Besides updating the help, I also decided to display it more prominently in the settings dialog (similarly to how the help is displayed in the field calculator or in Processing): So far, I haven’t started porting to QGIS 3 yet. If you are interested in TimeManager and want to help, please get in touch. On this note, let me leave you with a couple of animation inspirations from the Twitterverse: Here’s my attempt, Thanks @tjukanov for the inspiration and @dig_geo_com for the Tutorial. Distances traveled in 2,5,7,9,11,13,15 minutes in Cluj-Napoca, Romania. Made with @QGIS–#TimeManager,@openstreetmap, data from @here pic.twitter.com/tEhXfy6CAs — vincze istvan (@spincev) February 3, 2018 https://platform.twitter.com/widgets.js More typographic experiments with Danish ship dat …
  • Porting Processing scripts to QGIS3 28 janvier 2018
    I’ll start with some tech talk first. Feel free to jump to the usage example further down if you are here for the edge bundling plugin. As you certainly know, QGIS 3 brings a lot of improvements and under-the-hood changes. One of those changes affects all Python scripts. They need to be updated to Python 3 and the new PyQGIS API. (See the official migration guide for details.) To get ready for the big 3.0 release, I’ve started porting my Processing tools. The edge bundling script is my first candidate for porting to QGIS 3. I also wanted to use this opportunity to “upgrade” from a simple script to a plugin that integrates into Processing. I used Alexander Bruy’s “prepair for Processing” plugin as a template but you can also find an example template in your Processing folder. (On my system, it is located in C:\OSGeo4W64\apps\qgis-dev\python\plugins\processing\algs\exampleprovider.) Since I didn’t want to miss the advantages of a good IDE, I set up PyCharm as described by Heikki Vesanto. This will give you code completion for Python 3 and PyQGIS which is very helpful for refactoring and porting. (I also tried Eclipse with PyDev but if you don’t have a favorite IDE yet, I find PyCharm …
  • Creating reports in QGIS3 21 janvier 2018
    QGIS 3 has a new feature: reports! In short, reports are the good old Altas feature on steroids. Let’s have a look at an example project: To start a report, go to Project | New report. The report window is quite similar to what we’ve come to expect from Print Composer (now called Layouts). The most striking difference is the report panel at the left side of the screen. When a new report is created, the center of the report window is empty. To get started, we need to select the report entry in the panel on the left. By selecting the report entry, we get access to the Include report header and Include report footer checkboxes. For example, pressing the Edit button next to the Include report header option makes it possible to design the front page (or pages) of the report: Similarly, pressing Edit next to the Include report footer option enables us to design the final pages of our report. Now for the content! We can populate our report with content by clicking on the plus button to add a report section or a “field group”. A field group is basically an Atlas. For example, here I’ve added a field group that creates one page for each country in the Natural Earth countries layer that I ha …