Timeline is changing: Location and activity data analysis on your own infrastructure
if you've been using Google timeline to track places you went and the times you went there chances are that recently you received a message like this informing you that timeline is changing and you have to update some settings in Google maps to keep your visits the background is that Google is migrating location data from the cloud onto users devices they do so for a variety of reasons but one main reason is that they do not want to give authorities like law enforcement data about their user location and if they no longer keep that data on their servers they can rightfully claim that they don't have that information now some users use timeline just casually While others rely on it more and might even run sophisticated data analyzis on it if you receive this notification and you rely on the data in your timeline then please beware and don't start migrating your location data right away you will likely lose data in the process the reason is that location data as it is currently stored in the cloud appears to be far more granular than location data after the migration onto devices another reason is that after migrating your location data onto your device you may no longer be able to export it and use it in other services even if you do the format of that data will have changed and there are basically no tools out there at the moment that work with the on device location data so in this video I would like to tell you what you can can do if you want to preserve your data the first step is to head to Google takeout which is a service hosted under take out.google.com and download a copy of your location data the data will come in an archive file that you will have to extract after you extract it you will find a file in it that is called records. Json it is the file that contains most of the GPS or gnss data that was acquired from your devices and there are tools that you can use to import that data and basically host your own timeline you can import the data that Google provides and then use any GPS logging app or physical GPS logger to add data to it now that we have the data we can set up a service on our own machine that we can use to aggregate location data draw it on a map analyze it convert it into different formats annotate it with metadata Etc we will basically run our own location aggregation service which will replace timeline for that we will open up a terminal and then we will create a directory within our users home directory and navigate into it I will call this directory demo and we will carry out all further steps within it so I type MK D demo to make a directory and then I call CD demo to navigate into it the next thing we do is we grab the source code of the tool location visualizer it's available on GitHub we will use the following command to fetch it git clone https colon github.com / Andre pxx location minus visualizer dogit now that we got the source we can navigate into the directory that's just been created for us so we type CD location minus visualizer and now we can build the application by running the make Command it will will have a dependency to a graphics Library called Sydney which implements a special rendering process that's called abstract rendering the build system will automatically resolve this dependency we will need to have uh build tools for the programming language go installed so that the build will be successful so we type make and hit enter so in our case the build was successful if this fails for you then you will need to install additional build tools like make and the go compiler tool chain now that we have the application itself built we can create the TLs keys that we need for the web interface to run and for that you type make keys this will require opl to be installed now that we have TLS Keys we have to open up the configuration file in the text editor and make some adjustments for this we navigate into the configuration directory with the command CD config and then we open up the configuration file with the command Nano config.js you can use any text editor
you want to edit the file first we replace the entry US map false by US map true and now we have to configure a map server so that we can and have our data drawn on the Baseline map fortunately there's the open street map project that's run by volunteers and provides worldwide map data so we will have to go to this entry in the configuration file here where it says map server and then I will just put in HTTPS colon tile. openstreetmap.org and then we have to use placeholders so I do a forward slash and then a dollar sign an opening curly brace then a z then a closing curly brace slash then a dollar sign an opening opening curly brace X and a closing curly brace then again a slash dollar sign opening curly brace y closing curly brace dot PNG and then we can hit crol o to save the file and then contrl X to exit the editor note that I'm using a public T server here from the open street map project now the guys from the open street map project they don't really like you to use public T service since they have the costs of running them but will not get monetary compensation for it from people who use them so if you use this privately this might be okay but if you host location services for a larger user base you should definitely set up and run your own t server now the reasons I developed this application are manifold I do a lot of Endurance Sports mainly running and cycling I run Half Marathon distance Marathon distance and Beyond and I use location visualizer for tracking these activities I also make music I play the guitar myself both electric and acoustic I also often attend cultural events concerts Etc you know I've already seen over a hundred bands live I'm also active in the hacker and makeer community in my city I attend conferences both in Berlin and elsewhere where I can present my work and meet other open source developers I'm also active in the lgbti AQ community and in several voluntary and nonprofit projects I also travel a lot because I have a large social network and I know people from pretty much all over the world and there are also really diverse people with all sorts of different backgrounds which I find really enriching I know people from Berlin from other cities in Germany from other countries both inside and outside of Europe so I travel a lot so that I can visit people I also really like exploring Berlin as a city and and when I have visitors here then often I'm their guide and show them around you know people come here all the time for some event you know a concert or a conference they attend or just to visit the city and then I'm going to show them around professionally I work in research where Presence at conferences and stuff like that is also important for your reputation so I'm really on the move all the time and discover new places and I can keep track of all that with location visualizer now if we want to use location visualizer we will have to set up a user first thing we're going to do is go out of the config directory again with CD do Dot and then I will run the following command do/ loock this create minus user username this will create a user with name set to username now if you set this up you can of course choose a username of your choice and then we set a password so we type do/ loock this set minus password username secret and now let me just put the terminal into full screen mode so that we get rid of the line wrap the last command we entered set the password of the user with the name username to secret now obviously as I said if you want to run this as a public task server you choose a much stronger password here now we have to give the user some permissions so I type /log Bas add minus permission and then I type the name of the user so in our case username and then I type the name of the permission I want to give to the user and the first permission I want to give is called get minus tile and I will just hit the arrow up key and then I will get the last command again and then I will replace the permission with the permission render hit enter again and then I repeat this process next permission I give is called activity minus read then activity minus write then GB so geodb minus read GV minus write GB minus download now if we want we can also add the permission G udb minus clear then we can even clear the geographical database from within the web interface and now we're basically set and can run the service and for that we type /log Vis without any more parameters and hit enter so location visualiz now runs as a demon in the background and we can open a browser and connect to it so I will open a browser create a new tab and then navigate to Local Host colon 82 and now we have to accept a certificate warning and log in so we type username then we can hit enter and the cursor will advance into the password field and now we type the password which was secret and hit enter and then we will log in so in the background now there has been some sophisticated crypto going on if you want you can inspect the data that is exchanged during the login process using the network analyzer in your browser now we see the map background this is data from open street maps and now we can open the sidebar which is the thing over here and we see a lot of stuff we can do there first thing I'm going to do is put location visualizer into full screen mode you can see there are a lot of options for navigation and filtering the data and stuff like that you can see that we can change the intensity and color mapping for the map background so I would just demonstrate this you can adjust the intensity down and of course also upwards and you can see the brightness of the map change and we can also go to original and then it will just show the original map as it is coming from open street maps so it will not be mapped to these dark colors now let's get some data into this so click on the GB button here which opens the interface to the geographical database that's integrated into location visualizer and now you can select the data format you want to import we have support for open gudb which is a vender neutral binary file format with fixed size records for fast random access which can also be used by GPS or gnss logging devices as their internal storage format or also their wire format for transfer records over an interface like USB or isare C or whatever at the moment it's also the format that location visualizer uses internally for storage then we have CSV so comma separated values GPS exchange or GPX and Records Json this is not geojson it is a Json based format that is used by several online services like Google Maps or Google takeout and also by more modern GPS devices often in addition to the older XML based GPX format you know in general there's a tendency away from XML based formats which are often considered quite Legacy and heavy weight towards Json based formats which are often regarded as more lightweight and modern so we support all of this here and we can import from it and Export to it under this import strategy drop down you can select what data you want to import do you want to import all data points do you only want to import data points that are newer than the current latest or most recent data point in the database or do you not want to import any data at all the last option is useful if you just want to get some statistics on the data you upload since you will get an import report even if you didn't actually import anything so you can sort of preview your data before you actually modify the content of the database these are all options you can choose when importing data and down here are actions you can run on the geographical database you can sort the entries by timestamp you can remove duplicates and you can also clear the geographical database if you choose the last option then you will have to use this field down here to provide an sh 512 hash of the contents of the geographical database in open gudb format this serves as appr proof that you downloaded a backup of the database contents before clearing them and is here to prevent data loss it also means that should the database contents change between the point in time where you acquire the backup and the point in time where you decide to clear the database this discrepancy will get noticed since the hash will no longer match and the clear operation will get rejected okay so let's grab a file with Geo dat in it this one is in records Json format and let's import it since our database is empty we can just import all data points yeah and now you have some statistics here this is the import report you have some statistics on the database content before and after the import about the data you actually uploaded for import and the data points that were actually imported which is interesting of course if you decide not to import all data points let's close the import report and you also have some Statistics over here so you can see I've been recording for more than 9 years now and have over a million data points here so let's get back to the map view and I will just increase Point spreading a bit and now I will zoom into this area so this is a trip to Australia that you can see here I've been in Australia in 2019 and also in 2023 and now you can go over here and open the sidebar again and then you can see lots of Statistics here like where we currently are on the map in a variety of coordinate systems like geographical coordinates in degrees minutes and seconds including fractional seconds um dimensional as northing and easting and also northing and easting in kilometers and you also have boxes over here that you can use to filter the data based on time you can put ISO time stamps here for beginning and end so here you can see the uh trip that I did in 2019 and here you can see the one I did in 2023 and you can also put uh different Expressions here like uh instead of a full time stamp you can use um just a year and of course you can also leave one side open basically and you can also put um year and month so for example in 2019 I was there in September no that was the wrong expression yeah so that's the points from September and then I also have a few points in October so you can differentiate um between that and you can also also use [Music] um year month and day okay see we have nothing all right have the wrong month and and uh well you can also put a time zone here um so uh and there you can use UTC notation or GMT notation it really doesn't matter and uh yeah or you can use um a fulltime stamp and then you can use the plus minus notation or you can put the Z here for Zulu time which is uh equivalent to uh UTC so uh there's a lot of flexibility in here what you can do with uh these time stamps and uh with this uh filtering and every time you click this uh apply button then it will apply the respective filter and will only show you the data points that uh pass the uh filter criteria and over here you see this activities button here you can add metadata to your actual location data it will be stored separately from your location data but the two will be correlated by time stamps when we click on the add button we will insert what is called an activity group now the concept of activity groups is not that easy to explain and probably not that intuitive either but once you get the hang of it it is actually quite useful an activity group is a section in time in which activities are performed and unless you're dead you're pretty much always active in some way or another other since you will always do something you know whether that is running or cycling which are activities for which we have separate attributes here or whether that's something else like eating or sleeping or dancing or doing whatever you know unless you are dead you will always have some sort of metabolic raate going on which means your body and therefore you are doing something right that's the reason why an activity group only has a begin time stamp but no end time stamp the activity groups will automatically be sorted by their begin time stamps and they will implicitly end when the next activity group starts since unless you're dead there is no way you cannot at least do something activity groups therefore divide your entire timeline say your entire life or at least the time span you want to cover or capture here in location visualizer into segments and then for each of these segments you specify which amount of time you spent running or cycling and add some other attributes like the weight you had at some point within that activity group distances covered the number of steps taken the amount of energy consumed performing different activities and then for the rest of the duration of that segment you performed other activities which can be anything really so the only thing we can say about it is how much energy you consumed during the time that this segment or activity group covers where you were not performing a more specific specific activity namely running or cycling so if we fill some data in here we can create an activity group I'm just filling in some madeup data now and then we can add the activity group and you can see that the begin time stamp will now be a link and if we click on that link it will actually prefill the filter attributes and the filter so that on the map view we see exactly the time that is covered by that particular activity group and we also have aggregate Statistics over all activity groups down here so you can see which distances and times and so on you had in total so for me an activity group is always a day which might not necessarily be 24 hours since there might be daylight saving changes or time zone changes when it travel but really the concept of activity groups is flexible so if you run a marathon on one day let's say you could easily have an activity group covering the time before the run an activity group for the actual run and then an activity group for the rest of the day after the run it's really up to you an activity group is just a section of time for which you can specify certain attributes and down here you can see you can do a CSV export of your activity data for use with spreadsheet or data analysis applications and you also have the opportunity to import CSV data the CSV will have one row per activity group and then will contain the individual attributes of each activity group in the separate columns Now location visualizer has a lot more functionality that I did not cover in this tutorial for example you can export your map tiles from the tile cache to an archive or import map tiles from an archive into the tile cache and so on if you're Tex heavy I suggest you just read the documentation and Fiddle around with this there's actually a lot of stuff you can do with it some final notes from my side I started tracking my weight on 12th of May 2016 my activities on 6th of June 2016 and my location on 16th of June 2016 so I I'm close to 9 years of continuous records now up till 8th of December 2024 this amounts to almost 1.4 million unique GPS or gnss
data points more than 45,000 kilm so more than ones around the equator or more than a th000 marathons running and more than 10,000 kilm cycling I did the tracking with different apps and devices most of the apps that I use store data locally and I integrate and manage all my data with location visualizer this is open source software it's under a permissive license Apache License 2.0 to be precise I hope you enjoy my work if you have any questions or suggestions please feel free to comment of course you can also contribute to the project on GitHub if you want to use location visualizer as a replacement for the discontinued online Google timeline feature you can see it by importing data that Google takeout provides and then use different logging apps which can export to GPX for example and merge all that data into your local data storage location visualizer currently does not support importing or displaying the semantic location data that Google provides this data is also not very reliable it can change for example when the map changes and since Google does not keep a history of the map evolving it will try to match your previous locations captured years ago against a current map of your city which may lead to let's say interesting results for example when businesses close or move somewhere else existing buildings are torn down and new ones are constructed or even some roads might get relocated or moved around and since it seems that Google does not really keep track of the evolution of its map I wouldn't rely too much on the semantic data so providing the semantic data is not too much of a priority for us since in the end it does not provide um accurate or correct data anyhow also after Google discontinues its current service and use a switch to general purpose GPS logging apps or a physical GPS logging devices we would no longer be able to deduce that semantic data anyhow since that's something that Google does and requires its huge amount of data it has crowdsourced over the years so even if we do provide a way for you to import and display that data there will be no useful way to keep adding to that data after migrating away from timeline so yeah we may provide a way to import it someday but semantic location data really is a thing of the past I would say
2024-12-25 02:09
