Development Updates

SOURCE_FILE Scenarios

A new Scenario Type has been introduced, called SOURCE_FILE. After uploading a source, the source is deepcopied, and inserted as a parent of the SOURCE scenario. This happens after functions which fix zero duration or zero distance trips and creating opp stations at depots when needed. This source_file scenario should never be mutated. The SOURCE scenario is now mutable by a filter_scenario view, which permanently removes rotations/stations/lines. This is only possible as long as there are no other children of this scenario.

Data Scenarios

How to create public data scenarios. Admins can now easily add Scenarios which can be selected by users to run a scenarios without uploading data. Steps

1. Login as Superuser 2a. Upload a Scenarios using the standard way/trips.html OR 2b. Upload a Scenarios using the scenario_tee import
   • If the tree does not contain a Mutation-Scenario create one
     • Create a variant using the "Szenarien" Tab. Click on "Variante Erstellen" filter it the way you like using the wizards "Simulationsfilter"
2. Go to the admin panel, select the created mutation scenario and make the scenario a PUBLIC_DATA scenario

Admin Panel

Most Models have been added to the scenario panel. If you search for a specific item a query can be added to the model url,

vehicletype/?battery_capacity__gte=500&name__contains=MyVehicleType

Redis

To allow caching, enable permanent celery results storage and use the same stack as production. Redis is very recommended. Install redis and use the following .env settings

REDIS_URL=redis://127.0.0.1:6379
CELERY_BROKER_URL=redis://127.0.0.1:6379
CELERY_RESULT_BACKEND='redis://127.0.0.1:6379'

This removes rabbitmq or other brokers from the necessary dependencies. REDIS_URL is used for caching and needs to be redis. If no REDIS_URL is supplied, caching will be skipped. CELERY_BROKER_URL is the broker which manages celery task management. CELERY_RESULT_BACKEND is used to store results. CELERY_TASK_RESULT_EXPIRES can be used to define how long the results are stored (in seconds). default is 1 day.

Installation

1. Clone this git repository (or download a specific release)

   git clone git@github.com:rl-institut/django-simba.git

2. Install prerequisites

   1. The currently suggested Python version is 3.11.*, it may work with other versions, but this is not tested.
   2. This software requires GDAL, which can be installed
      • on Linux via the system's package manager (e.g. apt install gdal-bin on Ubuntu)
      • on macOS via Homebrew (brew install gdal)
      • on Windows via OSGeo4W (select the gdal package)
   3. The software requires a PostgreSQL database with the PostGIS package.
      • The software is found here and here or via your system's (or server's) package manager (e.g. apt install postgis)
      • When the elevation_api is used postgis_raster is needed
        • can be installed by connecting to your database and running

        create extension postgis_raster;

      • The credentials for the database are set in ebusdjango/settings.py in the DATABASESvariable. SECURITY WARNING: Do not commit your passwords to GitHub!
      • In order to set up PostGIS, the user you created needs to have 'superuser' privileges.
   4. The software can optionally use backend for Celery to be available. Its address is set in the CELERY_BROKER_URL in the .env file.
      • rabbitmq can be installed on ubuntu using apt install rabbitmq-server
      • A user can be added using the following commands (or the guest user can bs used):
        1. rabbitmqctl add_user $user $password
        2. rabbitmqctl set_permissions -p / $user ".*" ".*" ".*"
   5. Install poetry for environment and dependency management.
   6. The dependencies can be installed via

      poetry install

      For developers, use

      poetry install --with dev

   7. Django uses an .env file to read user specific data. This file has to be created by the user and is not shared through GitHub to make uploads of sensitive data impossible. Create a file named .env with the following input`

     DJANGO_SECRET_KEY=INSERT_YOUR_KEY_HERE
     DJANGO_DEBUG=True
     # Replace with your own database info
     DATABASE_URL=postgis://YOUR_DB_USERNAME:YOUR_PASSWORD@localhost/YOUR_DB_NAME
     # Should the simulation run with eflips-depot. Development setting which will be set to True in stable Versions.
     EFLIPS_USE=TRUE
     # Redis backend url. This allows caching. Can be the same as the celery broker url but does not have to be
     REDIS_URL=redis://127.0.0.1:6379
     # Should celery be used synchronously? In this case no celery process and CELERY_BROKER_URL is needed.
     CELERY_TASK_ALWAYS_EAGER=True
     # CELERY_BROKER_URL=pyamqp://guest@localhost//
     TILING_SERVICE_TOKEN=GET_YOUR_TOKEN_THROUGH_MAP_TILER
     TILING_SERVICE_STYLE_ID=basic-v2
     DJANGO_SETTINGS_MODULE=ebusdjango.settings
     # If you are developing locally set this to true. It disables production security features.
     DJANGO_LOCAL_DEVELOPMENT=True
     # for elevation API usage
     DJANGO_ELEVATION_TOKEN=YOUR_TOKEN_ELEVATION_TOKEN
     OPENELEVATION_URL=http://127.0.0.1:8000/elevation_api/YOUR_TOKEN_ELEVATION_TOKEN
     # If no admin user exists in the database, one will be autogenerated with this password
      DJANGO_ADMIN_PASSWORD=SOME_ADMIN_PASSWORD
   

3. Set up django (inside the virtual environment)

   1. Set up the database: python manage.py migrate
   2. Create admin account: python manage.py createsuperuser TODO: Is this necessary?

Running

1. Only if .envhas a celery broker listed, start a celery worker (in another terminal): celery -A ebusdjango worker -l info
   • on macOS OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES before the command may be necessary
2. Run the server: python manage.py runserver
3. For the design, Tailwind is used. Therefore, styles have to be compiled with python manage.py tailwind start. While developing and changing designs, this should run parallel to Django. Later, it might have to be called once to generate styles.

Development

For development dependencies can be installed via

poetry install --with dev

To run tests your PostgreSQL user needs SUPERUSER rights to be able to create test databases and delete them. Furthermore, for testing with selenium chromedriver is needed. Resources can be found here Latest Google Chrome and Chromedriver. Chromedriver has to be added to PATH.

Setup and Recreation of Database

During development, it might be necessary to delete the database and recreate it. The following instructions seem to work for Linux/Ubuntu go into terminal and log in as postgres superuser

sudo -i -u postgres

Get into sql

psql -U postgres

Drop the old database

DROP DATABASE your_database;

Create new database

CREATE DATABASE your_database;

go into database via

\c mydb

or exit with \q and connect directly from normal terminal

psql -U postgres -d mydb

Define settings and install postgis and show installation worked

CREATE EXTENSION IF NOT EXISTS postgis;
SELECT PostGIS_version();

Loading a SQL dump

A database dump can be used to fill the database. In your terminal navigate to the dump file ending with sql. If it is a text based dump use

psql -U YourProjectuser -h 127.0.0.1 YourDBName < DumpFileName.sql;

For some reason specifying the host seems to be needed on my machine. This has something to do how authentification seems to work.

Docker install

To build a docker locally install docker using https://docs.docker.com/engine/install/

Make sure to not have other dockers installed https://docs.docker.com/engine/install/ubuntu/#uninstall-old-versions

For me only installing from package worked https://docs.docker.com/engine/install/ubuntu/#install-from-a-package

wsl users can try following this https://docs.docker.com/desktop/wsl/

Docker build and run

Go into your django-simba root containing manage.py

With docker compose

Make sure your .env file reflects the docker-compose.yml properties

DATABASE_URL=postgis://myprojectuser:1234@my-docker-postgres:5432/mydb
CELERY_BROKER_URL=redis://my-docker-redis:6379/0

When running locally for development some security settings need to be applied by setting in the .env file

DJANGO_LOCAL_DEVELOPMENT=True

Navigate to your cloned repo of django-simba in a terminal. Now running the following line, builds and starts a docker container including creating a database

sudo docker compose up

After this django-simba will be available under http://127.0.0.1:8000/

To stop containers and remove containers, networks, volumes, and images created by up, run

sudo docker compose down

The build has to be repeated if dependencies change. This can be fixed by rebuilding the docker or removing containers, volumes and images and running again

sudo docker compose up

Changes in the source code or templates are reflected live, even while running the docker container.

Using Docker without compose

Create a network for your postgres and django-app to communicate

 docker network create mynetwork

Optional check your networks

 sudo docker network ls

run a postgis instance in this network. Set your database according to your settings

 sudo docker run --name my-docker-postgres -e POSTGRES_PASSWORD=1234 -e POSTGRES_USER=myprojectuser -e POSTGRES_DB=mydb -d --network=mynetwork postgis/postgis

Go into your Django .env file and make sure the host is the same as in the above db, e.g. my-docker-postgres. This replaces "localhost" in the database url, e.g.

DATABASE_URL=postgis://myprojectuser:1234@my-docker-postgres:5432/mydb

The .env used while building will define which configuration the dockerimage will use Build your Django-simba docker

 sudo docker build -t django-simba .

run the created docker in this network and expose the port

 sudo docker run -p 8000:8000 --network=mynetwork django-simba

Optional you can use the flag -d to start the container as detached. This means closing the terminal will NOT stop the container to stop the container

 sudo docker stop CONTAINER_ID

In the above options only docker compose uses a permanent storage for the database. By creating the volume postgres_data. In other words stopping the container and starting it again, the database will not have lost its data. At the same time, migrations have to respect the existing data as well.

Volumes can be checked using

sudo docker volume ls

or removed

sudo docker volume rm VOLUME_ID

but only if the docker is not running

data_scrapers

stations

• Search function provided with local db.
• Admin Areas from level 4 (Bundesland) to 8 (Gemeinden) for Germany are queried in a hierarchical manner
• bus_stops are queried with a guaranteed relation to the highest found admin_level up to level 8. The admin area hierarchy in Germany is complicated. Not every location is part of an admin area of level 8. In these cases the highest found admin area is used
• Introduces an API and map to find bus stops by name.
  • /data_scrapers/stations/map/?search_stations=S%2BU Alexanderplatz/Memhardstraße
  • /data_scrapers/stations/api/?search_stations=S%2BU Alexanderplatz/Memhardstraße
  • the API returns the results as JSON
  • if querying for multiple stations use '|' as delimiter
  • the keyword query param '&filter=true' can be added to use filters, which try to uniquely identify the searched stations. Default is filter=false
  • /data_scrapers/stations/api/?search_stations=Alexanderplatz
    • finds multiple Alexanderplatz bus_stops around Germany
  • /data_scrapers/stations/api/?search_stations=Alexanderplatz&filter=True
    • finds no bus_stop since the found stations relate to different bus_stops, since the distance between each other is above a threshold -/data_scrapers/stations/api/?search_stations=S%2BU%20Alexanderplatz/Memhardstra%C3%9Fe|Alexanderplatz&filter=false
    • finds the station at Memhardstraße and several Alexanderplatz stops around Germany -/data_scrapers/stations/api/?search_stations=S%2BU%20Alexanderplatz/Memhardstra%C3%9Fe|Alexanderplatz&filter=true
      • Finds the stop at Memhardstraße. Since this station is uniquely identified Alexanderplatz is found as well, since it's assumed that stations of a bus system are located next to each other.
  • Search Features:
    • Checks for Admin area names inside of searched name up to admin_level 8
      • /data_scrapers/stations/api/?search_stations=Berlin+Alexanderplatz
      • will find only the Alexanderplatz which is in Berlin
    • Fuzzy search with trigram similarity
      • Alekanderplatz finds bus_stop around Germany called Alexanderplatz
        • Only the closest matches of the fuzzy search are returned
      • If no unique stations can be found, previously found stations are used and:
        • a convex hull of found stations with a buffer zone is created. This area is checked for a uniquely identifiable station
        • admin_areas which contain found stations are fuzzily searched for a uniquely identifiable station
  • DB import and export are possible as superuser via /data_scrapers/stations/import/ and /export/
    • import is only supported for empty tables of BusStation and AdminArea

Translation

The website supports basic translation. Text is marked with get_text. To create translation files for all languages in the settings file run.

django-admin makemessages -a

The translation becomes available when the msgstr inside these .po files is set and running

django-admin compilemessages

Temperatures

The Website supports temperature statistics via DWD-data. Currently they are used in supporting the user deciding on a scenario temperature. The closest weatherstation in regards to the scenario centroid with sufficient data is searched. Found data is used to compare user input to historical data, but using temperature timeseries for simulation could be possible in the future. Data is NOT automatically filled into the db. Admins can upload temperature data to their liking via /temperatures/import/ The expected fileformat is a zip containing hourly, historical data of air_temperature found here

TU_Stundenwerte_Beschreibung_Stationen.txt
stundenwerte_TU_00003_19500401_20110331_hist.zip
stundenwerte_TU_00003_19500401_20110331_hist.zip
...

Temperature fetching uses caching. This requires a REDIS_URL. If no REDIS_URL is given, caching should be skipped.

Image Licensing Notice

Some visual assets in this project are licensed from Freepik under a Premium License. These assets are provided only for use within this project. They cannot be reused, redistributed, or modified outside of this context without obtaining a valid Freepik license.