pt_dm Introduction

Download manager for Gnip Historical PowerTrack. Written in Ruby for Windows 7, MacOS and Linux.

Introduction

This application automates the downloading and processing of Historical PowerTrack data files.

The application is made up of two distinct scripts: dm_ui and dm_process. dm_ui provides the user interface (currently developed in Ruby Tk) for entering account information, job details, and download options. The user interface also enables monitoring of the download progress, as well as launches the dm_process script. The dm_process executable manages the workflow as files are downloaded, converted and consolidated.

The pt_dm package helps manage the following processes:

• Downloading Historical PowerTrack files.
• Converting activities from JSON to CSV with reference to a "template" tweet in JSON format.
• Consolidating files into longer duration files:
  • 10-minute files to hourly and daily files.
  • All files can be consolidated into a single file (subject to configurable limits).
  • Currently supports JSON and CSV.

When deploying on Windows, the application is made up of two executables: dm_ui.exe and dm_process.exe. When deploying on MacOS or Linux, the Ruby interpreter is used to run the dm_ui.rb script.

Note: dm_process can also be used with a headless script, dm_script.rb (see script_v1 branch for a snapshot at the end of that mini-poc)..