iNetVisor

A SYSTRON Lab project
from the Department of Computer Science at the University of York

Table of Contents

1. About the project
2. Getting started
3. Collecting BGP data
4. Collecting traceroute data
5. Collecting metadata
6. Forming a topology
7. Publications
8. Contact

About the project

iNetVisor collects Internet routing and reachability data from BGP and traceroutes, and constructs a topology graph supported by Internet resource metadata from registries, PeeringDB and external sources.

(back to top)

Getting started

This repository contains tooling that integrates with RIPE Atlas to collect public traceroute data and instigate new traceroute measurements. You do not need a RIPE Access account or RIPE Atlas credits to collect publicly available data from RIPE Atlas as part of the traceroute collection. However, you do need an Access account and Atlas credits to instigate new measurements on the Atlas platform.

There are a few prerequisites before running this tool:

• Install rust-lang (this allows us to use BGPKit)

Install Python prerequisites

$ pip install -r requirements.txt

(back to top)

Collecting BGP data

You can simply run:

$ run_inetvisor.sh

Alternatively you can follow the steps below to run each step manually, rather than the automated script.

Find relevant route collectors

To obtain a file containing the relevant route collectors and metadata for the interval of interest, we use route_collectors.py, and supply a start time, duration and optionally, an output file destination. This produces a .json format.

# Specifying an output is optional, will print to terminal if -o is not supplied
$ python collectors/bgp/route_collectors.py -s "dd/mm/yyyy HH:MM:SS" -d "86400" -o "collectors.json"

Obtaining a list of paths

For each collector specified in the input file, we download and decompress the MRT file and explore the records, outputting a file containing pipe-separated paths followed by accompanying IPv4/IPv6 and timestamp metadata.

$ python collectors/bgp/mrt_to_paths.py -in collectors.json -o paths.txt

This produces a file of the format P|A|T|H/IPver|timestamp, for example:

123|456|789/v4|1704110400.0

Splitting into processable files (optional)

To make the large file size of paths.txt easier to manage, you can use the inbuilt Linux command:

split -C 20GB --numeric-suffixes paths.txt paths_part_

Separating clean and undesirable paths

This step cleans the available paths, removing prepending or route servers from valid paths, and removing paths containing loops or bogons. To do this, you must supply a PeeringDB dataset file as JSON or SQLite such that we can extract known route servers. This produces a number of files: clean_paths.txt containing paths without prepending or route servers, but otherwise valid; files starting formerly_ containing valid paths in uncleaned form (for reference); and finally files starting removed_ which have been identified as invalid. All exported paths continue to have the original metadata (IP version and timestamp).

python utils/path_cleaner.py -in list_of_path_files.txt -pdb peeringdb.json -o exportdir

Naive adjacencies

Simply extracting the unique adjacencies in the cleaned as_path.

python utils/naive_adjacencies.py -in list_of_files -o naive_adjacencies.txt

Run AS-Rank

Previous work in this area has resulted in the creation of differing versions of asrank.pl, including by Gao (Gao, 2001) and the CAIDA project (Luckie, 2013). We provide format_asrank.py to run on the same list of files to allow for direct comparisons with our data. This also enables support for other work, such as Problink (Jin, 2019).

python utils/format_asrank.py -in list_of_files -o path_data.txt
./asrank.pl path_data.txt > asrank_result.txt

Other notes

The files produced and processed by this tool can be quite large, depending on the volume of data collected. We note that using TAR to compress files reduces the size of the text files significantly (~500GB of a .txt becomes ~15GB of .tar.gz). We use text files as these are simple to process without adding extraction to the execution time of the tool.

(back to top)

Collecting traceroute data

Prepare the mapping of router IP addresses to ASNs

Traditional tools introduce circular dependency into IP to ASN mappings by relying on existing AS relationship models. Some alternative approaches use statistical modelling, which could introduce inaccuracies in the process. We want to attempt a best-possible approach to accuracy, instead relying on operational data from IXPs above existing approaches.

Therefore, the approach taken in this tool is first-match from:

1. Check IXPDB for the router IP address. If present, we prefer this ASN as the data source is relatively fresh and in most cases collected directly from the IXP. We collect this data using the 'List all ASNs' option in the IXPDB API.
2. Check PeeringDB for the router IP address. If present, we use this ASN, as self-reported operational data is still preferable to prefix matching. We find a discrepancy between IXPDB and PeeringDB data in around 0.24% of cases. Observing the link state using PCH and IXP looking glasses shows many of these discrepancies come from inaccessible peering states - so we prefer the more current IXPDB data, but it is unlikely a traceroute will contain the impacted IPs.

$ python collectors/traceroute/router_to_as.py -xdb data/ixpdb_asns.json -pdb data/peeringdb_data.json -o data/router2as.json

Supplement with best-guess (pyasn)

We use pyasn on the routing table to supplement our assured data, creating an ipasn.dat file we use for secondary lookups later. Read more about pyasn.

Collect public traceroutes

Using the RIPE Atlas API

This collects all publicly available traceroutes from RIPE Atlas in the specified time period, exporting key information (probe id, start time, source, destination, and hops) into a JSON file. Providing the optional --router2as argument returns an ASN value for each traceroute IP address, if a mapping exists.

$ python collectors/traceroute/collect.py -s "DD/MM/YYYY HH:MM:SS" -d 3600 -r2as data/router2as.json -asndb data/asndb.dat -o data/traceroute.json

Using Atlas Daily Dumps (most recent 30 days)

This collects publicly available traceroute data for the most recent 30-day period based on data available in the Atlas Daily Dumps. It comprises of two scripts (one to download the .bz2 files and another to extract and translate to our format) and a modified version of the collection Python script. Both require minor changes to adjust your data source (for the first, a .txt of desired files, and for the second, paths to your data/router2as.json and data/asndb.dat)

$ collectors/traceroute/01_daily_collect.sh
$ collectors/traceroute/02_daily_translate.sh

Transform traceroutes to AS adjacencies

This part of the tool takes as input the previously exported traceroute.json data and generates a new file containing a list of adjacencies as identified in the traceroute data. Notably, the function provided by this tool is strict: an AS is only considered adjacent where two ASNs are directly adjacent in the path, and so where an internal IP address, timeout/null response is seen, the ASNs on either side are not considered adjacent (contrary to the assumption used by some other tooling).

$ python utils/trace_to_adjacency.py -in data/traceroute.json -o data/adjacencies.txt

(back to top)

Collecting metadata

You can manually collect or generate the required and supplementary metadata files, each of which should be placed in the data/source directory.

Required files

• AS adjacency data (this should be a .txt file of ASN pairings acquired using the steps above)

Optional files

• AS relationship data: these are output as customer/provider or peer labels for the edges between AS nodes. You can generate this using by running asrank.pl on the data output from the utils/format_asrank.py script (as .txt), or alternatively download it a previously compiled version from the CAIDA AS Relationships Dataset (this should be a file of the form YYYYMMDD.asrel2.txt)
• Countries data: this is a file containing basic metadata about global countries in the format COUNTRY NAME|ISO 3166 ALPHA 2 CODE|LATITUDE (AVG)|LONGITUDE (AVG)|COLOUR, used for general country-level AS placement, providing expanded country names for ASes, and a base colour for graph plotting.
• AS hegemony data: this is calculated by using the Internet Health Report's AS Hegemony tooling, and results in two .csv files of hegemony values, one for IPv4 and one for IPv6 peering.
• PeeringDB data snapshot: we use the CAIDA UCSD PeeringDB Dataset, which is a file of the form peeringdb_2_dump_YYYY_MM_DD.json. The data can alternatively be sourced using the PeeringDB API but the codebase will require some modification.
• bgp.tools tags: this is acquired using the bgp.tools API. A template script for collecting this data can be found in collectors/metadata/bgptools_tags.sh (but will require adding an appropriate --user-agent).

(back to top)

Forming a topology

From all of the data above, we can run the iNetVisor data-to-topology tool by running the following script with appropriate arguments:

$ python topology/data_to_graph.py --adjacencies "data/working/ADJACENCIES.txt" --output "data/output/GRAPH.graphml"

For a more detailed metadata graph, it also accepts the following arguments:

• --relationships: the AS relationship data file.
• --countries: the countries data file.
• --asnregs: a cache of AS-to-country mappings from registry data (if not supplied, but a countries data file is present, this will be automatically generated and saved as data/working/asn_to_country.json)
• --hegemony4: IPv4 AS hegemony data file.
• --hegemony6: IPv6 AS hegemony data file.
• --peeringdb: PeeringDB data snapshot.
• --bgptools: two bgp.tools tag data files (the data/source/tags.txt list of tags, and the directory data/source/bgptools-tags containing the list of ASes for each tag).

For example, we can run the script with all information supplied using a command similar to:

$ python topology/data_to_graph.py -a 'data/working/AS_ADJACENCIES.txt' -r 'data/working/AS_RELATIONSHIPS.txt' -c 'data/source/countries_with_colours.csv' -h4 'data/working/IPv4_HEGEMONY.csv' -h6 'data/working/IPv6_HEGEMONY.csv' -p 'data/source/peeringdb_2_dump_2025_11_06.json' -b 'data/source/tags.txt' 'data/source/bgptools-tags' -o 'data/output/graph.graphml'

(back to top)

Publications

• (Preprint) Unveiling Internet Censorship: Analysing the Impact of Nation States’ Content Control Efforts on Internet Architecture and Routing Patterns

(back to top)

Contact

Josh Levett: @Levett_Josh / joshua.levett (at) york.ac.uk

(back to top)