Technical Description of the Data Processor App

This project provides a semi-automatic data cleansing solution for technical descriptions in database tables. It normalizes and standardizes technical terms, such as "n8n (workflows, data tables)", "ELT/ETL pipelines" to ensure uniform formatting and consistent nomenclature. The tool improves data quality, enables effective matching, and optimizes database usage for clearer presentation of technical descriptions.

Data cleansing

To clean the data, several rules need to be applied. Rules are imported dynamically from the rules folder and can be ordered how they should be applied.

Existing Rules:

• Labelled: e.g. "Data Science: Python, R, Julia"
• Concatenated: e.g. "n8n/n8n self-hosted"
• Distributed rule: e.g. "n8n (workflows, credentials, data tables)"
• Versioned: e.g. "n8n 1.114.4"

LLM static transformation:

For extremely long texts containing multiple technical descriptions and enumerations a Large Language Model (LLM) can be used to assist in the cleaning process by:

• Cleaning up or extracting technologies and technical terms from descriptive text using LLMs.
• Testing and documentation of edge cases and manual cleaning of descriptions that cannot be clarified automatically.
• Performing similarity comparisons between technical descriptions (e.g. using the similarity function or alternatives such as Levenshtein).

There is a preview code for using LLMs (LLM.py).

What needs to be cleaned?

The dataset contains several issues and inconsistencies in technical descriptions, as changes in naming conventions and formats have occurred over time.

Rules

The rules for data cleansing are defined in the rules/ folder. Each rule is implemented as a separate Python module and follows a specific structure. There is an example rule (rules/example_rule.py) to use as a template for creating new rules.

Tests

In folder tests several unit tests are located to test the rules (located in rules/). There is an example test for the example rule (tests/test_example_rule.py). The package pytest is used for testing.

Python - Streamlit app

Functionality:

• Semi-automatic data cleansing of technical descriptions
• Visualisation of technical description length distribution
• Order of rules to apply
• Manual adjustment of rules
• Documentation of the steps performed and the rules used

Setup

Do the following steps to set up the environment for the Python/Streamlit app:

# Installing the required packages
python -m pip install -r doc/requirements.txt

You need to store the OpenAI API key in .streamlit/secrets.toml:

[OPENAI]
OpenAI_API_KEY = "sk-xxx"

Running the app

To run the app from backend repo folder:

# Running the Python script
python -m streamlit run doc/dataprocessor.py