Intro
Currently we have unit tests for internal components (src/tests) and a few simple integration tests for public API (src/tests/public_api).
All these tests are non-real-time, i.e. they don't expect any action, like delivering a packet, to take any specific amount of time. Actually they mostly are not tied to time at all. It has both advantages and disadvantages:
-
Tests can run in any conditions. We run them on CI, which workers are often slow, and on valgrind, which also randomly slows down the test execution.
-
On the other hand, such tests can't check how well Roc behaves in real-time conditions. E.g. what is the service quality and the latency, are there underruns and overruns, etc.
To fill this gap, we have created this repo, rt-tests. It will contain various real-time integration tests for public API.
This repo will contain non-functional tests, i.e. tests which result is not just OK/failed, but instead it may be some metric (e.g. 90% loss) or even a plot (e.g. latency trend). We will periodically run these tests on real hardware and analyze results for regressions. In future we will also run this tests on real networks.
Unlike the main repo, rt-tests uses CMake, C++17, and Google Test.
Task
Currently rt-tests contains a skeleton of ServiceQuality test.
https://github.com/roc-streaming/rt-tests/blob/1212a60b0357c851282be3ec1cb1887e652ab22e/tests/test_service_quality.cpp
This issue is for completing that test.
The ServiceQuality test should start two threads. The first thread should write some predefined sequence of samples to the sender. The second should read samples from receiver and check the sequence.
There should be several sub-tests.
First we should check that the sample sequence is the same on sender and receiver (except possible leading zeros) when there is no packet loss.
Then we should introduce some amount of packet loss. It can be done by implementing a simple UDP-level proxy, like we do in our current integration test (but without using internal roc modules). Then check what is the service quality on different combinations of: FEC scheme, FEC block size, FEC redundancy level, and network loss level. We should test three FEC schemes: none, Reed-Solomon, LDPC-staircase.
The service quality can be measured as the percent of dropped samples, i.e. samples that are equal to zero instead of the expected value (on receiver). 100% service quality on 5% packet loss means that all 5% of lost packets were restored using FEC. 95% service quality on 10% of packet loss means that we restored 5% of packets, but was not able to restore another 5% of packets.
We can print a table with results (measured service quality values for different combinations of parameters) to stdout.
In future we will extend this test to run over real network to measure actual service quality on different network types.
Checks
This issue provides an example of how you can produce samples on sender and check them on receiver: roc-streaming/roc-go#101
That algorithm was implemented in this PR: roc-streaming/roc-go#111
These issue and PR are from roc-go repo (Go bindings for roc-toolkit), where we have a test that has some similarities with the one described here.
Info
See "Workflow" section in README: https://github.com/roc-streaming/rt-tests#workflow
Public API is documented here: https://roc-streaming.org/toolkit/docs/api.html
FEC is documented here: https://roc-streaming.org/toolkit/docs/internals/fec.html
Our current non-real-time integration test can be found here:
https://github.com/roc-streaming/roc-toolkit/blob/master/src/tests/public_api/test_sender_receiver.cpp
Intro
Currently we have unit tests for internal components (
src/tests) and a few simple integration tests for public API (src/tests/public_api).All these tests are non-real-time, i.e. they don't expect any action, like delivering a packet, to take any specific amount of time. Actually they mostly are not tied to time at all. It has both advantages and disadvantages:
Tests can run in any conditions. We run them on CI, which workers are often slow, and on valgrind, which also randomly slows down the test execution.
On the other hand, such tests can't check how well Roc behaves in real-time conditions. E.g. what is the service quality and the latency, are there underruns and overruns, etc.
To fill this gap, we have created this repo, rt-tests. It will contain various real-time integration tests for public API.
This repo will contain non-functional tests, i.e. tests which result is not just OK/failed, but instead it may be some metric (e.g. 90% loss) or even a plot (e.g. latency trend). We will periodically run these tests on real hardware and analyze results for regressions. In future we will also run this tests on real networks.
Unlike the main repo, rt-tests uses CMake, C++17, and Google Test.
Task
Currently rt-tests contains a skeleton of ServiceQuality test.
https://github.com/roc-streaming/rt-tests/blob/1212a60b0357c851282be3ec1cb1887e652ab22e/tests/test_service_quality.cpp
This issue is for completing that test.
The ServiceQuality test should start two threads. The first thread should write some predefined sequence of samples to the sender. The second should read samples from receiver and check the sequence.
There should be several sub-tests.
First we should check that the sample sequence is the same on sender and receiver (except possible leading zeros) when there is no packet loss.
Then we should introduce some amount of packet loss. It can be done by implementing a simple UDP-level proxy, like we do in our current integration test (but without using internal roc modules). Then check what is the service quality on different combinations of: FEC scheme, FEC block size, FEC redundancy level, and network loss level. We should test three FEC schemes: none, Reed-Solomon, LDPC-staircase.
The service quality can be measured as the percent of dropped samples, i.e. samples that are equal to zero instead of the expected value (on receiver). 100% service quality on 5% packet loss means that all 5% of lost packets were restored using FEC. 95% service quality on 10% of packet loss means that we restored 5% of packets, but was not able to restore another 5% of packets.
We can print a table with results (measured service quality values for different combinations of parameters) to stdout.
In future we will extend this test to run over real network to measure actual service quality on different network types.
Checks
This issue provides an example of how you can produce samples on sender and check them on receiver: roc-streaming/roc-go#101
That algorithm was implemented in this PR: roc-streaming/roc-go#111
These issue and PR are from roc-go repo (Go bindings for roc-toolkit), where we have a test that has some similarities with the one described here.
Info
See "Workflow" section in README: https://github.com/roc-streaming/rt-tests#workflow
Public API is documented here: https://roc-streaming.org/toolkit/docs/api.html
FEC is documented here: https://roc-streaming.org/toolkit/docs/internals/fec.html
Our current non-real-time integration test can be found here:
https://github.com/roc-streaming/roc-toolkit/blob/master/src/tests/public_api/test_sender_receiver.cpp