Skip to content

waldur-testing-guide.md

Latest commit

 

History

History
226 lines (172 loc) · 8.95 KB

waldur-testing-guide.md

File metadata and controls

226 lines (172 loc) · 8.95 KB
Raw

Waldur Testing Guide

Test Writing Best Practices

1. Understand Actual System Behavior

  • Always verify actual behavior before writing tests - Don't assume how the system should work
  • Test what the system actually does, not what you think it should do
  • Example: Basic permission queries don't automatically filter expired roles

2. Use Existing Fixtures and Factories

  • Always use established fixtures - Don't invent your own role names
  • Use CustomerRole.SUPPORT not CustomerRole.MANAGER (which doesn't exist)
  • Use fixtures.ProjectFixture() for consistent test setup with proper relationships
  • Use factories.UserFactory() for creating test users with proper defaults

3. Error Handling Reality Check

  • Test for actual exceptions, not ideal ones
  • If the system raises AttributeError for missing attributes, test for AttributeError
  • Only test for PermissionDenied when the system actually catches and converts errors

4. Mock Objects for Complex Testing

  • Use Mock objects effectively for nested permission paths
  • Create realistic mock structures: mock_resource.project.customer = self.customer
  • Test permission factory with multiple source paths: ["direct_customer", "project.customer"]
  • Mock objects help test complex scenarios without database overhead

5. Time-Based Testing Patterns

  • Understand explicit vs implicit time checking
  • Basic has_permission() doesn't check expiration times automatically
  • Test boundary conditions: exact expiration time, microseconds past expiration
  • Create roles with timezone.now() ± timedelta() for realistic time testing

6. Test Base Class Selection

Choose the right test base class for each test:

  • Default: test.APITestCase — uses transaction rollback, much faster
  • test.APITransactionTestCase is a last resort. It truncates every table between tests. Only two situations actually need it:
    1. Threading or multi-process database access (select_for_update under real concurrency)
    2. transaction.on_commit() callbacks must fire and wrapping the triggering call is impractical

Three reasons that look like they need it, but do not:

Looks like it needs TransactionTestCase What to do instead
transaction.on_commit() must fire Wrap the triggering call in self.captureOnCommitCallbacks(execute=True)
Deliberate IntegrityError Raise it inside transaction.atomic() — the savepoint keeps the wrapping transaction intact
responses.start() in setUp Register addCleanup(responses.reset) before addCleanup(responses.stop); leaked mocks come from a missing reset(), not from the base class 
# GOOD: Default to APITestCase
class MyTest(test.APITestCase):
    def test_something(self):
        ...

# GOOD: on_commit under APITestCase — capture the callbacks explicitly
class OrderProcessingTest(test.APITestCase):
    def test_order_triggers_task(self):
        with self.captureOnCommitCallbacks(execute=True):
            self.client.post(self.url, payload)
        mock_task.delay.assert_called_once()

# GOOD: a deliberate IntegrityError, contained by a savepoint
class UniqueConstraintTest(test.APITestCase):
    def test_duplicate_is_rejected(self):
        with self.assertRaises(IntegrityError), transaction.atomic():
            Model.objects.create(**duplicate)

A CI lint job (scripts/analyze_transaction_test_cases.py --ci --baseline 0) enforces this: any APITransactionTestCase class the analyzer cannot see a reason for fails the pipeline. When the reason lives in production code the analyzer cannot see — typically an on_commit() in a signal handler the test drives through the API — state it in a comment on the class:

# APITransactionTestCase required: the order handler dispatches the task
# from transaction.on_commit
class OrderNotificationTest(test.APITransactionTestCase):
    ...

7. Performance Testing Considerations

  • Include query optimization tests where appropriate
  • Use override_settings(DEBUG=True) to count database queries
  • Test with multiple users/roles to ensure performance doesn't degrade

8. System Role Protection

  • Test that system roles work correctly even when modified
  • System roles like CustomerRole.OWNER should maintain functionality
  • Test that role modifications don't break core functionality
  • Verify that predefined roles have expected permissions

9. Edge Case Testing

  • Test None values, missing attributes, and circular references
  • Handle AttributeError when accessing missing nested attributes
  • Test with inactive users, deleted roles, removed permissions
  • Verify behavior with complex nested object hierarchies

10. HTTP Mocking Patterns

Preferred: @responses.activate per method — fully isolated, no cleanup needed:

class MyTest(test.APITestCase):
    @responses.activate
    def test_external_call(self):
        responses.add(responses.GET, "https://api.example.com/data", json={"ok": True})
        result = my_function()
        self.assertEqual(result, {"ok": True})

Class-wide mocking with responses.start() — requires APITransactionTestCase:

class ExternalAPITest(test.APITransactionTestCase):
    """responses.start() in setUp leaks state across TestCase classes."""

    def setUp(self):
        super().setUp()
        responses.start()
        responses.add(responses.GET, "https://api.example.com/data", json={"ok": True})

    def tearDown(self):
        responses.stop()
        responses.reset()
        super().tearDown()

Using responses.start() in setUp with APITestCase causes leaked mock state across test classes because TestCase doesn't fully reset process-level state between classes.

11. Multiple Inheritance Pitfall

When combining APITransactionTestCase with a mixin that extends APITestCase, Python's MRO can silently break TransactionTestCase behavior:

# BAD: MRO puts TestCase._fixture_teardown first
class MyTest(test.APITransactionTestCase, SomeTestMixin):
    ...  # SomeTestMixin extends APITestCase — TransactionTestCase teardown is skipped

# GOOD: Ensure all parents use TransactionTestCase, or use standalone setup
class MyTest(test.APITransactionTestCase):
    def setUp(self):
        super().setUp()
        # Set up mocks directly instead of inheriting from a TestCase mixin

The declaration is already misleading — the class reads as a TransactionTestCase while running with TestCase semantics — and it turns into a hard error the moment the first base is migrated: a base class may not precede its own subclass, so class MyTest(test.APITestCase, SomeTestMixin) cannot be linearised and the whole module fails to import. Drop the redundant base rather than rewriting it:

# GOOD: SomeTestMixin already supplies APITestCase
class MyTest(SomeTestMixin):
    ...

analyze_transaction_test_cases.py fails CI on an unlinearisable base list and warns about a mixed one.

12. OpenStack Backend Test Patterns

When writing standalone backend tests that don't inherit from BaseBackendTestCase:

class StandaloneBackendTest(test.APITransactionTestCase):
    def setUp(self):
        super().setUp()
        self.fixture = openstack_fixtures.OpenStackFixture()
        # Mock all 5 OpenStack clients
        self.mock_admin = mock.patch("waldur_openstack.openstack_base.backend.AdminSession").start()
        self.mock_session = mock.patch("waldur_openstack.openstack_base.backend.SessionManager").start()
        self.mock_nova = mock.patch("waldur_openstack.openstack_base.backend.NovaClient").start()
        self.mock_neutron = mock.patch("waldur_openstack.openstack_base.backend.NeutronClient").start()
        self.mock_cinder = mock.patch("waldur_openstack.openstack_base.backend.CinderClient").start()

    def tearDown(self):
        mock.patch.stopall()
        super().tearDown()

Test Guidelines

  • Test behavior, not implementation
  • One assertion per test when possible
  • Clear test names describing scenario
  • Use existing test utilities/helpers
  • Tests should be deterministic

Debugging Complex Systems

When fixing performance or accuracy issues:

  1. Isolate the problem:
  • Run individual failing tests to understand specific issues
  • Use pytest -v -s for verbose output with print statements
  • Check if multiple tests fail for the same underlying reason
  1. Understand test expectations:
  • Read test comments carefully - they often explain intended behavior
  • Check if tests expect specific error types
  • Look for conflicting expectations between test suites
  1. Fix systematically:
  • Fix one root cause at a time
  • After each fix, run full test suite to check for regressions
  • Update related tests for consistency when changing behavior
  1. API changes require test updates:
  • When changing function signatures or default parameters, expect test failures
  • Update tests for consistency rather than reverting functional improvements
  • Document parameter behavior changes clearly