This project is a simplified server application inspired by Twitter (X) using NestJS, its TypeORM ,and GraphQL, focusing on implementing a permission system for tweets. It aims to replicate the permission functionalities of Twitter, where the author can control who can view or edit their tweets.
-
Default Behavior:
- A tweet inherits the parent tweet's permissions.
- If a tweet has no parent, it is accessible to everyone, and only the author can edit it.
-
Permissions:
- View:
- The author can add users or groups to the "View" permission list.
- Only listed users or group members can see the tweet. Others cannot.
- Auto-Inheritance: If enabled for a reply tweet, it will inherit the parent tweet's view permissions. Changes to the parent's permissions will reflect on this tweet.
- Edit:
- The author can add users or groups to the "Edit" permission list.
- Only listed users or group members can edit the tweet. Others cannot.
- Auto-Inheritance: Similar to "View" permissions, edit permissions can also inherit from the parent tweet.
- View:
The system leverages Redis as a cache mediator to minimize traffic to MySQL servers. If a requested item is not found in the cache, it is fetched from MySQL and subsequently cached.
-
In this design, groups are the primary mechanism for defining the visibility and editability of tweets. If a user wants to set specific view or edit permissions by assigning a combination of
UserIDsandGroupIDs, the system creates a new group that includes thoseUserIDsandGroupIDs. The newly created group's ID is then assigned to the tweet's permissions. This approach simplifies the overall design, making it more generalized and easier to manage. -
Secondly, I've considered that in the
UpdateTweetPermissionsmethod, IfUserIDsandGroupIDsare given but also inheritance is active, thoseUserIDsandGroupIDswill not be set for tweet permissions and only inheritance configuration will be saved for the tweet. This is true for both edit and view permissions. -
Thirdly, I've updated the design of the
UpdateTweetPermissionsmethod's inputs in the GraphQL schema to receivegroupIDsanduserIDsas separate arrays. This change was made because there was no specific rule defined to differentiate between auserIDand agroupIDwhen they were combined in a single array. -
I have converted
Datetypes to timestamps to eliminate the need to format dates for different timezones. This is because a timestamp doesn't have a timezone, and on the frontend side, it can be converted to a string based on the user's timezone. -
I haven't used Guard or any authentication mechanism to simplify testing.
-
The application's architecture is inspired by samples from the NestJS official repository. However, some design aspects may require improvements, which I have outlined in the
Future Enhancementssection. -
The application includes comprehensive unit tests, accounting for errors and exceptions from lower-level infrastructure. For convenience, errors and exceptions that meet the test expectations and do not cause test failures are suppressed using the
--silentoption configured in the test script ofpackage.json. If you'd like to view the full logs, you can either remove the--silentoption or run the unit tests manually. -
This application fetches tweets for the feed solely from the cache layer to ensure scalability, avoiding direct database queries for every user feed. The caches are populated through the
UpdateTweetPermissionsprocess. Additionally, when a new tweet is created, the caches related to the author are updated to include the user's self-created content.
In the event that Redis caches are cleared (e.g., the database is flushed), rebuild feed indexes with npm run cache:warm. Tweet body lookups fall back to MySQL on cache miss and re-populate Redis automatically.
In the future, the list of user-created groups can be displayed in the UI to avoid repeatedly creating groups with the same users or items. Additionally, server-side logic can be implemented to detect and prevent the creation of duplicate groups. If a user attempts to create a group with identical members to an existing one, the system could reuse that group instead of creating another.
Demo auth uses the x-user-id HTTP header (or GraphQL userId / authorId / creatorId arguments). This is intentionally simple for local testing — replace with real authentication before production.
Environment files are not committed. Copy .env.example to .env and test.env.example to test.env before running.
Rebuild Redis feed indexes after a flush with:
npm run cache:warm
I have used this caching design in the Quiz of Kings game, and it worked smoothly with over 30 million users and nearly 1 million daily active users.
-
Structure:
- Each tweet is serialized using JSON (but protobuf also could be used as a further improvement) and stored in Redis with a key:
cache:tweet:$tweetID. - A TTL (Time-To-Live) is assigned to cached tweets to ensure only relevant data is stored.
- Cache updates occur during tweet creation or modification.
- Each tweet is serialized using JSON (but protobuf also could be used as a further improvement) and stored in Redis with a key:
-
Cache Miss Handling:
If a tweet is not found in Redis, it is retrieved from MySQL and cached again. -
Cache Refresh Policy:
When a tweet is viewed, its TTL is extended to improve access speed for a period after the view.
- Retrieve Public Tweet IDs:
Fetch public tweets the user can see (paginated). - Retrieve Private Tweet IDs:
Fetch private tweets the user is permitted to access. - Merge and Sort:
Combine public and private tweets, sorted by timestamp. - Apply Filters:
Filter tweets based on criteria such as hashtags or categories. - Paginate the Feed:
Prepare the feed and deliver it to the user.
Instead of querying MySQL for every user feed request:
-
Use Redis ZSETs (Sorted Sets) for public tweets:
- Example query:
ZRANGEBYSCORE tweets:public $min $max WITHSCORES LIMIT $offset $count$minand$maxdefine the time range.$offsetand$counthandle pagination.
- Example query:
-
Chunking Keys:
Divide public tweet data into chunks for scalability. Examples:- Monthly:
tweets:public:2024_10 - Weekly:
tweets:public:2024_10_1 - Daily:
tweets:public:2024_10_01
- Monthly:
-
User Groups:
Redis Sets are used to cache group IDs for each user.- If the cache is empty, the group data is fetched from MySQL and stored in Redis.
- For new users, a placeholder key can indicate recent database queries to avoid repeated checks.
-
Private Tweet Retrieval:
Similar to public tweets, private tweets for groups are stored in Redis ZSETs.- Large datasets are chunked into separate keys based on time.
- Merge private and public tweet lists based on timestamps.
- Fetch the serialized tweet data (protobuf) from Redis for each tweet.
- Serve the final feed to the user with minimal MySQL dependency.
This project uses environment variables for configuration. The database and Redis credentials are stored in .env files:
.env: Contains the main environment variables for developmenttest.env: Contains environment variables for testing
These files are tracked in the repository for demonstration purposes, but in a real production environment, you should add them to .gitignore to keep sensitive information secure.
| Variable | Description |
|---|---|
| DB_HOST | MySQL database host |
| DB_PORT | MySQL database port |
| DB_USERNAME | MySQL database username |
| DB_PASSWORD | MySQL database password |
| DB_DATABASE | MySQL database name |
| REDIS_HOST | Redis host |
| REDIS_PORT | Redis port |
| TEST_DB_HOST | Test MySQL database host |
| TEST_DB_PORT | Test MySQL database port |
| TEST_DB_USERNAME | Test MySQL database username |
| TEST_DB_PASSWORD | Test MySQL database password |
| TEST_DB_DATABASE | Test MySQL database name |
- Adding "Seen" Feature: By introducing a "seen" feature with a
seentable (a relation betweenuser_idandtweet_id) and leveraging Redis caching mechanisms (using data structures like ZSET or SET), we can prevent the display of repetitive tweets that users have already seen.
To further enhance the system's scalability and performance, the following improvements can be implemented:
- Enhanced Chunking Strategies: Develop more advanced chunking methods for Redis keys to ensure efficient data retrieval and management.
- Optimized Feed Filtering: Refine feed filtering mechanisms to cater to advanced user preferences, such as custom hashtags, categories, or other criteria.
- Redis Clustering: Transition to Redis clustered deployments with replicas for each master node, ensuring higher availability and fault tolerance.
- MySQL Partitioning: Partition MySQL tables based on time, with a robust policy for removing old partitions after creating backups. This keeps database performance optimal and reduces the need for scaling resources.
- MySQL Query Indexing: Defining indexes which are aligned with defined and used queries in code
- Sharded and Replicated MySQL: Scale MySQL by implementing sharding and replication. Tools like Vitess can be used to manage large-scale MySQL deployments effectively.
- Use UUID in every where: In certain entities, such as
UserandGroup, I have used integer IDs for simplicity. However, in a production environment, using Universally Unique Identifiers (UUIDs) is a better practice for ensuring greater uniqueness and scalability. - Decoupling Modules: As seen here, due to the use of Nest ORM (TypeORM), even though we have separate modules such as
User,Group, andTweet, the relational nature of the data leads to some degree of coupling between these modules. This is because relationships must be defined within the ORM entities. This coupling can become problematic, especially in scenarios where you might want to use a different database (e.g., MongoDB) for theUsermodule. Such tight coupling can make future changes more cumbersome and harder to manage. - Implementing Error Codes: An essential improvement is to define distinct error codes for logical errors and internal errors, ensuring better error handling and debugging. Additionally, enhancing logging capabilities for internal processes is crucial. Leveraging NestJS Exception Filters can streamline error management and improve the overall system robustness.
- Using key caches to prevent hot key problem in Redis: If a tweet becomes so popular that it is being fetched hundreds of times per second, Redis may face a "Hot Key" problem, as each key is stored in a specific node. This can result in high pressure on certain Redis nodes. To prevent this issue, we can log the access rate of each key in a map on the application servers (a map of the 1000 most recently hot keys). For these keys, we can also cache their content in the RAM of each application server, helping to overcome the Redis hot key problem.
- Prevention of creating repetitious hashtags: In cases where a new hashtag becomes very popular, if two users use it and publish their tweets at the same time, it may result in the creation of duplicate hashtags. To prevent this, we could use distributed locks such as Redis locks or even MySQL locks.
Before running the project, ensure that both MySQL and Redis are up and running as defined in the docker-compose.yml file.
You can start them by running the following command:
docker-compose up -dThen you should install project dependencies via npm using the following command:
npm install# development
$ npm run start
# watch mode
$ npm run start:dev
# production mode
$ npm run start:prodIf you make any changes to the *.graphql files in any module, you will need to update the main (merged) auto-generated GraphQL schema. To do this, run the following command:
npm run generate:typingsmutation {
createUser(createUserInput: {
userName: "john_doe"
firstName: "John"
lastName: "Doe"
}) {
id
userName
firstName
lastName
}
}
mutation {
createTweet(createTweetInput: {
content: "Hello World!",
authorId: 1,
hashtags: ["#greeting", "#greetings"],
location: "Tehran",
category: "Tech"
}) {
id
content
authorId
hashtags
location
category
}
}
mutation {
createGroup(createGroupInput: {
name: "New Group"
userIds: [1, 2, 3]
parentGroupId: 1
creatorId: 1
}) {
id
name
users {
id
userName
}
parentGroup {
id
name
}
creator {
id
userName
}
createdAt
updatedAt
}
}
mutation UpdateTweetPermissions {
updateTweetPermissions(
id: "839f5b99-2abd-4f33-aa63-ca92ff40a435",
input: {
inheritViewPermissions: false,
inheritEditPermissions: false,
viewPermissionsUserIds: [1,2],
viewPermissionsGroupIds: [1,2],
editPermissionsUserIds: [1,2],
editPermissionsGroupIds: [1,2],
},
userId: 1
)
}
query CanEdit {
canEditTweet(
tweetId: "839f5b99-2abd-4f33-aa63-ca92ff40a435",
userId: 3
)
}
query PaginateTweets {
paginateTweets(userId: 1, limit: 10, page: 0) {
nodes {
id
createTime
updateTime
authorId
content
hashtags
parentTweetId
category
location
}
hasNextPage
}
}
query PaginateTweets {
paginateTweets(
userId: 1
limit: 10
page: 0
filter: {
authorId: "1"
hashtag: "#greetings"
parentTweetId: "parent123"
category: Tech
location: "Helsinki"
}
) {
nodes {
id
authorId
parentTweetId
hashtags
createTime
category
location
}
hasNextPage
}
}
# unit tests
$ npm run test
# e2e tests
$ npm run test:e2e
# test coverage
$ npm run test:covWhen you're ready to deploy your NestJS application to production, there are some key steps you can take to ensure it runs as efficiently as possible. Check out the deployment documentation for more information.
If you are looking for a cloud-based platform to deploy your NestJS application, check out Mau, our official platform for deploying NestJS applications on AWS. Mau makes deployment straightforward and fast, requiring just a few simple steps:
npm install -g mau
mau deployWith Mau, you can deploy your application in just a few clicks, allowing you to focus on building features rather than managing infrastructure.