HubMark has transitioned from a Markdown-first storage approach to a JSON-first architecture. This change provides better data integrity, reliable sync operations, and the ability to generate multiple presentation formats while maintaining a single source of truth.
- All bookmark data is stored in structured JSON format (
bookmarks.json) - Markdown files are generated views, never edited directly
- JSON schema validation ensures data integrity
- Deterministic bookmark IDs based on content hash (no timestamps)
- Consistent identification across devices and sync operations
- URL canonicalization for robust matching
- 3-way merge algorithm with configurable strategies
- Proper handling of concurrent modifications
- Exponential backoff retry on write conflicts
GitHub Repository:
├── bookmarks.json # 📊 Single source of truth (structured data)
├── README.md # 📖 Generated human-readable view
└── (future: additional generated views)
The canonical data structure follows this schema:
{
"schemaVersion": 1,
"generatedAt": "2023-09-05T12:00:00.000Z",
"bookmarks": [
{
"id": "hm_2f0b9c1f6d1e7f2b5c7a2b1a9d3c4e5f",
"title": "React Documentation",
"url": "https://reactjs.org",
"folder": "Development/Frontend",
"tags": ["react", "javascript"],
"notes": "Great reference for TS",
"dateAdded": 1693929600000,
"dateModified": 1693929600000,
"archived": false,
"favorite": false
}
],
"meta": {
"generator": "HubMark",
"generatorVersion": "0.1.0",
"lastSync": 1693929600000
}
}- Strict validation using Ajv JSON Schema validator
- Required fields:
schemaVersion,bookmarks(each bookmark requiresid,title,url,dateAdded,dateModified) - Type safety: Strong typing with TypeScript interfaces
- Additional properties: Rejected to prevent schema drift
Bookmark IDs are generated using a deterministic algorithm:
- URL Canonicalization: Normalize URL format for consistent matching
- Title Normalization: Clean up whitespace and formatting
- Composite Key: Combine canonical URL and normalized title
- Hash Generation: Create SHA-256 hash of composite key
- ID Format: Prefix with
hm_and truncate to 32 characters
// Input: "HTTP://www.Example.com/path/?utm_source=test&ref=social#section"
// Output: "https://example.com/path"Transformations:
- Convert hostname to lowercase
- Remove
www.prefix - Promote HTTP to HTTPS (configurable)
- Remove tracking parameters (
utm_*,gclid,fbclid, etc.) - Remove hash fragments
- Remove trailing slashes (except root)
- Consistency: Same URL variations produce identical IDs
- Stability: No timestamps means IDs never change for same content
- Conflict Resolution: Reliable matching across sync operations
- Deduplication: Automatic detection of duplicate bookmarks
graph TD
A[Read Remote JSON] --> B[3-Way Merge]
C[Local Changes] --> B
D[Base State] --> B
B --> E[Conflict Resolution]
E --> F[Write Complete Dataset]
F --> G[Generate Markdown Views]
-
Read Phase
- Fetch
bookmarks.jsonfrom GitHub - Handle 404 (new repository) gracefully
- Validate JSON schema
- Fetch
-
Merge Phase
- Create maps by bookmark ID for efficient lookup
- Apply local additions/modifications
- Apply deletions
- Detect conflicts using 3-way comparison
-
Conflict Resolution
- Latest-wins: Compare
dateModifiedtimestamps - Local-wins: Prefer local changes
- GitHub-wins: Prefer remote changes
- Manual: Surface conflicts for user resolution
- Latest-wins: Compare
-
Write Phase
- Validate merged dataset against schema
- Write complete bookmark set to
bookmarks.json - Retry on 409 conflicts with exponential backoff
- Generate markdown views only if content changed
interface BookmarkConflict {
id: string;
local: HubMarkBookmark; // Local version
remote: HubMarkBookmark; // Remote version
base?: HubMarkBookmark; // Last known common version
}3-Way Merge Logic:
- If only local changed: use local
- If only remote changed: use remote
- If both changed: apply conflict resolution strategy
- If content identical: no conflict
Base64 Encoding (replaces Node.js Buffer):
HubMark implements browser-safe base64 encoding to ensure MV3 service worker compatibility. See Browser-Safe Base64 Encoding for complete details.
import { encodeBase64, decodeBase64 } from '~/utils/base64';
// Encode for GitHub API (MV3 compatible)
const encoded = encodeBase64(jsonContent);
// Decode from GitHub API (handles Unicode correctly)
const decoded = decodeBase64(encodedContent);Key Features:
- ✅ No Node.js Buffer: Uses native Web APIs (
btoa,atob,encodeURIComponent) - ✅ Unicode Support: Handles emojis and international characters correctly
- ✅ MV3 Service Workers: Works in isolated service worker contexts
- ✅ GitHub API Compatible: Produces valid base64 for API operations
- ✅ Error Handling: Graceful handling of malformed input
Crypto Operations:
- Uses Web Crypto API (
crypto.subtle.digest) - No dependency on Node.js crypto module
- Works in service workers and main thread
Generated README includes:
- Header: Title and generation timestamp
- Statistics: Total bookmark count
- Folder Organization: Bookmarks grouped by folder
- Favorites: Special marking for favorite bookmarks ⭐
- Tags: Inline tag display with backticks
- Archived Section: Collapsible section for archived bookmarks
- Footer: Note about auto-generation
# My Bookmarks
*Generated by HubMark on 9/5/2023*
Total bookmarks: 42
## Development/Frontend
- [React Documentation](https://reactjs.org) ⭐ `react` `javascript`
> Great reference for TS
- [Vue.js Guide](https://vuejs.org/guide/)
## Archived (5)
<details>
<summary>Show archived bookmarks</summary>
- [Old Framework](https://example.com/old)
</details>
---
*This file is automatically generated from [bookmarks.json](./bookmarks.json). Do not edit directly.*For existing repositories with bookmarks.md:
- Detection: Check if
bookmarks.jsonexists - Parsing: Best-effort parse of existing Markdown
- Validation: Generate valid JSON structure
- Backup: Rename original to
bookmarks-legacy.md - Generation: Create new README from JSON
The extension has been updated to use MV3-compatible base64 encoding:
Before (broken in MV3):
// ❌ Crashes in service workers
const encoded = Buffer.from(content, 'utf8').toString('base64');
const decoded = Buffer.from(encoded, 'base64').toString('utf8');After (MV3 compatible):
// ✅ Works in all browser contexts
import { encodeBase64, decodeBase64 } from '~/utils/base64';
const encoded = encodeBase64(content);
const decoded = decodeBase64(encoded);Migration Benefits:
- ✅ MV3 Service Worker Support: No more
Buffer is not definederrors - ✅ Unicode Preservation: Handles international text and emojis correctly
- ✅ Performance: Native Web API calls are faster than polyfills
- ✅ GitHub API Compatibility: Produces valid base64 for all GitHub operations
- ✅ Error Handling: Graceful failure with detailed error messages
See Browser-Safe Base64 Encoding for complete migration guide and technical details.
Future schema versions will:
- Increment
schemaVersionfield - Maintain backward compatibility
- Provide migration utilities
- Validate against appropriate schema version
Main client for JSON-first GitHub operations:
const client = new JSONGitHubClient(githubConfig);
// Read bookmark data
const { data, sha } = await client.readBookmarkData();
// Write with conflict retry
const newSha = await client.writeBookmarkData(data, message, sha);
// 3-way merge
const result = await client.mergeBookmarks(base, local, remote, deletions, strategy);
// Generate views
await client.updateReadmeIfChanged(data);JSON schema validation:
import { schemaValidator } from '~/utils/json-schema';
// Validate data
const result = schemaValidator.validate(data);
if (!result.valid) {
console.error(result.errors);
}
// Validate and throw on error
schemaValidator.validateOrThrow(data);import { generateStableId, canonicalUrl, normalizeTitle } from '~/utils/stable-id';
// Generate stable bookmark ID
const id = await generateStableId(url, title);
// URL canonicalization
const canonical = canonicalUrl("HTTP://www.Example.com/path/?utm_source=test");
// Returns: "https://example.com/path"
// Title normalization
const normalized = normalizeTitle(" Multi Space Title ");
// Returns: "Multi Space Title"import { bookmarkContentDiffers, generateContentHash } from '~/utils/stable-id';
// Compare bookmark content (ignoring dates)
const differs = bookmarkContentDiffers(bookmark1, bookmark2);
// Generate content hash for quick comparison
const hash = await generateContentHash(bookmark);- Schema validation prevents corruption
- Atomic write operations
- Complete dataset writes (no partial updates)
- Deterministic conflict resolution
- 409 retry with exponential backoff
- 3-way merge prevents data loss
- Generate different Markdown formats
- Maintain single source of truth
- Consistent presentation across views
- Efficient JSON parsing vs. fragile Markdown parsing
- Quick content comparison via hashing
- Minimal API calls (write only when changed)
- Easy to add new bookmark fields
- Schema versioning for migrations
- Multiple output formats possible
- Browser-safe crypto operations
- No Node.js dependencies
- Service worker compatible
Comprehensive test coverage includes:
- Stable ID Generation: 29 test cases covering canonicalization, normalization, and consistency
- JSON Schema Validation: 17 test cases covering valid/invalid data scenarios
- Content Comparison: Edge cases for bookmark diffing
- URL Canonicalization: Various URL formats and edge cases
Run tests:
npm test utils/stable-id.test.ts
npm test utils/json-schema.test.ts- Conditional Writes: Only write files when content actually changes
- Content Hashing: Quick comparison without full object comparison
- Batch Operations: Combine multiple changes into single commit
- Caching: Memory cache of SHA values during sync operations
- Exponential Backoff: 250ms, 750ms, 2250ms on 409 conflicts
- Jitter: Random delay component to prevent thundering herd
- Minimal API Calls: Read current state only when necessary
- Private Repositories: Recommended for personal bookmarks
- Token Storage: GitHub token in
storage.local(not synced by default) - Content Warnings: User awareness of notes/tags in repository
- Schema Enforcement: Reject malformed data
- Input Sanitization: URL and title validation
- Size Limits: Prevent oversized data structures
- Per-Folder Files: Reduce conflict surface for large bookmark sets
- Content Compression: Gzip compression for large datasets
- Incremental Sync: Delta-based updates for efficiency
- Backup Strategies: Automated backup branches
- Import/Export: Standard bookmark format support
Version 2 considerations:
- Hierarchical folder structures
- Rich text notes with Markdown
- Custom metadata fields
- Bookmark relationships/linking