Skip to content
Navigation Menu
Sign in
Appearance settings
Platform
AI CODE CREATION
GitHub Copilot
Write better code with AI
GitHub Copilot app
Direct agents from issue to merge
MCP Registry
Integrate external tools
DEVELOPER WORKFLOWS
Actions
Automate any workflow
Codespaces
Instant dev environments
Issues
Plan and track work
Code Review
Manage code changes
Code Quality
Enforce quality at merge
APPLICATION SECURITY
GitHub Advanced Security
Find and fix vulnerabilities
Code security
Secure your code as you build
Secret protection
Stop leaks before they start
EXPLORE
Why GitHub
Documentation
Blog
Changelog
Marketplace
View all features
Solutions
BY COMPANY SIZE
Enterprises
Small and medium teams
Startups
Nonprofits
BY USE CASE
App Modernization
DevSecOps
DevOps
CI/CD
View all use cases
BY INDUSTRY
Healthcare
Financial services
Manufacturing
Government
View all industries
View all solutions
Resources
EXPLORE BY TOPIC
AI
Software Development
DevOps
Security
View all topics
EXPLORE BY TYPE
Customer stories
Events & webinars
Ebooks & reports
Business insights
GitHub Skills
SUPPORT & SERVICES
Documentation
Customer support
Community forum
Trust center
Partners
View all resources
Open Source
COMMUNITY
GitHub Sponsors
Fund open source developers
PROGRAMS
Security Lab
Maintainer Community
Accelerator
GitHub Stars
Archive Program
REPOSITORIES
Topics
Trending
Collections
Enterprise
ENTERPRISE SOLUTIONS
Enterprise platform
AI-powered developer platform
AVAILABLE ADD-ONS
GitHub Advanced Security
Enterprise-grade security features
Copilot for Business
Enterprise-grade AI features
Premium Support
Enterprise-grade 24/7 support
Pricing
Type
/
to search
Sign in
Sign up
Appearance settings
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Dismiss alert
{{ message }}
bzick
/
tokenizer
Public
Notifications
You must be signed in to change notification settings
Fork
11
Star
142
Code
Issues
5
Pull requests
4
Actions
Projects
Security and quality
0
Insights
Additional navigation options
Code
Issues
Pull requests
Actions
Projects
Security and quality
Insights
Actions: bzick/tokenizer
Actions
All workflows
Workflows
CodeQL
CodeQL
tokenizer
tokenizer
Show more workflows...
Management
Caches
All workflows
All workflows
Actions
Loading...
Loading
Sorry, something went wrong.
Uh oh!
There was an error while loading.
Please reload this page
.
will be ignored since log searching is not yet available
Showing runs from all workflows
will be ignored since log searching is not yet available
50 workflow runs
50 workflow runs
Workflow
Filter by Workflow
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching workflows.
Event
Filter by Event
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching events.
Status
Filter by Status
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching statuses.
Branch
Filter by Branch
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching branches.
Actor
Filter by Actor
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
No matching users.
Scheduled
CodeQL
#51:
by
github-advanced-security
AI
1m 22s
master
master
1m 22s
Scheduled
CodeQL
#50:
by
github-advanced-security
AI
1m 16s
master
master
1m 16s
test: add fuzz seed for a long token followed by another token
tokenizer
#96:
Commit
0356120
pushed by
bzick
49s
master
master
49s
View workflow file
Push on master
CodeQL
#49:
by
bzick
1m 22s
master
master
1m 22s
Add fuzz seed for a long token followed by another token
tokenizer
#95:
Pull request
#50
synchronize by
bzick
45s
fix/stream-token-longer-than-buffer
fix/stream-token-longer-than-buffer
45s
View #50
View workflow file
PR #50
CodeQL
#48:
by
bzick
1m 6s
refs/pull/50/head
refs/pull/50/head
1m 6s
PR #50
CodeQL
#47:
by
bzick
1m 1s
refs/pull/50/head
refs/pull/50/head
1m 1s
Fix parsing of ".E0" input
tokenizer
#94:
Commit
d0f617d
pushed by
bzick
53s
master
master
53s
View workflow file
Push on master
CodeQL
#46:
by
github-advanced-security
AI
1m 29s
master
master
1m 29s
Scheduled
CodeQL
#45:
by
github-advanced-security
AI
1m 13s
master
master
1m 13s
Also fix NextToken() when first token size >= chunk size
tokenizer
#93:
Commit
9c01dd5
pushed by
bzick
45s
master
master
45s
View workflow file
Push on master
CodeQL
#44:
by
bzick
1m 19s
master
master
1m 19s
Fix parsing stream with first token size >= parser's chunk size
tokenizer
#92:
Pull request
#45
synchronize by
n-peugnet
44s
n-peugnet:issue-44-long-first-token
n-peugnet:issue-44-long-first-token
44s
View #45
View workflow file
Scheduled
CodeQL
#43:
by
github-advanced-security
AI
1m 7s
master
master
1m 7s
docs: shorter commits, no signature rule
tokenizer
#91:
Commit
5c9aa30
pushed by
bzick
43s
master
master
43s
View workflow file
Push on master
CodeQL
#42:
by
bzick
1m 37s
master
master
1m 37s
Docs/readme overhaul
tokenizer
#90:
Pull request
#49
synchronize by
bzick
44s
docs/readme-overhaul
docs/readme-overhaul
44s
View #49
View workflow file
PR #49
CodeQL
#41:
by
bzick
52s
refs/pull/49/head
refs/pull/49/head
52s
PR #49
CodeQL
#40:
by
bzick
1m 0s
refs/pull/49/head
refs/pull/49/head
1m 0s
docs: fix and expand README
tokenizer
#89:
Commit
b19e4ac
pushed by
bzick
42s
master
master
42s
View workflow file
Push on master
CodeQL
#39:
by
bzick
1m 9s
master
master
1m 9s
docs: fix and expand README, add prose line-wrap rule
tokenizer
#88:
Commit
d3c3309
pushed by
bzick
44s
master
master
44s
View workflow file
Push on master
CodeQL
#38:
by
bzick
1m 12s
master
master
1m 12s
docs: fix and expand README
tokenizer
#87:
Pull request
#48
opened by
bzick
45s
docs/readme-overhaul
docs/readme-overhaul
45s
View #48
View workflow file
PR #48
CodeQL
#37:
by
bzick
1m 1s
refs/pull/48/head
refs/pull/48/head
1m 1s
Previous
1
2
Next
You can’t perform that action at this time.