-
-
Notifications
You must be signed in to change notification settings - Fork 751
fix(cluecode): detect glued Author:Name C++ tags #5236
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -428,6 +428,9 @@ def get_tokens(numbered_lines, splitter=re.compile(r'[\t =;]+').split): | |
| if TRACE_TOK: | ||
| logger_debug(' get_tokens: preped line: ' + repr(line)) | ||
|
|
||
| # Author:Frankie.Chu -> Author: Frankie.Chu for AUTH + name tokenization | ||
| line = re.sub(r'(?i)\b(authors?|@authors?)(:)(?=\S)', r'\1\2 ', line) | ||
|
|
||
| for tok in splitter(line): | ||
| # strip trailing quotes+comma | ||
| if tok.endswith("',"): | ||
|
|
@@ -2227,6 +2230,10 @@ def build_detection_from_node( | |
| # proper noun with some separator and trailing comma | ||
| (r'^[A-Z]+\.[A-Z][a-z]+,?$', 'NNP'), | ||
|
|
||
| # Mixed-case dotted name (Frankie.Chu). Not NNP: AUTHOR uses AUTH+NAME-DOT only. | ||
| # Exclude common org suffixes and license-prose glue tails (Assignment.This). | ||
| (r'^[A-Z][a-z0-9]+(?:\.(?!Org,?$|Com,?$|Net,?$|Edu,?$|Gov,?$|Inc,?$|Ltd,?$|Co,?$|This,?$|In,?$|All,?$|The,?$|Of,?$|And,?$|For,?$|By,?$|With,?$|From,?$|To,?$|As,?$|Or,?$|An,?$|At,?$|On,?$|Is,?$|Are,?$|Was,?$|Be,?$|If,?$|Not,?$|No,?$|Any,?$|Such,?$|That,?$|These,?$|Those,?$|When,?$|Where,?$|Which,?$|Who,?$|Will,?$|Shall,?$|May,?$|Can,?$|Must,?$|Should,?$|Would,?$|Could,?$|Rights,?$|Reserved,?$|Agreement,?$|Event,?$|Breach,?$|License,?$|Software,?$|Source,?$|Code,?$|File,?$|Files,?$|Notice,?$|Notices,?$|Subject,?$|Terms,?$|Conditions,?$|Section,?$|Clause,?$|Party,?$|Parties,?$)[A-Z][a-z0-9]+)+,?$', 'NAME-DOT'), | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is this blob? Where does it come from?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. NAME-DOT is a POS tag for one mixed-case dotted personal name token (e.g. Frankie.Chu). It is intentionally not generic NNP. AUTHOR stays AUTH + NAME-DOT so bare AUTH + any proper noun is not treated as an author. |
||
|
|
||
| # proper noun with apostrophe ': D'Orleans, D'Arcy, T'so, Ts'o | ||
| (r"^[A-Z][a-z]?'[A-Z]?[a-z]+[,\.]?$", 'NNP'), | ||
|
|
||
|
|
@@ -2552,6 +2559,8 @@ def build_detection_from_node( | |
|
|
||
| # NAME-YEAR starts or ends with a YEAR range | ||
| NAME-YEAR: {<YR-RANGE> <NNP> <NNP>+} #350 | ||
| NAME-YEAR: {<YR-RANGE> <NAME-DOT> <COMP|COMPANY|NNP|NN>+} #350.05 | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why tow lines?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Two NAME-YEAR rules on purpose: (1) year range + dotted name alone, (2) year range + dotted name + company/name tokens. Same split as the existing NNP NAME-YEAR forms, with NAME-DOT dual-homed for holders. I can fold them if you want a smaller grammar. |
||
| NAME-YEAR: {<YR-RANGE> <NAME-DOT>} #350.06 | ||
|
|
||
| COPYRIGHT: {<COPY> <YR-RANGE> <NNP> <NN> <NNP> <NNP> <NNP> <EMAIL>} #350.1 | ||
|
|
||
|
|
@@ -2699,6 +2708,8 @@ def build_detection_from_node( | |
| # Companies | ||
| COMPANY: {<NAME|NAME-EMAIL|NAME-YEAR|NNP>+ <OF> <NN>? <COMPANY|COMP> <NNP>?} #770 | ||
| COMPANY: {<NNP> <COMP|COMPANY> <COMP|COMPANY>} #780 | ||
| # NAME-DOT + company words only (not bare NAME-DOT) | ||
| COMPANY: {<NAME-DOT> <COMP|COMPANY>+} #780.1 | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you have an example?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. Holders that start with a dotted token then company words (e.g. Rocket.Chat followed by Technologies / Corp-style tails). Bare NAME-DOT as COMPANY was dropped; only NAME-DOT + COMP/COMPANY+ remains. |
||
| COMPANY: {<NN>? <COMPANY|NAME|NAME-EMAIL> <CC> <COMPANY|NAME|NAME-EMAIL>} #790 | ||
| COMPANY: {<COMP|COMPANY|NNP> <NN> <COMPANY|COMPANY> <NNP>+} #800 | ||
|
|
||
|
|
@@ -3475,6 +3486,8 @@ def build_detection_from_node( | |
| # author (Panagiotis Tsirigotis) | ||
| AUTHOR: {<AUTH> <NNP><NNP>+} #author Foo Bar | ||
|
|
||
| AUTHOR: {<AUTH> <NAME-DOT>} #author Frankie.Chu | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is "author Frankie.Chu" seen in the wild?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes — from #4229, this wild file still has it: That glued Author:Name line is what this PR detects. |
||
|
|
||
| # Author: Tim (xtimor@gmail.com) | ||
| AUTHOR: {<AUTH> <NNP>+ <EMAIL>+} #Author Foo joe@email.com | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| // Date:9 April,2012 | ||
| // Author:Frankie.Chu |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| what: | ||
| - authors | ||
| - authors_summary | ||
| authors: | ||
| - Frankie.Chu | ||
| authors_summary: | ||
| - value: Frankie.Chu | ||
| count: 1 | ||
| notes: Detect glued C++ Author:Name tags without space after colon (issue #4229) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you write a test for this? this is a really weird and complicated regex.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. Covered by tests/cluecode/data/authors/author_frankie_chu_cpp-tm1637.cpp and its expected yml. That fixture is the glued Author:Frankie.Chu case from #4229; detect returns author Frankie.Chu.