Skip to content

Invalid multibyte string in parsed FAERS drug table #24

Description

@InfiniteStryker0

While building a FAERS 2016–2025 dataset and ran into what looks like a character encoding issue in the parsed drug table. After downloading/parsing FAERS data with the package, calling View(drug) in RStudio failed with:

Error in nchar(col, type = "width") : 
  invalid multibyte string, element 16213314

Inspecting the offending row showed that the issue appears to come from the lot_num field:

drug[16213314]

Output:

    year quarter primaryid   caseid drug_seq role_cod drugname  prod_ai val_vbm        route
   <int>  <char>    <char>   <char>    <int>   <char>   <char>   <char>   <int>       <char>
1:  2019      q3 165897511 16589751        1       PS  KINERET ANAKINRA       1 Subcutaneous

   dose_vbm cum_dose_chr cum_dose_unit dechal rechal      lot_num exp_dt nda_num dose_amt
     <char>        <num>        <char> <char> <char>       <char> <char>  <char>   <char>
1:     <NA>           NA          <NA>   <NA>      Y 33298?\xad1B   <NA>  103950      100

   dose_unit dose_form dose_freq
      <char>    <char>    <char>
1:        MG      <NA>        QD

The problematic value appears to be:

lot_num = "33298?\xad1B"

This seems to contain a malformed or non-UTF-8 byte sequence / soft hyphen-like character, which causes downstream display/string-width operations to fail.

Would it make sense for faers_parse() or the relevant table parsing helper to normalize character columns to valid UTF-8 before returning the parsed tables? This would help avoid failures in View(), nchar(type = "width"), and potentially downstream export to Parquet or other formats that expect valid UTF-8 strings.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions