`occurrence` 'fields' param not returning fields as expected

[popovs]

Hello, I am trying to extract some data with the occurrence function, but it is not returning the expected fields.

E.g. -

tmp <- robis::occurrence(scientificname = "Raja miraletus",
                         fields = c("scientificName",
                                    "basisOfRecord",
                                    "decimalLatitude",
                                    "decimalLongitude",
                                    "bathymetry",
                                    "shoredistance",
                                    "sst",
                                    "sss")
                         )

will only return 5 fields:

> names(tmp)
[1] "basisOfRecord"    "decimalLatitude"  "decimalLongitude" "scientificName"   "id" 

Limiting the fields to just 'sst' or 'sss' returns only the id column:

tmp <- robis::occurrence(scientificname = "Raja miraletus",
                         fields = c("sst",
                                    "sss")
                         )

> names(tmp)
[1] "id" 

However, if I run the occurrence function with nothing specified in the 'fields' param, I get all 136 columns, including my other columns of interest (bathymetetry, sst, etc.).

[yuliaUU]

i have the same issue when i specify "flags" parameter

robis::occurrence(
      taxonid = 126316,
      fields = c(
        "aphiaID",
        "scientificName",
        "decimalLongitude",
        "decimalLatitude",
        "flags",
        "day",
        "month",
        "year"
      ) )

[sformel]

@popovs @yuliaUU sorry for the slow response. I think what you've run into here is a limitation of the API; not all fields are implemented as selectable. So I have some suggestions for improving documentation. Also tagging @pieterprovoost so he can weigh in on whether we have the capacity to develop this aspect of the API.

There are two unexpected behaviors here:

1. Not all fields are selectable.
2. If there are no values for a selected field in the results, the API does not return that column.

My suggested solutions for now:

1. create a helper function selectable_fields that prints out a list of fields that are valid for selection; add relevant info to the help documentation; add a warning message if you pass a field which isn't valid.
2. Add a warning message that notes columns which are not being returned because they are empty.

Do you have any suggestions for further improvement?

I think the list of valid fields for selection is below, but perhaps Pieter could confirm:

accessrights, aphiaid, areaid, associatedreferences, basisofrecord, catalognumber, collectioncode, coordinateuncertaintyinmeters, dataset_id, datasetid, date_day, date_month, date_year, decimallatitude, decimallongitude, depth, eventdate, flags, footprintwkt, geodeticdatum, id, individualcount, institutioncode, invalid, license, lifestage, locality, marine, missing, node_id, occurrenceid, recordedby, rightsholder, samplingprotocol, scientificname, sex, verbatimdepth, waterbody

[yuliaUU]

my workaround is loading all columns and then selecting:

robis::occurrence(
      taxonid = aphia) |> select(any_of(c(
        "aphiaID",
        "scientificName",
        "decimalLongitude",
        "decimalLatitude",
        "flags",
        "basisOfRecord",
        "day",
        "month",
        "year"
      )
    ))

[sformel]

Thanks for the feedback @yuliaUU. Obviously that's your only choice if you want to select fields which aren't supported for selection by the API. For larger queries I think the field selection makes a meaningful difference. For example, even this small query is 2-4 faster than retrieving all the fields and selecting afterward

robis::occurrence(
  geometry = "POLYGON ((-60.3816 12.2026, -59.4216 12.2026, -59.4216 14.1806, -60.3816 14.1806, -60.3816 12.2026))",
  fields = c(
    "aphiaID",
    "scientificName",
    "decimalLongitude",
    "decimalLatitude",
    "basisOfRecord",
    "day",
    "month",
    "year"
  )
  )

In the end, the best thing would be if we could develop support for selection of all fields. But for now, I think I'll add some documentation, including your suggested solution.

[pieterprovoost]

@sformel Please keep this open, I'll try to implement a fix at the API level.