Standardize Specimen Records

Overview

The barroso_std() function is the primary tool in the barRoso package for performing full data cleaning and standardization of biodiversity specimen records. It is designed to process raw data from herbaria, virtual repositories, and field collections, ensuring harmonized fields across taxonomic, geographic, and collector-related information.

Key Features

Normalize collector names and collection numbers
Normalize determiner names (identifiedBy)
Standardize geographic fields like country, state, and locality
Harmonize taxonomic names including family, genus, and species
Clean and normalize type status
Optionally remove unvouchered or duplicate records
Support for multilingual column names (e.g., Portuguese, Spanish)

Usage

library(barRoso)

df <- read.csv("raw_herbarium_data.csv")

standardized_df <- barroso_std(df,
                               unvouchered = TRUE,
                               delunkcoll = FALSE,
                               flag_missid = TRUE,
                               flag_duplicates = TRUE,
                               rm_duplicates = FALSE,
                               colname_recordedBy = "recordedBy",
                               colname_recordNumber = "recordNumber",
                               colname_identifiedBy = "identifiedBy",
                               colname_country = "country",
                               colname_stateProvince = "stateProvince",
                               colname_county = "county",
                               colname_municipality = "municipality",
                               colname_locality = "locality",
                               colname_collectionCode = "collectionCode",
                               colname_institutionCode = "institutionCode",
                               colname_typeStatus = "typeStatus",
                               colname_family = "family",
                               colname_genus = "genus",
                               colname_specificEpithet = "specificEpithet",
                               rm_original_column = TRUE)

Arguments

...: A data frame or list of data frames with raw biodiversity records
unvouchered: Remove records of wood, seed, fruit, pollen and spirit collections (e.g. INPAw, INPA-Carpoteca, BHZBcarpo), and observations without a specimen (basisOfRecord like HUMAN_OBSERVATION), before the herbarium codes are standardized
delunkcoll: Optionally delete records with unknown collectors
flag_duplicates, rm_duplicates: Detect and optionally remove duplicates
rm_original_column: Remove original (uncleaned) fields after standardization

Collector and determiner names

Collector names (recordedBy) are standardized with std_recordedBy() into initials followed by the surname, and additional collectors are moved to a new column addCollector (or stored as "et al." when there are more than two collectors). Determiner names (identifiedBy) are standardized with std_identifiedBy(), using the same rules, but all determiners are kept in the same cell. The original columns are kept with the suffix Original.

Original	Standardized
`Cardoso, D.B.O.S.`	`D. B. O. S. Cardoso`
`Padgurschi, MCG`	`M. C. G. Padgurschi`
`Martius, CFP von`	`C. F. P. von Martius`
`Uribe Uribe AL`	`A. L. Uribe-Uribe`
`Dusén`	`Dusén`
`A. Gómez Pompa, A. J. Sharp & P. Hernández`	`A. Gómez Pompa` (`addCollector`: `et al.`)
`Lobato, LC; Soares, CRA; Silva, CA` (determiners)	`L. C. Lobato, C. R. A. Soares & C. A. Silva`
`R.Barneby (NY) 1994-06` (determiner)	`R. Barneby`

Chinese names written in Chinese characters (e.g. 丁广奇、余莲) or as unicode escapes, as in some GBIF downloads (e.g. <U+674E><U+5149><U+7167>, 李光照), are converted into Latin names as Chinese authors cite themselves in publications: initials of the given name followed by the family name, e.g. G. Z. Li, T. S. Yi (易婷双) and R. Zhang (张荣). Family names of two characters (e.g. Ouyang) and family names with a special reading (e.g. 曾, Zeng) are also handled, and names of teams or groups (e.g. 植被组, vegetation group) are kept in full. When the name is also written in Latin, e.g. K.C.Yang 楊國禎 or Peng, Ching-I (彭鏡毅), the Latin name is kept (K. C. Yang, C.-I. Peng).

Identifiers like (Wikidata Q5850623), given names in lowercase (katherine Bravo) and abbreviations like Edm.Pereira are also cleaned, and names of teams and herbaria (e.g. Equipe do Jardim Botânico de Brasília, Belem Herbarium) are kept as they are.

Herbarium acronyms

Herbarium acronyms (collectionCode) are standardized with std_collection() into the acronyms of Index Herbariorum, using regular expressions rather than a list of every code:

Original `collectionCode`	`institutionCode` or `catalogNumber`	Standardized
`Tropicos`, `MO-Duplicates`	`NY`	`NY` (the herbarium of the specimen)
`Botany`, `Herbarium`, `Vascular Plants`	`catalogNumber` `U.1317533`	`U`
(missing, GenBank voucher)	`catalogNumber` `D. Cardoso 2118 (HUEFS)`	`HUEFS`
`F-Seedplants`, `BHCB-Fanerogamas`, `R-Tipos`, `UIS-H`		`F`, `BHCB`, `R`, `UIS`
`BSMvplantscoll`, `NL-Botany`, `CJBG-G-DC`		`M`, `L`, `G-DC`
`Herbario INPA`, `Herbario Nacional de Costa Rica (CR)`		`INPA`, `CR`
(missing)	`Instituto Florestal`, `Unicamp`	`SPSF`, `UEC`
`Observations`, `Dados de Pesquisa SISBio`, `Ecopetrol S.A`		`NA`

Codes of wood collections, like INPAw, are kept, as they are Index Herbariorum codes.

Places

Countries, states, counties and municipalities are standardized with std_place(), e.g. BRASIL into Brazil and BA into Bahia. Words for municipality are removed from county and municipality, leaving just the name, e.g. Manaus Mun., Mun. de Castelo and Município de Mucugê into Manaus, Castelo and Mucugê. Country codes become names, e.g. BR and GY into Brazil and Guyana. Missing countries are taken from the Brazilian state (e.g. Bahia), from a country name at the start or end of the locality (e.g. Rio Negro, Brasil), or from other records with the same state and county (e.g. Amazonas, Manaus); names shared by several countries, like the state Amazonas alone, are left empty. Missing municipalities are taken from the county, as GBIF keeps the municipalities of Brazil in county.

The function barroso_std() does not check geographic coordinates. Use std_coordinates() for that.

Output

Returns a cleaned and fully standardized data frame ready for:

Reconciliation and deduplication
Taxonomic and biogeographic analysis
Herbarium label generation workflows

Notes

Uses dynamic regular expressions for name harmonization rather than static dictionaries
Optimized for scalability and flexible input schema
Recommended to run before calling downstream functions like barroso_flag_duplicates() or barroso_labels()

Example

cleaned_data <- barroso_std(df,
                            colname_country = "pais",
                            colname_stateProvince = "estado",
                            rm_duplicates = TRUE)

For a complete list of supported fields, refer to the function documentation with ?barroso_std.

--- title: "Standardize Specimen Records" format: html --- ## Overview The `barroso_std()` function is the primary tool in the `barRoso` package for performing full data cleaning and standardization of biodiversity specimen records. It is designed to process raw data from herbaria, virtual repositories, and field collections, ensuring harmonized fields across taxonomic, geographic, and collector-related information. ## Key Features - Normalize collector names and collection numbers - Normalize determiner names (`identifiedBy`) - Standardize geographic fields like country, state, and locality - Harmonize taxonomic names including family, genus, and species - Clean and normalize type status - Optionally remove unvouchered or duplicate records - Support for multilingual column names (e.g., Portuguese, Spanish) ## Usage ```r library(barRoso) df <- read.csv("raw_herbarium_data.csv") standardized_df <- barroso_std(df, unvouchered = TRUE, delunkcoll = FALSE, flag_missid = TRUE, flag_duplicates = TRUE, rm_duplicates = FALSE, colname_recordedBy = "recordedBy", colname_recordNumber = "recordNumber", colname_identifiedBy = "identifiedBy", colname_country = "country", colname_stateProvince = "stateProvince", colname_county = "county", colname_municipality = "municipality", colname_locality = "locality", colname_collectionCode = "collectionCode", colname_institutionCode = "institutionCode", colname_typeStatus = "typeStatus", colname_family = "family", colname_genus = "genus", colname_specificEpithet = "specificEpithet", rm_original_column = TRUE) ``` ## Arguments - `...`: A data frame or list of data frames with raw biodiversity records - `unvouchered`: Remove records of wood, seed, fruit, pollen and spirit collections (e.g. `INPAw`, `INPA-Carpoteca`, `BHZBcarpo`), and observations without a specimen (`basisOfRecord` like `HUMAN_OBSERVATION`), before the herbarium codes are standardized - `delunkcoll`: Optionally delete records with unknown collectors - `flag_duplicates`, `rm_duplicates`: Detect and optionally remove duplicates - `rm_original_column`: Remove original (uncleaned) fields after standardization ## Collector and determiner names Collector names (`recordedBy`) are standardized with `std_recordedBy()` into initials followed by the surname, and additional collectors are moved to a new column `addCollector` (or stored as `"et al."` when there are more than two collectors). Determiner names (`identifiedBy`) are standardized with `std_identifiedBy()`, using the same rules, but all determiners are kept in the same cell. The original columns are kept with the suffix `Original`. | Original | Standardized | |---|---| | `Cardoso, D.B.O.S.` | `D. B. O. S. Cardoso` | | `Padgurschi, MCG` | `M. C. G. Padgurschi` | | `Martius, CFP von` | `C. F. P. von Martius` | | `Uribe Uribe AL` | `A. L. Uribe-Uribe` | | `Dusén` | `Dusén` | | `A. Gómez Pompa, A. J. Sharp & P. Hernández` | `A. Gómez Pompa` (`addCollector`: `et al.`) | | `Lobato, LC; Soares, CRA; Silva, CA` (determiners) | `L. C. Lobato, C. R. A. Soares & C. A. Silva` | | `R.Barneby (NY) 1994-06` (determiner) | `R. Barneby` | Chinese names written in Chinese characters (e.g. 丁广奇、余莲) or as unicode escapes, as in some GBIF downloads (e.g. `<U+674E><U+5149><U+7167>`, 李光照), are converted into Latin names as Chinese authors cite themselves in publications: initials of the given name followed by the family name, e.g. `G. Z. Li`, `T. S. Yi` (易婷双) and `R. Zhang` (张荣). Family names of two characters (e.g. `Ouyang`) and family names with a special reading (e.g. 曾, `Zeng`) are also handled, and names of teams or groups (e.g. 植被组, vegetation group) are kept in full. When the name is also written in Latin, e.g. `K.C.Yang 楊國禎` or `Peng, Ching-I (彭鏡毅)`, the Latin name is kept (`K. C. Yang`, `C.-I. Peng`). Identifiers like `(Wikidata Q5850623)`, given names in lowercase (`katherine Bravo`) and abbreviations like `Edm.Pereira` are also cleaned, and names of teams and herbaria (e.g. `Equipe do Jardim Botânico de Brasília`, `Belem Herbarium`) are kept as they are. ## Herbarium acronyms Herbarium acronyms (`collectionCode`) are standardized with `std_collection()` into the acronyms of [Index Herbariorum](https://sweetgum.nybg.org/science/ih/), using regular expressions rather than a list of every code: | Original `collectionCode` | `institutionCode` or `catalogNumber` | Standardized | |---|---|---| | `Tropicos`, `MO-Duplicates` | `NY` | `NY` (the herbarium of the specimen) | | `Botany`, `Herbarium`, `Vascular Plants` | `catalogNumber` `U.1317533` | `U` | | (missing, GenBank voucher) | `catalogNumber` `D. Cardoso 2118 (HUEFS)` | `HUEFS` | | `F-Seedplants`, `BHCB-Fanerogamas`, `R-Tipos`, `UIS-H` | | `F`, `BHCB`, `R`, `UIS` | | `BSMvplantscoll`, `NL-Botany`, `CJBG-G-DC` | | `M`, `L`, `G-DC` | | `Herbario INPA`, `Herbario Nacional de Costa Rica (CR)` | | `INPA`, `CR` | | (missing) | `Instituto Florestal`, `Unicamp` | `SPSF`, `UEC` | | `Observations`, `Dados de Pesquisa SISBio`, `Ecopetrol S.A` | | `NA` | Codes of wood collections, like `INPAw`, are kept, as they are Index Herbariorum codes. ## Places Countries, states, counties and municipalities are standardized with `std_place()`, e.g. `BRASIL` into `Brazil` and `BA` into `Bahia`. Words for municipality are removed from `county` and `municipality`, leaving just the name, e.g. `Manaus Mun.`, `Mun. de Castelo` and `Município de Mucugê` into `Manaus`, `Castelo` and `Mucugê`. Country codes become names, e.g. `BR` and `GY` into `Brazil` and `Guyana`. Missing countries are taken from the Brazilian state (e.g. `Bahia`), from a country name at the start or end of the locality (e.g. `Rio Negro, Brasil`), or from other records with the same state and county (e.g. `Amazonas`, `Manaus`); names shared by several countries, like the state `Amazonas` alone, are left empty. Missing municipalities are taken from the county, as GBIF keeps the municipalities of Brazil in `county`. The function `barroso_std()` does not check geographic coordinates. Use [`std_coordinates()`](/articles/fix_flag_coordinates.qmd) for that. ## Output Returns a cleaned and fully standardized data frame ready for: - Reconciliation and deduplication - Taxonomic and biogeographic analysis - Herbarium label generation workflows ## Notes - Uses dynamic regular expressions for name harmonization rather than static dictionaries - Optimized for scalability and flexible input schema - Recommended to run before calling downstream functions like `barroso_flag_duplicates()` or `barroso_labels()` ## Example ```r cleaned_data <- barroso_std(df, colname_country = "pais", colname_stateProvince = "estado", rm_duplicates = TRUE) ``` For a complete list of supported fields, refer to the function documentation with `?barroso_std`.