Major changes:

1. Implemented ability to calculate and track data quality (DQ) scores for FBS

   • DQ scores for Data Collection, Data Reliability, Geographical Correlation, Technological Correlation, Temporal Correlation
   • DQ scores based on EPA's Guidance on Data Quality Assessment for Life Cycle Inventory Data
   • DQ scores are beta version in this release and not included in all FBS methods
   • All FBS methods include scores for Geographical Correlation, Technological Correlation, and Temporal Correlation
   • Full DQ implementation will require updating FBA methods with Data Collection and Data Reliability scores
   • New functions added to calculate DQ
     • New adjust_dqi_reliability_collection_scores() to modify data reliability and data collection based on source and target sector levels
     • assign_temporal_correlation() assigns temporal DQ based on difference between year of data and target year of FBS
     • assign_geographical_correlation() assigns DQ for geoscale based on data geoscale vs target FBS geoscale
     • assign_technological_correlation() assigns DQ scores based on difference between source and target sectors

2. Modified how activities are mapped to sectors to enable proper accounting for Technological Correlation scores, which are based on the difference between the original activity to sector mapping and the target sector level.

   • First map all activities to the sector year identified in data crosswalk, then later convert to target sector year. Previously we immediately converted the crosswalk to target sector year, before matching on activities
   • We modified how NAICS are converted between NAICS years
     • We originally mapped all activities to NAICS6+ in the activity to sector crosswalk, then converted between NAICS years, then aggregated to target sector level, then merged NAICS to the activity-based data sets. This method is problematic when assigning DQ scores and unnecessary for FBS methods that are generated for more aggregated sector levels
     • Now to convert, we map the activities to the original sector year associated with that data. We then identify how many child NAICS there are (at NAICS 6) for each of the sectors and determine how many of those child sectors are converted to new sectors for the target sector year in generate_naics_crosswalk_conversion_ratios() and proportionally attribute the sectors to the new sectors for the target sector year.
       • For example, if we are converting NAICS4 across years, we identify all child NAICS6 for each NAICS4 and determine how those NAICS6 map between years. If there are five child NAICS6 and one child NAICS6 maps to a different parent NAICS4 in the target year, than 1/5 of the original NAICS4 parent value is mapped to a different NAICS4 in the target year
       • Conversion is not based on numeric values within the FBS because we might only have NAICS4 values, not NAICS6 and therefore do not have the data to create proportional conversions
   • New subset_sector_key()
     • Subsets sector key to return sector/industry that most closely maps activity/source sectors to target sectors – drops parent sectors within crosswalk and assigns tech corr scoring, modifies DatarReliability and DataCollection scores based on mapping
   • Modified NAICS conversion data check - originally checked if a sector-like activity was found in any NAICS year outside of the target year and if so, mapped to target year. This function did not always map correctly because the sector could be found in multiple NAICS years, and the NAICS years map differently to target year sectors
     • Revised function to check for the closest NAICS year to the target year and use that year to map to target NAICS

3. Updated default NAICS year in Employment FBS to NAICS 2017 (revised from NAICS 2012)

   • Changing NAICS year impacts the results of all FBS that use employment FBS as an allocation source
   • Previously, some BLS QCEW data were imported as NAICS 2012 left as NAICS 2012, while other data years were imported as NAICS 2017 and converted to NAICS 2012
   • Now, some BLS QCEW data are imported as NAICS 2012 and converted to NAICS 2017, while other data years are imported as NAICS 2017 and left as NAICS 2017
   • Many of these Employment datasets published as NAICS 2017 are later converted back to NAICS 2012 for use as allocation sources in other FBS methods. A conversion from NAICS 2012 -> NAICS 2017 -> NAICS 2012 occurs, which changes the employment results based on our conversion functions, resulting in changes to those relevant flows in the FBS methods.

4. Modified how data are merged on location so we can correctly merge state with county data

Minor changes:

• Correct error in attribute_flows_to_sectors()
  • Original group_total assignment was based on original df FlowAmount values, but we reset the index, so needed to base group_total on new index of the df
• Adds FIPS scale (1,3,5) to FIPS_Crosswalk
• Add NAICS 2002, 2007, 2022 crosswalks
• Expand NAICS_Crosswalk_TimeSeries to include NAICS 2022
• New NAICS_Year_Concordance which maps published 6-digit sectors across years
• New Sector_Levels .csv which labels sector level and sector length for all sectors
• Update BLS_QCEW NAICS years for 2011, 2022, and 2023 in source_catalog.yaml
• BLS QCEW estimate_suppressed_qcew()
  • Update the function to only estimate suppressed data up to max sector level. No longer estimate suppressed 6-digit sectors, when our target is 3-digit
• Consistent fips scale assignments. National = 5, state = 2, county = 1
• url updates to government FBA links

FBA Changes

• Generates new FBAs for EPA GHGI for 2019-2023
• Updated BEA FBAs (Supply, Use, GrossOutput) for 2012-2023

FBS Changes

• Updates to GHG FBS national (m1 and m2) for 2019 - 2023; drops 2012 - 2018 FBS which no longer will work with the latest FBAs
• New FBS method: Wages_national for 2017
• Updates Use and Supply tables in SUT format (see #453)

Includes PR:

#441
#452
#453
#455
#456

Full Changelog: v2.0.6...v2.1.0

What's Changed

• Updates StateGHGI FBS for 2024 release, includes updated StateIO FBAs, in #442
• Add 2020 census data set for urban/rural splits in #444
• Expands educational attainment and adds school enrollment to Census_ACS

New FBAs

• stateio_use_summary (2012 - 2023)
• EPA_StateGHGI (2012 - 2022)

New FBSs

• GHG_state_m1 (2012 - 2022)

Full Changelog: v2.0.4...v2.0.5

What's Changed

• Census Service Annual Survey in #421
• BEA Personal Consumption Expenditures by state in #420
• Census FBA datasets in #427
• Revised state level GHG data for CBEI in #428
• employment updates in #437

New FBAs

• Personal consumption expenditures by state (BEA_PCE)
• BLS Consumer Expenditures Survey (BLS_CES)
• Census American Community Survey (Census_ACS)
• Census County Business Patterns (Census_CBP), revised
• Census Economic Census, Class of Customer Statistics (Census_EC)
• Census Service Annual Survey (Census_SAS)
• State Inventory Tool (EPA_SIT); requires state data
• GHG Inventory data for select states to support EPAs Consumption Based Emissions Inventories
• Updated USDA ERS Farm Income and Wealth Statistics (USDA_ERS_FIWS)
• Updated USDA ERS Farm Income and Wealth Statistics (NOAA_FisheriesLandings)

New FBSs

• State_employment_m1 (added 2021-2023, updated all other years)
• National_employment_m1 (added 2023, updated all other years)

Full Changelog: v2.0.3...v2.0.4

Flow By Activity

• Updates EPA_GHGI through 2022 (2012-2022) in #406
• New BEA data for 2012-2022 (Summary & Gross Output), 2012 & 2017 Detail (using 2017 BEA schema)
• New CoA data (2022) and updated USGS_MYB_Lead (2020) in #405

Flow By Sector

• BEA_Detail FBS for 2013-2016, 2018-2022
• Updated GHG national FBS (m1 and m2) 2012-2022

Full Changelog: v2.0.2...v2.0.3

What's Changed

• add source publication dates to FBAs in #275
• option to specify git version/hash when returning an FBA via git_version in #399
• FBS metadata captures sequential FBAs in #399 (resolves #397)
• option to generate FBS that contains activity cols (retain_activity_columns=True) and sector name cols (append_sector_names=True) in #398
• Update 2017-2022 employment FBS in #410
• Updates Energy based datasets in #411
• add national CRHW methods in #414
• Global Materials Database in #415
• Enables calling multiple years at once for generating FBAs using call_all_years: True (#407)
• Allows skipping of standardize_units (#408)
• Limit numpy < 2.0.0 (see #418)

Flow-by-Activity

• Substantial updates to IEA Monthly Energy Review (EIA_MER)
• Adds UNEP Global Materials Flow Database (UNEP_IRP_GMFD)

Flow-by-Sector

• Updates Employment_national to 2017 NAICS schema, and adds 2021 and 2022 (#410)
• Updates CRHW_national to 2017 NAICS schema, and adds 2021 (#414)
• Adds Energy_fossil_national
• Adds Raw_Material_Extraction_national

Full Changelog: v2.0.1...v2.0.2

• new USEEIOv2 detail target schema
• fix broken FBAs (changed urls, changed excel tab names) for EIA_AEO, NOAA_FisheriesLandings, EIA_SEDS, EPA_CDDPath, USGS_MYB, EPA_WARMer
• reassign USDA_IWMS '111333' NAICS code to Berry Totals (from Orchards) to align with USDA_CoA_Cropland assignment c7c4c4f
• assign USGS_WU_Coef "Beef and other cattle" to "11213" in addition to "11212" a9c01c4
• update method_status.md to reflect current status of FBA/FBS errors when generating
• updates to stackedBarChart() to work in situations where df is already a collapsed FBS and where there are unique input parameters
• updates to FBSscatterplot() - add boxplot option
• update sector_aggregation() to work for collapsed FBS df
• generalize return_primary_activity_column() to return_primary_flow_column() so function works for both FBA and FBS 46ddf48
• new proxy_sector_data() to enable substituting an FBS sector value for a missing sector
• correct zenodo authorship
• update links to new data commons server
• rename "foss_landings.csv" to "NOAA_FisheriesLandings.csv" in external data folder
• update waste sector names, add 2 additional waste sectors 854eab3
• add warning when an FBS method uses "direct" when it should use "equal"
• update Land and Water FBS to use "equal" over "direct"
• edit log statements to make more concise

Changes to FBS

• Water_national_2015_m1, Water_national_2010_m1, Water_state_2015_m1 have new results to do reassigned activity to sector mapping for USGS_IWMS and USGS_WU_Coef

Full Changelog: v2.0.0...v2.0.1

Major updates:

• Turn FlowByActivity and FlowBySector into classes
• Create FlowBy class for functions used in both FBA and FBS classes
• FBS yamls revised to work for unlimited recursive and sequential attribution methods
• Update how suppressed data and parent sectors are equally attributed to child sectors, by equally attributing parent values to the next level of child values, rather than equally attributing parent values to target-level child sectors
• Ability to attribute dfs on non-sector columns
• Option to fill in primary source data columns with attribution columns
• Add support for 2017 NAICS codes
• Add mappings for 2017 BEA codes to 2017 NAICS
• All state FBS model results sum to national FBS model results

Minor Updates:

• Updates to Paths, which require most up-to-date versions of fedefelmflowlist, esupy, stewi
• Source_catalog.yaml updated - 'activity_schema' can be year-dependent
• New function compare_national_state_fbs() which compares aggregated results of state vs land FBS
• New github action to generate a single FBA
• Change BEA FBA names to align with useeior naming schema
• Updates to BLM FBAs - adds previously dropped state and national data for select activities
• Rename "CAP_HAP_national_2017" to "CAP_HAP_national_2017_m1"
• Add D.C. data to ERS MLU and NWIS WU
• Drop support for Python 3.8, add support for Python 3.11
• Rename sector_aggregation_levels from "aggregated" and "disaggregated" to "flat" and "Parent-completeChild" and "parent-incompleteChild"

New Flow-By-Sector Models

• CAP_HAP_Nonpoint 2014, 2017, 2020
• CAP_HAP_Nonroad 2014, 2017, 2020
• CAP_HAP_Onroad 2014, 2017, 2020
• CAP_HAP_national m1 and m2 2014, 2017, 2020
• CAP_HAP_state m1 2014, 2017, 2020
• CRHW_state 2013, 2015, 2019
• Detail Make, Supply, and Use tables
• Employment national 2002, 2016, 2019, 2020
• GHG national m1 and m2, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020
• GHG state m1 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020
• GRDREL national 2020
• GRDREL state 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020
• Land_state_2012
• TRI_DMR_state 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020

New Flow-By-Activity Models

• Annual BEA summary make and use tables
• Bureau of Transportation Statistics Airline fuel cost and consumption (BTS_Airlines) 2000-2021
• EPA State Inventory Tool (EPA_SIT) 2018-2019
• EPA State GHG Inventories (EPA_StateGHGI) 2020
• State GHGI for Maine (1990-2019), New York (2017-2019), Vermont (1990-2019)

Justifications for changes in FBS model results

• All FBS models have revised results
• Updates to how suppressed data are estimated and to how parent sector values are equally attributed to child sectors impacted all FBS models
• CNHW_national_2018, CNHW_state_2014 - changes due to employment FBS; fix error in missing F01000
• Food_waste_national_2018_m2 - changes due to new CNHW 2018 as primary data source
• Water_national_2015_m1 - Use state employment data for attribution instead of national employment data and reassigned an "Orchard" code to "Berry Totals"

Features removed:

• Generating Sankey diagrams (will be re-added in future version)
• Producing .bib files (will be re-added in future version)
• Appending material codes to sector codes (will be re-added in future version)
• Drop support for Water_national_m2 FBS
• Drop support for Electricity_gen_emissions FBS

What's Changed

• Seea recursive by @matthewlchambers in #243
• Adds display_tables() method to FBS by @matthewlchambers in #246
• allocate regional mecs data to states by @bl-young in #225
• update state ghg with develop by @catherinebirney in #257
• wrapped code in a function call so it won't all execute on import by @matthewlchambers in #289
• Override .astype() from DataFrame to fix issues introduced by pandas 1.5.0 by @matthewlchambers in #288
• Removes unused keys from certain FBS methods by @matthewlchambers in #291
• allow cached datasets to undergo futher selection by @bl-young in #294
• Add argument to FB constructor to prevent (when necessary) adding columns by @matthewlchambers in #295
• Pull master into state_ghg branch by @ericmbell1 in #299
• Several small fixes by @matthewlchambers in #312
• Document flowby functions by @catherinebirney in #331
• Update GHGI method to v2.0 by @bl-young in #332
• Update saving of file and metadata by @bl-young in #337
• update recursive-refac with sector_mapping (includes up-to-date develop branch) by @catherinebirney in #330
• drop attributionsources col in substitute_nonexistent_values() by @catherinebirney in #338
• Updates to support state methods on recursive branch by @bl-young in #339
• Seea time series by @matthewlchambers in #341
• expand exclusion_fields by @catherinebirney in #343
• update industry_spec_key() so it is not based on sector string length by @catherinebirney in #342
• Drop deprecated code and revise log handling by @bl-young in #347
• update NEI nonpoint data with 2020 by @bl-young in #348
• minor updates to support CAP_HAP FBS by @bl-young in #349
• specify NAICS year in activity_schema by @catherinebirney in #345
• Refactor with pathlib by @bl-young in #350
• Update cnhw by @catherinebirney in #352
• update recursive-refac with proportional_attribution by @catherinebirney in #354
• incorporate parent-incompleteChild sector hierarchy into map_to_sectors() by @catherinebirney in #355
• Gha error2 by @catherinebirney in #356
• Add action to create an FBA by @bl-young in #357
• update methods to estimate suppressed data by @catherinebirney in #360
• Set FBA columns and column dtypes by @catherinebirney in #362
• Split flowby.py into 3 python scripts by @catherinebirney in #363
• Updates to GHG state method by @bl-young in #365
• New GHG FBS method files by @ysrivas08 in #366
• update develop with recursive-refac (flowsa v2.0) by @catherinebirney in #364
• Bea refac by @catherinebirney in #372
• State ghgi v2 by @catherinebirney in #374
• update develop with changes to harmonize state and national fbs values by @catherinebirney in #375
• update flowsa2.0-release branch with develop by @catherinebirney in #376
• Flowsa2.0 release by @jbousquin in #381
• Update flowsa2.0 with updates from reviewer comments by @catherinebirney in #382
• 2017 Supply and Use tables from useeior by @WesIngwersen in #383
• update develop with Flowsa2.0 release branch by @catherinebirney in #386
• update state ghgi with develop/flowsa2.0-release branch by @catherinebirney in #387
• skip select_by_fields when using activity_sets by @bl-young in #392
• Revised approach to GHG models by @bl-young in #378
• Updates to generate 2014 CAP_HAP_Nonroad without memory errors by @bl-young in #393
• update 2.0-release branch with changes from develop by @catherinebirney in #394
• Flowsa2.0 release by @catherinebirney in #367

New Contributors

• @ysrivas08 made their first contribution in #366
• @jbousquin made their first contribution in #381

v2.0.0 reviewers

Thanks to David Graham and Justin Bousquin for reviewing FLOWSA for the v2.0.0 release.

Full Changelog: v1.3.2...v2.0.0

Pins final release supported by esupy<=0.2.2

Full Changelog: v1.3.1...v1.3.2

FLOWSA v1.3.1 release coincides with supply-chain-factors v1.2 release

Greenhouse Gas (GHG) Flow-By-Sector (FBS) Method changes:

• Updates GHG FBS m1 to equally allocate BEA and EIA MECS to sectors rather than use employment for attribution
• Updates GHG FBS m1 for 2016, 2017, 2018, and 2019 with latest inventory
• Use a common GHG FBS m1 yaml file as basis for all years
• Update some GHG FBS attribution source data years (MECS)
• Updates to GHGI activity names and activity to sector mapping

New Flow-By-Activity and Flow-By-Sector datasets:

• Adds 2020 GHGI FBA and GHG national FBS

Additional, minor changes:

• Update stackedbarchart() to use colors defined in visualizationessentials.csv, to include option to specify target sector level and to generalize attribution methods (direct vs attributed)

Major Updates:

• Option to append material codes to end of sector code in FBS
• New waste-related FBAs/FBSs, and 7/8 digit sector codes
• New food waste and concrete waste specific FBS
• New data visualization functions: Sankey, stackedBarChart
• Updates to methodology since v1.2.4 release changed the results for: CAP_HAP_national_2017, CNHW_national_2014, Land_national_2012, Water_national_2010_m2, Water_national_2015_m2, Water_national_2015_m3
• option to return an FBS with any mix of sector lengths, SPB and SCB columns no longer need to have matching sector lengths
• option for multiplication as allocation method
• update allocation methodology so if the allocation dataset is at a more aggregated geoscale than the primary FBA, that the primary df is not aggregated to match the allocation geoscale. Instead the more aggregated allocation dataset merges with the primary FBA on all related less aggregated geoscales
• Changes to github actions to address memory issues
• in equal_allocation() aggregate column before allocating to child naics - this impacts some NAICS6 results
• option to retain activity names in a primary data source after calling on those activities in an FBS, so the activity names can be used again
• requires plotly, kaleido

Minor Updates:

• new external_paths.env file to store local file paths
• rename load_api_key to load_env_file_key(), used for api keys and external_paths.env
• getFlowByActivity accepts flowclass of class or list
• new sector codes: S00203 - other state and local government enterprises, F040 - exports of goods and services, F050 - imports of goods and services, new 7/8 digit sectors for waste sectors (5622121, 5622191, 5622192, 5629201, 5629202, 5629203)
• New required columns in FBS: ProducedBySectorType, ConsumedBySectorType, AttributionSources (primary attribution source name)
• new selection_fields FBS parameter to subset FBA using column names/values
• new VisualizationEssentials.csv with standard colors for graphing sectors
• new csv "sector_2012_names" which includes additional names beyond official NAICS

New/Modified FBA:

• expand epa_cddpath sector mapping
• EPA_FactsAndFigures, EPA_REI, EPA_WARMer, EPA_WFR, Census_ASM, EPA_CDDPath (2018), EIA_AEO, EIA_SEDS

New/Modified FBS:

• CNHW_national_2018, Employment_national (2012, 2014, 2015), Food_waste_national_2018 (m1, m2), GHG_national_m1 (2016, 2017, 2018, 2019), REI_waste_national_2012