Parsing Tabular Data
Originally Contributed by: Clayton Barrows
Introduction
An example of how to parse tabular files (CSV) files similar to the format established in the RTS-GMLC and create a System using PowerSystems.jl
Dependencies
julia> using PowerSystemsjulia> using TimeSeriesjulia> using Dates
Fetch Data
PowerSystems.jl links to some test data that is suitable for this example. Let's get the test data using Artifacts. You can find the repository of the data of the GMLC system here:
julia> # include download methods"/home/runner/.julia/artifacts/2b7013812a985ce5884696ce276ca17b2719a57a/PowerSystemsTestData-2.0"julia> RTS_GMLC_DIR = joinpath(DATA_DIR, "RTS_GMLC")"/home/runner/.julia/artifacts/2b7013812a985ce5884696ce276ca17b2719a57a/PowerSystemsTestData-2.0/RTS_GMLC"
The tabular data format relies on a folder containing *.csv files and a user_descriptors.yaml file
First, we'll read the tabular data
julia> rawsys = PowerSystems.PowerSystemTableData( RTS_GMLC_DIR, 100.0, joinpath(RTS_GMLC_DIR, "user_descriptors.yaml"), timeseries_metadata_file = joinpath(RTS_GMLC_DIR, "timeseries_pointers.json"), generator_mapping_file = joinpath(RTS_GMLC_DIR, "generator_mapping_multi_start.yaml"), )[ Info: Parsing csv files in Reserves ... [ Info: Parsing csv data in DAY_AHEAD_regional_Flex_Down.csv ... [ Info: Parsing csv data in DAY_AHEAD_regional_Flex_Up.csv ... [ Info: Parsing csv data in DAY_AHEAD_regional_Reg_Down.csv ... [ Info: Parsing csv data in DAY_AHEAD_regional_Reg_Up.csv ... [ Info: Parsing csv data in DAY_AHEAD_regional_Spin_Up_R1.csv ... [ Info: Parsing csv data in DAY_AHEAD_regional_Spin_Up_R2.csv ... [ Info: Parsing csv data in DAY_AHEAD_regional_Spin_Up_R3.csv ... [ Info: Parsing csv data in REAL_TIME_regional_Reg_Down.csv ... [ Info: Parsing csv data in REAL_TIME_regional_Reg_Up.csv ... [ Info: Parsing csv data in REAL_TIME_regional_Spin_Up_R1.csv ... [ Info: Parsing csv data in REAL_TIME_regional_Spin_Up_R2.csv ... [ Info: Parsing csv data in REAL_TIME_regional_Spin_Up_R3.csv ... [ Info: Successfully parsed Reserves [ Info: Parsing csv data in branch.csv ... [ Info: Successfully parsed branch.csv [ Info: Parsing csv data in bus.csv ... [ Info: Successfully parsed bus.csv [ Info: Parsing csv data in dc_branch.csv ... [ Info: Successfully parsed dc_branch.csv [ Info: Parsing csv data in gen.csv ... [ Info: Successfully parsed gen.csv [ Info: Parsing csv data in reserves.csv ... [ Info: Successfully parsed reserves.csv [ Info: Parsing csv data in simulation_objects.csv ... [ Info: Successfully parsed simulation_objects.csv [ Info: Parsing csv data in storage.csv ... [ Info: Successfully parsed storage.csv [ Info: Parsing csv data in timeseries_pointers.csv ... [ Info: Successfully parsed timeseries_pointers.csv PowerSystemTableData: directory: /home/runner/.julia/artifacts/2b7013812a985ce5884696ce276ca17b2719a57a/PowerSystemsTestData-2.0/RTS_GMLC timeseries_metadata_file: /home/runner/.julia/artifacts/2b7013812a985ce5884696ce276ca17b2719a57a/PowerSystemsTestData-2.0/RTS_GMLC/timeseries_pointers.json base_power: 100.0 PowerSystems.InputCategoryModule.InputCategory.DC_BRANCH = 3: 1×60 DataFrame PowerSystems.InputCategoryModule.InputCategory.STORAGE = 8: 22×8 DataFrame PowerSystems.InputCategoryModule.InputCategory.RESERVE = 6: 7×7 DataFrame PowerSystems.InputCategoryModule.InputCategory.BRANCH = 1: 120×14 DataFrame PowerSystems.InputCategoryModule.InputCategory.BUS = 2: 73×15 DataFrame PowerSystems.InputCategoryModule.InputCategory.GENERATOR = 4: 158×58 DataFrame
Create a System
Next, we'll create a System from the rawsys data. Since a System is predicated on a time series resolution and the rawsys data includes both 5-minute and 1-hour resolution time series, we also need to specify which time series we want to include in the System. The time_series_resolution kwarg filters to only include time series with a matching resolution.
julia> sys = System(rawsys; time_series_resolution = Dates.Hour(1));[ Info: Unit System changed to UnitSystem.DEVICE_BASE = 1 ┌ Warning: Missing PowerSystems.InputCategoryModule.InputCategory.LOAD = 5 data. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:212 ┌ Warning: User-defined column name Startup Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Shutdown Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Status at Start is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Time at Status is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Cold is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Warm is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Hot is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Must Run is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: Missing PowerSystems.InputCategoryModule.InputCategory.LOAD = 5 data. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:212 [ Info: Adding contributing generators for Spin_Up_R1 by category ┌ Warning: User-defined column name Startup Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Shutdown Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Status at Start is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Time at Status is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Cold is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Warm is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Hot is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Must Run is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 [ Info: Adding contributing generators for Spin_Up_R2 by category ┌ Warning: User-defined column name Startup Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Shutdown Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Status at Start is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Time at Status is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Cold is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Warm is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Hot is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Must Run is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 [ Info: Adding contributing generators for Spin_Up_R3 by category ┌ Warning: User-defined column name Startup Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Shutdown Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Status at Start is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Time at Status is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Cold is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Warm is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Hot is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Must Run is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 [ Info: Adding contributing generators for Flex_Up by category ┌ Warning: User-defined column name Startup Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Shutdown Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Status at Start is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Time at Status is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Cold is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Warm is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Hot is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Must Run is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 [ Info: Adding contributing generators for Flex_Down by category ┌ Warning: User-defined column name Startup Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Shutdown Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Status at Start is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Time at Status is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Cold is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Warm is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Hot is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Must Run is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 [ Info: Adding contributing generators for Reg_Up by category ┌ Warning: User-defined column name Startup Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Shutdown Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Status at Start is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Time at Status is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Cold is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Warm is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Hot is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Must Run is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 [ Info: Adding contributing generators for Reg_Down by category ┌ Warning: User-defined column name Startup Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Shutdown Ramp Rate MW/min is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Status at Start is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Time at Status is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Cold is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Warm is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Start Cost Hot is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413 ┌ Warning: User-defined column name Must Run is not in dataframe. └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/power_system_table_data.jl:1413julia> horizon = 24;julia> interval = Dates.Hour(24);julia> transform_single_time_series!(sys, horizon, interval);┌ Warning: There is only one forecast window. Setting interval = 0 seconds └ @ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/9btGb/src/component.jl:779julia> sysSystem ┌───────────────────┬─────────────┐ │ Property │ Value │ ├───────────────────┼─────────────┤ │ Name │ │ │ Description │ │ │ System Units Base │ DEVICE_BASE │ │ Base Power │ 100.0 │ │ Base Frequency │ 60.0 │ │ Num Components │ 525 │ └───────────────────┴─────────────┘ Static Components ┌─────────────────────┬───────┬────────────────────────┬───────────────┐ │ Type │ Count │ Has Static Time Series │ Has Forecasts │ ├─────────────────────┼───────┼────────────────────────┼───────────────┤ │ ACBus │ 73 │ false │ false │ │ Arc │ 109 │ false │ false │ │ Area │ 3 │ false │ false │ │ GenericBattery │ 1 │ false │ false │ │ HydroDispatch │ 20 │ true │ true │ │ Line │ 105 │ false │ false │ │ LoadZone │ 3 │ true │ true │ │ PowerLoad │ 51 │ true │ true │ │ RenewableDispatch │ 30 │ true │ true │ │ RenewableFix │ 31 │ true │ true │ │ TapTransformer │ 15 │ false │ false │ │ ThermalMultiStart │ 76 │ false │ false │ │ TwoTerminalHVDCLine │ 1 │ false │ false │ │ VariableReserve │ 5 │ true │ true │ │ VariableReserve │ 2 │ true │ true │ └─────────────────────┴───────┴────────────────────────┴───────────────┘ Time Series Summary ┌──────────────────────────────────┬─────────────────────┐ │ Property │ Value │ ├──────────────────────────────────┼─────────────────────┤ │ Components with time series data │ 142 │ │ Total StaticTimeSeries │ 182 │ │ Total Forecasts │ 182 │ │ Resolution │ 60 minutes │ │ First initial time │ 2020-01-01T00:00:00 │ │ Last initial time │ 2020-01-01T00:00:00 │ │ Horizon │ 24 │ │ Interval │ 0 minutes │ │ Forecast window count │ 1 │ └──────────────────────────────────┴─────────────────────┘