Parsing Data
PowerSystems.jl
supports the creation of a System
from a variety of common data formats:
- MATPOWER (code copied with permission from
PowerModels.jl
) - PSS/e RAW Files (code copied with permission from
PowerModels.jl
) - PSS/e DYR Files
- PowerSystems table data (CSV Files)
MATPOWER / PSS/e
The following code will create a System from a MATPOWER or PSS/e file:
julia> using PowerSystems
julia> file_dir = joinpath(pkgdir(PowerSystems), "docs", "src", "tutorials", "tutorials_data")
"/home/runner/work/PowerSystems.jl/PowerSystems.jl/docs/src/tutorials/tutorials_data"
julia> sys = System(joinpath(file_dir, "case5.m"))
[ Info: Correcting vm in bus 1 to 1.07762 to match generator set-point [ Info: Correcting vm in bus 3 to 1.1 to match generator set-point [ Info: Correcting vm in bus 4 to 1.06414 to match generator set-point [ Info: Correcting vm in bus 10 to 1.06907 to match generator set-point [ Info: extending matpower format with data: areas 1x3 [ Info: reversing the orientation of branch 6 (4, 3) to be consistent with other parallel branches [ Info: removing 1 cost terms from generator 5: [1000.0, 0.0] [ Info: removing 1 cost terms from generator 4: [4000.0, 0.0] [ Info: removing 1 cost terms from generator 2: [1500.0, 0.0] [ Info: removing 1 cost terms from generator 3: [3000.0, 0.0] [ Info: removing 1 cost terms from generator 1: [1400.0, 0.0] ┌ Info: Constructing System from Power Models │ data["name"] = "nesta_case5_pjm" └ data["source_type"] = "matpower" [ Info: Reading bus data [ Info: Reading Load data in PowerModels dict to populate System ... [ Info: Reading LoadZones data in PowerModels dict to populate System ... [ Info: Reading generator data [ Info: Reading branch data [ Info: Reading shunt data [ Info: Reading DC Line data [ Info: Reading storage data System ┌───────────────────┬─────────────┐ │ Property │ Value │ ├───────────────────┼─────────────┤ │ Name │ │ │ Description │ │ │ System Units Base │ SYSTEM_BASE │ │ Base Power │ 100.0 │ │ Base Frequency │ 60.0 │ │ Num Components │ 28 │ └───────────────────┴─────────────┘ Static Components ┌──────────────────────────┬───────┐ │ Type │ Count │ ├──────────────────────────┼───────┤ │ ACBus │ 5 │ │ Arc │ 6 │ │ Area │ 1 │ │ Line │ 5 │ │ LoadZone │ 1 │ │ PhaseShiftingTransformer │ 2 │ │ PowerLoad │ 3 │ │ ThermalStandard │ 5 │ └──────────────────────────┴───────┘
PSS/e dynamic data parsing
PSS/e's dynamic model library is extensive, we currently support parsing a limited amount of models out of the box.
Machine models | AVR Models | Prime Movers | PSS models |
---|---|---|---|
GENSAE | IEEET1 | HYGOV | IEEEST |
GENSAL | ESDC1A | IEEEG1 | |
GENROE | ESAC1A | GGOV1 | |
GENCLS | ESST4B | ||
GENROU | EXAC2 | ||
EXPIC1 | |||
ESAC6A | |||
EXAC1 | |||
SCRX | |||
ESDC2A |
Creating a Dynamic System using .RAW and .DYR data
A PowerSystems.jl
system can be created using a .RAW and a .DYR file. In this example we will create the following three bus system using the following RAW file:
0, 100, 33, 0, 0, 60 / 24-Apr-2020 19:28:39 - MATPOWER 7.0.1-dev
101, 'BUS 1 ', 138, 3, 1, 1, 1, 1.02, 0, 1.1, 0.9, 1.1, 0.9
102, 'BUS 2 ', 138, 2, 1, 1, 1, 1.0142, 0, 1.1, 0.9, 1.1, 0.9
103, 'BUS 3 ', 138, 2, 1, 1, 1, 1.0059, 0, 1.1, 0.9, 1.1, 0.9
0 / END OF BUS DATA, BEGIN LOAD DATA
101, 1, 1, 1, 1, 100, 20, 0, 0, 0, 0, 1, 1, 0
102, 1, 1, 1, 1, 70, 10, 0, 0, 0, 0, 1, 1, 0
103, 1, 1, 1, 1, 50, 10, 0, 0, 0, 0, 1, 1, 0
0 / END OF LOAD DATA, BEGIN FIXED SHUNT DATA
0 / END OF FIXED SHUNT DATA, BEGIN GENERATOR DATA
101, 1, 20, 0, 100, -100, 1.02, 0, 100, 0, 0, 0, 0, 1, 1, 100, 318, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1
102, 1, 100, 0, 100, -100, 1.0142, 0, 100, 0, 0.7, 0, 0, 1, 1, 100, 318, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1
103, 1, 100, 0, 100, -100, 1.0059, 0, 100, 0, 0.2, 0, 0, 1, 1, 100, 318, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1
0 / END OF GENERATOR DATA, BEGIN BRANCH DATA
101, 103, 1, 0.01000, 0.12, 0.0, 250, 250, 250, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1
101, 102, 1, 0.01000, 0.12, 0.0, 250, 250, 250, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1
102, 103, 1, 0.01000, 0.12, 0.0, 250, 250, 250, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1
0 / END OF BRANCH DATA, BEGIN TRANSFORMER DATA
0 / END OF TRANSFORMER DATA, BEGIN AREA DATA
0 / END OF AREA DATA, BEGIN TWO-TERMINAL DC DATA
0 / END OF TWO-TERMINAL DC DATA, BEGIN VOLTAGE SOURCE CONVERTER DATA
0 / END OF VOLTAGE SOURCE CONVERTER DATA, BEGIN IMPEDANCE CORRECTION DATA
0 / END OF IMPEDANCE CORRECTION DATA, BEGIN MULTI-TERMINAL DC DATA
0 / END OF MULTI-TERMINAL DC DATA, BEGIN MULTI-SECTION LINE DATA
0 / END OF MULTI-SECTION LINE DATA, BEGIN ZONE DATA
0 / END OF ZONE DATA, BEGIN INTER-AREA TRANSFER DATA
0 / END OF INTER-AREA TRANSFER DATA, BEGIN OWNER DATA
0 / END OF OWNER DATA, BEGIN FACTS CONTROL DEVICE DATA
0 / END OF FACTS CONTROL DEVICE DATA, BEGIN SWITCHED SHUNT DATA
0 / END OF SWITCHED SHUNT DATA, BEGIN GNE DEVICE DATA
0 / END OF GNE DEVICE DATA, BEGIN INDUCTION MACHINE DATA
0 / END OF INDUCTION MACHINE DATA
Q
This system is a three bus system with three generators, three loads and three branches. The dynamic data for the generators is provided in the DYR file:
101 'GENROE' 1 8.000000 0.030000 0.400000 0.050000 6.500000 0.000000 1.800000
1.700000 0.300000 0.550000 0.250000 0.200000 0.039200 0.267200 /
101 'ESST1A' 1 1 1 0.01 99 -99 1 10 1 1 200 0 4 -4 4 -4 0 0 1 0 3 /
102 'GENCLS' 1 0.0 0.0 /
103 'GENCLS' 1 3.1 2.0 /
That assigns a GENROU generator and a ESST1A voltage regulator at the generator located at bus 101, while classic machine models for the generators located at bus 102 and 103.
To create the system we can do it passing both files directories:
julia> RAW_dir = joinpath(file_dir, "ThreeBusNetwork.raw")
"/home/runner/work/PowerSystems.jl/PowerSystems.jl/docs/src/tutorials/tutorials_data/ThreeBusNetwork.raw"
julia> DYR_dir = joinpath(file_dir, "TestGENCLS.dyr")
"/home/runner/work/PowerSystems.jl/PowerSystems.jl/docs/src/tutorials/tutorials_data/TestGENCLS.dyr"
julia> dyn_system = System(RAW_dir, DYR_dir, runchecks = false)
[ Info: The PSS(R)E parser currently supports buses, loads, shunts, generators, branches, transformers, and dc lines [ Info: The PSS(R)E parser currently supports buses, loads, shunts, generators, branches, transformers, and dc lines [ Info: Parsing PSS(R)E Bus data into a PowerModels Dict... [ Info: Parsing PSS(R)E Load data into a PowerModels Dict... [ Info: Parsing PSS(R)E Shunt data into a PowerModels Dict... [ Info: Parsing PSS(R)E Generator data into a PowerModels Dict... [ Info: Parsing PSS(R)E Branch data into a PowerModels Dict... [ Info: Parsing PSS(R)E Transformer data into a PowerModels Dict... [ Info: Parsing PSS(R)E Two-Terminal and VSC DC line data into a PowerModels Dict... ┌ Warning: This PSS(R)E parser currently doesn't support Storage data parsing... └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/pm_io/psse.jl:998 ┌ Warning: This PSS(R)E parser currently doesn't support Switch data parsing... └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/pm_io/psse.jl:1004 [ Info: angmin and angmax values are 0, widening these values on branch 1 to +/- 60.0 deg. [ Info: angmin and angmax values are 0, widening these values on branch 2 to +/- 60.0 deg. [ Info: angmin and angmax values are 0, widening these values on branch 3 to +/- 60.0 deg. ┌ Info: Constructing System from Power Models │ data["name"] = "threebusnetwork" └ data["source_type"] = "pti" [ Info: Reading bus data [ Info: Reading Load data in PowerModels dict to populate System ... [ Info: Reading LoadZones data in PowerModels dict to populate System ... [ Info: Reading generator data [ Info: Reading branch data [ Info: Reading shunt data [ Info: Reading DC Line data [ Info: Reading storage data [ Info: Generators provided in .dyr, without a generator in .raw file will be skipped. [ Info: Machine at bus 102, id 1 has zero inertia. Modeling it as Voltage Source System ┌───────────────────┬─────────────┐ │ Property │ Value │ ├───────────────────┼─────────────┤ │ Name │ │ │ Description │ │ │ System Units Base │ SYSTEM_BASE │ │ Base Power │ 100.0 │ │ Base Frequency │ 60.0 │ │ Num Components │ 19 │ └───────────────────┴─────────────┘ Static Components ┌─────────────────┬───────┐ │ Type │ Count │ ├─────────────────┼───────┤ │ ACBus │ 3 │ │ Arc │ 3 │ │ Area │ 1 │ │ Line │ 3 │ │ LoadZone │ 1 │ │ Source │ 1 │ │ StandardLoad │ 3 │ │ ThermalStandard │ 2 │ └─────────────────┴───────┘ Dynamic Components ┌─────────────────────────────────────────────────────────────────────────────── │ Type ⋯ ├─────────────────────────────────────────────────────────────────────────────── │ DynamicGenerator{BaseMachine, SingleMass, AVRFixed, TGFixed, PSSFixed} ⋯ │ DynamicGenerator{RoundRotorExponential, SingleMass, ESST1A, TGFixed, PSSFixe ⋯ └─────────────────────────────────────────────────────────────────────────────── 2 columns omitted
Common Issues
Please note that while PSS/e does not enforce unique bus names, PowerSystems.jl
does. To reparse bus names to comply with this requirement the bus_name_formatter
*kwarg can be used in System()
as shown in the example below:
julia> dyn_system = System(RAW_dir, DYR_dir; bus_name_formatter = x -> strip(string(x["name"])) * "-" * string(x["index"]))
[ Info: The PSS(R)E parser currently supports buses, loads, shunts, generators, branches, transformers, and dc lines [ Info: The PSS(R)E parser currently supports buses, loads, shunts, generators, branches, transformers, and dc lines [ Info: Parsing PSS(R)E Bus data into a PowerModels Dict... [ Info: Parsing PSS(R)E Load data into a PowerModels Dict... [ Info: Parsing PSS(R)E Shunt data into a PowerModels Dict... [ Info: Parsing PSS(R)E Generator data into a PowerModels Dict... [ Info: Parsing PSS(R)E Branch data into a PowerModels Dict... [ Info: Parsing PSS(R)E Transformer data into a PowerModels Dict... [ Info: Parsing PSS(R)E Two-Terminal and VSC DC line data into a PowerModels Dict... ┌ Warning: This PSS(R)E parser currently doesn't support Storage data parsing... └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/pm_io/psse.jl:998 ┌ Warning: This PSS(R)E parser currently doesn't support Switch data parsing... └ @ PowerSystems ~/work/PowerSystems.jl/PowerSystems.jl/src/parsers/pm_io/psse.jl:1004 [ Info: angmin and angmax values are 0, widening these values on branch 1 to +/- 60.0 deg. [ Info: angmin and angmax values are 0, widening these values on branch 2 to +/- 60.0 deg. [ Info: angmin and angmax values are 0, widening these values on branch 3 to +/- 60.0 deg. ┌ Info: Constructing System from Power Models │ data["name"] = "threebusnetwork" └ data["source_type"] = "pti" [ Info: Reading bus data [ Info: Reading Load data in PowerModels dict to populate System ... [ Info: Reading LoadZones data in PowerModels dict to populate System ... [ Info: Reading generator data [ Info: Reading branch data [ Info: Reading shunt data [ Info: Reading DC Line data [ Info: Reading storage data [ Info: Generators provided in .dyr, without a generator in .raw file will be skipped. [ Info: Machine at bus 102, id 1 has zero inertia. Modeling it as Voltage Source ┌ Warning: struct DynamicGenerator does not exist in validation configuration file, validation skipped └ @ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/4ZGA8/src/validation.jl:51 ┌ Warning: struct DynamicGenerator does not exist in validation configuration file, validation skipped └ @ InfrastructureSystems ~/.julia/packages/InfrastructureSystems/4ZGA8/src/validation.jl:51 System ┌───────────────────┬─────────────┐ │ Property │ Value │ ├───────────────────┼─────────────┤ │ Name │ │ │ Description │ │ │ System Units Base │ SYSTEM_BASE │ │ Base Power │ 100.0 │ │ Base Frequency │ 60.0 │ │ Num Components │ 19 │ └───────────────────┴─────────────┘ Static Components ┌─────────────────┬───────┐ │ Type │ Count │ ├─────────────────┼───────┤ │ ACBus │ 3 │ │ Arc │ 3 │ │ Area │ 1 │ │ Line │ 3 │ │ LoadZone │ 1 │ │ Source │ 1 │ │ StandardLoad │ 3 │ │ ThermalStandard │ 2 │ └─────────────────┴───────┘ Dynamic Components ┌─────────────────────────────────────────────────────────────────────────────── │ Type ⋯ ├─────────────────────────────────────────────────────────────────────────────── │ DynamicGenerator{BaseMachine, SingleMass, AVRFixed, TGFixed, PSSFixed} ⋯ │ DynamicGenerator{RoundRotorExponential, SingleMass, ESST1A, TGFixed, PSSFixe ⋯ └─────────────────────────────────────────────────────────────────────────────── 2 columns omitted
In this example the anonymous function x -> strip(string(x["name"])) * "-" * string(x["index"])
takes the bus name and index from PSSe and concatenates them to produce the name.
PowerSystems Table Data
This is a custom format that allows users to define power system component data by category and column with custom names, types, and units.
Categories
Components for each category must be defined in their own CSV file. The following categories are currently supported:
- branch.csv
- bus.csv (required)
- columns specifying
area
andzone
will create a corresponding set ofArea
andLoadZone
objects. - columns specifying
max_active_power
ormax_reactive_power
will createPowerLoad
objects when nonzero values are encountered and will contribute to thepeak_active_power
andpeak_reactive_power
values for the
LoadZone
object. - columns specifying
- dc_branch.csv
- gen.csv
- load.csv
- reserves.csv
- storage.csv
These must reside in the directory passed when constructing PowerSystemTableData.
Adding Time Series Data
PowerSystems requires a metadata file that maps components to their time series data in order to be able to automatically construct time_series from raw data files. The following fields are required for each time array:
simulation
: User description of simulationresolution
: Resolution of time series in secondsmodule
: Module that defines the abstract type of the componentcategory
: Type of component. Must map to abstract types defined by the "module" entry (Bus, ElectricLoad, Generator, LoadZone, Reserve)component_name
: Name of componentname
: User-defined name for the time series data.normalization_factor
: Controls normalization of the data. Use 1.0 for pre-normalized data. Use 'Max' to divide the time series by the max value in the column. Use any float for a custom scaling factor.scaling_factor_multiplier_module
: Module that defines the accessor function for the
scaling factor
scaling_factor_multiplier
: Accessor function of the scaling factordata_file
: Path to the time series data file
Notes:
- The "module", "category", and "component_name" entries must be valid arguments to retrieve
a component using get_component(${module}.${category}, sys, $name)
.
- The "scalingfactormultipliermodule" and the "scalingfactor_multiplier" entries must
be sufficient to return the scaling factor data using ${scaling_factor_multiplier_module}.${scaling_factor_multiplier}(component)
.
PowerSystems supports this metadata in either CSV or JSON formats. Refer to RTS_GMLC for an example.
Performance considerations
By default PowerSystems stores time series data in HDF5 files. It does not keep all of the data in memory. This means that every time you access a timeseries PowerSystems will have to read the data from storage, which will add latency. If you know ahead of time that all of your data will fit in memory then you can change this behavior by passing `timeseriesinmemory = true` when you create the System.
If the time series data is stored in HDF5 then PowerSystems will use the tmp filesystem by default. You can change this by passing time_series_directory = X
when you create the System. This is required if the time series data is larger than the amount of tmp space available. You can also override the location by setting the environment variable SIENNATIMESERIES_DIRECTORY to another directory.
Customization
The tabular data parser in PowerSystems.jl
can be customized to read a variety of datasets by configuring:
- which type of generator (
<:Generator
) to create based on the fuel and prime mover specifications - property names, units, and per units conversions](@ref csvperunit) in *.csv files
Here is an example of how to construct a System with all customizations listed in this section:
data_dir = "/data/my-data-dir"
base_power = 100.0
descriptors = "./user_descriptors.yaml"
timeseries_metadata_file = "./timeseries_pointers.json"
generator_mapping_file = "./generator_mapping.yaml"
data = PowerSystemTableData(
data_dir,
base_power,
descriptors;
timeseries_metadata_file = timeseries_metadata_file,
generator_mapping_file = generator_mapping_file,
)
sys = System(data, time_series_in_memory = true)
Examples configuration files can be found in the RTS-GMLC repo:
CSV Data Configurations
Custom construction of generators
PowerSystems supports custom construction of subtypes of the abstract type Generator based on fuel
and type
. The parsing code detects these fields in the raw data and then constructs the concrete type listed in the passed generator mapping file. The default file is src/parsers/generator_mapping.yaml
. You can override this behavior by specifying your own file when constructing PowerSystemTableData.
Column names
PowerSystems provides am input mapping capability that allows you to keep your own column names.
For example, when parsing raw data for a generator the code expects a column called name
. If the raw data instead defines that column as GEN UID
then you can change the custom_name
field under the generator
category to GEN UID
in your YAML file.
To enable the parsing of a custom set of csv files, you can generate a configuration file (such as user_descriptors.yaml
) from the defaults, which are stored in src/descriptors/power_system_inputs.json
.
python ./bin/generate_config_file.py ./user_descriptors.yaml
Next, edit this file with your customizations.
Note that the user-specific customizations are stored in YAML rather than JSON to allow for easier editing. The next few sections describe changes you can make to this YAML file. Do not edit the default JSON file.
Per-unit conversion
For more info on the per-unit conventions in PowerSystems.jl
, refer to the per-unit section of the system documentation.
PowerSystems defines whether it expects a column value to be per-unit system base, per-unit device base, or natural units in power_system_inputs.json
. If it expects a per-unit convention that differs from your values then you can set the unit_system
in user_descriptors.yaml
and PowerSystems will automatically convert the values. For example, if you have a max_active_power
value stored in natural units (MW), but power_system_inputs.json
specifies unit_system: device_base
, you can enter unit_system: natural_units
in user_descriptors.yaml
and PowerSystems will divide the value by the value of the corresponding entry in the column identified by the base_reference
field in power_system_inputs.json
. You can also override the base_reference
setting by adding base_reference: My Column
to make device base per-unit conversion by dividing the value by the entry in My Column
. System base per-unit conversions always divide the value by the system base_power
value instantiated when constructing a System
.
Unit conversion
PowerSystems provides a limited set of unit conversions. For example, if power_system_inputs.json
indicates that a value's unit is degrees but your values are in radians then you can set unit: radian
in your YAML file. Other valid unit
entries include GW
, GWh
, MW
, MWh
, kW
, and kWh
.