Manipulating Datasets

PowerSystems provides function interfaces to all data, and in this tutorial we will explore how to do this using the show_components, get_component/ get_components, and getter (get_*) and setter (set_*) functions for component fields.

Viewing Components in the System

We are going to begin by loading in a test case System from PowerSystemCaseBuilder.jl:

julia> using PowerSystems;
julia> using PowerSystemCaseBuilder;
julia> sys = build_system(PSISystems, "c_sys5_pjm")[ Info: Loaded time series from storage file existing=/home/runner/.julia/packages/PowerSystemCaseBuilder/Eeywj/data/serialized_system/614e094ea985a55912fc1321256a49b755f9fc451c0f264f24d6d04af20e84d7/c_sys5_pjm_time_series_storage.h5 new=/tmp/jl_9ZOPHq compression=CompressionSettings(false, CompressionTypes.DEFLATE = 1, 3, true)
System
┌───────────────────┬─────────────┐
│ Property          │ Value       │
├───────────────────┼─────────────┤
│ Name              │             │
│ Description       │             │
│ System Units Base │ SYSTEM_BASE │
│ Base Power        │ 100.0       │
│ Base Frequency    │ 60.0        │
│ Num Components    │ 27          │
└───────────────────┴─────────────┘

Static Components
┌───────────────────┬───────┐
│ Type              │ Count │
├───────────────────┼───────┤
│ ACBus             │ 5     │
│ Arc               │ 6     │
│ Line              │ 6     │
│ PowerLoad         │ 3     │
│ RenewableDispatch │ 2     │
│ ThermalStandard   │ 5     │
└───────────────────┴───────┘

Time Series Summary
┌───────────────────┬────────────────┬──────────────────┬───────────────────────
│ owner_type        │ owner_category │ time_series_type │ time_series_category ⋯
│ String            │ String         │ String           │ String               ⋯
├───────────────────┼────────────────┼──────────────────┼───────────────────────
│ PowerLoad         │ Component      │ SingleTimeSeries │ StaticTimeSeries     ⋯
│ RenewableDispatch │ Component      │ SingleTimeSeries │ StaticTimeSeries     ⋯
└───────────────────┴────────────────┴──────────────────┴───────────────────────
                                                               3 columns omitted

Notice that the print statement for the System already includes a basic summary of the components, including 5 ThermalStandard components. We can use the show_components function to get more details:

julia> show_components(ThermalStandard, sys)ThermalStandard
┌───────────┬───────────┐
│ name      │ available │
├───────────┼───────────┤
│ Solitude  │ true      │
│ Park City │ true      │
│ Alta      │ true      │
│ Brighton  │ true      │
│ Sundance  │ true      │
└───────────┴───────────┘

We can see the names and availability are the standard fields returned when using show_components.

We can also view specific fields within components using the show_components function. For example, we can view the type of fuel the thermal generators are using, and their current active_power and reactive_power for a power flow case:

julia> show_components(ThermalStandard, sys, [:fuel, :active_power, :reactive_power])ThermalStandard
┌───────────┬───────────┬───────────────────────┬──────────────┬────────────────┐
│ name      │ available │ fuel                  │ active_power │ reactive_power │
├───────────┼───────────┼───────────────────────┼──────────────┼────────────────┤
│ Solitude  │ true      │ ThermalFuels.COAL = 1 │ 5.2          │ 1.0            │
│ Park City │ true      │ ThermalFuels.COAL = 1 │ 1.7          │ 0.2            │
│ Alta      │ true      │ ThermalFuels.COAL = 1 │ 0.4          │ 0.01           │
│ Brighton  │ true      │ ThermalFuels.COAL = 1 │ 6.0          │ 1.5            │
│ Sundance  │ true      │ ThermalFuels.COAL = 1 │ 2.0          │ 0.4            │
└───────────┴───────────┴───────────────────────┴──────────────┴────────────────┘

Notice all our thermal generators are currently fueled by coal.

Accessing and Updating a Component in a System

We can access a component in our system using the get_component function. For example, if we are interested in accessing a ThermalStandard component we can do so using the component's name and Type from PowerSystems.jl's Type Tree. From above we know the names of the thermal generators.

julia> solitude = get_component(ThermalStandard, sys, "Solitude")ThermalStandard: Solitude:
   name: Solitude
   available: true
   status: true
   bus: ACBus: nodeC
   active_power: 5.2
   reactive_power: 1.0
   rating: 5.2
   active_power_limits: (min = 0.0, max = 5.2)
   reactive_power_limits: (min = -3.9, max = 3.9)
   ramp_limits: (up = 0.062400000000000004, down = 0.062400000000000004)
   operation_cost: ThermalGenerationCost composed of variable: CostCurve{LinearCurve}
   base_power: 100.0
   time_limits: (up = 3.0, down = 2.0)
   must_run: false
   prime_mover_type: PrimeMovers.ST = 20
   fuel: ThermalFuels.COAL = 1
   services: 0-element Vector{Service}
   time_at_status: 10000.0
   dynamic_injector: nothing
   ext: Dict{String, Any}()
   InfrastructureSystems.SystemUnitsSettings:
      base_value: 100.0
      unit_system: UnitSystem.SYSTEM_BASE = 0
   has_supplemental_attributes: false
   has_time_series: false

Notice that all of Solitude's fields are pretty-printed with the return statment for quick reference. However, what is returned is a ThermalStandard object we can manipulate:

julia> typeof(solitude)ThermalStandard

If we are interested in accessing a particular field, we can use a get_* function, also known as a getter, on this object. For example, if we are interested in the fuel we can use get_fuel:

julia> get_fuel(solitude)ThermalFuels.COAL = 1

You can see a ThermalFuels option returned.

To recap, get_component will return a component object, but we can use a specific get_* function to return the data in a particular field.

Warning

Using the "dot" access to get a field value from a component is actively discouraged, use get_* functions instead. Julia syntax enables access to this data using the "dot" access (e.g., solitude.fuel), however this is discouraged for two reasons:

We make no guarantees on the stability of component structure definitions. We will maintain version stability on the accessor methods.
Per-unit conversions are made in the return of data from the accessor functions. (see the per-unit section for more details)

To update a field we can use a specific set_*, or setter function, which are defined for each component field. We can use set_fuel! to update the fuel field of Solitude to natural gas.

julia> set_fuel!(solitude, ThermalFuels.NATURAL_GAS)ThermalFuels.NATURAL_GAS = 7

We can use show_components again to check that the Solitude fuel has been updated to ThermalFuels.NATURAL_GAS:

julia> show_components(ThermalStandard, sys, [:fuel])ThermalStandard
┌───────────┬───────────┬──────────────────────────────┐
│ name      │ available │ fuel                         │
├───────────┼───────────┼──────────────────────────────┤
│ Solitude  │ true      │ ThermalFuels.NATURAL_GAS = 7 │
│ Park City │ true      │ ThermalFuels.COAL = 1        │
│ Alta      │ true      │ ThermalFuels.COAL = 1        │
│ Brighton  │ true      │ ThermalFuels.COAL = 1        │
│ Sundance  │ true      │ ThermalFuels.COAL = 1        │
└───────────┴───────────┴──────────────────────────────┘

Similarly, you can updated the active_power field using its specific get_* and set_* functions. We can access this field by using get_active_power:

julia> get_active_power(solitude)5.2

We can then update it using set_active_power!:

julia> set_active_power!(solitude, 4.0)4.0

We can see that our active_power field has been updated to 4.0.

Accessing and Updating Multiple Components in the System at Once

We can also update more than one component at a time using the get_components and set_* functions.

Let's say we were interested in updating the base_voltage field for all of the ACBus. We can see that currently the base_voltages are:

julia> show_components(ACBus, sys, [:base_voltage])ACBus
┌───────┬──────────────┐
│ name  │ base_voltage │
├───────┼──────────────┤
│ nodeA │ 230.0        │
│ nodeC │ 230.0        │
│ nodeE │ 230.0        │
│ nodeB │ 230.0        │
│ nodeD │ 230.0        │
└───────┴──────────────┘

But what if we are looking to correct them to 250.0 kV? Let's start by getting an iterator for all the buses using get_components:

julia> buses = get_components(ACBus, sys)ACBus Counts:
ACBus: 5

See that the pretty-print summarizes this set of components, but let's check what was actually returned:

julia> typeof(buses)InfrastructureSystems.FlattenIteratorWrapper{ACBus, Vector{Base.ValueIterator}}

Tip

Notice that get_components and similar functions return Julia iterators, which allows you to access and manipulate data without a large memory allocation that might occur for very large data sets. Use collect to gather the data to a vector instead, but be aware of your dataset size.

Now using the set_base_voltage! function and a for loop we can update the voltage:

julia> for i in buses
           set_base_voltage!(i, 250.0)
       end

We could use show_components to verify the results, but this time let's use a get_* function and Julia's dot notation over our bus iterator to get the data for a specific field from multiple components:

julia> get_base_voltage.(buses)5-element Vector{Float64}:
 250.0
 250.0
 250.0
 250.0
 250.0

We can see that all of the buses now have a base_voltage of 250.0 kV.

If we are interested in updating the fuel in all the thermal generators, we would use a similar approach. We begin by grabbing an iterator for all the components in ThermalStandard.

julia> thermal_gens = get_components(ThermalStandard, sys)ThermalStandard Counts:
ThermalStandard: 5

Now, using the set_fuel! and a for loop, we will update the fuel to NATURAL_GAS.

julia> for i in thermal_gens
           set_fuel!(i, ThermalFuels.NATURAL_GAS)
       end

We can verify that the fuel types for all the thermal generators has been updated, using dot notation again to access the fuel fields:

julia> get_fuel.(get_components(ThermalStandard, sys))5-element Vector{ThermalFuels}:
 ThermalFuels.NATURAL_GAS = 7
 ThermalFuels.NATURAL_GAS = 7
 ThermalFuels.NATURAL_GAS = 7
 ThermalFuels.NATURAL_GAS = 7
 ThermalFuels.NATURAL_GAS = 7

See that we linked two functions here with Julia's dot notation – this is a very convenient way of quickly getting the data you need.

Filtering Specific Data

We have seen how to update a single component, and all the components of a specific type, but what if we are interested in updating only particular components? We can do this using filter functions.

For example, let's say we are interested in updating all the active_power values of the thermal generators except Solitude. Let's start by seeing the current active_power values.

julia> show_components(ThermalStandard, sys, [:active_power])ThermalStandard
┌───────────┬───────────┬──────────────┐
│ name      │ available │ active_power │
├───────────┼───────────┼──────────────┤
│ Solitude  │ true      │ 4.0          │
│ Park City │ true      │ 1.7          │
│ Alta      │ true      │ 0.4          │
│ Brighton  │ true      │ 6.0          │
│ Sundance  │ true      │ 2.0          │
└───────────┴───────────┴──────────────┘

Let's grab an iterator for the all the thermal generators except Solitude by adding a filter function in another version of the get_components function defined with Julia's multiple dispatch:

julia> thermal_not_solitude = get_components(x -> get_name(x) != "Solitude", ThermalStandard, sys)ThermalStandard Counts:
ThermalStandard: 4

We can see that only four ThermalStandard components are returned, as expected. Now let's update the active_power field of these four thermal generators using the set_active_power! function.

julia> for i in thermal_not_solitude
           set_active_power!(i, 0.0)
       end

Let's check the update using show_components:

julia> show_components(ThermalStandard, sys, [:active_power])ThermalStandard
┌───────────┬───────────┬──────────────┐
│ name      │ available │ active_power │
├───────────┼───────────┼──────────────┤
│ Solitude  │ true      │ 4.0          │
│ Park City │ true      │ 0.0          │
│ Alta      │ true      │ 0.0          │
│ Brighton  │ true      │ 0.0          │
│ Sundance  │ true      │ 0.0          │
└───────────┴───────────┴──────────────┘

We can see that all the active_power values are 0.0, except Solitude.

We can filter on any component field. Similarly, let's filter all of the thermal generators that now have an active_power of 0.0, and also set their availability to false.

julia> for i in get_components(x -> get_active_power(x) == 0.0, ThermalStandard, sys)
           set_available!(i, 0)
       end

Getting Available Components

The get_available_components function is a useful short-hand function with a built-in filter for grabbing all the components of a particular type that are available.

For example, if we are interested in grabbing all the available ThermalStandard:

julia> get_available_components(ThermalStandard, sys)ThermalStandard Counts:
ThermalStandard: 1

We only retrieved one component, because we just set the rest to unavailable above:

julia> show_components(ThermalStandard, sys)ThermalStandard
┌───────────┬───────────┐
│ name      │ available │
├───────────┼───────────┤
│ Solitude  │ true      │
│ Park City │ false     │
│ Alta      │ false     │
│ Brighton  │ false     │
│ Sundance  │ false     │
└───────────┴───────────┘

Getting Buses

We can retrieve the ACBus components using get_buses, by ID number or Area or LoadZone.

Let's begin by accessing the ID numbers associated with the ACBus components using the get_bus_numbers function.

julia> get_bus_numbers(sys)5-element Vector{Int64}:
 1
 2
 3
 4
 5

We can see that these bus IDs are numbered 1 through 5. Now let's specifically grab buses 2 and 3 using the get_buses function:

julia> high_voltage_buses = get_buses(sys, Set(2:3))2-element Vector{ACBus}:
 ACBus(3, nodeC, ACBusTypes.PV = 2, 0.0, 1.0, (min = 0.9, max = 1.05), 250.0, nothing, nothing, Dict{String, Any}())
 ACBus(2, nodeB, ACBusTypes.PQ = 1, 0.0, 1.0, (min = 0.9, max = 1.05), 250.0, nothing, nothing, Dict{String, Any}())

and update their base voltage to 330 kV:

julia> for i in high_voltage_buses
           set_base_voltage!(i, 330.0)
       end

As usual, we can review the updated data with show_components:

julia> show_components(ACBus, sys, [:number, :base_voltage])ACBus
┌───────┬────────┬──────────────┐
│ name  │ number │ base_voltage │
├───────┼────────┼──────────────┤
│ nodeA │ 1      │ 250.0        │
│ nodeC │ 3      │ 330.0        │
│ nodeE │ 5      │ 250.0        │
│ nodeB │ 2      │ 330.0        │
│ nodeD │ 4      │ 250.0        │
└───────┴────────┴──────────────┘

Updating Component Names

We can also access and update the component name field using the get_name and set_name! functions.

Recall that we created an iterator called thermal_gens for all the thermal generators. We can use the get_name function to access the name field for these components with dot notation:

julia> get_name.(thermal_gens)5-element Vector{String}:
 "Solitude"
 "Park City"
 "Alta"
 "Brighton"
 "Sundance"

To update the names we will use the set_name! function.

Warning

Specifically when using set_name! to modify multiple components accessed through an iterator, it is important to note that this not only changes the field name, but also changes the iterator itself as you are iterating over it, which will result in unexpected outcomes and errors.

Therefore, rather than using set_name! on an iterator, only use it after first calling collect on the iterator to get a vector of the components:

julia> for thermal_gen in collect(get_components(ThermalStandard, sys))
           set_name!(sys, thermal_gen, get_name(thermal_gen) * "-renamed")
       end

Now we can check the names using the get_name function again.

julia> get_name.(get_components(ThermalStandard, sys))5-element Vector{String}:
 "Brighton-renamed"
 "Solitude-renamed"
 "Sundance-renamed"
 "Alta-renamed"
 "Park City-renamed"

Be aware again that accessing components through a vector using collect might cause large memory allocations, based on your dataset size.

Next Steps & Links

In this tutorial, we explored a dataset using show_components to summarize data and accessed particular groups of components with get_components, get_buses, and get_available_components. We used specific get_* functions and set_* functions to see and update the fields in ThermalStandard and ACBus components, but remember that these getters and setters are available for each data field for components of all Types in PowerSystems.jl.

Follow the next tutorials to learn how to work with time series.