Modifying a case file¶

The Dispatch API allows users to perform numerical experiments. By augmenting a selected case file parameter, while keeping all other parameters constant, the relationship between dispatch outcomes and the parameter under investigation can be examined.

The following sections discuss strategies that can be used to modify case files. While this notebook uses Python, it's possible to use other programming languages and workflows. The Dispatch API simply expects a case file in XML or JSON format - so long as the inputs are correctly structured, the API is agnostic as to the technology used to edit and submit the case file.

Imports¶

In [1]:

import requests
import xmltodict
import pandas as pd
from jsonpath_ng.ext import parse

# Base URL endpoint for the Dispatch API
base_url = 'http://localhost:8000/'

import requests import xmltodict import pandas as pd from jsonpath_ng.ext import parse # Base URL endpoint for the Dispatch API base_url = 'http://localhost:8000/'

Approach¶

Case files contain tens of thousands of parameters. While it's possible to design a completely customised case file, users should note the Dispatch API expects case files to be submitted in a standard format. Errors introduced when constructing a case file will almost certainly result in the model failing to return a solution.

For now let's use data from a historical case file and only modify selected components. We can proceed by first loading a case file.

In [2]:

def convert_casefile(path_to_file):
    """Load a historical NEMDE case file and convert it to a dict"""
    
    # Read case file contents
    with open(path_to_file, 'r') as f:
        casefile = f.read()
    
    # Force these nodes to always return lists
    force_list = ('Trade', 'TradeTypePriceStructure',)

    return xmltodict.parse(casefile, force_list=force_list)


# Load case file and convert to JSON
casefile = convert_casefile('../../data/NEMSPDOutputs_2021040100100.loaded')

def convert_casefile(path_to_file): """Load a historical NEMDE case file and convert it to a dict""" # Read case file contents with open(path_to_file, 'r') as f: casefile = f.read() # Force these nodes to always return lists force_list = ('Trade', 'TradeTypePriceStructure',) return xmltodict.parse(casefile, force_list=force_list) # Load case file and convert to JSON casefile = convert_casefile('../../data/NEMSPDOutputs_2021040100100.loaded')

Method 1 - Traverse dictionary¶

The simplest strategy is to traverse the case file dictionary and update parameters directly. The parameter reference page can be used to see which parameters can be meaningfully updated.

Let's update the Demand Forecast (@DF) parameter for South Australia as an example. This parameter corresponds to the amount by which demand is expected to change over the dispatch interval. From the parameter reference page we can see the path to this parameter is as follows:

NEMSPDCaseFile.NemSpdInputs.PeriodCollection.Period.RegionPeriodCollection.RegionPeriod[?(@RegionID="{region_id}")].@DF

Using this path we can traverse nodes within the dictionary:

In [3]:

regions = (casefile.get('NEMSPDCaseFile').get('NemSpdInputs')
           .get('PeriodCollection').get('Period')
           .get('RegionPeriodCollection').get('RegionPeriod'))
regions

regions = (casefile.get('NEMSPDCaseFile').get('NemSpdInputs') .get('PeriodCollection').get('Period') .get('RegionPeriodCollection').get('RegionPeriod')) regions

Out[3]:

[{'@RegionID': 'NSW1',
  '@DF': '50.11376953125',
  '@DemandForecast': '6575.22802734375',
  '@Suspension_Schedule_Energy_Price': '34.66',
  '@Suspension_Schedule_R6_Price': '1.01',
  '@Suspension_Schedule_R60_Price': '1.41',
  '@Suspension_Schedule_R5_Price': '0.76',
  '@Suspension_Schedule_RReg_Price': '11.18',
  '@Suspension_Schedule_L6_Price': '0.45',
  '@Suspension_Schedule_L60_Price': '1.19',
  '@Suspension_Schedule_L5_Price': '0.39',
  '@Suspension_Schedule_LReg_Price': '9.72'},
 {'@RegionID': 'QLD1',
  '@DF': '1.39823298286458',
  '@DemandForecast': '5362.80249',
  '@Suspension_Schedule_Energy_Price': '35.01',
  '@Suspension_Schedule_R6_Price': '1.01',
  '@Suspension_Schedule_R60_Price': '1.41',
  '@Suspension_Schedule_R5_Price': '0.76',
  '@Suspension_Schedule_RReg_Price': '11.18',
  '@Suspension_Schedule_L6_Price': '0.45',
  '@Suspension_Schedule_L60_Price': '1.19',
  '@Suspension_Schedule_L5_Price': '0.39',
  '@Suspension_Schedule_LReg_Price': '9.72'},
 {'@RegionID': 'SA1',
  '@DF': '5.0419807434082',
  '@DemandForecast': '1075.27025222778',
  '@Suspension_Schedule_Energy_Price': '32.23',
  '@Suspension_Schedule_R6_Price': '1.01',
  '@Suspension_Schedule_R60_Price': '1.41',
  '@Suspension_Schedule_R5_Price': '0.76',
  '@Suspension_Schedule_RReg_Price': '11.18',
  '@Suspension_Schedule_L6_Price': '0.39',
  '@Suspension_Schedule_L60_Price': '2.93',
  '@Suspension_Schedule_L5_Price': '0.37',
  '@Suspension_Schedule_LReg_Price': '9.7'},
 {'@RegionID': 'TAS1',
  '@DF': '2.3228759765625',
  '@DemandForecast': '997.246337890625',
  '@Suspension_Schedule_Energy_Price': '26.84',
  '@Suspension_Schedule_R6_Price': '9.46',
  '@Suspension_Schedule_R60_Price': '8.99',
  '@Suspension_Schedule_R5_Price': '0.91',
  '@Suspension_Schedule_RReg_Price': '11.16',
  '@Suspension_Schedule_L6_Price': '1.27',
  '@Suspension_Schedule_L60_Price': '2.21',
  '@Suspension_Schedule_L5_Price': '0.35',
  '@Suspension_Schedule_LReg_Price': '9.15'},
 {'@RegionID': 'VIC1',
  '@DF': '49.33447265625',
  '@DemandForecast': '4157.650390625',
  '@Suspension_Schedule_Energy_Price': '22.54',
  '@Suspension_Schedule_R6_Price': '1.01',
  '@Suspension_Schedule_R60_Price': '1.41',
  '@Suspension_Schedule_R5_Price': '0.76',
  '@Suspension_Schedule_RReg_Price': '11.18',
  '@Suspension_Schedule_L6_Price': '0.45',
  '@Suspension_Schedule_L60_Price': '1.19',
  '@Suspension_Schedule_L5_Price': '0.39',
  '@Suspension_Schedule_LReg_Price': '9.72'}]

Once we reach the RegionPeriod node we encounter a list of dictionaries describing parameters for each region. We can loop through the list and update the @DF parameter for South Australia (i.e. the dictionary with @RegionID == 'SA1').

In [4]:

# Updating the @DF parameter for South Australia
for i in regions:
    if i.get('@RegionID') == 'SA1':
        i['@DF'] = 20

# Updating the @DF parameter for South Australia for i in regions: if i.get('@RegionID') == 'SA1': i['@DF'] = 20

Note: By default all values within a case file are of type 'string'. Strings, floats or integers can be used when updating case file parameters as types are converted in a preprocessing step before formulating a mathematical program from the inputs. Strings should be used when updating flags e.g. parameters that take on a value of '1' or '0'.

Checking the value has been updated.

In [5]:

(casefile.get('NEMSPDCaseFile').get('NemSpdInputs')
 .get('PeriodCollection').get('Period')
 .get('RegionPeriodCollection').get('RegionPeriod'))

(casefile.get('NEMSPDCaseFile').get('NemSpdInputs') .get('PeriodCollection').get('Period') .get('RegionPeriodCollection').get('RegionPeriod'))

Out[5]:

[{'@RegionID': 'NSW1',
  '@DF': '50.11376953125',
  '@DemandForecast': '6575.22802734375',
  '@Suspension_Schedule_Energy_Price': '34.66',
  '@Suspension_Schedule_R6_Price': '1.01',
  '@Suspension_Schedule_R60_Price': '1.41',
  '@Suspension_Schedule_R5_Price': '0.76',
  '@Suspension_Schedule_RReg_Price': '11.18',
  '@Suspension_Schedule_L6_Price': '0.45',
  '@Suspension_Schedule_L60_Price': '1.19',
  '@Suspension_Schedule_L5_Price': '0.39',
  '@Suspension_Schedule_LReg_Price': '9.72'},
 {'@RegionID': 'QLD1',
  '@DF': '1.39823298286458',
  '@DemandForecast': '5362.80249',
  '@Suspension_Schedule_Energy_Price': '35.01',
  '@Suspension_Schedule_R6_Price': '1.01',
  '@Suspension_Schedule_R60_Price': '1.41',
  '@Suspension_Schedule_R5_Price': '0.76',
  '@Suspension_Schedule_RReg_Price': '11.18',
  '@Suspension_Schedule_L6_Price': '0.45',
  '@Suspension_Schedule_L60_Price': '1.19',
  '@Suspension_Schedule_L5_Price': '0.39',
  '@Suspension_Schedule_LReg_Price': '9.72'},
 {'@RegionID': 'SA1',
  '@DF': 20,
  '@DemandForecast': '1075.27025222778',
  '@Suspension_Schedule_Energy_Price': '32.23',
  '@Suspension_Schedule_R6_Price': '1.01',
  '@Suspension_Schedule_R60_Price': '1.41',
  '@Suspension_Schedule_R5_Price': '0.76',
  '@Suspension_Schedule_RReg_Price': '11.18',
  '@Suspension_Schedule_L6_Price': '0.39',
  '@Suspension_Schedule_L60_Price': '2.93',
  '@Suspension_Schedule_L5_Price': '0.37',
  '@Suspension_Schedule_LReg_Price': '9.7'},
 {'@RegionID': 'TAS1',
  '@DF': '2.3228759765625',
  '@DemandForecast': '997.246337890625',
  '@Suspension_Schedule_Energy_Price': '26.84',
  '@Suspension_Schedule_R6_Price': '9.46',
  '@Suspension_Schedule_R60_Price': '8.99',
  '@Suspension_Schedule_R5_Price': '0.91',
  '@Suspension_Schedule_RReg_Price': '11.16',
  '@Suspension_Schedule_L6_Price': '1.27',
  '@Suspension_Schedule_L60_Price': '2.21',
  '@Suspension_Schedule_L5_Price': '0.35',
  '@Suspension_Schedule_LReg_Price': '9.15'},
 {'@RegionID': 'VIC1',
  '@DF': '49.33447265625',
  '@DemandForecast': '4157.650390625',
  '@Suspension_Schedule_Energy_Price': '22.54',
  '@Suspension_Schedule_R6_Price': '1.01',
  '@Suspension_Schedule_R60_Price': '1.41',
  '@Suspension_Schedule_R5_Price': '0.76',
  '@Suspension_Schedule_RReg_Price': '11.18',
  '@Suspension_Schedule_L6_Price': '0.45',
  '@Suspension_Schedule_L60_Price': '1.19',
  '@Suspension_Schedule_L5_Price': '0.39',
  '@Suspension_Schedule_LReg_Price': '9.72'}]

Submitting a modified case file¶

The same steps outlined in the previous tutorial can be followed to submit the modified case file to the /solve endpoint.

In [6]:

def solve_casefile(base_url, casefile):
    """Submit a case file to the Dispatch API and return the solution"""

    url = base_url + 'solve'

    # Passing the case file via 'json' sets the 'Content-Type: application/json' header
    response = requests.post(url=url, json=casefile)
    response.raise_for_status()

    return response.json()


# Solve the modified case file
solution_modified = solve_casefile(base_url=base_url, casefile=casefile)

def solve_casefile(base_url, casefile): """Submit a case file to the Dispatch API and return the solution""" url = base_url + 'solve' # Passing the case file via 'json' sets the 'Content-Type: application/json' header response = requests.post(url=url, json=casefile) response.raise_for_status() return response.json() # Solve the modified case file solution_modified = solve_casefile(base_url=base_url, casefile=casefile)

The response only contains the solution, so to see the update's impact we can compare it with the solution obtained using the original case file. The modifications above were made to casefile in place, so the original case file is loaded again from disk.

In [7]:

# Load and solve the original, unmodified case file
casefile_original = convert_casefile('../../data/NEMSPDOutputs_2021040100100.loaded')
solution_original = solve_casefile(base_url=base_url, casefile=casefile_original)

# Load and solve the original, unmodified case file casefile_original = convert_casefile('../../data/NEMSPDOutputs_2021040100100.loaded') solution_original = solve_casefile(base_url=base_url, casefile=casefile_original)

Let's compare selected results for South Australia.

In [8]:

def get_region_solution(solution, region_id):
    """Extract the solution for a given region"""

    return next(i for i in solution.get('RegionSolution') if i.get('@RegionID') == region_id)


# Compare South Australia's results for the original and modified case files
columns = ['@FixedDemand', '@ClearedDemand', '@NetExport', '@EnergyPrice']
pd.DataFrame({
    'original': get_region_solution(solution=solution_original, region_id='SA1'),
    'modified': get_region_solution(solution=solution_modified, region_id='SA1'),
}).loc[columns]

def get_region_solution(solution, region_id): """Extract the solution for a given region""" return next(i for i in solution.get('RegionSolution') if i.get('@RegionID') == region_id) # Compare South Australia's results for the original and modified case files columns = ['@FixedDemand', '@ClearedDemand', '@NetExport', '@EnergyPrice'] pd.DataFrame({ 'original': get_region_solution(solution=solution_original, region_id='SA1'), 'modified': get_region_solution(solution=solution_modified, region_id='SA1'), }).loc[columns]

Out[8]:

	original	modified
@FixedDemand	1065.295629	1080.253648
@ClearedDemand	1077.473676	1092.754301
@NetExport	-264.539133	-279.497148
@EnergyPrice	37.246542	37.800731

Increasing South Australia's @DF parameter increases the region's fixed demand by the same amount (approximately 15 MW). This additional demand is met by importing more power from Victoria (net exports become more negative), and the region's energy price increases slightly. We can see our update is reflected in the solution returned by the API.

Method 2 - Using JSON path syntax¶

While the previous method is quite intuitive, it is not very robust - it's easy to lose track of which values have been updated when using loops. An alternative is to search and update the case file dictionary using JSON path syntax. Rather than loop through a list, expressions can be specified to find and update specific elements. See jsonpath-ng to learn more about the syntax.

An expression targeting the @DF parameter for South Australia can be formulated as follows:

In [9]:

# Path to South Australia region period parameters
expression = ("NEMSPDCaseFile \
              .NemSpdInputs \
              .PeriodCollection \
              .Period \
              .RegionPeriodCollection \
              .RegionPeriod[?(@RegionID=='SA1')] \
              .@DF")

# Path to South Australia region period parameters expression = ("NEMSPDCaseFile \ .NemSpdInputs \ .PeriodCollection \ .Period \ .RegionPeriodCollection \ .RegionPeriod[?(@RegionID=='SA1')] \ .@DF")

Note this expression corresponds to the path outlined on the parameter reference page. When seeking to update parameters users can consult this document to find paths corresponding to parameters of interest.

The following functions can be used to get and update parameters using a JSON path expression.

In [10]:

def get_casefile_parameter(casefile, expression):
    """
    Get parameter given a case file and JSON path expression
    
    Parameters
    ----------
    casefile : dict
        System parameters
    
    expression : str
        JSON path expression to value or object that should be
        extracted
        
    Returns
    -------
    Value corresponding to JSON path expression.    
    """

    jsonpath_expr = parse(expression)
    values = [match.value for match in jsonpath_expr.find(casefile)]
    
    # Check only one match found
    if len(values) != 1:
        raise Exception(f'Expected 1 match, encountered {len(values)}')
        
    return values[0]


def update_casefile_parameter(casefile, expression, new_value):
    """
    Update case file parameter
    
    Parameters
    ----------
    casefile : dict
        System parameters
    
    expression : str
        JSON path to value or object that should be updated
    
    new_value : str, float, or int
        New value for parameter
    
    Returns
    -------
    casefile : dict
        Updated case file
    """

    jsonpath_expr = parse(expression)
    values = [match.value for match in jsonpath_expr.find(casefile)]
    
    # Check only one match found
    if len(values) != 1:
        raise Exception(f'Expected 1 match, encountered {len(values)}')
    
    # Update case file
    jsonpath_expr.update(casefile, new_value)
    
    return casefile

def get_casefile_parameter(casefile, expression): """ Get parameter given a case file and JSON path expression Parameters ---------- casefile : dict System parameters expression : str JSON path expression to value or object that should be extracted Returns ------- Value corresponding to JSON path expression. """ jsonpath_expr = parse(expression) values = [match.value for match in jsonpath_expr.find(casefile)] # Check only one match found if len(values) != 1: raise Exception(f'Expected 1 match, encountered {len(values)}') return values[0] def update_casefile_parameter(casefile, expression, new_value): """ Update case file parameter Parameters ---------- casefile : dict System parameters expression : str JSON path to value or object that should be updated new_value : str, float, or int New value for parameter Returns ------- casefile : dict Updated case file """ jsonpath_expr = parse(expression) values = [match.value for match in jsonpath_expr.find(casefile)] # Check only one match found if len(values) != 1: raise Exception(f'Expected 1 match, encountered {len(values)}') # Update case file jsonpath_expr.update(casefile, new_value) return casefile

Let's get the value of South Australia's @DF parameter.

In [11]:

get_casefile_parameter(casefile=casefile, expression=expression)

get_casefile_parameter(casefile=casefile, expression=expression)

Out[11]:

20

Similarly, we can update values given an expression.

In [12]:

# Update @DF parameter for SA1 - set @DF = 60
casefile = update_casefile_parameter(casefile=casefile, expression=expression, new_value=60)

# Check the value has been updated
get_casefile_parameter(casefile=casefile, expression=expression)

# Update @DF parameter for SA1 - set @DF = 60 casefile = update_casefile_parameter(casefile=casefile, expression=expression, new_value=60) # Check the value has been updated get_casefile_parameter(casefile=casefile, expression=expression)

Out[12]:

60

Summary¶

We've explored two ways to update case file parameters. The first method can be useful if seeking to explore a case file's structure, and augment parameters in an ad hoc manner. The second method is more precise in its ability to target specific parameters within a case file as it avoids the use of loops. The following tutorials will build upon these tools when conducting scenario analyses using the Dispatch API.