Running a model¶

The Dispatch API is a web service that you run yourself (see Running the API on the home page). A case file describing the system's state is sent to the API in the body of a request. The API uses these inputs to formulate and solve a mathematical program that approximates the NEMDE's operation, and returns the solution in its response.

This notebook describes how to load a case file, submit it to the API, and examine the results.

Imports¶

In [1]:

import requests
import xmltodict
import pandas as pd

import IPython.display as display

import requests import xmltodict import pandas as pd import IPython.display as display

First specify the base URL for API calls. This assumes the API is running locally on port 8000.

In [2]:

# Base URL endpoint for the Dispatch API
base_url = 'http://localhost:8000/'

# Base URL endpoint for the Dispatch API base_url = 'http://localhost:8000/'

Load case file¶

Case files contain parameters used by the NEMDE when determining dispatch targets for generators and loads. The API uses selected inputs from these case files when formulating a mathematical program that approximates the NEMDE's operation (see the parameter reference page for a description of the inputs used).

Each case file is identified by a case ID, e.g. 20201101001, which has the following format:

{year}{month}{day}{interval_id}

Interval IDs range from 001-288 and are used to identify each 5 minute interval within a 24 hour period.

As each trading day begins at 4.00am, the case ID 20201101001 corresponds to the trading interval which starts at 2020-11-01 04:00:00 and ends at 2020-11-01 04:05:00. When NEMDE is run at approximately 04:00:00 it produces dispatch targets that generators and loads should meet at 04:05:00. The following table illustrates the mapping between case IDs and the start and end times for selected dispatch intervals.

Case ID	Interval start	Interval end
20201101001	2020-11-01 04:00:00	2020-11-01 04:05:00
20201101002	2020-11-01 04:05:00	2020-11-01 04:10:00
20201101003	2020-11-01 04:10:00	2020-11-01 04:15:00
...	...	...
20201101287	2020-11-02 03:50:00	2020-11-02 03:55:00
20201101288	2020-11-02 03:55:00	2020-11-02 04:00:00

The following function loads a case file in XML format and converts it to a Python dictionary.

In [3]:

def convert_casefile(path_to_file):
    """Load a historical NEMDE case file and convert it to a dict"""
    
    # Read case file contents
    with open(path_to_file, 'r') as f:
        casefile = f.read()
    
    # Force these nodes to always return lists
    force_list = ('Trade', 'TradeTypePriceStructure',)

    return xmltodict.parse(casefile, force_list=force_list)

casefile = convert_casefile('../../data/NEMSPDOutputs_2021040100100.loaded')

def convert_casefile(path_to_file): """Load a historical NEMDE case file and convert it to a dict""" # Read case file contents with open(path_to_file, 'r') as f: casefile = f.read() # Force these nodes to always return lists force_list = ('Trade', 'TradeTypePriceStructure',) return xmltodict.parse(casefile, force_list=force_list) casefile = convert_casefile('../../data/NEMSPDOutputs_2021040100100.loaded')

Case file components¶

Case files describe the state of the system at the start of a dispatch interval, and also include forecasts for demand and intermittent generation at the end of the interval. A nested data structure, in this case a Python dictionary, organises data into logical components.

Those familiar with NEMDE case files in XML format may recognise the following data structure. In fact, the dictionary is obtained by converting a NEMDE case file in XML format to a Python dictionary. See this tutorial to learn how to convert your own NEMDE XML files into a format that can be consumed by the Dispatch API.

At the dictionary's root there is a single key, NEMSPDCaseFile:

In [4]:

casefile.keys()

casefile.keys()

Out[4]:

dict_keys(['NEMSPDCaseFile'])

The dictionary can be traversed by 'getting' the value for a given key, in this case NEMSPDCaseFile, and looking at its constituent components. Here we can see there are three nested keys:

In [5]:

casefile.get('NEMSPDCaseFile').keys()

casefile.get('NEMSPDCaseFile').keys()

Out[5]:

dict_keys(['NemSpdInputs', 'NemSpdOutputs', 'SolutionAnalysis'])

Key	Description
NemSpdInputs	Parameters describing the system's state
NemSpdOutputs	NEMDE solution for each trader (generator / load), generic constraint, interconnector, and region
SolutionAnalysis	Price setting results

While NEMDE case files provide a convenient data structure describing parameters used to set dispatch targets, there are limitations associated with their design. For instance, some parameters are duplicated, while others may be ignored. Users seeking to modify case files should consult the parameter reference page to see which parameters can be meaningfully modified when using the Dispatch API.

Solving a case file¶

For now let's run the case file without modifying any of its components. The case file dictionary is sent as JSON in the body of a POST request to the /solve endpoint:

http://localhost:8000/solve

The API formulates and solves the model, then returns the solution in its response. There is no queue, so the request simply waits until the solution is ready, which usually takes well under a minute.

The API also accepts the original XML file directly if the Content-Type header is set to application/xml. Here we send the dictionary instead, as later tutorials modify its values before submitting it.

In [6]:

def solve_casefile(base_url, casefile):
    """Submit a case file to the Dispatch API and return the solution"""

    url = base_url + 'solve'

    # Passing the case file via 'json' sets the 'Content-Type: application/json' header
    response = requests.post(url=url, json=casefile)
    response.raise_for_status()

    return response.json()


# Solve the case file and inspect the solution's components
solution = solve_casefile(base_url=base_url, casefile=casefile)
solution.keys()

def solve_casefile(base_url, casefile): """Submit a case file to the Dispatch API and return the solution""" url = base_url + 'solve' # Passing the case file via 'json' sets the 'Content-Type: application/json' header response = requests.post(url=url, json=casefile) response.raise_for_status() return response.json() # Solve the case file and inspect the solution's components solution = solve_casefile(base_url=base_url, casefile=casefile) solution.keys()

Out[6]:

dict_keys(['CaseSolution', 'PeriodSolution', 'RegionSolution', 'TraderSolution', 'InterconnectorSolution', 'ConstraintSolution', 'ObjectiveBreakdown', 'SolverInfo'])

The solution contains the following components. The first six follow the structure of the NemSpdOutputs section of a NEMDE case file.

Key	Description
CaseSolution	Total constraint violations for the case
PeriodSolution	Objective function value and total constraint violations for the dispatch interval
RegionSolution	Prices and dispatch results for each region
TraderSolution	Energy and FCAS targets for each trader (generator / load)
InterconnectorSolution	Flows and losses for each interconnector
ConstraintSolution	Right-hand side value and deficit for each generic constraint
ObjectiveBreakdown	The objective function value broken down by component
SolverInfo	Solver status and solve times for each pass of the solution algorithm

It's good practice to check the model solved to optimality before examining results.

In [7]:

solution.get('SolverInfo')

solution.get('SolverInfo')

Out[7]:

{'first_pass': {'termination_condition': 'optimal',
  'solver_status': 'ok',
  'solve_seconds': 2.087},
 'second_pass': {'termination_condition': 'optimal',
  'solver_status': 'ok',
  'solve_seconds': 2.137},
 'fast_start_units_starting': [],
 'solve_seconds': 4.224,
 'termination_condition': 'optimal'}

Examining results¶

Let's use Pandas to examine the output.

In [8]:

region_solution = solution.get('RegionSolution')

# Convert to markdown to display results
region_solution_md = pd.DataFrame(region_solution).to_markdown(index=False)
display.Markdown(region_solution_md)

region_solution = solution.get('RegionSolution') # Convert to markdown to display results region_solution_md = pd.DataFrame(region_solution).to_markdown(index=False) display.Markdown(region_solution_md)

Out[8]:

@RegionID	@CaseID	@Intervention	@EnergyPrice	@DispatchedGeneration	@DispatchedLoad	@FixedDemand	@NetExport	@SurplusGeneration	@R1Dispatch	@R6Dispatch	@R60Dispatch	@R5Dispatch	@R5RegDispatch	@L1Dispatch	@L6Dispatch	@L60Dispatch	@L5Dispatch	@L5RegDispatch	@ClearedDemand
NSW1	20210401001	0	38.2172	5127.54	200	6330.79	-1403.26	0	0	261	251	141	88	0	91	142	67	33	6576.14
QLD1	20210401001	0	32.25	6278.72	0	5322.13	956.591	0	0	73.8294	53	107.915	23.2	0	0	0	0	59.875	5362.13
SA1	20210401001	0	37.2465	810.756	10	1065.3	-264.539	0	0	149	107	121	54.8	0	137	69	86	57.125	1077.47
TAS1	20210401001	0	31.7702	1224.95	0	991.98	232.971	0	0	33	82.2457	0	49	0	56.166	152.495	50	50	996.513
VIC1	20210401001	0	34.4678	4729.57	0	4134.2	595.377	0	0	100	123.584	98	5	0	25	31	52.5805	10	4159.28

In [9]:

interconnector_solution = solution.get('InterconnectorSolution')

# Convert to markdown to display results
interconnector_solution_md = pd.DataFrame(interconnector_solution).to_markdown(index=False)
display.Markdown(interconnector_solution_md)

interconnector_solution = solution.get('InterconnectorSolution') # Convert to markdown to display results interconnector_solution_md = pd.DataFrame(interconnector_solution).to_markdown(index=False) display.Markdown(interconnector_solution_md)

Out[9]:

@InterconnectorID	@CaseID	@Intervention	@Flow	@Losses	@Deficit
N-Q-MNSP1	20210401001	0	-97	5.32865	0
NSW1-QLD1	20210401001	0	-819.585	65.3249	0
T-V-MNSP1	20210401001	0	228.438	4.53231	0
V-S-MNSP1	20210401001	0	63	0.102114	0
V-SA	20210401001	0	203.717	8.27106	0
VIC1-NSW1	20210401001	0	532.012	24.9029	0

In [10]:

trader_solution = solution.get('TraderSolution')

# Convert to markdown to display results
trader_solution_md = pd.DataFrame(trader_solution).head().to_markdown(index=False)
display.Markdown(trader_solution_md)

trader_solution = solution.get('TraderSolution') # Convert to markdown to display results trader_solution_md = pd.DataFrame(trader_solution).head().to_markdown(index=False) display.Markdown(trader_solution_md)

Out[10]:

@TraderID	@CaseID	@Intervention	@EnergyTarget	@R1Target	@R6Target	@R60Target	@R5Target	@R5RegTarget	@L1Target	@L6Target	@L60Target	@L5Target	@L5RegTarget	@R6Violation	@R60Violation	@R5Violation	@R5RegViolation	@L6Violation	@L60Violation	@L5Violation	@L5RegViolation	@RampUpRate	@RampDnRate	@FSTargetMode
AGLHAL	20210401001	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	720	720	0
AGLSOM	20210401001	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	480	480	0
ANGAST1	20210401001	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	840	840	nan
APD01	20210401001	0	0	0	31	67.5837	45	0	0	0	0	0	0	0	0	0	0	0	0	0	0	nan	nan	nan
ARWF1	20210401001	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1200	600	nan

Summary¶

This tutorial demonstrates the basic functionality of the Dispatch API. Two key components have been introduced: the ability to interact with historical case files, and the /solve endpoint which returns the model's solution for a given case file. Future tutorials will discuss how to modify case files, perform scenario analyses, and also introduce more advanced workflows using additional Dispatch API features.