{ "cells": [ { "cell_type": "markdown", "id": "e6341b89", "metadata": {}, "source": [ "# Viable Pathway API: transitioning from v1 to v2\n", "\n", "This notebook runs the same payload through Calculation API **v1** (`POST /api/calculate`, frozen methodology) and **v2** (`POST /api/v2/calculate`, current methodology), then compares the responses programmatically: response keys, nested shapes, totals and pathways.\n", "\n", "It also covers the optional v2 features: pinning an External Trends pack, migrating your payload to the v2 shape, and sending several activity rows that share one catalogue activity.\n", "\n", "Partner guide: [https://viablepathway.net/Resources/api-access/v1-to-v2/](https://viablepathway.net/Resources/api-access/v1-to-v2/)\n", "\n", "Run the cells from top to bottom. If a call fails, the cell prints the HTTP status and message, and later cells skip the parts that depend on it." ] }, { "cell_type": "markdown", "id": "22d18718", "metadata": {}, "source": [ "## 1. Setup\n", "\n", "Viable Pathway inserts your API key into `API_KEY` below before sharing this notebook. **Keep the key private:** don't commit it to a repository, paste it into public forums, or share a notebook that contains it.\n", "\n", "All calculation calls in this notebook use `?savePayload=false`, so your payload isn't stored." ] }, { "cell_type": "code", "execution_count": null, "id": "a4dbb749", "metadata": {}, "outputs": [], "source": [ "%pip install --quiet requests matplotlib" ] }, { "cell_type": "code", "execution_count": null, "id": "b1c598dd", "metadata": {}, "outputs": [], "source": [ "import copy\n", "import json\n", "import os\n", "\n", "import matplotlib.pyplot as plt\n", "import requests\n", "\n", "HOST = \"https://app.viablepathway.net\"\n", "API_KEY = \"PASTE_YOUR_API_KEY\"\n", "\n", "# Optional environment overrides (used for local testing). Edit the two lines above instead.\n", "HOST = os.environ.get(\"VIABLE_API_HOST\", HOST).rstrip(\"/\")\n", "API_KEY = os.environ.get(\"VIABLE_API_KEY\", API_KEY)\n", "\n", "CALCULATE_STYLE_SUFFIXES = (\"/calculate\", \"/scenarioanalysis\", \"/methodology\")\n", "TIMEOUT_SECONDS = 180\n", "\n", "\n", "def call_api(path, data=None, method=\"POST\", params=None):\n", " \"\"\"Call the Viable Pathway API and return (status_code, json_body).\n", "\n", " status_code is None if the request never reached the API (network error, timeout).\n", " savePayload=false is always added for calculate-style endpoints.\n", " \"\"\"\n", " params = dict(params or {})\n", " if path.rstrip(\"/\").endswith(CALCULATE_STYLE_SUFFIXES):\n", " params[\"savePayload\"] = \"false\"\n", " headers = {\"Content-Type\": \"application/json\", \"X-API-Key\": API_KEY}\n", " try:\n", " response = requests.request(\n", " method,\n", " HOST + path,\n", " headers=headers,\n", " params=params,\n", " json=None if method == \"GET\" else (data if data is not None else {}),\n", " timeout=TIMEOUT_SECONDS,\n", " )\n", " except requests.RequestException as error:\n", " print(f\"{method} {path}: request failed ({type(error).__name__}: {error})\")\n", " return None, None\n", " try:\n", " body = response.json()\n", " except ValueError:\n", " body = {\"raw\": response.text[:500]}\n", " if not response.ok:\n", " if isinstance(body, dict):\n", " message = body.get(\"message\") or body.get(\"error\") or body\n", " else:\n", " message = body\n", " print(f\"{method} {path} -> HTTP {response.status_code}: {message}\")\n", " return response.status_code, body\n", "\n", "\n", "def succeeded(status, body):\n", " return status == 200 and isinstance(body, dict)\n", "\n", "\n", "if API_KEY == \"PASTE_YOUR_API_KEY\":\n", " print(\"API_KEY is still the placeholder: calls will return 401 until you paste your key.\")\n", "print(\"HOST:\", HOST)" ] }, { "cell_type": "code", "execution_count": null, "id": "34ab1c6d", "metadata": {}, "outputs": [], "source": [ "# Connectivity check: list the published External Trends packs (used again in section 7).\n", "status, trend_versions = call_api(\"/api/data/externalTrendsVersions\", method=\"GET\")\n", "if succeeded(status, trend_versions):\n", " print(\"Latest pack:\", trend_versions.get(\"latestId\"))\n", " for version in trend_versions.get(\"versions\", []):\n", " print(f\" {version.get('id'):<10} {version.get('status', ''):<10} {version.get('summary', '')}\")" ] }, { "cell_type": "markdown", "id": "50601d28", "metadata": {}, "source": [ "## 2. A small v1 payload\n", "\n", "A fictional organisation with one org unit, five activities in the v1 **nested** `activitydata` shape (`category -> name@@region -> leaf`), and two internal actions. The fleet action has a **balancing activity**: diesel car km removed by the action are replaced by battery electric car km.\n", "\n", "Organisation and projection settings sit at the root of the payload, as in v1. Nothing in this payload is v2-specific." ] }, { "cell_type": "code", "execution_count": null, "id": "01843077", "metadata": {}, "outputs": [], "source": [ "V1_PAYLOAD = {\n", " \"organisationid\": \"org_riverbend_bakeries_demo\",\n", " \"name\": \"Riverbend Bakeries (fictional)\",\n", " \"hqcountry\": \"australia\",\n", " \"industrytype\": \"c – manufacturing: 11 – food product manufacturing\",\n", " \"currency\": \"aud\",\n", " \"revenue\": 30,\n", " \"grossassets\": 20,\n", " \"employees\": 90,\n", " \"desiredgrowth\": 3,\n", " \"projectionstartyear\": 2025,\n", " \"projectionendyear\": 2050,\n", " \"electricityapproach\": \"location\",\n", " \"neartermsbtitarget\": None,\n", " \"neartermsbtitargetyear\": None,\n", " \"userdefinedtargetbaseyear\": 2024,\n", " \"userdefinedtarget\": {\n", " \"targetKind\": \"absolute_per_scope\",\n", " \"neartermyear\": 2030,\n", " \"neartermscope1\": -30,\n", " \"neartermscope2\": -40,\n", " \"neartermscope3\": -20,\n", " \"netzeroyear\": 2050,\n", " \"netzeroscope1\": -90,\n", " \"netzeroscope2\": -90,\n", " \"netzeroscope3\": -70,\n", " },\n", " \"orgunits\": {\n", " \"orgunit_bakery\": {\n", " \"id\": \"orgunit_bakery\",\n", " \"name\": \"Parramatta Bakery\",\n", " \"country\": \"australia\",\n", " \"region\": \"au-nsw\",\n", " \"currency\": \"aud\",\n", " \"historicemissionsunit\": \"tCO2e\",\n", " \"historicemissionsyears\": [2022, 2023, 2024],\n", " \"historicemissions\": {\n", " \"1\": [85, 83, 82],\n", " \"2\": {\"location\": [222, 218, 214], \"market\": [300, 296, 294]},\n", " \"3\": [30, 29, 29],\n", " },\n", " \"activitydata\": {\n", " \"companyvehicle\": {\n", " \"car_diesel@@au-nsw\": {\n", " \"catalogactivityid\": \"companyvehicle_car_diesel\",\n", " \"emissionfactorname\": \"companyvehicle_car_diesel\",\n", " \"region\": \"au-nsw\",\n", " \"activitydatavalue\": 180000,\n", " \"activityuom\": \"km\",\n", " \"ghgpscope\": 1,\n", " \"ghgpscopecategory\": \"1-2 Direct Emissions from Mobile Combustion\",\n", " \"emissionfactorvalue\": 0.2113,\n", " \"emissionfactorunit\": \"kgCO2e\",\n", " \"emissionfactorsource\": \"desnz-uk\",\n", " \"operationalcostperunit\": 0.3,\n", " \"description\": \"Diesel delivery cars\",\n", " },\n", " \"car_bev@@au-nsw\": {\n", " \"catalogactivityid\": \"companyvehicle_car_bev\",\n", " \"emissionfactorname\": \"companyvehicle_car_bev\",\n", " \"region\": \"au-nsw\",\n", " \"activitydatavalue\": 0,\n", " \"activityuom\": \"km\",\n", " \"ghgpscope\": 2,\n", " \"ghgpscopecategory\": \"2-1 Indirect Emissions from Purchased Electricity\",\n", " \"emissionfactorvalue\": 0.05794,\n", " \"emissionfactorunit\": \"kgCO2e\",\n", " \"emissionfactorsource\": \"desnz-uk\",\n", " \"operationalcostperunit\": 0.1,\n", " \"description\": \"Battery electric delivery cars\",\n", " },\n", " },\n", " \"utilities\": {\n", " \"electricity_grid@@au-nsw\": {\n", " \"catalogactivityid\": \"utilities_electricity_grid\",\n", " \"emissionfactorname\": \"utilities_electricity_grid\",\n", " \"region\": \"au-nsw\",\n", " \"activitydatavalue\": 320000,\n", " \"activityuom\": \"kWh\",\n", " \"ghgpscope\": 2,\n", " \"ghgpscopecategory\": \"2-1 Indirect Emissions from Purchased Electricity\",\n", " \"emissionfactorvalue\": 0.67,\n", " \"emissionfactorvalue_location\": 0.67,\n", " \"emissionfactorvalue_market\": 0.92,\n", " \"emissionfactorunit\": \"kgCO2e\",\n", " \"emissionfactorsource\": \"dcceew-aus\",\n", " \"operationalcostperunit\": 0.3,\n", " \"description\": \"Bakery grid electricity (ovens, refrigeration)\",\n", " },\n", " },\n", " \"FuelStationary\": {\n", " \"EnergyHeat_Dieseloil@@au-nsw\": {\n", " \"catalogactivityid\": \"FuelStationary_EnergyHeat_Dieseloil\",\n", " \"emissionfactorname\": \"fuelliquid_stationary_dieseloil\",\n", " \"region\": \"au-nsw\",\n", " \"activitydatavalue\": 500,\n", " \"activityuom\": \"GJ\",\n", " \"ghgpscope\": 1,\n", " \"ghgpscopecategory\": \"1-1 Direct Emissions from Fuels (Stationary Combustion)\",\n", " \"emissionfactorvalue\": 87.5,\n", " \"emissionfactorunit\": \"kgCO2e\",\n", " \"emissionfactorsource\": \"dcceew-aus\",\n", " \"operationalcostperunit\": 75,\n", " \"description\": \"Backup diesel generator\",\n", " },\n", " },\n", " \"commuting\": {\n", " \"car_petrol@@au-nsw\": {\n", " \"catalogactivityid\": \"commuting_car_petrol\",\n", " \"emissionfactorname\": \"commuting_car_petrol\",\n", " \"region\": \"au-nsw\",\n", " \"activitydatavalue\": 150000,\n", " \"activityuom\": \"km\",\n", " \"ghgpscope\": 3,\n", " \"ghgpscopecategory\": \"3-7 Employee Commuting\",\n", " \"emissionfactorvalue\": 0.191309503,\n", " \"emissionfactorunit\": \"kgCO2e\",\n", " \"emissionfactorsource\": \"epa-usa\",\n", " \"description\": \"Staff commuting by petrol car\",\n", " },\n", " },\n", " },\n", " \"internalactions\": {\n", " \"action_fleet_ev\": {\n", " \"actionname\": \"Switch delivery cars to EVs\",\n", " \"actiontype\": \"Company Vehicles\",\n", " \"activity\": \"companyvehicle_car_diesel@@au-nsw\",\n", " \"multiplier\": -0.5,\n", " \"absolutechange\": 0,\n", " \"balancingactivity\": \"companyvehicle_car_bev@@au-nsw\",\n", " \"balancingactivityuom\": \"km\",\n", " \"balancingmultiplier\": 1,\n", " \"balancingabsolutechange\": 0,\n", " \"capitalcostcalculationmethod\": \"absolute\",\n", " \"capitalcost\": 90000,\n", " \"capitalcostperunit\": None,\n", " \"startyear\": 2027,\n", " \"description\": \"Replace 50% of diesel car km with battery electric cars\",\n", " \"source\": \"\",\n", " },\n", " \"action_led\": {\n", " \"actionname\": \"LED lighting and oven controls\",\n", " \"actiontype\": \"Energy Efficiency\",\n", " \"activity\": \"utilities_electricity_grid@@au-nsw\",\n", " \"multiplier\": -0.2,\n", " \"absolutechange\": 0,\n", " \"balancingactivity\": \"\",\n", " \"balancingactivityuom\": \"\",\n", " \"balancingmultiplier\": 0,\n", " \"balancingabsolutechange\": 0,\n", " \"capitalcostcalculationmethod\": \"absolute\",\n", " \"capitalcost\": 30000,\n", " \"capitalcostperunit\": None,\n", " \"startyear\": 2026,\n", " \"description\": \"Cut bakery electricity use by 20%\",\n", " \"source\": \"\",\n", " },\n", " },\n", " }\n", " },\n", "}\n", "\n", "for unit_id, unit in V1_PAYLOAD[\"orgunits\"].items():\n", " leaves = [f\"{category}.{name}\" for category, rows in unit[\"activitydata\"].items() for name in rows]\n", " print(f\"{unit_id}: {len(leaves)} activities, {len(unit['internalactions'])} internal actions\")\n", " for leaf in leaves:\n", " print(\" \", leaf)" ] }, { "cell_type": "markdown", "id": "702f3933", "metadata": {}, "source": [ "## 3. Call v1 (standard and extended)\n", "\n", "`?extended=true` returns the full detailed results (per-activity projections, MAC curve slices, trend contributions). Without it you get the trimmed standard payload." ] }, { "cell_type": "code", "execution_count": null, "id": "bb384d8b", "metadata": {}, "outputs": [], "source": [ "ENDPOINTS = {\"v1\": \"/api/calculate\", \"v2\": \"/api/v2/calculate\"}\n", "\n", "\n", "def calculate(version, payload, extended=False, label=None):\n", " \"\"\"POST a payload to v1 or v2. Returns the response body on HTTP 200, otherwise None.\"\"\"\n", " params = {\"extended\": \"true\"} if extended else None\n", " status, body = call_api(ENDPOINTS[version], payload, params=params)\n", " name = label or f\"{version} {'extended' if extended else 'standard'}\"\n", " print(f\"{name:<28} HTTP {status}\")\n", " return body if succeeded(status, body) else None\n", "\n", "\n", "v1_standard = calculate(\"v1\", V1_PAYLOAD)\n", "v1_extended = calculate(\"v1\", V1_PAYLOAD, extended=True)" ] }, { "cell_type": "markdown", "id": "220c231d", "metadata": {}, "source": [ "## 4. Call v2 with the same payload\n", "\n", "This is the **essential** change: only the URL changes, from `/api/calculate` to `/api/v2/calculate`. Same API key, same query parameters, same request body." ] }, { "cell_type": "code", "execution_count": null, "id": "94d00d1d", "metadata": {}, "outputs": [], "source": [ "v2_standard = calculate(\"v2\", V1_PAYLOAD)\n", "v2_extended = calculate(\"v2\", V1_PAYLOAD, extended=True)" ] }, { "cell_type": "markdown", "id": "13dc8e9a", "metadata": {}, "source": [ "## 5. What changed in the response shape?\n", "\n", "The helpers below compare keys and value types, so the output reflects whatever the API returns today rather than a hard-coded list.\n", "\n", "Nested items are matched by `actionid` (v1 and v2 may order internal actions differently)." ] }, { "cell_type": "code", "execution_count": null, "id": "4d1636d6", "metadata": {}, "outputs": [], "source": [ "def type_name(value):\n", " if value is None:\n", " return \"null\"\n", " if isinstance(value, bool):\n", " return \"boolean\"\n", " if isinstance(value, (int, float)):\n", " return \"number\"\n", " if isinstance(value, str):\n", " return \"string\"\n", " if isinstance(value, list):\n", " return \"array\"\n", " return \"object\"\n", "\n", "\n", "def shape(obj):\n", " return {key: type_name(value) for key, value in obj.items()} if isinstance(obj, dict) else {}\n", "\n", "\n", "def key_diff(left, right):\n", " left_keys, right_keys = set(left or []), set(right or [])\n", " return sorted(left_keys - right_keys), sorted(right_keys - left_keys)\n", "\n", "\n", "def print_shape_diff(label, v1_obj, v2_obj):\n", " v1_shape, v2_shape = shape(v1_obj), shape(v2_obj)\n", " print(f\"\\n{label}\")\n", " if not v1_shape and not v2_shape:\n", " print(\" (not present in either response)\")\n", " return\n", " if not v1_shape or not v2_shape:\n", " print(f\" present in v1: {bool(v1_shape)}, present in v2: {bool(v2_shape)}\")\n", " only_v1, only_v2 = key_diff(v1_shape, v2_shape)\n", " shared = sorted(set(v1_shape) & set(v2_shape))\n", " changed = {key: (v1_shape[key], v2_shape[key]) for key in shared if v1_shape[key] != v2_shape[key]}\n", " print(f\" only in v1 ({len(only_v1)}): {only_v1 or '-'}\")\n", " print(f\" only in v2 ({len(only_v2)}): {only_v2 or '-'}\")\n", " print(f\" shared keys: {len(shared)}\")\n", " if changed:\n", " print(f\" type changed (v1, v2): {changed}\")\n", "\n", "\n", "def find_action(items, action_id):\n", " for item in items or []:\n", " if isinstance(item, dict) and item.get(\"actionid\") == action_id:\n", " return item\n", " return None\n", "\n", "\n", "def first_action_id(body):\n", " for item in (body or {}).get(\"internalActionsContributions\") or []:\n", " if isinstance(item, dict) and item.get(\"actionid\"):\n", " return item[\"actionid\"]\n", " return None\n", "\n", "\n", "def macc_slice(body, endpoint=None):\n", " \"\"\"Return (endpoint_year, items) for a maccByEndpoint slice (first endpoint by default).\"\"\"\n", " macc = (body or {}).get(\"maccByEndpoint\")\n", " if not isinstance(macc, dict) or not macc:\n", " return None, []\n", " endpoint = endpoint if endpoint in macc else sorted(macc)[0]\n", " items = macc[endpoint]\n", " return endpoint, items if isinstance(items, list) else []\n", "\n", "\n", "def first_orgunit(body):\n", " units = (body or {}).get(\"orgunitResults\")\n", " return units[0] if isinstance(units, list) and units else None\n", "\n", "\n", "for mode, v1_body, v2_body in ((\"standard\", v1_standard, v2_standard), (\"extended\", v1_extended, v2_extended)):\n", " print(\"=\" * 80)\n", " print(f\"{mode.upper()} response\")\n", " if not (v1_body and v2_body):\n", " print(\" skipped: one of the calls failed\")\n", " continue\n", " print_shape_diff(\"Top-level keys\", v1_body, v2_body)\n", "\n", " action_id = first_action_id(v1_body)\n", " print_shape_diff(\n", " f\"internalActionsContributions item (actionid={action_id})\",\n", " find_action(v1_body.get(\"internalActionsContributions\"), action_id),\n", " find_action(v2_body.get(\"internalActionsContributions\"), action_id),\n", " )\n", "\n", " endpoint, v1_items = macc_slice(v1_body)\n", " _, v2_items = macc_slice(v2_body, endpoint)\n", " if endpoint is None and macc_slice(v2_body)[0] is None:\n", " print(\"\\nmaccByEndpoint\\n (not in this response; extended results only)\")\n", " else:\n", " macc_action = (v1_items[0] or {}).get(\"actionid\") if v1_items else None\n", " print_shape_diff(\n", " f\"maccByEndpoint['{endpoint}'] item (actionid={macc_action})\",\n", " find_action(v1_items, macc_action),\n", " find_action(v2_items, macc_action),\n", " )\n", "\n", " print_shape_diff(\"orgunitResults[0]\", first_orgunit(v1_body), first_orgunit(v2_body))" ] }, { "cell_type": "code", "execution_count": null, "id": "7848a2df", "metadata": {}, "outputs": [], "source": [ "# New v2 metadata: which External Trends pack was used, and any warnings.\n", "for label, body in ((\"v2 standard\", v2_standard), (\"v2 extended\", v2_extended)):\n", " if body:\n", " print(f\"{label}:\")\n", " print(\" externalTrendsPack:\", json.dumps(body.get(\"externalTrendsPack\"), indent=2).replace(\"\\n\", \"\\n \"))\n", " print(\" warnings:\", body.get(\"warnings\"))\n", "for label, body in ((\"v1 standard\", v1_standard), (\"v1 extended\", v1_extended)):\n", " if body:\n", " print(f\"{label}: has externalTrendsPack={'externalTrendsPack' in body}, has warnings={'warnings' in body}\")" ] }, { "cell_type": "markdown", "id": "2637c4a6", "metadata": {}, "source": [ "## 6. Compare totals and pathways\n", "\n", "`ghgBAUsumperyear`, `ghgEXTsumperyear` and `ghgINTsumperyear` are arrays aligned with `projectionyears`, so a year's value is read by its index in `projectionyears`.\n", "\n", "- **BAU**: business as usual (organisational growth only).\n", "- **EXT**: BAU plus external trends (grid decarbonisation, EV uptake and similar market changes).\n", "- **INT**: EXT plus your internal actions." ] }, { "cell_type": "code", "execution_count": null, "id": "a0698710", "metadata": {}, "outputs": [], "source": [ "PATHWAYS = {\"BAU\": \"ghgBAUsumperyear\", \"EXT\": \"ghgEXTsumperyear\", \"INT\": \"ghgINTsumperyear\"}\n", "COMPARE_YEARS = [2030, 2040, 2050]\n", "\n", "\n", "def series(body, field):\n", " \"\"\"Return (years, values) for a per-year series aligned with projectionyears.\"\"\"\n", " years = (body or {}).get(\"projectionyears\") or []\n", " values = (body or {}).get(field)\n", " if isinstance(values, dict):\n", " return years, [values.get(str(year), values.get(year)) for year in years]\n", " if isinstance(values, list) and len(values) == len(years):\n", " return years, values\n", " return years, []\n", "\n", "\n", "def value_at(body, field, year):\n", " years, values = series(body, field)\n", " return values[years.index(year)] if values and year in years else None\n", "\n", "\n", "def print_totals(left, right, left_label=\"v1\", right_label=\"v2\", years=COMPARE_YEARS):\n", " print(f\"{'Pathway':<8}{'Year':>6}{left_label:>14}{right_label:>14}{'Change':>12}{'Change %':>10} (tCO2e)\")\n", " for name, field in PATHWAYS.items():\n", " for year in years:\n", " a, b = value_at(left, field, year), value_at(right, field, year)\n", " if isinstance(a, (int, float)) and isinstance(b, (int, float)):\n", " pct = f\"{(b - a) / a * 100:+.1f}%\" if a else \"n/a\"\n", " print(f\"{name:<8}{year:>6}{a:>14.1f}{b:>14.1f}{b - a:>+12.1f}{pct:>10}\")\n", " else:\n", " print(f\"{name:<8}{year:>6}{str(a):>14}{str(b):>14}\")\n", "\n", "\n", "if v1_standard and v2_standard:\n", " print_totals(v1_standard, v2_standard)\n", "else:\n", " print(\"Skipped: need both v1 and v2 standard responses.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "55254930", "metadata": {}, "outputs": [], "source": [ "if v1_standard and v2_standard:\n", " fig, ax = plt.subplots(figsize=(10, 5))\n", " colours = {\"BAU\": \"grey\", \"EXT\": \"tab:orange\", \"INT\": \"tab:green\"}\n", " for name in (\"BAU\", \"EXT\", \"INT\"):\n", " for label, body, style in ((\"v1\", v1_standard, \"--\"), (\"v2\", v2_standard, \"-\")):\n", " years, values = series(body, PATHWAYS[name])\n", " if values:\n", " ax.plot(years, values, linestyle=style, color=colours[name], label=f\"{name} {label}\",\n", " alpha=0.5 if name == \"BAU\" else 1)\n", " ax.set_title(\"Emissions pathways: v1 (dashed) vs v2 (solid)\")\n", " ax.set_xlabel(\"Year\")\n", " ax.set_ylabel(\"tCO2e\")\n", " ax.grid(alpha=0.3)\n", " ax.legend(ncol=3)\n", " plt.show()\n", "else:\n", " print(\"Skipped: need both v1 and v2 standard responses.\")" ] }, { "cell_type": "markdown", "id": "cf2edca5", "metadata": {}, "source": [ "### Why the numbers change\n", "\n", "v1 is a frozen snapshot of the methodology and reference data at the time v2 launched. v2 runs the current engine, so the same payload can give different emissions, MAC and pathway values. Typical reasons:\n", "\n", "- **Newer External Trends packs.** v2 resolves a versioned pack (see `externalTrendsPack`), which can change trend rates or which activities a trend applies to.\n", "- **Current reference data** for emission factors, costs and internal-action templates.\n", "- **Methodology fixes**, for example in how balancing activities grow over time, and `absoluteChangeGrows` defaulting to `true` for absolute-change actions (fixed absolute changes scale with organisational growth unless you set it to `false` on the action).\n", "\n", "**Re-baseline:** store a fresh v2 baseline for each organisation rather than diffing v2 results against historic v1 results. The BAU pathway often stays the same (it only depends on your activity data and growth), while EXT and INT move with the trends and methodology." ] }, { "cell_type": "markdown", "id": "c5e933c0", "metadata": {}, "source": [ "## 7. Pin the External Trends pack (recommended)\n", "\n", "Without `meta.externalTrendsVersion`, v2 uses the latest pack and adds a warning. Pin the pack id so your results don't move when Viable Pathway publishes a new pack. Here we take the id from the v2 response above.\n", "\n", "You can add `meta` to a v1-shaped payload: nested `meta` values win over flat root keys, and anything not in `meta` is still read from the root." ] }, { "cell_type": "code", "execution_count": null, "id": "a8cf229a", "metadata": {}, "outputs": [], "source": [ "pack_id = ((v2_standard or {}).get(\"externalTrendsPack\") or {}).get(\"id\")\n", "pinned_standard = None\n", "if pack_id:\n", " pinned_payload = copy.deepcopy(V1_PAYLOAD)\n", " pinned_payload[\"meta\"] = {\"externalTrendsVersion\": pack_id}\n", " pinned_standard = calculate(\"v2\", pinned_payload, label=f\"v2 pinned to {pack_id}\")\n", " if pinned_standard:\n", " print(\"externalTrendsPack.id:\", (pinned_standard.get(\"externalTrendsPack\") or {}).get(\"id\"))\n", " print(\"warnings before pinning:\", v2_standard.get(\"warnings\"))\n", " print(\"warnings after pinning: \", pinned_standard.get(\"warnings\", \"(no warnings key)\"))\n", " same = all(series(pinned_standard, f)[1] == series(v2_standard, f)[1] for f in PATHWAYS.values())\n", " print(\"Totals identical to the unpinned call:\", same,\n", " \"(expected while the pinned pack is still the latest)\" if same else \"\")\n", "else:\n", " print(\"Skipped: the v2 response had no externalTrendsPack id.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "23563bae", "metadata": {}, "outputs": [], "source": [ "# An unknown pack id is rejected with 400 and the list of valid ids.\n", "unknown_payload = copy.deepcopy(V1_PAYLOAD)\n", "unknown_payload[\"meta\"] = {\"externalTrendsVersion\": \"1999Q1\"}\n", "status, body = call_api(ENDPOINTS[\"v2\"], unknown_payload)\n", "print(\"HTTP\", status)\n", "if isinstance(body, dict):\n", " print(json.dumps({key: body.get(key) for key in (\"error\", \"latestId\", \"availableIds\", \"path\") if key in body}, indent=2))" ] }, { "cell_type": "markdown", "id": "da24a622", "metadata": {}, "source": [ "## 8. Optional: migrate the payload to the v2 shape\n", "\n", "v2 accepts your v1 payload as is. If you want to adopt the shape the Viable Pathway web app uses, `POST /api/migrate-payload` converts it:\n", "\n", "- organisation and projection settings move into a root `meta` object, targets into `target`;\n", "- `datastructureversion: 2`;\n", "- `activitydata` becomes a flat map keyed by **activity instance id**, each row with a `catalogactivityid`, and `emissionfactorid` replaces `emissionfactorname`;\n", "- internal actions reference instance ids.\n", "\n", "Query options: `?activitydata=false` or `?metatarget=false` skip either part. Don't send the v2 shape to the v1 endpoint." ] }, { "cell_type": "code", "execution_count": null, "id": "ebc9a1cc", "metadata": {}, "outputs": [], "source": [ "status, migrated = call_api(\"/api/migrate-payload\", V1_PAYLOAD)\n", "if not succeeded(status, migrated):\n", " migrated = None\n", " print(\"Migration failed; sections 8 and 9 will be skipped.\")\n", "else:\n", " print(\"Top-level keys:\", sorted(migrated))\n", " print(\"datastructureversion:\", migrated.get(\"datastructureversion\"))\n", " print(\"meta keys: \", sorted((migrated.get(\"meta\") or {}).keys()))\n", " print(\"target keys:\", sorted((migrated.get(\"target\") or {}).keys()))\n", " moved = sorted(key for key in V1_PAYLOAD if key not in migrated)\n", " print(\"Root keys moved out of the root:\", moved)\n", " for unit_id, unit in (migrated.get(\"orgunits\") or {}).items():\n", " print(f\"\\n{unit_id} activitydata (flat, keyed by instance id):\")\n", " for instance_id, row in (unit.get(\"activitydata\") or {}).items():\n", " print(f\" {instance_id:<46} catalogactivityid={row.get('catalogactivityid')!s:<38} emissionfactorid={row.get('emissionfactorid')}\")\n", " print(f\"{unit_id} internal action references:\")\n", " for action_id, action in (unit.get(\"internalactions\") or {}).items():\n", " print(f\" {action_id:<18} activity={action.get('activity')!r} balancingactivity={action.get('balancingactivity')!r}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "3d6fc3e9", "metadata": {}, "outputs": [], "source": [ "# Field-level differences between a v1 leaf and the migrated v2 row (matched by catalogactivityid).\n", "if migrated:\n", " v1_leaves = {\n", " leaf.get(\"catalogactivityid\"): leaf\n", " for unit in V1_PAYLOAD[\"orgunits\"].values()\n", " for rows in unit[\"activitydata\"].values()\n", " for leaf in rows.values()\n", " }\n", " removed, added = set(), set()\n", " for unit in migrated[\"orgunits\"].values():\n", " for row in unit[\"activitydata\"].values():\n", " leaf = v1_leaves.get(row.get(\"catalogactivityid\"))\n", " if leaf:\n", " only_v1, only_v2 = key_diff(leaf, row)\n", " removed.update(only_v1)\n", " added.update(only_v2)\n", " print(\"Activity fields only in v1 rows:\", sorted(removed) or \"-\")\n", " print(\"Activity fields only in v2 rows:\", sorted(added) or \"-\")" ] }, { "cell_type": "code", "execution_count": null, "id": "e075225d", "metadata": {}, "outputs": [], "source": [ "migrated_standard = migrated_extended = None\n", "if migrated:\n", " migrated_standard = calculate(\"v2\", migrated, label=\"v2 standard (migrated)\")\n", " migrated_extended = calculate(\"v2\", migrated, extended=True, label=\"v2 extended (migrated)\")\n", "\n", "\n", "def max_abs_difference(left, right):\n", " worst = 0.0\n", " for field in PATHWAYS.values():\n", " a, b = series(left, field)[1], series(right, field)[1]\n", " if len(a) != len(b):\n", " return None\n", " worst = max([worst] + [abs(x - y) for x, y in zip(a, b) if isinstance(x, (int, float)) and isinstance(y, (int, float))])\n", " return worst\n", "\n", "\n", "if migrated_standard and v2_standard:\n", " diff = max_abs_difference(v2_standard, migrated_standard)\n", " print(f\"\\nLargest BAU/EXT/INT difference vs the un-migrated v2 call, any year: {diff} tCO2e\")\n", " if diff is not None and diff < 1e-6:\n", " print(\"Same results: migration only changes the payload shape.\")\n", " else:\n", " print(\"Results differ; compare the tables below.\")\n", " print_totals(v2_standard, migrated_standard, \"v2 (v1 shape)\", \"v2 migrated\")" ] }, { "cell_type": "markdown", "id": "2a2e0d69", "metadata": {}, "source": [ "The two calls should give the same totals: v2 applies the same External Trends to nested v1 `activitydata` and to the migrated flat rows, including the balancing activities a trend auto-inserts (for example the alternatives a commuting trend shifts petrol-car km into). The next cell lists the applied trends and auto-inserted rows for both calls, and any EXT emissions that differ.\n", "\n", "Auto-inserted rows get a new `act_…` id on every call, so the comparison matches rows by emission factor rather than by id." ] }, { "cell_type": "code", "execution_count": null, "id": "94d2f7d7", "metadata": {}, "outputs": [], "source": [ "def auto_inserted(body):\n", " return sorted(\n", " (str(row.get(\"emissionfactorname\")), str(row.get(\"reference\")))\n", " for row in (body or {}).get(\"autoInsertedActivities\") or []\n", " )\n", "\n", "\n", "def applied_trend_ids(body):\n", " unit = first_orgunit(body) or {}\n", " return sorted({str(row.get(\"activitytrendid\")) for row in unit.get(\"externalActivityTrendsContributions\") or []})\n", "\n", "\n", "def ext_emissions_by_factor(body, year):\n", " \"\"\"EXT emissions in `year` per emission factor (stable across calls, unlike auto-inserted act_ ids).\"\"\"\n", " unit = first_orgunit(body) or {}\n", " years = unit.get(\"projectionyears\") or (body or {}).get(\"projectionyears\") or []\n", " if year not in years:\n", " return {}\n", " index, totals = years.index(year), {}\n", " for row in unit.get(\"projectedEXTGHG\") or []:\n", " key = row.get(\"emissionfactorname\") or row.get(\"activity\")\n", " totals[key] = totals.get(key, 0) + (row.get(\"values\") or [0] * len(years))[index]\n", " return totals\n", "\n", "\n", "if v2_extended and migrated_extended:\n", " print(\"Applied activity trend ids: v1 shape\", applied_trend_ids(v2_extended), \"| migrated\", applied_trend_ids(migrated_extended))\n", " only_nested, only_migrated = key_diff(auto_inserted(v2_extended), auto_inserted(migrated_extended))\n", " print(\"Auto-inserted (emission factor, trend id) only with the v1 shape:\", only_nested or \"-\")\n", " print(\"Auto-inserted (emission factor, trend id) only when migrated: \", only_migrated or \"-\")\n", " year = 2040\n", " nested_rows, migrated_rows = ext_emissions_by_factor(v2_extended, year), ext_emissions_by_factor(migrated_extended, year)\n", " print(f\"\\nEXT emissions in {year} by emission factor where the two calls differ (tCO2e):\")\n", " for key in sorted(set(nested_rows) | set(migrated_rows)):\n", " a, b = nested_rows.get(key, 0), migrated_rows.get(key, 0)\n", " if abs(a - b) > 1e-6:\n", " print(f\" {key:<40} v1 shape {a:>9.2f} migrated {b:>9.2f} change {b - a:>+9.2f}\")\n", "else:\n", " print(\"Skipped: needs both v2 extended responses (v1 shape and migrated).\")" ] }, { "cell_type": "markdown", "id": "24e9a6da", "metadata": {}, "source": [ "## 9. Optional: several activity rows for one catalogue activity\n", "\n", "In v1, activities were effectively one row per catalogue activity. In v2 each row has its own **instance id**, so several rows can share a `catalogactivityid`, for example two sites on the same grid. Each row gets its own projections, and an internal action can target one instance.\n", "\n", "Here we add a second electricity row (`act_warehouse_electricity`) and a solar action that only affects that site." ] }, { "cell_type": "code", "execution_count": null, "id": "35c17208", "metadata": {}, "outputs": [], "source": [ "multi_payload = multi_extended = None\n", "if migrated:\n", " multi_payload = copy.deepcopy(migrated)\n", " if pack_id:\n", " multi_payload.setdefault(\"meta\", {})[\"externalTrendsVersion\"] = pack_id\n", " unit = next(iter(multi_payload[\"orgunits\"].values()))\n", " grid_id = next((iid for iid, row in unit[\"activitydata\"].items()\n", " if row.get(\"catalogactivityid\") == \"utilities_electricity_grid\"), None)\n", " if grid_id:\n", " unit[\"activitydata\"][\"act_warehouse_electricity\"] = {\n", " **unit[\"activitydata\"][grid_id],\n", " \"activityname\": \"Warehouse grid electricity\",\n", " \"activitydatavalue\": 120000,\n", " \"description\": \"Second site on the same grid\",\n", " }\n", " unit[\"internalactions\"][\"action_warehouse_solar\"] = {\n", " \"actionname\": \"Warehouse rooftop solar\",\n", " \"actiontype\": \"Renewable Electricity Installation\",\n", " \"activity\": \"act_warehouse_electricity\",\n", " \"multiplier\": -0.3,\n", " \"absolutechange\": 0,\n", " \"balancingactivity\": \"\",\n", " \"balancingactivityuom\": \"\",\n", " \"balancingmultiplier\": 0,\n", " \"balancingabsolutechange\": 0,\n", " \"capitalcostcalculationmethod\": \"absolute\",\n", " \"capitalcost\": 60000,\n", " \"capitalcostperunit\": None,\n", " \"startyear\": 2027,\n", " \"description\": \"Offset 30% of warehouse grid electricity\",\n", " \"source\": \"\",\n", " }\n", " print(f\"Rows sharing catalogactivityid 'utilities_electricity_grid': {grid_id}, act_warehouse_electricity\")\n", " multi_extended = calculate(\"v2\", multi_payload, extended=True, label=\"v2 extended (multi-instance)\")\n", " else:\n", " print(\"Skipped: no utilities_electricity_grid row found in the migrated payload.\")\n", "else:\n", " print(\"Skipped: needs the migrated payload from section 8.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "feb81e66", "metadata": {}, "outputs": [], "source": [ "if multi_extended:\n", " catalogue = {iid: row.get(\"catalogactivityid\")\n", " for unit in multi_payload[\"orgunits\"].values()\n", " for iid, row in unit[\"activitydata\"].items()}\n", " for row in multi_extended.get(\"autoInsertedActivities\") or []:\n", " catalogue.setdefault(row.get(\"activity\"), f\"(auto-inserted by trend {row.get('reference')}: {row.get('emissionfactorname')})\")\n", " unit_result = first_orgunit(multi_extended) or {}\n", "\n", " print(\"projectionstartyearactivitydata (one row per instance; auto-inserted trend rows start at 0):\")\n", " for row in unit_result.get(\"projectionstartyearactivitydata\") or []:\n", " print(f\" {row.get('activity')!s:<46} {catalogue.get(row.get('activity'))!s:<60} {row.get('values')}\")\n", "\n", " print(\"\\nemissions2030INT for the electricity rows:\")\n", " for row in unit_result.get(\"emissions2030INT\") or []:\n", " if catalogue.get(row.get(\"activity\")) == \"utilities_electricity_grid\":\n", " print(f\" {row.get('activity')!s:<46} {row.get('displayName')!s:<30} {row.get('emissions')} tCO2e\")\n", "\n", " print(\"\\nInternal action contributions and the instance they target:\")\n", " for item in multi_extended.get(\"internalActionsContributions\") or []:\n", " print(f\" {item.get('actionid')!s:<24} activity={item.get('activity')!r:<46} netImpact={item.get('netImpact')}\")" ] }, { "cell_type": "markdown", "id": "2d3146e7", "metadata": {}, "source": [ "## 10. Extended results: MAC slices, contribution activity keys and activity metadata\n", "\n", "Only relevant if you call `?extended=true`." ] }, { "cell_type": "code", "execution_count": null, "id": "b403a0d7", "metadata": {}, "outputs": [], "source": [ "# MAC curve slices: fields per endpoint year, v1 vs v2 (matched by actionid).\n", "for label, key in ((\"top level\", None), (\"orgunitResults[0]\", \"orgunit\")):\n", " v1_body = first_orgunit(v1_extended) if key else v1_extended\n", " v2_body = first_orgunit(v2_extended) if key else v2_extended\n", " if not (v1_body and v2_body):\n", " continue\n", " print(f\"maccByEndpoint ({label}) endpoint years: v1={sorted((v1_body.get('maccByEndpoint') or {}).keys())}, \"\n", " f\"v2={sorted((v2_body.get('maccByEndpoint') or {}).keys())}\")\n", " print(f\" maccByEndpointChronological present: v1={'maccByEndpointChronological' in v1_body}, \"\n", " f\"v2={'maccByEndpointChronological' in v2_body}\")\n", "\n", "if v1_extended and v2_extended:\n", " endpoint, v1_items = macc_slice(v1_extended)\n", " _, v2_items = macc_slice(v2_extended, endpoint)\n", " for v1_item in v1_items:\n", " v2_item = find_action(v2_items, v1_item.get(\"actionid\"))\n", " if v2_item:\n", " only_v1, only_v2 = key_diff(v1_item, v2_item)\n", " print(f\"\\nmaccByEndpoint['{endpoint}'] {v1_item.get('actionid')}: fields only in v2: {only_v2 or '-'}; only in v1: {only_v1 or '-'}\")\n", " for field in only_v2:\n", " value = v2_item[field]\n", " print(f\" {field}: {json.dumps(value)[:120]}\")\n", " break" ] }, { "cell_type": "code", "execution_count": null, "id": "6bb9653b", "metadata": {}, "outputs": [], "source": [ "# Which activity key does each contribution use? (catalogue-style ids vs instance ids)\n", "responses = {\n", " \"v1 extended\": v1_extended,\n", " \"v2 extended (v1 shape)\": v2_extended,\n", " \"v2 extended (migrated)\": migrated_extended,\n", " \"v2 extended (multi-instance)\": multi_extended,\n", "}\n", "for label, body in responses.items():\n", " if not body:\n", " continue\n", " print(label)\n", " for item in body.get(\"internalActionsContributions\") or []:\n", " print(f\" {item.get('actionid')!s:<24} activity={item.get('activity')!r:<46} balancingactivity={item.get('balancingactivity')!r}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "d227ab04", "metadata": {}, "outputs": [], "source": [ "# Activity metadata: top-level activityMetadata map and per-contribution activitymetadata.\n", "for label, body in responses.items():\n", " if not body:\n", " continue\n", " metadata = body.get(\"activityMetadata\")\n", " contributions = body.get(\"internalActionsContributions\") or []\n", " with_metadata = sum(1 for item in contributions if isinstance(item, dict) and \"activitymetadata\" in item)\n", " keys = sorted(metadata) if isinstance(metadata, dict) else []\n", " auto_ids = {row.get(\"activity\") for row in body.get(\"autoInsertedActivities\") or []}\n", " print(f\"{label}:\")\n", " print(f\" activityMetadata: {type_name(metadata)}, entries: {[key for key in keys if key not in auto_ids]}\"\n", " f\" + {sum(1 for key in keys if key in auto_ids)} auto-inserted trend rows\")\n", " print(f\" contributions with activitymetadata: {with_metadata}/{len(contributions)}\")\n", "\n", "# Fields of one activityMetadata entry present in both v1 and v2.\n", "v1_meta = (v1_extended or {}).get(\"activityMetadata\") or {}\n", "v2_meta = (v2_extended or {}).get(\"activityMetadata\") or {}\n", "shared_id = next((key for key in v1_meta if key in v2_meta), None)\n", "if shared_id:\n", " print_shape_diff(f\"activityMetadata['{shared_id}']\", v1_meta[shared_id], v2_meta[shared_id])\n", "warehouse_meta = ((multi_extended or {}).get(\"activityMetadata\") or {}).get(\"act_warehouse_electricity\")\n", "if warehouse_meta:\n", " print(\"\\nactivityMetadata['act_warehouse_electricity']:\", json.dumps(warehouse_meta)[:300])" ] }, { "cell_type": "markdown", "id": "f2e88a90", "metadata": {}, "source": [ "## 11. Checklist\n", "\n", "**Standard results (no `extended=true`)**\n", "\n", "| | What to do |\n", "|---|---|\n", "| Essential | Change the URL: `/api/calculate` → `/api/v2/calculate` (and `/api/scenarioanalysis`, `/api/methodology` → `/api/v2/...`). Keep `?savePayload=false`. |\n", "| Essential | Re-baseline: store fresh v2 results; don't compare them with historic v1 numbers. |\n", "| Recommended | Pin `meta.externalTrendsVersion` to the `externalTrendsPack.id` you validated, and log `warnings`. |\n", "| Recommended | Ignore or store the new `externalTrendsPack` and `warnings` keys; existing keys keep their shape. |\n", "| Optional | Migrate to the v2 payload shape with `POST /api/migrate-payload` (nested `meta` / `target`, flat `activitydata`). |\n", "\n", "**Extended results (`extended=true`)**: everything above, plus:\n", "\n", "| | What to do |\n", "|---|---|\n", "| Essential | Re-check parsers for the MAC slice fields listed in section 10 (new cost fields such as `marginalCost`, `annualisedMarginalCost`, present-value and `nominal` fields) and any keys that are only in v1. |\n", "| Recommended | Key activity lookups by the `activity` value returned in contributions and projections (instance ids), not by catalogue id. |\n", "| Optional | Send several rows per catalogue activity (section 9) and target actions at one instance. |" ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }