IODE and numpy
IODE offers several ways to interact with numpy ndarray objects.
Using numpy ndarray objects is not recommended as there is no compatibility check between for the names and periods. The result is not guaranteed to be correct. This possibility is provided for speed reasons (when dealing with large subsets/databases).
Lets start with necessary imports and loading the sample data:
[1]:
import numpy as np
from iode import (SAMPLE_DATA_DIR, comments, equations, identities, lists, scalars,
tables, variables, Sample, NA)
[2]:
# ---- load equations, identities, scalars and variables ----
# Note: test binary and ASCII 'fun' files are located in the 'SAMPLE_DATA_DIR'
# directory of the 'iode' package
comments.load(f"{SAMPLE_DATA_DIR}/fun.cmt")
equations.load(f"{SAMPLE_DATA_DIR}/fun.eqs")
identities.load(f"{SAMPLE_DATA_DIR}/fun.idt")
lists.load(f"{SAMPLE_DATA_DIR}/fun.lst")
scalars.load(f"{SAMPLE_DATA_DIR}/fun.scl")
tables.load(f"{SAMPLE_DATA_DIR}/fun.tbl")
variables.load(f"{SAMPLE_DATA_DIR}/fun.var")
# ---- print the number of objects present in the above workspaces ----
len(comments), len(equations), len(identities), len(lists), len(scalars), len(tables), len(variables)
Loading C:\soft\Miniconda3\Lib\site-packages\iode\tests\data/fun.cmt
317 objects loaded
Loading C:\soft\Miniconda3\Lib\site-packages\iode\tests\data/fun.eqs
274 objects loaded
Loading C:\soft\Miniconda3\Lib\site-packages\iode\tests\data/fun.idt
48 objects loaded
Loading C:\soft\Miniconda3\Lib\site-packages\iode\tests\data/fun.lst
17 objects loaded
Loading C:\soft\Miniconda3\Lib\site-packages\iode\tests\data/fun.scl
161 objects loaded
Loading C:\soft\Miniconda3\Lib\site-packages\iode\tests\data/fun.tbl
46 objects loaded
Loading C:\soft\Miniconda3\Lib\site-packages\iode\tests\data/fun.var
394 objects loaded
[2]:
(317, 274, 48, 17, 161, 46, 394)
Add one variable:
[3]:
values = list(range(variables.nb_periods))
values[0] = NA
values[-1] = np.nan
data = np.asarray(values)
variables["A3"] = data
variables["A3"]
[3]:
Workspace: Variables
nb variables: 1
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 1960Y1:2015Y1
mode: LEVEL
name 1960Y1 1961Y1 1962Y1 1963Y1 1964Y1 1965Y1 ... 2009Y1 2010Y1 2011Y1 2012Y1 2013Y1 2014Y1 2015Y1
A3 na 1.00 2.00 3.00 4.00 5.00 ... 49.00 50.00 51.00 52.00 53.00 54.00 na
Update one variable.
Update all values of a variable:
[4]:
values = list(range(variables.nb_periods))
values[0] = NA
values[-1] = np.nan
data = np.asarray(values)
variables["AOUC"] = data
variables["AOUC"]
[4]:
Workspace: Variables
nb variables: 1
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 1960Y1:2015Y1
mode: LEVEL
name 1960Y1 1961Y1 1962Y1 1963Y1 1964Y1 1965Y1 ... 2009Y1 2010Y1 2011Y1 2012Y1 2013Y1 2014Y1 2015Y1
AOUC na 1.00 2.00 3.00 4.00 5.00 ... 49.00 50.00 51.00 52.00 53.00 54.00 na
Set the values for range of (contiguous) periods:
[5]:
# variable[t:t+x] = numpy array
values = [1.0, NA, 3.0, np.nan, 5.0]
data = np.asarray(values)
variables["ACAG", "1991Y1:1995Y1"] = data
variables["ACAG", "1991Y1:1995Y1"]
[5]:
Workspace: Variables
nb variables: 1
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 1991Y1:1995Y1
mode: LEVEL
name 1991Y1 1992Y1 1993Y1 1994Y1 1995Y1
ACAG 1.00 na 3.00 na 5.00
Update multiple variables at once
[6]:
data = [[28.89, 31.90, 36.66, 42.13, 9.92],
[np.nan, -39.96, -42.88, -16.33, -41.16],
[1.023, np.nan, 1.046, np.nan, 1.064]]
data = np.asarray(data)
data
[6]:
array([[ 28.89 , 31.9 , 36.66 , 42.13 , 9.92 ],
[ nan, -39.96 , -42.88 , -16.33 , -41.16 ],
[ 1.023, nan, 1.046, nan, 1.064]])
[7]:
variables["ACAF, ACAG, AOUC", "1991Y1:1995Y1"] = data
variables["ACAF, ACAG, AOUC", "1991Y1:1995Y1"]
[7]:
Workspace: Variables
nb variables: 3
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 1991Y1:1995Y1
mode: LEVEL
name 1991Y1 1992Y1 1993Y1 1994Y1 1995Y1
ACAF 28.89 31.90 36.66 42.13 9.92
ACAG na -39.96 -42.88 -16.33 -41.16
AOUC 1.02 na 1.05 na 1.06
Arithmetic Operations On Variables With numpy ndarray objects
IODE variables can be used in arithmetic operations with numpy arrays.
Let’s first reload the variables database to start from a clean state. Then we will select a subset of variables for the examples below:
[8]:
# reload variables to start from a clean state
variables.load(f"{SAMPLE_DATA_DIR}/fun.var")
# select a subset of variables for the examples below
vars_subset = variables["A*", "1991Y1:1995Y1"]
vars_subset
Loading C:\soft\Miniconda3\Lib\site-packages\iode\tests\data/fun.var
394 objects loaded
[8]:
Workspace: Variables
nb variables: 5
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 1991Y1:1995Y1
mode: LEVEL
name 1991Y1 1992Y1 1993Y1 1994Y1 1995Y1
ACAF 26.24 30.16 34.66 8.16 -13.13
ACAG -30.93 -40.29 -43.16 -16.03 -41.85
AOUC 1.02 1.03 1.03 1.05 1.05
AOUC_ 0.96 0.97 0.98 0.99 1.00
AQC 1.06 1.11 1.15 1.16 1.16
[9]:
data = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
# multiply the values of a single variable by
# the values of a numpy 1D ndarray
updated_ACAF = vars_subset["ACAF"] * data
updated_ACAF
[9]:
Workspace: Variables
nb variables: 1
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 1991Y1:1995Y1
mode: LEVEL
name 1991Y1 1992Y1 1993Y1 1994Y1 1995Y1
ACAF 26.24 60.32 103.99 32.64 -65.65
[10]:
# multiply the values of a subset corresponding to a single period
# by the values of a numpy 1D ndarray
vars_subset_1995Y1 = vars_subset[:, "1995Y1"] * data
vars_subset_1995Y1
[10]:
Workspace: Variables
nb variables: 5
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 1995Y1:1995Y1
mode: LEVEL
name 1995Y1
ACAF -13.13
ACAG -83.69
AOUC 3.15
AOUC_ 3.98
AQC 5.81
[11]:
# multiply the values of a subset of variables
# by the values of a numpy 2D ndarray
data = np.array([[1.0, 2.0, 3.0, 4.0, 5.0],
[6.0, 7.0, 8.0, 9.0, 10.0],
[11.0, 12.0, 13.0, 14.0, 15.0],
[16.0, 17.0, 18.0, 19.0, 20.0],
[21.0, 22.0, 23.0, 24.0, 25.0]])
new_vars_subset = vars_subset * data
new_vars_subset
[11]:
Workspace: Variables
nb variables: 5
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 1991Y1:1995Y1
mode: LEVEL
name 1991Y1 1992Y1 1993Y1 1994Y1 1995Y1
ACAF 26.24 60.32 103.99 32.64 -65.65
ACAG -185.60 -282.00 -345.26 -144.26 -418.46
AOUC 11.27 12.38 13.40 14.65 15.75
AOUC_ 15.43 16.56 17.62 18.80 19.91
AQC 22.32 24.43 26.53 27.77 29.04
Import/Export IODE Variables workspace from/to numpy ndarray
To import / export the content of the Variables workspace (or a subset of it) from/to a numpy ndarray object, use the from_numpy and to_numpy methods.
[12]:
len(variables)
[12]:
394
[13]:
variables.sample
[13]:
Sample("1960Y1:2015Y1")
[14]:
variables.nb_periods
[14]:
56
Export to numpy
Export the whole Variables workspace to a numpy ndarray:
[15]:
# export the whole Variables workspace to a numpy ndarray (394 variables x 56 periods)
data = variables.to_numpy()
data.shape
[15]:
(394, 56)
[16]:
data[5, 40]
[16]:
442.26441085858613
[17]:
variables.i[5, 40]
[17]:
442.26441085858613
Export a subset of names:
[18]:
# export a subset of names
vars_subset = variables["A*"]
vars_subset.names
[18]:
['ACAF', 'ACAG', 'AOUC', 'AOUC_', 'AQC']
[19]:
vars_subset
[19]:
Workspace: Variables
nb variables: 5
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 1960Y1:2015Y1
mode: LEVEL
name 1960Y1 1961Y1 1962Y1 1963Y1 1964Y1 1965Y1 ... 2009Y1 2010Y1 2011Y1 2012Y1 2013Y1 2014Y1 2015Y1
ACAF na na na na na na ... -37.46 -37.83 -44.54 -55.56 -68.89 -83.34 -96.41
ACAG na na na na na na ... 27.23 28.25 29.28 30.32 31.37 32.42 33.47
AOUC na 0.25 0.25 0.26 0.28 0.29 ... 1.29 1.31 1.33 1.36 1.39 1.42 1.46
AOUC_ na na na na na na ... 1.23 1.25 1.27 1.30 1.34 1.37 1.41
AQC 0.22 0.22 0.22 0.23 0.24 0.25 ... 1.45 1.46 1.48 1.51 1.56 1.61 1.67
[20]:
data = vars_subset.to_numpy()
data.shape
[20]:
(5, 56)
[21]:
# values of the 'ACAF' variable
data[0]
[21]:
array([ nan, nan, nan, nan,
nan, nan, nan, nan,
nan, nan, 1.2130001 , 5.2020001 ,
9.184 , 8.0790005 , 11.332 , 13.518001 ,
15.784 , 16.544001 , 21.489 , 20.281 ,
21.277 , 32.417999 , 24.446999 , 27.025002 ,
24.504 , 27.560999 , 25.542 , 27.499001 ,
25.353001 , 17.165001 , 23.771 , 26.240999 ,
30.159 , 34.661999 , 8.1610022 , -13.130997 ,
32.171001 , 39.935799 , 29.645657 , 13.53040492,
10.04661079, 2.86792274, -0.92921251, -6.09156499,
-14.58209446, -26.53878957, -28.98728798, -33.37842578,
-38.40951778, -37.46350964, -37.82742883, -44.54479263,
-55.55928982, -68.89465432, -83.34062511, -96.41041983])
[22]:
# values of the 'AQC' variable
data[-1]
[22]:
array([ 0.21753037, 0.21544869, 0.22228125, 0.22953896, 0.23653506,
0.24732406, 0.26255098, 0.26907021, 0.27206925, 0.27986595,
0.29396999, 0.31906503, 0.3426649 , 0.36655167, 0.42489415,
0.49478459, 0.53812659, 0.5841772 , 0.61441606, 0.64528418,
0.68947881, 0.73596764, 0.77532566, 0.82384807, 0.85829282,
0.90006256, 0.92794591, 0.93221092, 0.92874223, 0.9445076 ,
1. , 1.0628064 , 1.1102825 , 1.1532652 , 1.1571276 ,
1.1616869 , 1.1580297 , 1.201328 , 1.2031082 , 1.34296997,
1.33860286, 1.37918825, 1.40881647, 1.41970458, 1.40065206,
1.39697298, 1.39806354, 1.40791334, 1.42564488, 1.44633167,
1.46286837, 1.48227361, 1.51366598, 1.55803879, 1.61318117,
1.67429058])
Export a subset of names and periods:
[23]:
# export a subset of names and periods
vars_subset = variables["A*", "2000Y1:2010Y1"]
vars_subset
[23]:
Workspace: Variables
nb variables: 5
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 2000Y1:2010Y1
mode: LEVEL
name 2000Y1 2001Y1 2002Y1 2003Y1 2004Y1 2005Y1 2006Y1 2007Y1 2008Y1 2009Y1 2010Y1
ACAF 10.05 2.87 -0.93 -6.09 -14.58 -26.54 -28.99 -33.38 -38.41 -37.46 -37.83
ACAG -41.53 18.94 19.98 21.02 22.07 23.11 24.13 25.16 26.19 27.23 28.25
AOUC 1.12 1.14 1.16 1.17 1.17 1.18 1.20 1.22 1.26 1.29 1.31
AOUC_ 1.10 1.14 1.15 1.16 1.15 1.16 1.19 1.20 1.21 1.23 1.25
AQC 1.34 1.38 1.41 1.42 1.40 1.40 1.40 1.41 1.43 1.45 1.46
[24]:
data = vars_subset.to_numpy()
data.shape
[24]:
(5, 11)
[25]:
data
[25]:
array([[ 10.04661079, 2.86792274, -0.92921251, -6.09156499,
-14.58209446, -26.53878957, -28.98728798, -33.37842578,
-38.40951778, -37.46350964, -37.82742883],
[-41.53478657, 18.93980114, 19.98081488, 21.02050218,
22.06647552, 23.10796216, 24.12963715, 25.16090905,
26.19211148, 27.22995512, 28.25392898],
[ 1.11623762, 1.14047639, 1.15716928, 1.17048954,
1.16767464, 1.1815207 , 1.19946163, 1.21933288,
1.26280574, 1.28713178, 1.3071099 ],
[ 1.1019572 , 1.13624426, 1.15021519, 1.16082895,
1.14802147, 1.16412337, 1.18589708, 1.19516611,
1.21383423, 1.23185399, 1.25016433],
[ 1.33860286, 1.37918825, 1.40881647, 1.41970458,
1.40065206, 1.39697298, 1.39806354, 1.40791334,
1.42564488, 1.44633167, 1.46286837]])
import from numpy
To update a subset of the Variables workspace, use the from_numpy method.
[26]:
vars_names = variables.get_names("A*")
vars_names
[26]:
['ACAF', 'ACAG', 'AOUC', 'AOUC_', 'AQC']
[27]:
first_period = "2000Y1"
last_periods = "2010Y1"
sample = Sample(first_period, last_periods)
nb_periods = sample.nb_periods
nb_periods
[27]:
11
[28]:
# save original values to restore them later
original_values = variables["A*", "2000Y1:2010Y1"].to_numpy()
original_values
[28]:
array([[ 10.04661079, 2.86792274, -0.92921251, -6.09156499,
-14.58209446, -26.53878957, -28.98728798, -33.37842578,
-38.40951778, -37.46350964, -37.82742883],
[-41.53478657, 18.93980114, 19.98081488, 21.02050218,
22.06647552, 23.10796216, 24.12963715, 25.16090905,
26.19211148, 27.22995512, 28.25392898],
[ 1.11623762, 1.14047639, 1.15716928, 1.17048954,
1.16767464, 1.1815207 , 1.19946163, 1.21933288,
1.26280574, 1.28713178, 1.3071099 ],
[ 1.1019572 , 1.13624426, 1.15021519, 1.16082895,
1.14802147, 1.16412337, 1.18589708, 1.19516611,
1.21383423, 1.23185399, 1.25016433],
[ 1.33860286, 1.37918825, 1.40881647, 1.41970458,
1.40065206, 1.39697298, 1.39806354, 1.40791334,
1.42564488, 1.44633167, 1.46286837]])
[29]:
# create the numpy ndarray containing the values to copy into the Variables database
data = np.zeros((len(vars_names), nb_periods), dtype=float)
for i in range(len(vars_names)):
for j in range(nb_periods):
data[i, j] = i * nb_periods + j
data
[29]:
array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.],
[ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.],
[ 22., 23., 24., 25., 26., 27., 28., 29., 30., 31., 32.],
[ 33., 34., 35., 36., 37., 38., 39., 40., 41., 42., 43.],
[ 44., 45., 46., 47., 48., 49., 50., 51., 52., 53., 54.]])
[30]:
variables["A*", "2000Y1:2010Y1"]
[30]:
Workspace: Variables
nb variables: 5
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 2000Y1:2010Y1
mode: LEVEL
name 2000Y1 2001Y1 2002Y1 2003Y1 2004Y1 2005Y1 2006Y1 2007Y1 2008Y1 2009Y1 2010Y1
ACAF 10.05 2.87 -0.93 -6.09 -14.58 -26.54 -28.99 -33.38 -38.41 -37.46 -37.83
ACAG -41.53 18.94 19.98 21.02 22.07 23.11 24.13 25.16 26.19 27.23 28.25
AOUC 1.12 1.14 1.16 1.17 1.17 1.18 1.20 1.22 1.26 1.29 1.31
AOUC_ 1.10 1.14 1.15 1.16 1.15 1.16 1.19 1.20 1.21 1.23 1.25
AQC 1.34 1.38 1.41 1.42 1.40 1.40 1.40 1.41 1.43 1.45 1.46
[31]:
# copy the numpy ndarray into the Variables database (overriding the existing values)
variables.from_numpy(data, vars_names, first_period, last_periods)
variables["A*", "2000Y1:2010Y1"]
[31]:
Workspace: Variables
nb variables: 5
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 2000Y1:2010Y1
mode: LEVEL
name 2000Y1 2001Y1 2002Y1 2003Y1 2004Y1 2005Y1 2006Y1 2007Y1 2008Y1 2009Y1 2010Y1
ACAF 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00
ACAG 11.00 12.00 13.00 14.00 15.00 16.00 17.00 18.00 19.00 20.00 21.00
AOUC 22.00 23.00 24.00 25.00 26.00 27.00 28.00 29.00 30.00 31.00 32.00
AOUC_ 33.00 34.00 35.00 36.00 37.00 38.00 39.00 40.00 41.00 42.00 43.00
AQC 44.00 45.00 46.00 47.00 48.00 49.00 50.00 51.00 52.00 53.00 54.00
If you already work on the subset you whish to update the values, you can skip to specify the value for the parameters vars_names, first_period and last_period:
[32]:
vars_subset = variables["A*", "2000Y1:2010Y1"]
vars_subset.from_numpy(original_values)
vars_subset
[32]:
Workspace: Variables
nb variables: 5
filename: c:\soft\Miniconda3\Lib\site-packages\iode\tests\data\fun.var
description: Modèle fun - Simulation 1
sample: 2000Y1:2010Y1
mode: LEVEL
name 2000Y1 2001Y1 2002Y1 2003Y1 2004Y1 2005Y1 2006Y1 2007Y1 2008Y1 2009Y1 2010Y1
ACAF 10.05 2.87 -0.93 -6.09 -14.58 -26.54 -28.99 -33.38 -38.41 -37.46 -37.83
ACAG -41.53 18.94 19.98 21.02 22.07 23.11 24.13 25.16 26.19 27.23 28.25
AOUC 1.12 1.14 1.16 1.17 1.17 1.18 1.20 1.22 1.26 1.29 1.31
AOUC_ 1.10 1.14 1.15 1.16 1.15 1.16 1.19 1.20 1.21 1.23 1.25
AQC 1.34 1.38 1.41 1.42 1.40 1.40 1.40 1.41 1.43 1.45 1.46