Controversial handling of CG indexes by VEDA

[Antti-L]

Dear VEDA developers (and VEDA users),

I think the handling of CGs for TIMES attributes that have a single CG index (and no commodity index) is in certain cases controversial and unexpected, such that it can also be viewed inconsistent.

Example test case below, where I define two almost identical processes (TESTP1, TESTP2) in a FI_T table:
     
In the example, all those attributes that have only a single CG index are defined with the CG index specified in the column header, and as such they are in my view not in any way ambiguous: the CG index for the values below the column should only be taken from the column header. For the TESTP2 process the only difference is that I added an excess heat output from using the second fuel input (FUEL2), where the source group (FUEL2) is specified in the CommGrp column.

Unexpectedly, the resulting parameters are, however, vastly different for the two test processes, and are in my view quite unexpected for TESTP2, as shown below. 
     
I understand that those resulting parameters for the TESTP2 process are caused by the convention in VEDA, that whenever there is ambiguity about where to take the CG index, the CommGrp column takes the highest precedence (if present and is filled-in).  In the same way for commodity indexes, the CommName column takes the highest precedence (if present and filled-in).  However, for attributes with commodity indexes, any index in CommName will not override the commodity specified in the column header, which is indeed as one would expect. 

In other words, attributes with a CG index seem to be handled quite differently, where the index specified in the CommGrp column is always overriding the CG index explicitly given in the column header, although the values in the column are clearly defined specifically for the CG index in the column header, and so there should in my view be no ambiguity about what the user wants: it is the CG given in the column header. In my view the handling is inconsistent in this respect.

One can additionally see that the attributes for the second process look even quite messy in their indexes (commodity + group), which I think might potentially cause problems for TFM_UPD/MIG or TS filtering operations.  And one can also see a duplicate ACT_EFF parameter for TESTP2, which would be causing a GAMS error.

I understand that there could be differing views about it as well, but I would hope the VEDA developers could explain why the CG index in the column header is not respected when the attribute has a single CG index (and no commodity index), and whether they could consider changing this behavior. Other users are also welcome to give their opinion, as to whether they are satisfied with the current behavior, or see it undesirable like I do.

[AKanudia]

Dear Antti,

Thanks for exploring and exposing the intricacies of Veda.

Veda supports elements from only the following dimensions in column headers: attribute, commodity, currency, lim_type, region, side, time_slice, and year. The reason is Veda is able to determine the dimension from the element. For users to be able to manage potential overlaps, dimensions are picked up in the following order: attribute, currency, lim_type, region, side, time_slice, year, commodity. For example, ACT from a col header will be read as region, if defined in the model, rather than a commodity.

For TESTP2, what you intended as CGs were read as commodities and ignored due to the explicit CG specification. 

Does this help?

Of course, I understand that this has probably not been clearly documented anywhere. FI_T tag documentation will be updated soon in the online Veda doc.

[Antti-L]

Well, what you don't seem to see is that commodities are always CGs, and so explicit commodities in the column headers are explicit and unambiguous CGs. Hence, I don't see how that would explain the fact that VEDA throws away CG1 and replaces it by CG4, even though the user has specifically associated the parameter values with CG1.

So, when the user defines:

   NCAP_AFC(r,y,p,CG1,tsl) = value;
   ACT_EFF(r,y,p,CG2,ts) = value;
   PRC_ACTFLO(r,y,p,CG3) = value;

VEDA does not respect that, but replaces all those CGs by some CG4, which is only used for another parameter and has not been explicitly associated with these parameters by the user:

   NCAP_AFC(r,y,p,CG4,tsl) = value;
   ACT_EFF(r,y,p,CG4,ts) = value;
   PRC_ACTFLO(r,y,p,CG4) = value;

Do you really consider that reasonable?  Can you still elaborate on the justification, why VEDA thinks that it is ok to do such a replacement, even though the commodity groups have been explicitly specified by the user?  The values in a column with a CG explicitly specified in the header are valid only for that CG, and not for some other CG.  So, because I think it is obvious that VEDA is doing the wrong thing here, why should VEDA be doing that wrong thing?  What would be the reasoning behind deciding that CG4 is the correct CG instead of CG1 given in the column header?

I would consider even an error message more appropriate than replacing the user-defined explicit CG with the VEDA-determined implicit CG4 in such cases.  And I guess those cases would be easy to catch, by looking up those messy parameter instances having an extra commodity field spuriously filled in.

[Antti-L]

> For users to be able to manage potential overlaps, dimensions are picked up in the following order: attribute, currency, lim_type, region, side, time_slice, year, commodity. For example, ACT from a col header will be read as region, if defined in the model, rather than a commodity.

Thanks Amit, this is very useful information. 
It also prompted me to make a small follow-up test, see TESTP3 specification below:
     
As you can see, TESTP3 has FUEL1 explicitly specified in the CommName column, and therefore, following your explanation, if things should work consistently for commodity attributes, I would expect that commodities specified in the value column headers should be ignored due to the CommName specification (because CGs in the value column headers were ignored for CG attributes due to a CommGrp specification). However, I was glad to see that it did not work that way, see the resulting browse below:
     
As you can see, FUEL1 is not getting used for any attributes.  Instead, I can see the commodity groups NRG, MAT, NRGI and MATO being treated as commodity indexes (which would lead to GAMS errors, because they are NOT commodities).  So, the value column headers were all respected in this case, while the CommName was ignored.  I am quite happy with that, despite the erroneous NRG, MAT, NRGI and MATO "commodities", and I would only hope that for CG attributes the value column headers would be likewise respected.

Moreover, I can see that MEuro05 (a currency in the model) and W (a timeslice in the model) are also getting read as commodities, even though you said that the dimensions are picked up in the following order: attribute, currency, lim_type, region, side, time_slice, year, commodity. I feel that this would deserve some additional explanation: Why does VEDA define FLO_COST with MEuro05 as the commodity, and why does VEDA define FLO_PKCOI with W as the commodity?  In the order, currency and timeslice appear before commodity, and so why would they getting read as commodities?

[AKanudia]

Can you please send me this model?

[Antti-L]

I assume the reason for asking the model is the test I made about the ability to “manage potential overlaps”, which you explained above.  I have sent you the test model by email.

Based on my understanding of it, there would be no such overlaps unless some items are defined to be members of multiple dimensions, am I correct?  Therefore, to test this ability to manage overlaps, I had to define some of such items, and I thought it was already obvious that I chose to test with defining MEuro05 and W as commodities. According to your statement, I understood that they should have been still read only as currency and timeslice when used in value column headers, based on the order you stated.

Could you thus explain why MEuro05 and W were read both as a currency/timeslice and as a commodity?

[AKanudia]

https://www.dropbox.com/scl/fi/y05yqxiu9bzewbgovigyt/Veda2.0_localhost_3.1.3.1.7z?rlkey=g1z6i3fjsr4us4bvi2ljxezxy&dl=0

Thanks for creating these cases. Please see if this version resolves the overlap issues.

[Antti-L]

> Thanks for creating these cases. Please see if this version resolves the overlap issues.

Thanks Amit.  Yes, according to my tests that version does remove the overlap behavior I reported.  Does that mean that the current version v3.1.3.0 does not work correctly, i.e. there was a bug?

However, I was able to identify some regression in the 3.1.3.1 version you provided:
  • User-defined CGs defined also as commodities now cause invalid/unrecognized column error when using them in value column headers for Commodity or CG attributes (current version worked fine)
  • As a curiosity, if I define NRGI/NRGO/MATI/MATO etc. as a user-defined CG and use such in a value column header for a CG attribute, I also get invalid/unrecognized column error (current version works fine).

In addition, both the current version and v3.1.3.1 manifest some additional issues in my new tests:
  • When there are multiple commodities in the Comm-Out column, VEDA generates duplicate parameter instances for a CG attribute if CommGrp is specified, due to the messy Commodity field;
  • When there are multiple value columns for a CG attribute with different CGs specified, VEDA generates duplicate (and controversial) parameter instances for a CG attribute if CommGrp is specified, due to the messy Commodity field;
  • In these messy Commodity field cases, I was also able to verify that TS filtering does not work correctly: The time-series is not getting filtered out.

And finally, the controversial CG replacement issue still remains (explicit CG specified in value column is replaced by a CG in CommGrp if specified, although the explicit CG in value column header should take precedence, or at least should trigger a warning for the user, if replaced by VEDA).

I would hope the regression would not be adopted, and the duplicate and TS filtering errors should also be easy to fix, and so I hope they can be all taken care of.

[AKanudia]

Thanks Antti. Please send the test model once again.

[Antti-L]

Ok Amit, I have sent you my new test model.

I think there would be none of those duplicate parameter nor TS filtering issues if the CGs in the value column headers would be respected by VEDA2 (and the commodity field would be cleaned up).  I guess just cleaning up the commodity field and removing the duplicates should probably also fix these duplicate and TS filtering issues (but would not fix the controversial CG replacement issue).

[AKanudia]

https://www.dropbox.com/scl/fi/dfunda5rostywt7lsc3bu/Veda2.0_localhost_3.1.3.2.7z?rlkey=i1mx3ydgwmkjb12ofz46k1op4&dl=0

Thanks for creating these cases Antti. Indeed, cleaning up the commodity field after handling the CG declaration was key. Please see how this version performs.

[Antti-L]

> Please see how this version performs.

Thanks Amit, I will look into it as soon as possible. I am sure I will find it performing quite well!

However, if I understand correctly, you have not yet considered the controversial CG replacement issue, because (based on your email) you may have misunderstood that I would be suggesting those CG attribute indexes not currently considered at all by VEDA when filling in the dimensions from value column headers, to be also filled in from those headers (such as the CG index in FLO_SHAR). But I have not been suggesting any that kind of a change!

I am only suggesting to fix the current inconsistency in VEDA, with respect to the precedence between the items specified in the index columns versus those explicitly specified in the value column headers. Apparently, for all other dimensions, the items specified in the value column headers take highest precedence, as intuitively expected, but for the single-CG attributes (e.g. ACT_EFF, NCAP_AFC, PRC_ACTFLO), a CG in the CommGrp index column currently overrides the CG specified in the value column header, which I consider inconsistent, non-intuitive and error-prone. So, I am only suggesting to improve VEDA in that respect: to fix the inconsistency in the precedence when filling in those CG indexes that are already currently supported through value column headers.

[Antti-L]

Ok, Amit, I have now tested with that v3.1.3.2, and it indeed appears to fix the regression, duplicate and TS filtering issues, which great!

Hence, at this point my only remaining concern is the controversial CG replacement issue.  It has caused me notable headache on a few occasions due to wasting time as a result of such a CG replacement, which has been left for some time unnoticed, despite the related incorrect model behavior. Therefore, I would really welcome at least a warning when the CG explicitly specified in value column header (which should take precedence) is getting replaced by VEDA.

[AKanudia]

We have found a way to implement the behaviour you expected. Dimension allocation for column header elements was being done independent of the attributes, as they can apply to multiple attributes.

We will have a version for testing on Monday. It will also reinstate support for the other UC sets (T and TS).

[AKanudia]

https://www.dropbox.com/scl/fi/wo5d4lma9rpinrbvxmfex/Veda2.0_localhost_3.1.3.3.7z?rlkey=4u2269mcmbzp7i1w3uoc627le&dl=0

This version will give top priority to CG declarations in col headers for attributes that have one CG index. And the UC sets related to T and TS are supported again.