User SETs

[janis]

My question concerns user sets in VEDA.
Historically, when transitioning from MARKAL to TIMES, some technology sets from MARKAL have remained in data input tables, such as HLK and LNK.



Moreover, these sets work in the VEDA – Tools > Sets > Sets Editor; for example, if they are assigned in the ~TFM_Psets table under the pset_set column. Like specifing User Set “Industry”.



Specifically, when pressing "View in SETS Editor", it selects PRE processes that have an output commodity starting with IND, as well as LNK and HLK processes that also have an output commodity starting with IND.
However, when navigating to the Results tab in VEDA and checking the user set "Industry", only PRE processes are selected, and the CSV files contain only PRE processes.

I've run into this issue multiple times ?, so my question is: What is the correct behavior?
Should it work in both places or in neither, or am I missing some subtle nuance?

Thanks!!!

[rohit]

I think this is because sets on the input side see all the sets that have been used in proc/comm tables, even if they are not valid TIMES sets.
The VDS file which has only the valid TIMES sets, is involved in creating process and commodity lists in results processing.
So in the results tab, only TIMES sets will be present.

[Antti-L]

@Rohit: 
> So in the results tab, only TIMES sets will be present.

I am quite confused by this statement. 
I think one can create any number of process and commodity sets for results tables under VEDA2, just like it was under VEDA-BE.  These sets are not TIMES sets.  But they should work nonetheless, shouldn't they?  That's why there was even a tool for converting those results sets from VEDA-BE to VEDA2, no?  Therefore, I think the explanation to the problem must be something different... maybe it is so that the Markal set names for some reason do not work under results?  But if that is so, why are they then shown in the sets editor (and should thus be usable for defining new sets, like the TIMES sets can be used)?

[AKanudia]

Even though Veda 2.0 consolidates input and results management in a single interface, it's important to note that the results section does not use the "active" input data directly. This design choice is because VD files can be imported independently (e.g., at different times or on different machines), and relying on input data could cause inconsistencies in results processing. Results are processed based only on the set definitions that come from the VDS and Veda user-defined sets definition files.

The key distinction here is that sets like `HLK`, `LNK`, etc., are not standard TIMES sets, but rather labels embedded in the `sets` column of process or commodity definitions. Content of the sets column is parsed during input processing to update process `type` and `subtype`, and the superfluous labels just stay in the sets column. Now, the "problem" is that Veda’s input module (including the Sets Editor) uses the string in sets column, along with the `type` and `subtype` to resolve user-defined sets. However, the non-TIMES labels are not propagated to the results module, since they're not part of the `.VDS` file (that communicates the sets that are not included in the UD sets definition file).

In essence, user-defined sets based on these legacy (or any other) labels will behave differently between the input and results modules. This seems to be a byproduct of the the legacy MARKAL-style definitions and the unbridled flexibility that Veda provides.

How about a warning when a user-defined set relies on non-TIMES sets?

[Antti-L]

> How about a warning when a user-defined set relies on non-TIMES sets?

But all the numerous VEDA results sets are non-TIMES sets (those defined in the Sets*.xlsx). If you would issue a warning always when a user-defined set is relying on those VEDA sets, it would cause vast amounts of unnecessary warnings.  I think it is those Markal sets that are the only ones relying on input data only, no?  At least under VEDA-BE the VEDA sets were relying on results data only, has that been changed?

I am thus heavily using non-TIMES sets when defining additional results sets, and I would not expect to get warnings about them, because I think they should be working well, like they worked under VEDA-BE.

[AKanudia]

Of course, you are right. I am only referring to the non-TIMES sets that have been used in the sets col of process/commodity definition tables.

[Antti-L]

Ok, thanks Amit, that's clear now.

However, there are also sets that VEDA does in fact declare as TIMES Sets, but are coming neither from the VDS nor the Veda user-defined sets definition files. Do you think TIMES should start defining some of these in the VDS files, as VEDA seems to think they are TIMES sets? A few examples below:
     
As I see "CEN" and "PRC" there, it seems that the Markal sets may actually be getting declared as TIMES Sets by VEDA, which may thus also need to be corrected?

[janis]

Thank you all for the discussion and responses in this thread in deed!
I also see MARKAL SETs—or as Amit noted, 'labels'—in my SET table, including the previously mentioned LNK and HLK.



I would really like to use SETs defined in this way within the results module as well, as it would align with how I define the model input data. I find filtering based on names (or in/out comm) using wildcards to select processes or commodities for user-defined SETs quite frustrating (I brought up this issue because I encountered the same problem again! ?). While filtering is undoubtedly a powerful tool, there are always exceptions during the workflow, such as new outputs/inputs or name changes. Additionally, I find it difficult to anticipate the full impact of this approach on results processing.

Perhaps a workaround for me would be to define labels within process and commodity descriptions and use those as a basis for filtering, as explicitly defined labels within descriptions provide more control and help mitigate these risks.