Observations & Questions

[Antti-L]

I have encountered some more differences between VEDA-FE and VEDA2, unexplained in the VEDA2 online documentation, section "Things that are different". Because I have neither found other documentation on these features that would explain the behaviour under VEDA2, I am left unsure whether they are bugs or by new design. Therefore I'd hope the VEDA developers can comment on the following observations:

Obs1: If I have a TFM_UPD row that matches with the same parameter specified in TFM_INS in the same scenario, VEDA-FE did use that data from TFM_INS  as the source data for the update. In VEDA2 this no longer seems to be the case.  Consequently, if TFM_UPD finds matching data from another scenario (when no SourceScen is used), any matching data generated by TFM_INS in the same scenario would be left ignored. 

Q1:  Is the above observation correct, and is that change intended?  If it is, shouldn't this difference be mentioned in the "Things that are different" section?

Obs2: If I use several negative (exclude) filters in a TFM_INS, VEDA-FE applied those exclude filters by combining them with AND (by default).  However, VEDA2 appears to apply those exclude filters by combining them with OR (by default).

Q2:  Is the above observation correct, and is that change intended?  If it is, shouldn't this difference be mentioned in the "Things that are different" section?

Obs3:  If I try to update interpolation options in SysSettings by using an TFM_INS table, with year=0,  in VEDA-FE that worked well (with or without an actual "seed" value). However, in VEDA2 no records are generated by the TFM_INS table.

Q3: Is the above observation correct, and is that change intended?  If it is, why does the "Things that are different" section specifically say the following (i.e. what is the meaning of the second remark below):
  • Interpolation settings table in SysSettings needs to be converted to a MIG table.
    ◦ if updating with year=0 (without an actual seed value) then need to convert it to in INS

[AKanudia]

Thanks for raising these issues, and apologies for the long delay.

#1 Your observation is correct and it is an intended change. It has been added to the "Things that are different" section of Veda doc.
Reason: Removing awkward exceptions was one of our targets while rewriting the data processing queries. INS/DINS used to provide seed values in the same scenario, but UPD/MIG did not. I thought that it is clearer if seed values are not seen from the same scenario at all, rather than being visible from some tags and not others. Further, I didn't encounter any cases where I actually wanted to update values that were inserted in the same scenario. In fact, sometimes I had to prevent the inserted values from being updated by using sourcescen in the UPD tables.

#2 Your observation is correct and it is an intended change. It has been added to the "Things that are different" section of Veda doc.
Reason: I think OR is what is needed in a vast majority of cases when one uses negative filters in multiple columns. I decided to change the default to the most commonly used value.

#3 Can you please give me an example that did not work for you? INS/UPD/MIG work with exactly the same logic in SysSettings as they do in other files. In summary, you have to use MIG to generate interpolation options based on seed values, and INS for generating them regardless of seed values. One pertinent exception is the following: Year=0 records are ignored in all fetch queries (UPD/MIG/FILL) unless year=0 is explicitly specified in the table.

[Antti-L]

Many thanks for the clarifications for my questions 1 and 2.

Concerning 3, a very simplistic example is given in the picture below.  The purpose of the TFM_INS is to update some IE options (with year=0), just like it is described  in the "Things that are different".  Is this not what is meant by that statement there (the second remark below):

  • Interpolation settings table in SysSettings needs to be converted to a MIG table.
   ◦ if updating with year=0 (without an actual seed value) then need to convert it to in INS

Note: Any that kind of TFM_INS worked in VEDA-FE, but it does not work in VEDA2. Nothing is generated by that table under VEDA2. So, is that change intended?
     

[AKanudia]

I created this table in SysSettings of DemoS_012 and I see the records as expected. The third tech needed to be ELCTNOIL01 though.

Can you please send me a model where I can reproduce this problem?

[Antti-L]

I suspect that you forgot that the whole purpose was to update some existing IE options?

Anyway, please find attached an example demonstrating nothing being generated.

[Antti-L]

Could you, in return, please show a full working example of updating IE options as described in "Things that are different":
  ◦ if updating with year=0 (without an actual seed value) then need to convert it to in INS

As mentioned, thus far I have not been able to make that work under VEDA2 (but it did under VEDA-FE).

[AKanudia]

Apologies! Like you said, *updating* is the key.

The undocumented rule you have discovered is: Ignore any interpolation options generated by SysSettings (via any tag) that are already defined in Base or SubRES.

So, updating has been explicitly prohibited for options declared close to the original process/commodity. The underlying thinking is that SysSettings is meant to provide *default* interpolation options. If a user has declared it at the time of creating the process/commodity, then she must want to stick with it.

Note:
1. Your INS table will work from any scenario file.
2. The current INS table (in SysSettings) will work if you try updating interpolation options defined outside Base/SubRES.

Would you propose any change in these rules or just explaining them in the doc is enough?

[Antti-L]

Thanks for clarifying it. 
NO, I do not want changes into the underlying thinking where SysSettings provides *default* interpolation options.
But VEDA-FE was able to do that, while also allowing to use TFM_INS for updating any IE options in Base or SubRES.

So, do I understand correctly, that this is an intentional change in that thinking, and in the design of VEDA2?
Should it then be mentioned in "Things that are different"?
And perhaps need to clarify the statement "if updating with year=0 (without an actual seed value) then need to convert it to in INS"?

[AKanudia]

More things are coming back... didn't you and I discuss this "default" nature of SysSettings and that there should be a way for the "original" declarations to prevail?

I have updated the doc. Please let me know if things are in order now.

[Antti-L]

> didn't you and I discuss this "default" nature of SysSettings and that there should be a way for the "original" declarations to prevail?

We sure did, a long time ago. 
That's one reason I have been confused by the change in the thinking adopted for VEDA2. The goal of the design back then (for VFE) was that all the IE options defined in the TFM_UPD table in SysSettings should never override any options specified elsewhere. And so they truly represented default IE options.  I think it worked quite well in VEDA-FE.

In VEDA2, the default IE options are now defined in a TFM_MIG table instead. However, in VEDA2 there is a significant change in thinking, causing any IE options defined in TFM_INS tables also to be considered "default" options, while in VEDA-FE, default options could only be generated by TFM_UPD, i.e. only for any existing time-series instances. In VEDA-FE, TFM_INS thus worked in the normal way in SysSettings, which is no longer the case in VEDA2.

This change in VEDA2, where only default options can now be generated by both MIG and INS tables in SySettings, thus deviates from the original design, and that change was unexpected to me, and has been causing some confusion for a long time.

The online "Things that are different" section now makes the following statement, which I find incorrect or at least misleading:
> VEDA_FE used to generate interpolation option data from SysSettings irrespective of existing declarations in Base/SubRES/other scenario files.

This would be only true for TFM_INS, which was of course never used for defining default IE options in VEDA-FE, and I think it would very rarely be used for that purpose in VEDA2 either.  I think this statement should thus be corrected:

Suggestion: "VEDA_FE used to allow defining interpolation options in SysSettings also with TFM_INS tables, which could be used for generating IE options irrespective of existing declarations in Base/SubRES/other scenario files.  This is no longer possible in VEDA2. Veda2.0 ignores all options that have been defined in Base/SubRES."

Nonetheless, after these final clarifications, all my three questions have been well answered, thank you!

[AKanudia]

I am glad