I have come across an issue that I am finding a hard time troubleshooting. It might have an easy fix to it but I haven’t found it so far.
I am running my TIMES model in a couple configs: ·Config A (scenario name: iemm_restart~0003): This configuration simply runs the model from Veda. ·Config B (scenario name: iemm_restart~0003_230713_130313): This configuration generates a restart folder, uses a Python wrapper to add data to the problem via the AddData.Gms file, and then executes the .CMD file. The AddData.gms file updates solar and wind capex and fixed O&M among other things.
The issue I am seeing is that config B leads to spiking power prices around 2050 and then super low prices around 2070 in most regions. This does not happen in config A.
I have compared VAR_NcapR for power capacities in both configs and found that solar and wind lcoes are higher in 2050 in config A relative to config B. However, the capex and fixed o&m of solar and wind are not that different in both scenarios. I have also compared fossil fuel prices, demands, and other factors, but I have not been able to identify any discrepancies that would explain the difference in power prices.
I have added reproducible cases to Dropbox here: https://www.dropbox.com/scl/fo/mb35pxqgkx0ucp2r22xk9/h?rlkey=0zlfaqdnxvywpyiz41ua22w0b&dl=0. Please let me know if you would like to have access to these for troubleshooting purposes.
I would be grateful if you could take a look at this problem and let me know if you have any ideas about what might be going on.
Thank you in advance for your help.
Sincerely,
Mahmoud
13-07-2023, 10:12 PM (This post was last modified: 13-07-2023, 10:26 PM by Mahmoud.)
Thank you for suggesting that, Amit! I did, the only difference there is TLIFE of some power pricesses, which I voluntarily updated but isn't causing the issue with prices. Does AddData.gms add data directly to the datagdx file?
13-07-2023, 10:15 PM (This post was last modified: 13-07-2023, 11:02 PM by Antti-L.)
The model statistics are notably different, can you explain why:
Without ADDATA:
-----------------------
MODEL STATISTICS
BLOCKS OF EQUATIONS 108 SINGLE EQUATIONS 2,309,645
BLOCKS OF VARIABLES 17 SINGLE VARIABLES 2,594,240 188 projected
NON ZERO ELEMENTS 33,151,586
-----------------------
With ADDATA:
-----------------------
MODEL STATISTICS
BLOCKS OF EQUATIONS 108 SINGLE EQUATIONS 2,244,473
BLOCKS OF VARIABLES 28 SINGLE VARIABLES 2,527,385 188 projected
NON ZERO ELEMENTS 29,725,726
-----------------------
I took a look at ADDATA, and I only see some cost and a few demand attributes there. Do they explain the reduction in the number of constraints and non-zeros? I don't think so...
Thanks for pointing this out, Antti! I have yet to check that. Unfortunately, I don't have any explanation for this. the difference between config A and config B is that for config B, I am using the restart option in Veda to generate the .g00 file and updating demands and costs via AddData.gms. Is there a way to have a look at the problem's matrix by modifying cplex options?
Sure, you can use the writelp option to get all the model equations written out. But it would be a BIG file...
I can also see that you are using GAMS 24.2 (a ten years old version) for compiling the restart file, and then using GAMS 43.4 for running it with ADDATA. I would not recommend such. Compiling of the TIMES code with a GAMS version so much older might not result in the compiled GAMS code being correctly executed with a much newer GAMS version. I would suggest trying to use the same GAMS version for both.
Perhaps the reason for that might be using a different server machine for running the second case, or...?
13-07-2023, 11:23 PM (This post was last modified: 13-07-2023, 11:50 PM by Antti-L.)
(13-07-2023, 09:26 PM)AKanudia Wrote: compare the datagdx from the two runs?
Ha! The data gdx files were almost identical in size, which fooled me.
But indeed, there were important differences, related to NCAP_TLIFE: The run without addata used 200 years' lifetimes for numerous power plants, including solar PV and wind turbines. So, no wonder the results were different, and that may also explain the differences in the model size as well.
Ok, so it seems possible that running a model compiled with a 10 years old GAMS might still be executed correctly with a new GAMS (although I am a bit surpised if that is true). Possibly GAMS looks up the version of the compiled code from the restart file and then executes the code taking that information into account, which would be great for maintaining compatibility.
(13-07-2023, 10:48 PM)Antti-L Wrote: Sure, you can use the writelp option to get all the model equations written out. But it would be a BIG file...
I can also see that you are using GAMS 24.2 (a ten years old version) for compiling the restart file, and then using GAMS 43.4 for running it with ADDATA. I would not recommend such. Compiling of the TIMES code with a GAMS version so much older might not result in the compiled GAMS code being correctly executed with a much newer GAMS version. I would suggest trying to use the same GAMS version for both.
Perhaps the reason for that might be using a different server machine for running the second case, or...?
Indeed, my local machine has an old gams license that only supports gams 24.2 or older. And for config B, I comile the code on my local machine using that old version and do run the model on GAMS engine. That's why you see different GAMS versions used for compilation and execution. I will renew my GAMS version and retry running the model.
(13-07-2023, 10:48 PM)Antti-L Wrote: Sure, you can use the writelp option to get all the model equations written out. But it would be a BIG file...
I can also see that you are using GAMS 24.2 (a ten years old version) for compiling the restart file, and then using GAMS 43.4 for running it with ADDATA. I would not recommend such. Compiling of the TIMES code with a GAMS version so much older might not result in the compiled GAMS code being correctly executed with a much newer GAMS version. I would suggest trying to use the same GAMS version for both.
Perhaps the reason for that might be using a different server machine for running the second case, or...?
GAMS confirmed they ensure all new GAMS versions are able to read save files generated by older gams systems. One question I have on this is whether the TIMES code v4.7.0 uses any GAMS feature that was introduced after 24.2. I am working towards renewing my license but just wanted to confirm on that.
14-07-2023, 04:06 PM (This post was last modified: 14-07-2023, 04:12 PM by Mahmoud.)
(13-07-2023, 11:23 PM)Antti-L Wrote:
(13-07-2023, 09:26 PM)AKanudia Wrote: compare the datagdx from the two runs?
Ha! The data gdx files were almost identical in size, which fooled me.
But indeed, there were important differences, related to NCAP_TLIFE: The run without addata used 200 years' lifetimes for numerous power plants, including solar PV and wind turbines. So, no wonder the results were different, and that may also explain the differences in the model size as well.
Ok, so it seems possible that running a model compiled with a 10 years old GAMS might still be executed correctly with a new GAMS (although I am a bit surpised if that is true). Possibly GAMS looks up the version of the compiled code from the restart file and then executes the code taking that information into account, which would be great for maintaining compatibility.
On this, I should've shared the right data. My bad. I've just updated the dropbox data I shared earlier (https://www.dropbox.com/scl/fo/mb35pxqgkx0ucp2r22xk9/h?rlkey=0zlfaqdnxvywpyiz41ua22w0b&dl=0). Now, the configs A and B have the exact same datagdx, model statistics are slightly different (blocks of equations??). I wonder if updating costs and demands via AddData adds more blocks of variables to the model and how this would impact the results.
------------------------
config A
------------------------
MODEL STATISTICS
BLOCKS OF EQUATIONS 108 SINGLE EQUATIONS 2,244,473
BLOCKS OF VARIABLES 17 SINGLE VARIABLES 2,527,385 188 projected
NON ZERO ELEMENTS 29,725,726
------------------------
Config B
------------------------
MODEL STATISTICS
BLOCKS OF EQUATIONS 108 SINGLE EQUATIONS 2,244,473
BLOCKS OF VARIABLES 28 SINGLE VARIABLES 2,527,385 188 projected
NON ZERO ELEMENTS 29,725,726
14-07-2023, 04:53 PM (This post was last modified: 14-07-2023, 08:47 PM by Antti-L.)
(14-07-2023, 03:35 PM)Mahmoud Wrote: GAMS confirmed they ensure all new GAMS versions are able to read save files generated by older gams systems. One question I have on this is whether the TIMES code v4.7.0 uses any GAMS feature that was introduced after 24.2. I am working towards renewing my license but just wanted to confirm on that.
Ok, that would be great, but I am not sure they understood the nature of the restart files in this case. There are basically two kinds of restart files: Those that do not have any active code remaining but only GAMS data structures, and those that have active compiled code as well. Your restart files are of the latter type: They include the whole TIMES code, compiled by the old GAMS version, which is getting executed only when the restart file is read. Surely, over time GAMS may improve compilation of the code, and that would easily create compatibility issues. However, as I already mentioned, it is of course possible that those compatibility issues have indeed been addressed, by using version information stored in the restart file. Hopefully that is the case.
Yes, TIMES uses some features introduced after 24.2, but those are used only if detected to be supported at compile time, and therefore that creates no problem here.
(14-07-2023, 04:06 PM)Mahmoud Wrote: Now, the configs A and B have the exact same datagdx, model statistics are slightly different (blocks of equations??). I wonder if updating costs and demands via AddData adds more blocks of variables to the model and how this would impact the results.
Ok, that's an interesting difference, which, however, looks purely cosmetic to me. In other words, the number of blocks reported should not have an impact here, unless it is actually caused by execution errors due to compatibility issues between the GAMS versions. It would be interesting to know why more variable "blocks" have been listed in the second case, but I am pretty sure they are all empty blocks.
Got it. Thanks for your detailed answer, Antti! It's super helpful to know that the added blocs could be empty. I will let you know if updating my license solves the issue.
Quick update: for config B, instead of compiling with v24.2, adding data through ADDDATA.gms and running with v43.4, I created a suppxls scen file where I added the data I want to update (demands and prices) and ran the model from Veda (with the restart option deactivated). I then got my power prices to behave properly (no weird spikes around 2050).
This leads me to think that either there is an error instigated by the difference in versions used for compilation and execution or I am incorrectly passing the data to AddData.gms.
14-07-2023, 07:43 PM (This post was last modified: 14-07-2023, 07:45 PM by Antti-L.)
Well, in the first place, I see that you do have an oversight in your ADDDATA.gms:
You would need to use $ONEPS, like in all VEDA-generated DD files, to get the zero values taken into account. Could you please try that, and see if the results are then the same as with your suppxls scen.
But if that still does not help, then there is indeed something to investigate...
