Using GAMS Engine

[olexandr]

Dear all,

By any chance, has anyone already used GAMS Engine via Veda?
These are my observations after the first trial:

• run cancelled after a few minutes (model too large?)
  


• could not find how to specify my GAMS Engine credentials (default Veda credentials for GAMS Engine are always used?)
  

Cheers,
Olex

[AKanudia]

In the recent versions of Veda, you will find a GAMS Engine tab under Tools - User Options. You can provide your credentials and also control the size of the machine that is used for your runs. The default being too small is one possible explanation.

The other possibility is $170 warnings. I have noticed that the GAMS engine does not tolerate them, while they go through in local runs.

[olexandr]

Thanks a lot, Amit! I'll do some tests...

[Antti-L]

The other possibility is $170 warnings. I have noticed that the GAMS engine does not tolerate them, while they go through in local runs.

Hmm... that sounds strange to me. Could you by any chance provide a listing file from a TIMES model run that failed due to $170 warnings? I suppose you do get a listing file from the GAMS engine? Olex, if you encounter such, could you pass me the listing file?

[Antti-L]

Ok, as there were no responses, I tested it now myself with the VEDA default credentials.

It turned out that the statement "The other possibility is $170 warnings. I have noticed that the GAMS engine does not tolerate them, while they go through in local runs." was false.  In other words, the GAMS engine does in fact tolerate $170 warnings just fine.

The problem actually appears to be related to the TIMES setting $SET DATAGDX YES:  When using this setting, TIMES filters out any domain violations via a GDX file, such that GAMS works well even when having domain violations. This filtering, by writing and then re-reading all the input data via a GDX file works very well under local GAMS installations, and eliminates the problems related to domain violations.

However, for some reason, the GAMS engine shows jobs being canceled when using $SET DATAGDX YES and with domain violations in the input data. But no information whatsoever is disclosed by the GAMS engine about the reason for canceling these jobs. That is obviously badly lacking any transparency needed for troubleshooting the issue. It seems that reading TIMES model input data from a GDX file is not fully supported by the GAMS engine.

Therefore, for running with $170 warnings, you should remove the DATAGDX YES setting if using the GAMS engine, because reading the model input data from a GDX file does not seem to work under the GAMS engine, which is a disappointing limitation.

[AKanudia]

Apologies for not responding... I could not create this case quickly enough. And thanks a lot for investigating and getting to the bottom of this.

[olexandr]

Thanks Antti, thanks Amit!
I believe my runs got cancelled due to the model size, as I normally try not to leave domain violations or other errors unattended. Besides the runs went through with a smaller model (same model, but shorter modelling horizon).
I am now trying to get a large model solved using my own GAMS Engine credentials. As far as I can see, I should be able to get the logs out through the web interface.

[Fred_Fiand]

I tried to reproduce the issue that occurs with $170 warnings and DATAGDX=YES. Running such an example through Engine I get

Code:

[...]
--- ... gdxfilter.gms(10) 4 Mb
sh: 1: Syntax error: word unexpected (expecting ")")
--- ... gdxfilter.gms(15) 4 Mb 1 Error
[...]

in the log. Is this what you get as well?
This issue is not specific to Engine but to running under Linux in general. It is related to the following line in gdxfilter.gms

Code:

$ hiddencall sed "/^\(Scalar\|[^$(]*([^,]*)\|[^$].*empty *$\)/{N;d;}; /^\([^$]\|$\)/d; s/\$LOAD.. /\$LOADR /I" _dd_.dmp > _dd_.dd

I guess, trouble is how single/double quotes are interpreted under Linux (see e.g. here). Replacing double quotes by single quotes as below, I get the desired behavior under Linux.

Code:

$ hiddencall sed '/^\(Scalar\|[^$(]*([^,]*)\|[^$].*empty *$\)/{N;d;}; /^\([^$]\|$\)/d; s/\$LOAD.. /\$LOADR /I' _dd_.dmp > _dd_.dd

I also tested this through Engine. I manually registered a model with id 'latest_fix20221020' where I changed this particular line. I will leave this online for while, in case this is useful for further tests. Not sure though if/how Veda supports the selection of different registered TIMES versions. Note that when TIMES version x.y.z is released, we always register this new version with Engine under the id 'timesxyz' and also update the model with id 'latest' to the latest version. I guess by default Veda submits to the registered model with id 'latest'.

I hope this helps!

[Antti-L]

Thanks for looking into it.

> I get [...] in the log. Is this what you get as well?

As I said, I was getting nothing else but a message showing "run canceled" when submitting such a case to the GAMS engine from within VEDA. No information whatsoever about the reason: No log file, no listing file, nothing else but that message.  So, as I tried to explain, I could not see any way of troubleshooting the issue.

With your help we can now try what you suggested. But shouldn't the log file and listing file be automatically provided to the users?  I was getting those files back only when running without DATAGDX YES (i.e. when the run was not getting canceled).

[Fred_Fiand]

But shouldn't the log file and listing file be automatically provided to the users?

Yes. The Information exists on the Engine side. I'll check with Amit how to get those files to the VEDA users in case of a failed job.

Thanks for bringing this issue to our attention.

[olexandr]

Thanks a lot Fred!

I have a few more questions to Amit about Veda / GAMS Engine interaction. Specifically, I am wondering what would happen under the following scenarios when submitting jobs to GAMS Engine using the default veda user:
1) If a user cancels a run in Veda, would Veda kill the corresponding job at the Engine side?
2) If a user closes Veda (and e.g. shuts down the PC) would Veda check on the job status and import the results on the next launch?

I've also noticed that Veda deletes job logs and results from GAMS Engine after downloading them. Could this behaviour be made user controlled?

[VictorG]

Would it be possible to choose a calculation server other than gams engine?

I think the possibility of outsourcing calculations is useful in the case of large models. I usually perform calculations on another device with better specifications than the one veda is installed on. However, my workflow is not very optimal.

-Generate a file by selecting write dd only
-Send a copy of the file to my calculation server.
-start a simulation by clicking on vtrun.cmd
-post-process results with the vd file

Having the ability to send calculations to another machine directly from veda would automate the process a little.

If anyone has a similar workflow and has managed to automate it, I'd be interested to know how?