We are building a model where we wish to use flexible time-slice definition, where the used time-slices depend on need. We have built a small model that works fine with 48 time-slices (syncing and running), but we are not able to sync the same model with 8736 time-slices.
We get the following error message as shown below. What is causing this and how can it be solved? The time-slices are named 1 - 2 - ....- 8736.
I have also seen this problem several times in the past. I have understood that this error can be caused by either the temporary JET file(s) exceeding 2GB, or the amount of file locks being too high. The temporary file size problem would be difficult to address, but the file lock limit can be increased.
You could try editing the MaxLocksPerFile setting under the following Registry key:
Change the value for MaxLocksPerFile, setting the value to something greater than what it is (it should be 9500 by default). Years ago, I changed it myself to 20000, and since then my problems with this error have indeed been very rare, and so I think changing it made a difference. You can always change it back to what it was, if it does not help you.
On a side note, could you also share a bit of your approach of using "flexible time-slice definition, where the used time-slices depend on need". It seems something that may be of interest to many other users. You mention that the time-slices are named 1 - 2 - ....- 8736, and so are they all SEASONs, or perhaps a DAYNITE cycle?
Thank you for sharing your experience on this issue.
First, after changing the MaxLocksPerFile, the model actually synced. However, in this case some of the COM_FR to the end-use demand were not imported and I did not really understand why.
Nevertheless, today (with MaxLocksPerFile = 20 000), I get the same error twice in a row when syncing the model.
Do you think the naming of the scenarios will solve this problem? Now, I have only defined 1 – 2 – 3 ---…- 8736 daily time-slices (just to test), but will test if it works better with 52 weeks and 168 daily time-slices.
We are in a starting phase of developing a model that enables the use of a flexible time-slice definition. This is because we wish to run the model with a coarse time resolution and run the final version of model analysis with a finer temporal resolution. We also wish to use the flexible time-slice structure to test what is the necessary time-slice resolution for different types of analyses.
The naming of the time-slices is not final yet, but the basis is the production and demand profiles on an hourly level. If we use 8736 hours of a year, these hours will further be grouped into weeks and the weeks can be grouped into seasons. To implement this, we plan to gather all the production and demand profiles into one excel-sheet.
Sorry to hear that the problem persists.
Of course you could try still higher values for MaxLocksPerFile (the suggestions I have seen typically cite 200,000 as a value to try), but otherwise I think you could consider giving the test model to KanOrs to see if they can do anything about it, e.g. by checking that using a high number of timeslices does not lead to wasteful query executions.
Concerning the timeslice definition, to avoid any possible mix-up with the TIMES system sets using such integer labels, I would suggest not to use pure integer labels 1, 2,…, 8736 for the timeslices, but use some prefix like S1,…,S8736.
In addition, bear in mind that TIMES assumes by default that there are 365 days in a year, and that each DAYNITE cycle forms a representative day, i.e. 24 hours. Thus, if you have 24 equal length timeslices as a DAYNITE cycle under any parent timeslice, then TIMES assumes that they are all exactly one hour in length. Correspondingly, if you would have 8736 equal length timeslices as a DAYNITE cycle under ANNUAL, then TIMES assumes that they are all of 8760/(365*8736) hours = 9.9 seconds in length. So, if necessary, make sure that you override the default G_CYCLE(tslvl) appropriately.
have you found out anything new on the topic, so what exactly causes the error and how to solve it?
Unfortunately, I experience the same issue, but with VEDA-BE in this case:
(The error message is in German, but telling the same thing as the English one posted by Pernille; the given long integer is also equal.)
This error occures when I attempt to export flow variables (both VAR_FIn and VAR_FOut) into a CSV file of model runs with more than, ~1.1 million records.
Since the model consists of 192 timeslices, 8 periods, 30 transformation processes and 29 regions, such a number of records can easily occur (192*8*30*29=1.34e+6), especially when two or more attributes and several scenarios are being exported...
As Antti suggested, I set the MaxLocksPerFile setting to 200,000 but the error persists. I tried it on two different machines which both have plenty of free working memory.
So far, I can only help myself to split the number of records into several different files and append those manually via Notepad++ which can be quite cumbersome...
Depending on what you are doing with these CSV files (and how), the Aggregation options of VBE might make your life easier. See page 44 of TIMES doc Part V.
Basically, you can split with ENUM1/2/3 and you can aggregate the dimensions you don't need with COLLAPSE.
Thank you for this hint, Amit.
I tried this option for SCENARIO with ENUM1 and the tables get indeed generated for each scenario in the VEDA TABLES window.
However, when I click the "Export to CSV (as flat file)" button, the above mentioned error arises again
Since I use python scripts for automated post-processing, generation of indicators and a broad variety of plotting exercises (e.g. results onto geographical maps, ternary plots etc.), COLLAPSE is no option for me. I really need the data in an unmended form, just as the CSV export does.
Any further ideas?
Could you maybe provide an option to export the results directly from the underlying MS-Access databases without the need to process them through VBE?
13-03-2018, 06:34 PM (This post was last modified: 13-03-2018, 06:34 PM by Antti-L.)
For single scenarios, you can easily produce CSV files with the GDX2VEDA utility. Just make a new flows.vdd file containing the following: * TIMES GDX2VEDA Set Directives
[DataBaseName]
TIMES
[Dimensions]
Attribute attr
Commodity c
Process p
Period t
Region r
Vintage v
TimeSlice s
UserConstraint uc_n
[Options]
not-0 var_fin var_fout
[DataEntries]
* VEDA Attr GAMS - indexes -
VAR_FIn f_in r v t p c s
VAR_FOut f_out r v t p c s
Then you can run run gdx2veda as follows: gdx2veda <filename>.gdx flows.vdd <filename>.csv
GDX2VEDA.exe is located in the GAMS system folder. There is the GDX2VEDA.exe you have been using for generating the .vd files for BE. Get rid of that old version from 2005, which you for some reason now took into use. As you can see, you have not been using it (it does not work with your version of GAMS).
