11-07-2017, 06:56 PM
Some years ago, we had a lot of problems with our 37 regions JRC-EU-TIMES, caused by Access limitations. Whenever you change something in an Access database, the size of the database grows and after compacting it goes back to the original size. Because of such behaviour, the limitation of 2GB can be reached quickly. Today we have much less problems and one reason seems to be that the newer versions of VEDA FE have a more intelligent and frequent (?) compacting of the databases.
Another important change is that we paid much more attention to how JRC-EU-TIMES is configured. The "Model Stats" Tool in VEDA FE is a first good start to detect very large Subres or scenario files. We are convinced that many scenario files in big models are possible candidates for a SepDb structure because no other scenario file is dependent on any datavalue. Many modellers might not be familiar with SepDb. For that reason, it would be a great help to share on this Forum an extended version of the advice Amit already shared within the ETSAP mailing list: (1) use DINS, (2) use SepDb and (3) keep SysSettings as light as possible.
On a similar note, Amit and I have been exchanging thoughts on multi-regions models. The idea was introduced to design the database so that there can be region specific and non-region specific data input. Region independent data would be applicable to all regions. This approach only requires 1 value rather than for example 37 in our case. We did a test with our newest JRC-EU-TIMES and counted 80% of the datavalues to be duplicate values after removing the region dimension (not including topology etc..). Regardless of the platform, such feature would tremendously improve the synchronisation speed and reduce the model's size. The implementation probably involves major efforts, that is to be further analysed.
Another important change is that we paid much more attention to how JRC-EU-TIMES is configured. The "Model Stats" Tool in VEDA FE is a first good start to detect very large Subres or scenario files. We are convinced that many scenario files in big models are possible candidates for a SepDb structure because no other scenario file is dependent on any datavalue. Many modellers might not be familiar with SepDb. For that reason, it would be a great help to share on this Forum an extended version of the advice Amit already shared within the ETSAP mailing list: (1) use DINS, (2) use SepDb and (3) keep SysSettings as light as possible.
On a similar note, Amit and I have been exchanging thoughts on multi-regions models. The idea was introduced to design the database so that there can be region specific and non-region specific data input. Region independent data would be applicable to all regions. This approach only requires 1 value rather than for example 37 in our case. We did a test with our newest JRC-EU-TIMES and counted 80% of the datavalues to be duplicate values after removing the region dimension (not including topology etc..). Regardless of the platform, such feature would tremendously improve the synchronisation speed and reduce the model's size. The implementation probably involves major efforts, that is to be further analysed.