Efficiency and database sizes

[AKanudia]

Several users have raised this as a high priority issue in a recent survey conducted by ETSAP. I will put my initial comments here, and invite other users to report specific issues and make concrete suggestions on this thread, so that I can make a concrete proposal to ETSAP at the next meeting.

~~~~~~
I fully agree that ACCESS imposes a lot of limitations and creates efficiency issues, but even with all this, a LOT depends on how the model is configured in VEDA. We all know that there are multiple ways of doing things in VEDA. Most users facing serious efficiency issues should be able to improve the situation significantly by being more creative with the various options that VEDA offers already.

What we can do to improve the situation:
1. Users should report their particularly inefficient scenarios or tables on the forum, so that VEDA experts can suggest efficient alternatives. This could also result in fine tuning of VEDA-FE. We continuously enhance performance, but based almost entirely on cases that I encounter in my own work. User reports could expose new cases.
2. I could prepare a special demo model that demonstrates setting up a large model efficiently.
3. We certainly need to move to SQL server or some other database platform eventually. But remember that VEDA applications have to work on (humble) desktops and portables as well. We cannot adopt a platform that will not work on these.
~~~~~~

[Pernille.S]

For me it would be helpful with:
- a faster synchronization in VEDA_FE
- a faster import in VEDA_BE

I do not know how this can be implemented or of it feasible at all.

Often when importing large models into VEDA_BE, the program freezes and I need to restart it.

[wnijs]

Some years ago, we had a lot of problems with our 37 regions JRC-EU-TIMES, caused by Access limitations. Whenever you change something in an Access database, the size of the database grows and after compacting it goes back to the original size. Because of such behaviour, the limitation of 2GB can be reached quickly. Today we have much less problems and one reason seems to be that the newer versions of VEDA FE have a more intelligent and frequent (?) compacting of the databases.

Another important change is that we paid much more attention to how JRC-EU-TIMES is configured. The "Model Stats" Tool in VEDA FE is a first good start to detect very large Subres or scenario files. We are convinced that many scenario files in big models are possible candidates for a SepDb structure because no other scenario file is dependent on any datavalue. Many modellers might not be familiar with SepDb. For that reason, it would be a great help to share on this Forum an extended version of the advice Amit already shared within the ETSAP mailing list: (1) use DINS, (2) use SepDb and (3) keep SysSettings as light as possible.

On a similar note, Amit and I have been exchanging thoughts on multi-regions models. The idea was introduced to design the database so that there can be region specific and non-region specific data input. Region independent data would be applicable to all regions. This approach only requires 1 value rather than for example 37 in our case. We did a test with our newest JRC-EU-TIMES and counted 80% of the datavalues to be duplicate values after removing the region dimension (not including topology etc..). Regardless of the platform, such feature would tremendously improve the synchronisation speed and reduce the model's size. The implementation probably involves major efforts, that is to be further analysed.