Very long sync time

[kogexo]

Hello everyone,

I have been working on a spatially detailed model these days. I tried to synchronise a model with 100 or 277 zones and encountered some problems with sync time: after two hours generally, VEDA FE starts getting white and I have a Windows error message (AppHangB1) asking whether I should stop it or wait for it to respond.
   

Usually, if I wait long enough after VEDA freezing like that, the sync will finish and VEDA will go back to its normal state.

I tried with BY only at first and it took a few hours (can't remember how many). I then added the SubRES New Techs and it took 6 hours.
   

I then added a few constraints scenarios and it took 13 hours. The time have been increasing and I don't really understand why.

The databases files have shown an increasing size, up to 600 Mo whereas they end up at 180 Mo. Is it normal?
   

Have I been doing something wrong? Is 100 zones too many? Is it normal that even if my model is quite simple (you can see it on the stats on the ended sync), it takes so much time? Can you help me find a solution or should I try to reduce to less zones?

I use VEDA FE 4.5.8, Excel 2010, a lot of RAM (32 Gb I guess). I have no problem with my model when it has 19 zones (usually).

Thank you in advance for your help!
Matthieu

[Ravinder]

Hello everyone,

I have been working on a spatially detailed model these days. I tried to synchronise a model with 100 or 277 zones and encountered some problems with sync time: after two hours generally, VEDA FE starts getting white and I have a Windows error message (AppHangB1) asking whether I should stop it or wait for it to respond.


Usually, if I wait long enough after VEDA freezing like that, the sync will finish and VEDA will go back to its normal state.

I tried with BY only at first and it took a few hours (can't remember how many). I then added the SubRES New Techs and it took 6 hours.


I then added a few constraints scenarios and it took 13 hours. The time have been increasing and I don't really understand why.

The databases files have shown an increasing size, up to 600 Mo whereas they end up at 180 Mo. Is it normal?


Have I been doing something wrong? Is 100 zones too many? Is it normal that even if my model is quite simple (you can see it on the stats on the ended sync), it takes so much time? Can you help me find a solution or should I try to reduce to less zones?

I use VEDA FE 4.5.8, Excel 2010, a lot of RAM (32 Gb I guess). I have no problem with my model when it has 19 zones (usually).

Thank you in advance for your help!
Matthieu

Hi Matthieu,

First of all, you have to update the Veda_FE application. You are currently using a very old version of VEDA_FE.

The Current version of the VEDA_FE application is 4.5.828.

Kindly download it from the below link
http://licensing.kanors-emr.org/thanksdl.php

Please try to run your model after updating the application. After the installation, If you got any issue please let us know.

[kogexo]

Hi Matthieu,

First of all, you have to update the Veda_FE application. You are currently using a very old version of VEDA_FE.

The Current version of the VEDA_FE application is 4.5.828.

Kindly download it from the below link
http://licensing.kanors-emr.org/thanksdl.php

Please try to run your model after updating the application. After the installation, If you got any issue please let us know.

Thank you, I will try right now

[Ravinder]

Matthieu,

If updated VEDA_FE application stuck on splash screen while reading the supported tags... Kindly read the below post
http://forum.kanors-emr.org/showthread.php?tid=722&highlight=VedaTablesStructure

[kogexo]

Matthieu,

If updated VEDA_FE application stuck on splash screen while reading the supported tags... Kindly read the below post
http://forum.kanors-emr.org/showthread.php?tid=722&highlight=VedaTablesStructure

Is there a changelog of 828 compared to 8 somewhere?

[kogexo]

I launched it a dozen minutes ago and everything seems fine right now. See you hopefully in a few hours (or in 13 hours )!

[AKanudia]

Thank you very much for this post. I have always encouraged users to use this forum when they encounter such debilitating performance issues.

Number of regions alone is not important. What really matters is the total number of data points you are putting into Veda, and the templates/tables that are used to do so. Data explosion can happen along other dimensions too - processes, years and timeslices are typical candidates.

it is normal for database sizes to increase during processing and then shrink at the end, when databases are compacted. But 600/180 Mb is not a large database at all; the main problem is the number of BY templates.

The core philosophy of Veda is to not have to specify the same structural or numerical information multiple times. TIAM and PET models have a separate set of Base-year templates for each model region because different analysts were responsible for different regions when these models were created. In general, it would be much more efficient to have multiple (or all) regions in each base-year template, by sector. The very old DEMO model that comes with Veda has this structure. And whenever possible, you should use DINS tables in BY_trans or SubRES_Trans files to specify parameters.

If you upload sample tables from a couple regions, I can help you restructure the base-year templates.

[Antti-L]

Amit: Thanks for the good advice!

However, looking at the screenshot provided by kogexo, it seems that out of the 3 hours total sync time VEDA spends almost two hours in the phase "Transfer Data: [B-Y] + [SR]".  I think this means transferring the data from VFE_Work to ActiveDB, no?  How can transferring the data (<150 Mb) take such a long time, even if for each B-Y template there is a separate table to be transferred? And then it is also interesting that about one hour is spent in "VEDA To TIMES II", where I think the number of templates should no longer play any role?

[kogexo]

Hi everyone,

A few updates since yesterday. Upgrading to 4.5.828 did the work and I could sync my BY alone in a little bit more than 1h30 yesterday evening. To be sure, I tried again this morning with the whole BY, the SubRES and new scenarios files. It also worked and took almost 3h.

I managed to take a screenshot of the status before it finished (is it possible to have the detailed schedule of the steps after the sync finished?).

   

And here you have after the sync:

   



Thank you for your advice Amit, I will try in a few days to gather all 100 zones in one file per sector instead of 100 files (which is obviously larger to import). But I agree with Antti: the steps that seem to take much time are steps which shouldn't depend on my file structure anymore, e.g. VEDA To TIMES II took 1h in my last sync as you can see from the screenshots above.

Looking at the MDB files sizes, I am a little bit puzzled.

   

Do you want me to run another few tests? Maybe syncing the BY first, then the SubRES, then the Scenarios?

EDIT :
Sorry, I am confused, I forgot that the SYNC times were in SYNC_Tasks.log, here is the file for my model!

  SYNC_Tasks.txt (Size: 45.33 KB / Downloads: 5)

[Vikrant]

Matthieu, Can you check the written messages on the bottom of the VEDA application window screen when the application is stuck in the following tasks?

Tasks (I can understand 2nd is not easy to see but Ist will appear in a few minutes.)
1. Transfer Data: [B-Y] + [SR]
2. VEDA To TIMES II...

I have also attached an image for your reference about message positions.
   

[kogexo]

Matthieu, Can you check the written messages on the bottom of the VEDA application window screen when the application is stuck in the following tasks?

Tasks (I can understand 2nd is not easy to see but Ist will appear in a few minutes.)
1. Transfer Data: [B-Y] + [SR]
2. VEDA To TIMES II...

I have also attached an image for your reference about message positions.

I will try as soon as possible. I have been trying for the last few hours to solve a case (with only one zone for example) but I keep getting Unexpected termination of run. I'm looking into it!