Slow import time in VEDA-FE

[kramea]

Hi,

I have been facing this issue with VEDA-FE when I am doing the "start from scratch" for some of the large files I have. The import time is really large for these, sometimes even 10-12 hours. I understand it's because VEDA is converting these files to ASCII files for future use. Is there any way or shortcut to cut down on this time?

Thanks,
Kalai

[AKanudia]

That's too long. Lets schedule a web meeting to a look at your model.

But here are some general guidelines to keep SYNC times in check:

• SYNC efficiency is something we improve continuously, as our own models grow bigger and more complex. So, you should use the latest version of VEDA.
• Some users (who have UC-heavy models) should see a significant improvement in UC processing in the update that will be released next week.
• Start from scratch was needed quite often till a few years ago, but now it should be done only if you change the region/sector/timeslice definition or end up with corrupted databases. VEDA is able to reimport quite efficiently starting from any state.
• Interpolation options should be used to avoid injecting dense timeseries data.
• If the post-SYNC database sizes are above 1.2 GB, then the model should be broken up and used with the attach model feature (to be released soon).

[kramea]

Thanks Amit! I reduced the size of the model for now, and following up on the previous question, I get a CPLEX error now "1001: Out of Memory", when I run the model.

I googled the error, and tried the 'conserve memory on' toggle in the CPLEX options, and this solves the problem for this model. However when I progressively increase the size of the model, I hit the same error at some point. 

Do you think it's because of the computer RAM size or it's the limitations of CPLEX?

Thanks!
Kalai

[AKanudia]

This is a RAM issue.

did you succeed in reducing the import times significantly? If yes, please mention things that helped.

[kramea]

Thanks Amit. 

I didn't solve the import time issue. I reduced the model size quite a bit for easier analysis, and even with this I have been getting the CPLEX memory issue. Will look into the RAM options. 

Thanks!
Kalai