Empty Group in FLO_SUM/FLO_FUNC/FLO_SHAR

[canismajoris]

Dear all, 

I have a question regarding an error I get from the QA_Check.log file. 
According this thread having "Empty group in FLO_SUM/FLO_FUNC/FLO_SHARE" means that some FLO_SUM/FLO_FUNC/FLO_SHAR has been specified but the process does not have the flows that are referred to in the parameter. 

I have this problem with commodities that are inputs to energy efficiency processes. the error appears in the QA_Check file as follows : 
*** Empty Group in FLO_SUM/FLO_FUNC/FLO_SHAR

 *01 WARNING       -     R=Mt           P=Rdw_DetH     CG=Bldg_DETPre09-Cei
 *01 WARNING       -     R=Mt           P=Rdw_DetH     CG=Bldg_DETPre09-Wal
 *01 WARNING       -     R=Mt           P=Rdw_DetH     CG=Bldg_DETPre09-Win
 *01 WARNING       -     R=Mt           P=Rdw_DetH     CG=Bldg_DETPre79-Cei
...

Although, the commodities Bldg_DETPre****** are well definied and linked the Rdw_DetH process. 

Any idea about the origin of any of this warning / error , and how we could deal with them ?  that would be really appreciated.

Many thanks, 
Mahmoud

[Antti-L]

Not easy to see the cause with so limited information. Can you possibly provide all the *.DD files and the *.RUN file for this run case (via Dropbox or equivalent; you could give the download link in a private message)?  

If you can provide these files, it will be easy to explain what is wrong there.

[canismajoris]

Of course, here is a link to the dropbox link containing DD & RUN files of the underlying study case:
https://www.dropbox.com/home/mahmoud%20m...Carbon.zip

Thank you Antti for your help,
Mahmoud

[Antti-L]

Ok, thanks for the files (was a huge ZIP file including many unnecessary files, but managed to download it).

The commodities Bldg_DETPre09-Cei, Bldg_DETPre09-Wal, etc, which trigger these errors, are not defined in the process topology.  The process topology includes only the following outputs for Rdw_DetH and Rdw_DetH-70:

'MT'.'Rdw_DetH'.'Bldg_DETPre45-Cei'.'OUT'
'MT'.'Rdw_DetH'.'Bldg_DETPre45-Wal'.'OUT'
'MT'.'Rdw_DetH'.'Bldg_DETPre45-Win'.'OUT'
'MT'.'Rdw_DetH'.'Bldg_DETPre69-Cei'.'OUT'
'MT'.'Rdw_DetH'.'Bldg_DETPre69-Wal'.'OUT'
'MT'.'Rdw_DetH'.'Bldg_DETPre69-Win'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre09-Cei'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre09-Wal'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre09-Win'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre79-Cei'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre79-Wal'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre79-Win'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre89-Cei'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre89-Wal'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre89-Win'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre99-Cei'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre99-Wal'.'OUT'
'MT'.'Rdw_DetH-70'.'Bldg_DETPre99-Win'.'OUT'

Therefore, the errors are caused by you defining FLO_EMIS for commodities not in the topology.  The "empty group" message indicates just that: The group used in the attribute has no members in the process topology.  Note that derived commodities used in FLO_EMIS are automatically added to the topology by TIMES only when they are of type ENV (so are "emissions"), as outputs.  But your Bldg* commodities are of type NRG, and it is considered too error-prone to add NRG commodities automatically into the topology, just on the basis of some parameter.

However, I see that VEDA nonetheless shows them in the topology, apparently because they are used as the derived commodity in FLO_EMIS. But I am personally very happy that VEDA does not automatically add them to the topology, because I really think that would be too error-prone for NRGs. Thus, you can just fix these modeling errors yourself by adding them to the topology yourself (or change the commodity type to ENV).

[canismajoris]

Thank you Antti for your clear answer. That helped me a lot; just added the underlying commodities to the process topology and the error disappeared.

[canismajoris]

Dear all, I still have problems with my model, especially when CO2 reduction constraint is quite binding (net emissions by the EOH). 

I proceed as follows: 
  1) run a BAU until 2050
  2) run another scenario in which CO2 constraint (0 target by 2050) and I fix the results to BAU up to 2020. 

By doing so, I got numerical difficulties. Cplex reports "Solution found but not proven optimal due to numerical difficulties". I followed some recommendations cited here in order to find out what constraints were causing infeasibilities. I also relaxed the eprhs parameter. 

I am struggling to grassp what does it mean that EQ_OBJFIX, EQ8OBJINV and EQ_OBJVAR are infeasible. I get tat these constraints are inherent ones but don't understand why aren't they feasible! Is it because some user constraints are infeasible. I also get some infeasibilities with some constraints that limit the activity of some processes (dwellings) to the installed capacity. 


Could anyone help clarifying these points ? I highly appreciate your help. 
Thanks in advance, 

here are my xplex options: 

bardisplay 1
scaind 0
rerun yes
iis yes
lpmethod 4
baralg 1
barcrossalg -1 (running with crossover increases the run time of my model)
barorder 0
THREADS=7
aggind 3
Names no
datacheck 2
feasopt 0
eprhs 1e-2

[Antti-L]

The screen capture you posted shows an extract of the Equation listing.  The "INFES" reported there are not related to the solution (or infeasibilities in the solution), they are just the initial equation levels calculated for information. So, with the initial levels of the variables, the EQE_CAPACT equations are infeasible.  That's quite normal.

If you have numerical difficulties, as the first thing to try I would suggest disabling the dummy imports (if you have enabled them):

* Disable dummy imports to improve numerical stability
 NCAP_START(R,'IMPNRGZ')=2200;
 NCAP_START(R,'IMPMATZ')=2200;
 NCAP_START(R,'IMPDEMZ')=2200;

Of course, disabling them requires that your model is in fact feasible.

[Antti-L]

See also this thread on the ETSAP Forum for several options that can be used for troubleshooting "Solution available but not proven optimal due to numerical difficulties":
https://iea-etsap.org/forum/showthread.php?tid=162

Nonetheless, getting rid of all dummy imports has in many cases proven to be the easiest and effective way of improving numerical stability.

[canismajoris]

Thank you Antti for your reply!! 

I was using PRC_FOFF between BOH and EOH on dummy imports. I switched to NCAP_START = 2200 for these dummies. Now I don't get the "solution found but not proven optimal" message but the model runs without converging and is scaled infeasible during all iterations. I activated IIS (on 1) in order to get the conflicting constraints but the problem is that the run never ends so that I can see where the problem comes from. 

I red here that IIS parameter is no longer available for Cplex v10 ore more recent versions and one should use conflictalg to find out what constraints are causing infeasibility if Cplex version is >= 10. 

here's the message I get from .LST file regarding my cplex options: 

Reading parameter(s) from "D:\VEDA\GAMS_WrkTIMES\ZeroCarbon\cplex.opt"
>>  bardisplay 1
>>  scaind 0
>>  rerun yes
>>  conflictdisplay 2
*** Error line 4: Unknown option "conflictdisplay"
>>  conflictalg 1
*** Error line 5: Unknown option "conflictalg"
>>  lpmethod 4
>>  baralg 1
>>  barcrossalg -1
>>  barorder 0
>>  THREADS=-1
>>  aggind 3
>>  Names no
>>  datacheck 2
>>  eprhs 1e-2

which means conflictdisplay and conflictalg are not supported by cplex. 


and here's a copied part from the .LST file regarding the iterations: 

Iteration: 297931    Scaled infeas =       2227848.856795
Elapsed time = 657.31 sec. (134720.27 ticks, 297931 iterations)
Iteration: 298662    Scaled infeas =       2226559.410346
Iteration: 299489    Scaled infeas =       2223998.707930
Iteration: 300478    Scaled infeas =       2209902.964034
Iteration: 301492    Scaled infeas =       2204511.255930
Iteration: 302487    Scaled infeas =       2200783.082545
Iteration: 303689    Scaled infeas =       2188626.991034
Elapsed time = 718.33 sec. (146230.58 ticks, 303689 iterations)
Iteration: 304885    Scaled infeas =       2185137.388240
Iteration: 306129    Scaled infeas =       2183047.766942
Iteration: 307253    Scaled infeas =       2169318.618399
Iteration: 308445    Scaled infeas =       2162373.622056
Elapsed time = 785.72 sec. (156622.63 ticks, 308445 iterations)
Iteration: 309524    Scaled infeas =       2146746.291152


My question is: is there any configuration of the cplex options that makes it easy to find out conflicting constraints without waiting for a run that might run infinitely? 

Thank you again for you support.

[canismajoris]

In addition, can I kindly ask how one could identify a variable or a constraint from its allocated number in the prompt ? 
As you can see in my cplex options above, I activated datacheck in order to get extensive details about the coefficients that may cause infeasibilities. But Cplex only reports the number of the constraints and not the constraints themselves. SO how can one identify those based on the messages from the prompt? 

Thanks in advance, 
Mahmoud

[Antti-L]

Well, I think your tests clearly demonstrate that you have a dummy import issue.
 
  1) With your original settings Cplex found a solution, but could not prove optimality;
  2) When you disabled dummy imports, Cplex was no longer able to find a solution.

It think this clearly suggests that you have dummy imports flows in the solution when running with your original settings. Therefore, the first thing to do is to eliminate those dummy imports; I think you should be able to see what they are in VEDA-BE, looking at the results from 1), and then you can fix your model accordingly.

If you want to see the names of constraints instead of numbers of constraints in the Cplex reports, please remove the setting names no from your Cplex.opt.

[canismajoris]

Thank you Antti for your answer!

I am trying to solve this issue on my hyge problem. I have lots of very small dummy flows into a bunch of commodities.
I have another question about the control of dummy variables under veda 2.0:
I tried to deactivate those from veda interface (from tools tab) and still get them when no scenario is used to eliminate them. Is it normal?

regards,
Mahmoud

[AKanudia]

The dummy import settings under Tools menu apply only for Sync operation. To be sure,you can start from scratch after deactivating them. But like Antti said, the model might be infeasible after that.

[Antti-L]

Like Amit said, your earlier attachment showed that the model was detected infeasible by Cplex when you disabled dummy imports:

> LP status(3): infeasible

Thus, there is not much point trying to run without dummy imports until you have corrected the model in that respect. But if you fix that issue, you would be much better off resolving also the issue with the numerical problems.

[canismajoris]

Update:
I managed to remove almost all dummy imports in the model, but still I have a question about allowing dummy variables to satisfy UCs.
I have defined a technology (stock of dwellings) in the base and set 200 year lifetime and defined PASTI for those dwellings. In a ReScen file, I defined a UC to force a demolition rate of x% of this technology per year.
I activated "retire" option and still got negative dummy variables in the output. the value of these negative dummy variables in year y is equal to the initial stock * demolition rate ^(y- start year of demolition) ; Which means that the technology isn't retiring (I verified afterwards) and that the model is using negative dummies to satisfy my UC.

Could you help understand why the technology isn't retiring?
Thanks in advance