We’ve learnt quite a bit about the intricacies of Excel; this page documents some of our more unusual findings. These comments refer to Excel 2007 unless we state otherwise.
Duplicate Cells in Ranges and Range Unions
Excel’s union operator can produce ranges with duplicate cells!? Chip Pearson describes the problem here; it is also discussed on Daily Dose of Excel (with some interesting code). We haven’t implemented Chip’s solution yet in OpenSolver as it may slow things down, but will do if it causes problems. I also realised that Excel allows ranges (such as for the decision variables) like “A1:A5,A2:A6″ that internally overlap; OpenSolver now internally fixes ranges like this when they occur using the following code:
Function MergeRangesCellByCell(ByVal r1 As Range, ByVal r2 As Range) As Range ' This merges range r2 into r1 cell by cell. ' This shoulsd be fastest if range r2 is smaller than r1 Dim result As Range, cell As Range result = r1 For Each cell In r2 result = Union(result, cell) Next cell MergeRangesCellByCell = result End Function Function RemoveRangeOverlap(ByVal r As Range) As Range ' This creates a new range from r which does not contain any multiple repetitions of cells ' This works around the fact that Excel allows range like "A1:A2,A2:A3", which has a .count of 4 cells ' The Union function does NOT remove all overlaps; call this after the union to If r.Areas.count = 1 Then RemoveRangeOverlap = r Exit Function End If Dim s As Range, a As Range, i As Integer s = r.Areas(1) For i = 2 To r.Areas.count If Intersect(s, r.Areas(i)) Is Nothing Then ' Just take the standard union s = Union(s, r.Areas(i)) Else ' Merge these two ranges cell by cell; this seems to remove the overlap in my tests, but also see http://www.cpearson.com/excel/BetterUnion.aspx ' Merge the smaller range into the larger If s.count < r.Areas(i).count Then s = MergeRangesCellByCell(r.Areas(i), s) Else s = MergeRangesCellByCell(s, r.Areas(i)) End If End If Next i RemoveRangeOverlap = s End Function
Named Ranges can contain cells from multiple sheets, but Ranges cannot
The built in Name Manager allows named ranges to be created that contain cells from multiple sheets. However, this “named range” cannot be turned into a range because a range must old only cells from a single sheet. As a consequence, all your decision cells must be on the same sheet (which does not have to be the active sheet).
Name Definitions not Localised
Solver stores its constraint LHS and RHS values as defined Names. Excel users have probably used Named Ranges, such as Test=$D$1. Names can store not only ranges, but also constants, such as Test=5.3, and formulae, such as Test=SQRT(2)*D4.
Solver stores its integer tolerance in a name solver_tol, with a value such as 0.05 meaning a 5% tolerance. Like all names, this is simply stored as a string that is NOT affected by regional settings, and so in VBA this must be interpreted using the Val() function which always uses the full stop, and not the comma, as the decimal point. We didn’t get this right in OpenSolver until version 1.3beta.
You can see this if you switch your regional settings to French, use Excel’s Name Manager to define a value such as Test = 0,25 (ie one quarter), and then in VBA’s immediate window type ? Range(“Test”). The output is 0.25.
Excel is consistent in this regard in that formulae are always stored using English settings. So, if if 1,23 is entered into A1 on a French system, ? Range(“A1”).value gives 1,23, but ? Range(“A1”).Formula gives 1.23.
VBA, however, is a bit more tricky. If you simply assign a string to a double in VBA, then that string is interpreted using the current locale, and so using this approach to get constants from names does not work. This means that this otherwise excellent code by Chip Pearson does not work on non-English systems. See OpenSolver’s code for an updated version.
Localisation and Evaluate
Excel has an evaluate function, allowing you to type the following into the immediate window:
? Evaluate(“1,3+1,6”)
Surprisingly, even if the system settings are set to French, this fails. However, typing
? Evaluate(“1.3+1.6”)
on a French system gives “2,9” (with a comma) as the output, showing again that formulae must always be internally in English. Fortunately, we use Evaluate to calculate values for formulae stored in defined names (which are always stored in US format), so this all “just works” for us.
Excellent Localisation Site
There is some excellent information on localisation issues available here. A copy of this in Word is also available, and the full book. Also look at: http://www.rondebruin.nl/international.htm.
RefEdit Control
To let the user select a range, we use either an InputBox or a RefEdit control. Both of these have bugs which make them challenging to use! For the input box, we note:
The RefEdit control has its own problem detailed here and here.
To let the user select cells on the spreadsheet, we now use the RefEdit control. This has caused us more grief than anything else! We believe the fundamental issue is that a RefEdit control implements its own simplified event loop whenever it has the focus, and so you should not do anything complicated while this RefEdit event loop is running. For example, closing a form or doing Application.Calculate within this event loop can crash Excel. Note also that VBA breakpoints are all ignored within this loop.
To help avoid the RefEdit bugs, we suggest you look at the Peltier Tech Blog which notes that
The RefEdit control has some strange and wonderful idiosyncrasies which must be considered when designing a dialog. There are (at least) four issues that are important to keep in mind.
- RefEdits must be placed directly on the UserForm itself. If you put a RefEdit in a frame or on a multipage, strange things will happen, including bizarre Excel crashes.
- RefEdits must not be used on modeless forms. RefEdits on modeless forms will result in bizarre Excel crashes.
- RefEdit event procedures should be avoided. RefEdit events do not behave reliably, and they may result in VBA errors which are difficult to debug.
- References to RefEdits must be removed. When a UserForm is added to a VB project, Excel adds a reference to the Microsoft Forms 2.0 Object Library. This reference is required for proper operation of the UserForms in your project. To see what references your project has, select the project in the Project Explorer, then select References from the Tools menu. [We have not needed to do this.]
- When a RefEdit is added to a UserForm, Excel sometimes adds a reference to the Ref Edit Control. This was fine in earlier versions of Excel, but there appear to be incompatibilities in recent versions of the RefEdit library. These problems often are associated with an Unspecified but Painfully Frustrating Error.
- It is also helpful to process the workbook or add-in’s code through an earlier version of Excel. I use a VM with Office 2000 installed on it for this purpose. Open Excel 2000, open the workbook or add-in, run Rob Bovey’s free Code Cleaner application, then compile the code and save the file.
- It may also be necessary to remove RefEdit.exd files from a user’s computer if problems occur and unchecking the reference doesn’t fix them. these exd files are temporary files that contain instructions Excel uses to build RefEdits, and the exd may conflict with the RefEdits in the workbook. If this doesn’t work, the next step is to run Detect and Repair on Excel, then reinstall Office, then sacrifice the computer at the next full moon.
After a day of anguish, we can also add:
- If you are enabling/disabling controls on a form, make sure that the last item you enable is not a RefEdit. Our code that was doing this would crash Excel on the second time around when it called Application.Calculate.
- We can now be a bit more specific, thanks in part to this comment on tab-stops. If a refedit control has a .TabStop value, then when another control is disabled, the focus can move to the RefEdit (as the next item in the tab-stop sequence). This means the RefEdit grabs the focus, and in doing so, pauses the VBA execution. This can be avoided by removinbg the tab stops from the RefEdits, or managing the focus to stop RefEdits getting focus unexpectedly (by, for example, when disabling a control, first setting the focus to an item that is not being disabled).
There are some useful RefEdit tips here.
See also this knowledge base article for Excel 2002: XL2002: VBA Error on Macro Startup If You Open Multiple Excel Instances with RefEdit Control.
We have found that the VBA debugger is not good at tracking events associated with RefEdit’s. For example, RefEdit events may fire and run ok, but without triggering any breakpoints set in the event handler. We have seen errors get thrown (and a dialog appear on the screen), even though our event handler has an On Error handlers; On Error handlers (and errors in general) are perhaps best avoided. (Given the tricks the RefEdits do, it is likely that they run their own main event loop which allows them to process clicks in Excel windows. We suspect that differences between this main event loop standard VBA main event loop leads to things operating differently in subtle but dangerous ways!)
This page suggests that you trace the refedit’s exit event (by debug.print, not breakpoints), and states that it MUST occur before you hide or close your userform. The following code is given to help ensure this:
Private Sub UserForm_QueryClose(ByVal Cancel As Integer, ByVal CloseMode As Integer) If Me.ActiveControl Is Me.RefEdit1 Then Me.Frame1.SetFocus() DoEvents() End If Cancel = True Me.Hide() End Sub
The source for a C#.net RefEdit replacement is available on Gabhan Berry’s MSDN Excel Programming blog.
The RefEdit.Text returns, as might be expected, the text associated with a range. For ranges with multiple areas, Excel uses a “,” as the separator in English locales, and a “;” in German, for example. You can see this is you switch your “Current Format” to German, and type the following into VBA’s immediate window:? range(“A1,B4”).AddressLocal
$A$1;$B$4 <- Note Excel’s AddressLocal has replaced the , by a ;
Function IsAmericanNumber(ByVal s As String, Optional ByVal i As Integer = 1) As Boolean ' Check this is a number like 3.45 or +1.23e-34 ' This does NOT test for regional variations such as 12,34 ' This code exists because ' val("12+3") gives 12 with no error ' Assigning a string to a double uses region-specific translation, so x="1,2" works in French ' IsNumeric("12,45") is true even on a US English system (and even worse, eg IsNumeric("($1,23,,3.4,,,5,,E67$)")=True! See http://www.eggheadcafe.com/software/aspnet/31496070/another-vba-bug.aspx) Dim MustBeInteger As Boolean, SeenDot As Boolean, SeenDigit As Boolean MustBeInteger = i > 1 ' We call this a second time after seeing the "E", when only an int is allowed IsAmericanNumber = False ' Assume we fail If Len(s) = 0 Then Exit Function ' Not a number If Mid(s, i, 1) = "+" Or Mid(s, i, 1) = "-" Then i = i + 1 ' Skip leading sign For i = i To Len(s) Select Case Asc(Mid(s, i, 1)) Case Asc("E"), Asc("e") If MustBeInteger Or Not SeenDigit Then Exit Function ' No exponent allowed (as must be a simple integer) IsAmericanNumber = IsAmericanNumber(s, i + 1) ' Process an int after the E Exit Function Case Asc(".") If SeenDot Then Exit Function SeenDot = True Case Asc("0") To Asc("9") SeenDigit = True Case Else Exit Function ' Not a valid char End Select Next i ' i As Integer, AllowDot As Boolean IsAmericanNumber = SeenDigit End Function
Gaghan Berry has a good (but old, around 2008) blog on programming Excel, part of the Microsoft MSDN site. We are also a big fan of Chip Pearsons’s Excel site. There are some very interesting projects (including source code) showing XLL add-in examples, VBA COM code and C# add-ins at Codematic.com.
Compiling CBC v2.7.6 – our experiences
To compile CBC.exe ready for use with OpenSolver, you need to download the latest zip CBC source code. Our experiences below come from compiling version 2.7.6. We tried compiling under the free version of Visual Studio 2010, using the “v10” folder in the CBC download. This compiled successfully, but did not appear to produce a CBC.exe file. Instead, we compiled using a full copy of Visual Studio v9 using the “v9” folder in the CBC download (Cbc-2.7.6Cbc-2.7.6CbcMSVisualStudiov9), working with the default project “cbcExamplesSample2”. This generated multiple errors, but did produce a CBC.exe file in the “Release” folder (after switching from Debug to Release). We then re-compiled after changing the properties for all the projects (ie with them all selected) to set the “C/C++ : Code Generation : Runtime Library” to “Multi-Threaded” (so we don’t need to distribute any DLL’s), and “General : Common Language Runtime support” to “No common language runtime support”. To check that there are no dependencies, you can use Dependency Walker (depends.exe) to open CBC.exe, and check that it only requires Kernel32.dll instead of all the library files you normally see.
AddConstraint – model building using Application.Run with OpenSolver API
(Applies to Excel 2016 – other versions may differ)
When calling OpenSolver API subs and functions with the Application.Run
method, normal VBA conventions about the scopes of subs and functions do not always apply. According to an article from wellsr and a StackOverflow question, Application.Run
is able to run both ‘Public’ and ‘Private’ subs.
In OpenSolver.xlam
, the sub AddConstraint
is defined in two places – in the OpenSolverAPI module and the SolverFileNL module. Its definition in the API module is Public
and in SolverFileNL it is Private
.
However, if AddConstraint
is called using Application.Run
, it seems that these scopes are ignored and there are effectively conflicting or duplicate definitions for the sub – it is now ambiguous as to whether OpenSolverAPI.AddConstraint
or SolverFileNL.AddConstraint
should be run.
Thus, in order to build up a model using the OpenSolver API while using Application.Run
, the following function call should be made e.g.:
Application.Run "OpenSolver.xlam!OpenSolverAPI.AddConstraint", LHSRange, rel, RHSRange
instead of calling:
Application.Run "OpenSolver.xlam!AddConstraint", LHSRange, rel, RHSRange
if there are a lot of if statements the problem is non linear.. the only open solver engine that goes around this is nomad, but the number of variables it can handle (before degrading performance) is limited.
Thanks for the quick reply. Andrew
Hi all, very interesting site!! I have a question, because I’m not so expert in all these tools. I have a huge excel with many formulas (few thousands of variables), in some cells (as variables) we have some binary entries and some integer , on other different cells calculated through excel formulas and on the variables we have the constraints. Most of the excel functions are not linear (“IF”, etc.). I tried to use bonmin and couenne but I have problems with the parsing. I changed many formulas (avoiding volatiles) and it went on and stopped again and finally I realize that also “IF” with arguments is a problem and I have many of them. Have you any suggestion for me to use the Opensolver?
Thanks a lot!!
Michele