Programming

We’ve learnt quite a bit about the intricacies of Excel; this page documents some of our more unusual findings. These comments refer to Excel 2007 unless we state otherwise.

Duplicate Cells in Ranges and Range Unions
Excel’s union operator can produce ranges with duplicate cells!? Chip Pearson describes the problem here; it is also discussed on Daily Dose of Excel (with some interesting code). We haven’t implemented Chip’s solution yet in OpenSolver as it may slow things down, but will do if it causes problems. I also realised that Excel allows ranges (such as for the decision variables) like “A1:A5,A2:A6″ that internally overlap; OpenSolver now internally fixes ranges like this when they occur using the following code:

Function MergeRangesCellByCell(ByVal r1 As Range, ByVal r2 As Range) As Range
    ' This merges range r2 into r1 cell by cell.
    ' This shoulsd be fastest if range r2 is smaller than r1
    Dim result As Range, cell As Range
    result = r1
    For Each cell In r2
        result = Union(result, cell)
    Next cell
    MergeRangesCellByCell = result
End Function

Function RemoveRangeOverlap(ByVal r As Range) As Range
    ' This creates a new range from r which does not contain any multiple repetitions of cells
    ' This works around the fact that Excel allows range like "A1:A2,A2:A3", which has a .count of 4 cells
    ' The Union function does NOT remove all overlaps; call this after the union to
    If r.Areas.count = 1 Then
        RemoveRangeOverlap = r
        Exit Function
    End If
    Dim s As Range, a As Range, i As Integer
    s = r.Areas(1)
    For i = 2 To r.Areas.count
        If Intersect(s, r.Areas(i)) Is Nothing Then
            ' Just take the standard union
            s = Union(s, r.Areas(i))
        Else
            ' Merge these two ranges cell by cell; this seems to remove the overlap in my tests, but also see http://www.cpearson.com/excel/BetterUnion.aspx
            ' Merge the smaller range into the larger
            If s.count < r.Areas(i).count Then
                s = MergeRangesCellByCell(r.Areas(i), s)
            Else
                s = MergeRangesCellByCell(s, r.Areas(i))
            End If
        End If
    Next i
    RemoveRangeOverlap = s
End Function

 

Named Ranges can contain cells from multiple sheets, but Ranges cannot
The built in Name Manager allows named ranges to be created that contain cells from multiple sheets. However, this “named range” cannot be turned into a range because a range must old only cells from a single sheet. As a consequence, all your decision cells must be on the same sheet (which does not have to be the active sheet).

Name Definitions not Localised
Solver stores its constraint LHS and RHS values as defined Names. Excel users have probably used Named Ranges, such as Test=$D$1. Names can store not only ranges, but also constants, such as Test=5.3, and formulae, such as Test=SQRT(2)*D4.

Solver stores its integer tolerance in a name solver_tol, with a value such as 0.05 meaning a 5% tolerance. Like all names, this is simply stored as a string that is NOT affected by regional settings, and so in VBA this must be interpreted using the Val() function which always uses the full stop, and not the comma, as the decimal point. We didn’t get this right in OpenSolver until version 1.3beta.

You can see this if you switch your regional settings to French, use Excel’s Name Manager to define a value such as Test = 0,25 (ie one quarter), and then in VBA’s immediate window type ? Range(“Test”). The output is 0.25.

Excel is consistent in this regard in that formulae are always stored using English settings. So, if if 1,23 is entered into A1 on a French system, ? Range(“A1”).value gives 1,23, but ? Range(“A1”).Formula gives 1.23.

VBA, however, is a bit more tricky. If you simply assign a string to a double in VBA, then that string is interpreted using the current locale, and so using this approach to get constants from names does not work. This means that this otherwise excellent code by Chip Pearson does not work on non-English systems. See OpenSolver’s code for an updated version.

Localisation and Evaluate
Excel has an evaluate function, allowing you to type the following into the immediate window:

? Evaluate(“1,3+1,6”)

Surprisingly, even if the system settings are set to French, this fails. However, typing

? Evaluate(“1.3+1.6”)

on a French system gives “2,9” (with a comma) as the output, showing again that formulae must always be internally in English. Fortunately, we use Evaluate to calculate values for formulae stored in defined names (which are always stored in US format), so this all “just works” for us.

Excellent Localisation Site
There is some excellent information on localisation issues available here. A copy of this in Word is also available, and the full book. Also look at: http://www.rondebruin.nl/international.htm.

RefEdit Control
To let the user select a range, we use either an InputBox or a RefEdit control. Both of these have bugs which make them challenging to use! For the input box, we note:

It may be tempting to use type 8 to indicate a range object, but there is an obscure glitch that causes it to fail when the worksheet contains conditional formatting conditions that use formulas in their definitions. Fortunately using Type 0 to specify a formula works just fine. (From here)

The RefEdit control has its own problem detailed here and here.

To let the user select cells on the spreadsheet, we now use the RefEdit control. This has caused us more grief than anything else!  We believe the fundamental issue is that a RefEdit control implements its own simplified event loop whenever it has the focus, and so you should not do anything complicated while this RefEdit event loop is running. For example, closing a form or doing Application.Calculate within this event loop can crash Excel. Note also that VBA breakpoints are all ignored within this loop.

To help avoid the RefEdit bugs, we suggest you look at the Peltier Tech Blog which notes  that

The RefEdit control has some strange and wonderful idiosyncrasies which must be considered when designing a dialog. There are (at least) four issues that are important to keep in mind.

  • RefEdits must be placed directly on the UserForm itself. If you put a RefEdit in a frame or on a multipage, strange things will happen, including bizarre Excel crashes.
  • RefEdits must not be used on modeless forms. RefEdits on modeless forms will result in bizarre Excel crashes.
  • RefEdit event procedures should be avoided. RefEdit events do not behave reliably, and they may result in VBA errors which are difficult to debug.
  • References to RefEdits must be removed. When a UserForm is added to a VB project, Excel adds a reference to the Microsoft Forms 2.0 Object Library. This reference is required for proper operation of the UserForms in your project. To see what references your project has, select the project in the Project Explorer, then select References from the Tools menu. [We have not needed to do this.]
  • When a RefEdit is added to a UserForm, Excel sometimes adds a reference to the Ref Edit Control. This was fine in earlier versions of Excel, but there appear to be incompatibilities in recent versions of the RefEdit library. These problems often are associated with an Unspecified but Painfully Frustrating Error.
  • It is also helpful to process the workbook or add-in’s code through an earlier version of Excel. I use a VM with Office 2000 installed on it for this purpose. Open Excel 2000, open the workbook or add-in, run Rob Bovey’s free Code Cleaner application, then compile the code and save the file.
  • It may also be necessary to remove RefEdit.exd files from a user’s computer if problems occur and unchecking the reference doesn’t fix them. these exd files are temporary files that contain instructions Excel uses to build RefEdits, and the exd may conflict with the RefEdits in the workbook. If this doesn’t work, the next step is to run Detect and Repair on Excel, then reinstall Office, then sacrifice the computer at the next full moon.

After a day of anguish, we can also add:

  • If you are enabling/disabling controls on a form, make sure that the last item you enable is not a RefEdit. Our code that was doing this would crash Excel on the second time around when it called Application.Calculate.
  • We can now be a bit more specific, thanks in part to this comment on tab-stops. If a refedit control has a .TabStop value, then when another control is disabled, the focus can move to the RefEdit (as the next item in the tab-stop sequence). This means the RefEdit grabs the focus, and in doing so, pauses the VBA execution. This can be avoided by removinbg the tab stops from the RefEdits, or managing the focus to stop RefEdits getting focus unexpectedly (by, for example, when disabling a control, first setting the focus to an item that is not being disabled).
In OpenSolver, Excel seems to crash most often when users press <return> instead of clicking on a button (eg on the Continue button), or when the user presses <escape> instead of clicking Cancel (which we now disable). To try to catch these, we have made sure that the form is not closed while a RefEdit control has the focus, and put in lots of DoEvents around anything involving a RefEdit control. These checks seem to work, mostly, but the RefEdit is still somewhat unreliable.
In version 1.3beta, we no longer associate pressing the return key with the next Continue button. When this was allowed (eg in version 1.2 beta), selecting a blank objective cell, and then pressing return caused an error dialog to appear (even tho’ there was an On Error top catch this error) and also caused a floating RefEdit dialog to show; this sometimes then resulted in crashes. Debugging this was difficult as breakpoints in this code were ignored, perhaps because RefEdit was confusing the debugger. (I suspect that RefEdit runs its own event loop to do its clever stuff, and this causes all these problems.) We do, however, still allow Escape to ‘press’ the Cancel button (via Cancel=true for this button); this code again runs without stopping at breakpoints, but does not seem to cause crashes.
Note also that, as detailed here, RefEdit does not fire the AfterUpdate and BeforeUpdate events, and setting the .Font property of cells (eg to colour selected cells blue) on the Enter or Exit events crashes Excel when the focus leaves the RefEdit control. Both the .Text and .Range properties of the RefEdit give the range selected as a string.

There are some useful RefEdit tips here.

See also this knowledge base article for Excel 2002: XL2002: VBA Error on Macro Startup If You Open Multiple Excel Instances with RefEdit Control.

We have found that the VBA debugger is not good at tracking events associated with RefEdit’s. For example, RefEdit events may fire and run ok, but without triggering any breakpoints set in the event handler. We have seen errors get thrown (and a dialog appear on the screen), even though our event handler has an On Error handlers; On Error handlers (and errors in general) are perhaps best avoided. (Given the tricks the RefEdits do, it is likely that they run their own main event loop which allows them to process clicks in Excel windows. We suspect that differences between this main event loop standard VBA main event loop leads to things operating differently in subtle but dangerous ways!)

This page suggests that you trace the refedit’s exit event (by debug.print, not breakpoints), and states that it MUST occur before you hide or close your userform. The following code is given to help ensure this:

Private Sub UserForm_QueryClose(ByVal Cancel As Integer, ByVal CloseMode As Integer)
    If Me.ActiveControl Is Me.RefEdit1 Then
        Me.Frame1.SetFocus()
        DoEvents()
    End If
    Cancel = True
    Me.Hide()
End Sub

 

The source for a C#.net RefEdit replacement is available on Gabhan Berry’s MSDN Excel Programming blog.

RefEdit Range Localisation
The RefEdit.Text returns, as might be expected, the text associated with a range. For ranges with multiple areas, Excel uses a “,” as the separator in English locales, and a “;” in German, for example. You can see this is you switch your “Current Format” to German, and type the following into VBA’s immediate window:? range(“A1,B4”).AddressLocal
$A$1;$B$4  <- Note Excel’s AddressLocal has replaced the , by a ;
This case can be detected by observing that Application.International(xlListSeparator) is “;” instead of “:”. The next version of OpenSolver will translate formulae (and ranges) from the current language to US by putting the item to translate into a cell, making use of .FormulaLocal and .Formula
Testing a String for an English Number – IsNumeric() fails miserably
OpenSolver needs to test if a defined name contains a formula, a constant or a range. It is surprisingly difficult  in VBA to test for a constant value expressed in English numerical style in a fashion that works in different locales (such as French with commas instead of full stops for decimal points). The obvious IsNumeric() has its problems, such as IsNumeric(“($1,23,,3.4,,,5,,E67$)”) giving True! This is the code we wrote for this; it will detect English numbers (as appear in defined names) even if the locale is non-English.
Function IsAmericanNumber(ByVal s As String, Optional ByVal i As Integer = 1) As Boolean
    ' Check this is a number like 3.45  or +1.23e-34
    ' This does NOT test for regional variations such as 12,34
    ' This code exists because
    '   val("12+3") gives 12 with no error
    '   Assigning a string to a double uses region-specific translation, so x="1,2" works in French
    '   IsNumeric("12,45") is true even on a US English system (and even worse, eg IsNumeric("($1,23,,3.4,,,5,,E67$)")=True! See http://www.eggheadcafe.com/software/aspnet/31496070/another-vba-bug.aspx)
    Dim MustBeInteger As Boolean, SeenDot As Boolean, SeenDigit As Boolean
    MustBeInteger = i > 1   ' We call this a second time after seeing the "E", when only an int is allowed
    IsAmericanNumber = False    ' Assume we fail
    If Len(s) = 0 Then Exit Function ' Not a number
    If Mid(s, i, 1) = "+" Or Mid(s, i, 1) = "-" Then i = i + 1 ' Skip leading sign
    For i = i To Len(s)
        Select Case Asc(Mid(s, i, 1))
            Case Asc("E"), Asc("e")
                If MustBeInteger Or Not SeenDigit Then Exit Function ' No exponent allowed (as must be a simple integer)
                IsAmericanNumber = IsAmericanNumber(s, i + 1)   ' Process an int after the E
                Exit Function
            Case Asc(".")
                If SeenDot Then Exit Function
                SeenDot = True
            Case Asc("0") To Asc("9")
                SeenDigit = True
            Case Else
                Exit Function   ' Not a valid char
        End Select
    Next i
    ' i As Integer, AllowDot As Boolean
    IsAmericanNumber = SeenDigit
End Function

 

Excel Programming and Advanced Use
Gaghan Berry has a good (but old, around 2008) blog on programming Excel, part of the Microsoft MSDN site. We are also a big fan of Chip Pearsons’s Excel site. There are some very interesting projects (including source code) showing XLL add-in examples, VBA COM code and C# add-ins at Codematic.com.
User Defined Functions We would like to make OpenSolver accessible as a function entered into a cell. Normally a formula needs to be exntered (using Shift-Ctrl-Enter) into multiple cells if they are all to be set as a result of the function call. However, this CodeProject article shows how the Calculate event can be used to allow data to be written to a number of cells outside the actual formula cell. This could be an easy way to write the decision variable cells to the sheet using a function such as =OpenSolver.Solver(righHandSideCells) that is triggered whenever a right hand side is changed.

Compiling CBC v2.7.6 – our experiences
To compile CBC.exe ready for use with OpenSolver, you need to download the latest zip CBC source code. Our experiences below come from compiling version 2.7.6. We tried compiling under the free version of Visual Studio 2010, using the “v10” folder in the CBC download. This compiled successfully, but did not appear to produce a CBC.exe file. Instead, we compiled using a full copy of Visual Studio v9 using the “v9” folder in the CBC download (Cbc-2.7.6Cbc-2.7.6CbcMSVisualStudiov9), working with the default project “cbcExamplesSample2”. This generated multiple errors, but did produce a CBC.exe file in the “Release” folder (after switching from Debug to Release). We then re-compiled after changing the properties for all the projects (ie with them all selected) to set the “C/C++ : Code Generation : Runtime Library” to “Multi-Threaded” (so we don’t need to distribute any DLL’s), and “General : Common Language Runtime support” to “No common language runtime support”. To check that there are no dependencies, you can use Dependency Walker (depends.exe) to open CBC.exe, and check that it only requires Kernel32.dll instead of all the library files you normally see.

42 thoughts on “Programming”

  1. Hi all, very interesting site!! I have a question, because I’m not so expert in all these tools. I have a huge excel with many formulas (few thousands of variables), in some cells (as variables) we have some binary entries and some integer , on other different cells calculated through excel formulas and on the variables we have the constraints. Most of the excel functions are not linear (“IF”, etc.). I tried to use bonmin and couenne but I have problems with the parsing. I changed many formulas (avoiding volatiles) and it went on and stopped again and finally I realize that also “IF” with arguments is a problem and I have many of them. Have you any suggestion for me to use the Opensolver?
    Thanks a lot!!
    Michele

Leave a Reply

Your email address will not be published. Required fields are marked *