View Issue Details

Jump to Notes
IDProjectCategoryView StatusDate SubmittedLast Update
0014957MMW v4Framework: Scripts/Extensionspublic2018-07-10 09:142018-07-18 18:56
Reporterzvezdan Assigned To 
PrioritynoneSeverityminorReproducibilitysometimes
Status feedbackResolutionopen 
PlatformWindowsOS-OS Version10
Summary0014957: Slow startup because of the MatchCollection object
DescriptionHere is the short script that you could use to test the slow down during startup caused by the MatchCollection object, as I described in the related Issue (https://www.ventismedia.com/mantis/view.php?id=14820#c50347).

It is executing during startup after which it shows one dialog box telling the running time. After that you could also start it from the Search toolbar by click on the "RegExp Test" button and click on the Test button in the new dialog box. There is also one spin button in that dialog with which you could set the number of iterations for the test - 10000 iterations are running approx. 3 seconds during startup and 2 seconds after that on my computer, but I am not experiencing the problem on my computer, most probably because it is not Win10.

However, here is the report about this test script which I got from one user that had the mentioned problem:
Load MM 10,000 iterations: 119.94
   MM running:
       10,000 iterations: 1.17
       50,000 iterations: 5.83
       100,000 iterations: 11.65

Load MM 5,000 iterations: 49.13 and a second time of 56.76 after running another application
    MM running:
       5,000 iterations: 0.59
     10,000 iterations: 1.16
     50,000 iterations: 5.81
   100,000 iterations: 11.59

As you could see, he is experiencing approx. 100 times slower executing during startup than latter when started from the toolbar for the same number of iterations.

As I already explained in the related Issue, this is all happening with Set oMatches = oRegEx.Execute("..."), but only during startup. The same code is running fine latter.

I have also attached the packed script as the add-on ready to use.
Steps To ReproduceOption Explicit

Dim iCount

Sub OnStartUp()
    Dim oMenuItem

    Set oMenuItem = SDB.UI.AddMenuItem(SDB.UI.Menu_TbSearch, 0, 0)
    With oMenuItem
        .Caption = "RegEx Test"
        .OnClickFunc = "Dialog"
        .UseScript = Script.ScriptPath
    End With
    Set SDB.Objects("RegExTest") = oMenuItem

    With SDB.IniFile
        If Len(.StringValue("RegExTest", "Iterations")) = 0 Then
            .IntValue("RegExTest", "Iterations") = 10000
        End If
        iCount = .IntValue("RegExTest", "Iterations")
    End With

    Test
End Sub

Sub Test()
    Dim oRegEx
    Dim oProgress
    Dim oMatches
    Dim dtTime
    Dim i

    Set oRegEx = New RegExp
    oRegEx.IgnoreCase = True
    oRegEx.Global = True
    oRegEx.Multiline = False
    Set oProgress = SDB.Progress

    dtTime = Timer
    oRegEx.Pattern = "((?:^(?:Preset\d+ *= *)?|, *)ReplaceWith *: *)(\d+|(?:""(?:""""|[^""])*?"")|[^,]+)(?=( *,|$))"
    For i = 0 To iCount
        Set oMatches = oRegEx.Execute("Preset230=Menu: ""Partial manipulations"", Name: ""Move right-specified part of <From Field> between (and including) specified strings to <Into Field>..."", Description: ""This preset modifies two fields at once! - - If Assign is on, Opening string is """"("""", Closing string is """")"""", Part number = 2 (counting from the right), Copy opening, Copy closing, Remove opening and Remove closing are on: - The Wall (CD2) (41:55) [1979] (source field) -> The Wall (41:55) [1979] (source field) and (CD2) (destination field) - - If Copy opening and Copy closing are off: - CD2 (destination field) - - If Remove opening and Remove closing are off: - The Wall () (41:55) [1979] (source field) - - If Use RegEx option is on, Opening string is """"[([]"""" and Closing string is """"[)\]]"""", it will be copied parts either between """"("""" and """")"""" or """"["""" and """"]""""."", Shortcut: """", Icon: """", Toolbar: 0, FindWhat: ""<If Caption=""""Assign to destination"""" Value=1 ID=1><If Caption=""""Only if destination is empty"""" Value=0 ID=3>^$<Else>^.*<End If><Else Caption=""""Append to:""""><If Caption=""""Only if destination is not empty"""" Value=0 ID=4><End If><If Caption=""""the start of destination"""" Value=1 ID=2>^<If ID:4>(?!$)<End If><Else Caption=""""the end of destination""""><If ID:4>(?!^)<End If>$<End If><End If>"", FindInto: ""Custom 1"", FindRegExp: 1, WholeWord: 0, ReplaceWith: ""LetVar(0, RegSub(<From Field>, """".*("""" & SetVar(1, <If Caption=""""Use RegEx to specify strings"""" Value=0 ID=5><Else>RegSub(<End If>""""<String Caption=""""Opening string"""" Value=""""("""">""""<If ID:5><Else>, """"[$^*()+[\]\\{}|.?]"""", """"\$$&"""")<End If>) & """")(.*)("""" & SetVar(2, <If ID:5><Else>RegSub(<End If>""""<String Caption=""""Closing string"""" Value="""")"""">""""<If ID:5><Else>, """"[$^*()+[\]\\{}|.?]"""", """"\$$&"""")<End If>) & """")(?:.*?"""" & GetVar(1) & """".*?"""" & GetVar(2) & """"){"""" & <Number Caption=""""Part number"""" Value=""""2"""" MinValue=""""1"""" ID=6> - 1 & """"}.*?$<If Caption=""""Move entire source if it doesn't contain specified strings"""" Value=""""0""""><Else>|.*<End If>"""", """"<If Caption=""""Copy opening string"""" Value=""""1"""">$1<End If>$2<If Caption=""""Copy closing string to destination"""" Value=""""1"""">$3<End If>"""")) & <If ID=1>GetVar(0)<Else><If ID:2>GetVar(0) & <End If>IIf(Len(""""$_"""") > 0 And Len(GetVar(0)) > 0, """"<String Caption=""""Separator"""" Value="""" - """">"""", """""""")<If ID:2><Else> & GetVar(0)<End If><End If> & Execute(""""<If ID:1><If ID:3>If Len(<Into Field>) = 0 Then <End If><Else><If ID:4>If Len(<Into Field>) > 0 Then <End If><End If><If Value=""""bReplacing""""><From Field><Else>sResultFrom<End If> = RegSub(<From Field>, """"""""(.*)("""""""" & GetVar(1) & """""""").*("""""""" & GetVar(2) & """"""""<If Caption=""""Remove opening string"""" Value=1 ID=-7>\s*<End If>)(?=(?:.*?"""""""" & GetVar(1) & """""""".*?"""""""" & GetVar(2) & """"""""){"""" & <ID:6> - 1 & """"}.*?$)"""""""", """"""""$1<If ID:-7><Else>$2<End If><If Caption=""""Remove closing string from source"""" Value=1><Else>$3<End If>"""""""")"""")"", ReplaceFrom: ""Title"", ReplaceRegExp: 0, ReplaceVBScr: 1, MatchCase: 0")
        If i / 10 = i \10 Then oProgress.Text = i
    Next
    SDB.MessageBox FormatNumber(Timer - dtTime, 2) & " sec.", mtInformation, Array(mbOk)
End Sub

Sub Dialog(oMenuItem)
    Dim oForm
    Dim lblTest
    Dim spnTest
    Dim btnClose
    Dim btnTest

    Set oForm = SDB.UI.NewForm
    With oForm
        .Common.SetRect 100, 100, 337, 130
        .BorderStyle = 3
        .FormPosition = 4
        .Caption = oMenuItem.Caption
    End With

    Set lblTest = SDB.UI.NewLabel(oForm)
    With lblTest
        .Common.SetRect 10, 12, 130, 17
        .Caption = "Number of iterations:"
    End With

    iCount = SDB.IniFile.IntValue("RegExTest", "Iterations")
    Set spnTest = SDB.UI.NewSpinEdit(oForm)
    With spnTest
        .Common.SetRect 160, 10, 150, 20
        .MinValue = -1000000
        .MaxValue = 1000000
        .Value = iCount
    End With
    Script.RegisterEvent spnTest, "OnChange", "spnTest_OnChange"

    Set btnTest = SDB.UI.NewButton(oForm)
    With btnTest
        .Caption = SDB.Localize("&Test")
        .Common.SetRect 90, 60, 77, 25
    End With
    Script.RegisterEvent btnTest, "OnClick", "btnTest_OnClick"

    Set btnClose = SDB.UI.NewButton(oForm)
    With btnClose
        .Caption = SDB.Localize("&Close")
        .Common.SetRect 170, 60, 77, 25
        .Cancel = True
        .ModalResult = 2
    End With
    oForm.showModal
    SDB.IniFile.IntValue("RegExTest", "Iterations") = iCount
End Sub

Sub spnTest_OnChange(oCtrl)
    iCount = oCtrl.Value
End Sub

Sub btnTest_OnClick()
    Test
End Sub
Additional Informationlog: v.4.1.21.1875 - ID: 3E1F
TagsNo tags attached.
Attached Files
RegExTest.mmip (3,143 bytes)
Fixed in build

Relationships

related to 0014820 closedLudek Loading time is 60 seconds when Real time protection is active (Windows Defender) 

Activities

zvezdan

2018-07-18 18:56

updater   ~0050786

I would like to close this issue since it is not a problem of MediaMonkey. Here is a part of message that I reported to Microsoft at:
https://answers.microsoft.com/en-us/protect/forum/protect_defender-protect_scanning-windows_10/last-update-of-windows-defender-causes-terribly/c71cc133-5c5f-447e-95fb-e2641085836d?tm=1527000700529

I made one short VBscript which you could try directly from the command line:

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

Option Explicit

Const iCount = 10000
Dim oRegEx
Dim oMatches
Dim dtTime
Dim i

Set oRegEx = New RegExp
oRegEx.IgnoreCase = True
oRegEx.Global = True
oRegEx.Multiline = False
dtTime = Timer
oRegEx.Pattern = "((?:^(?:Preset\d+ *= *)?|, *)ReplaceWith *: *)(\d+|(?:""(?:[^""]+|"""")*""))(?=( *,|$))"
For i = 0 To iCount
    Set oMatches = oRegEx.Execute("Preset230=Menu: ""Partial manipulations"", Name: ""Move right-specified part of <From Field> between (and including) specified strings to <Into Field>..."", Description: ""This preset modifies two fields at once! - - If Assign is on, Opening string is """"("""", Closing string is """")"""", Part number = 2 (counting from the right), Copy opening, Copy closing, Remove opening and Remove closing are on: - The Wall (CD2) (41:55) [1979] (source field) -> The Wall (41:55) [1979] (source field) and (CD2) (destination field) - - If Copy opening and Copy closing are off: - CD2 (destination field) - - If Remove opening and Remove closing are off: - The Wall () (41:55) [1979] (source field) - - If Use RegEx option is on, Opening string is """"[([]"""" and Closing string is """"[)\]]"""", it will be copied parts either between """"("""" and """")"""" or """"["""" and """"]""""."", Shortcut: """", Icon: """", Toolbar: 0, FindWhat: ""<If Caption=""""Assign to destination"""" Value=1 ID=1><If Caption=""""Only if destination is empty"""" Value=0 ID=3>^$<Else>^.*<End If><Else Caption=""""Append to:""""><If Caption=""""Only if destination is not empty"""" Value=0 ID=4><End If><If Caption=""""the start of destination"""" Value=1 ID=2>^<If ID:4>(?!$)<End If><Else Caption=""""the end of destination""""><If ID:4>(?!^)<End If>$<End If><End If>"", FindInto: ""Custom 1"", FindRegExp: 1, WholeWord: 0, ReplaceWith: ""LetVar(0, RegSub(<From Field>, """".*("""" & SetVar(1, <If Caption=""""Use RegEx to specify strings"""" Value=0 ID=5><Else>RegSub(<End If>""""<String Caption=""""Opening string"""" Value=""""("""">""""<If ID:5><Else>, """"[$^*()+[\]\\{}|.?]"""", """"\$$&"""")<End If>) & """")(.*)("""" & SetVar(2, <If ID:5><Else>RegSub(<End If>""""<String Caption=""""Closing string"""" Value="""")"""">""""<If ID:5><Else>, """"[$^*()+[\]\\{}|.?]"""", """"\$$&"""")<End If>) & """")(?:.*?"""" & GetVar(1) & """".*?"""" & GetVar(2) & """"){"""" & <Number Caption=""""Part number"""" Value=""""2"""" MinValue=""""1"""" ID=6> - 1 & """"}.*?$<If Caption=""""Move entire source if it doesn't contain specified strings"""" Value=""""0""""><Else>|.*<End If>"""", """"<If Caption=""""Copy opening string"""" Value=""""1"""">$1<End If>$2<If Caption=""""Copy closing string to destination"""" Value=""""1"""">$3<End If>"""")) & <If ID=1>GetVar(0)<Else><If ID:2>GetVar(0) & <End If>IIf(Len(""""$_"""") > 0 And Len(GetVar(0)) > 0, """"<String Caption=""""Separator"""" Value="""" - """">"""", """""""")<If ID:2><Else> & GetVar(0)<End If><End If> & Execute(""""<If ID:1><If ID:3>If Len(<Into Field>) = 0 Then <End If><Else><If ID:4>If Len(<Into Field>) > 0 Then <End If><End If><If Value=""""bReplacing""""><From Field><Else>sResultFrom<End If> = RegSub(<From Field>, """"""""(.*)("""""""" & GetVar(1) & """""""").*("""""""" & GetVar(2) & """"""""<If Caption=""""Remove opening string"""" Value=1 ID=-7>\s*<End If>)(?=(?:.*?"""""""" & GetVar(1) & """""""".*?"""""""" & GetVar(2) & """"""""){"""" & <ID:6> - 1 & """"}.*?$)"""""""", """"""""$1<If ID:-7><Else>$2<End If><If Caption=""""Remove closing string from source"""" Value=1><Else>$3<End If>"""""""")"""")"", ReplaceFrom: ""Title"", ReplaceRegExp: 0, ReplaceVBScr: 1, MatchCase: 0")
Next
MsgBox FormatNumber(Timer - dtTime, 2) & " sec."
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

I don't have Windows 10, but here are the reports of some users of my add-on that tried this script. The first user reported this:
"Using your code with 10,000 iterations the results:
10,000 iterations
   11.70 secs
   11.41 secs
   11.14 secs
   11.34 secs
Using your code but changing 10,000 iterations to 1,000 iterations
1,000 iterations
   1.13 secs
   1.15 secs
   1.16 secs
   1.15 secs
NO R-T Protection for 10,000 (10k) iterations is 1.83 and 1.84 secs - I ran 4 times. WITH R-T Protection I get 11.53 and 11.45 secs respectively for same iterations."

The same user even tried the same script with the much shorter string for the Execute command, but got just slightly shorter execution:
"Test 1 - shorter Execute: 2 tests 9.64 and 9.57 secs respectively."

Here is that shorter string:
"Preset17=Menu: ""Basic manipulations"", Name: ""Increment Play counter by 1"", Description: """", Shortcut: """", Icon: """", Toolbar: 1, FindWhat: ""^.*"", FindInto: ""Played #"", FindRegExp: 1, WholeWord: 0, ReplaceWith: ""<Into Field> + 1"", ReplaceFrom: ""Track #"", ReplaceRegExp: 0, ReplaceVBScr: 1, MatchCase: 0"

The other user reported this:
"RegExTest2.vbs time is 40.52 seconds"

And the report of the third user:
"I don't have an option to turn of the Win Defender Realtime Protection.
I guess my Kaspersky Antivirus disabled it. But I tried with and without
my antivirus, and it turned out with Kaspersky enabled, the script was
running about 39 secs and with Kaspersky disabled it only took 2 seconds
to complete.

Being curious I tested it on my laptop running Win 10 too but having
Avira Antivirus installed. Interestingly it runs even with Avira RT
Protection enabled in 1.5 secs! Disabling Avira makes it runs in 1.3 secs.

I went further and ran the script on a virtual machine in VirtalBox I'm
using now and then. With Windows Defender enabled it took about 16 secs
to finish, disabling the Win Defender RT Protection made it finish
within 1.6 secs."

I don't know what is going on in the background with Kaspersky AV, but I suppose that it has not disabled Windows Defender and because of that it is slow. Maybe it just removed it from the GUI, but I guess that Defender service is still running.