[Bro-Dev] [JIRA] (BIT-1547) broctl sets the same state variables over and over

[Justin Azoff (JIRA)]

    [ https://bro-tracker.atlassian.net/browse/BIT-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=24701#comment-24701 ] 

Justin Azoff commented on BIT-1547:
-----------------------------------

Before the change: real    1m32.978s
After the change: real    0m6.413s

It set options 1192 times. this is what it sets (count + key, value):

{code}
      1 commiting set of configchksum to "8a94f1078550bc5bcfbb78228b673d75"
      1 commiting set of confignodechksum to "761eb1e973f7bdbc94120ae522b76db5"
     34 commiting set of manager-port to 47760
     34 commiting set of manager-port to 47761
     34 commiting set of nids-test1a-1-port to 47763
     34 commiting set of nids-test1a-2-port to 47764
     34 commiting set of nids-test1a-3-port to 47765
     34 commiting set of nids-test1a-4-port to 47766
     34 commiting set of nids-test1a-5-port to 47767
     34 commiting set of nids-test1a-6-port to 47768
     34 commiting set of nids-test1a-7-port to 47769
     34 commiting set of nids-test1a-8-port to 47770
     34 commiting set of nids-test1b-1-port to 47771
     34 commiting set of nids-test1b-2-port to 47772
     34 commiting set of nids-test1b-3-port to 47773
     34 commiting set of nids-test1b-4-port to 47774
     34 commiting set of nids-test1b-5-port to 47775
     34 commiting set of nids-test1b-6-port to 47776
     34 commiting set of nids-test1b-7-port to 47777
     34 commiting set of nids-test1b-8-port to 47778
     34 commiting set of nids-test2a-1-port to 47779
     34 commiting set of nids-test2a-2-port to 47780
     34 commiting set of nids-test2a-3-port to 47781
     34 commiting set of nids-test2a-4-port to 47782
     34 commiting set of nids-test2a-5-port to 47783
     34 commiting set of nids-test2a-6-port to 47784
     34 commiting set of nids-test2a-7-port to 47785
     34 commiting set of nids-test2a-8-port to 47786
     34 commiting set of nids-test2b-1-port to 47787
     34 commiting set of nids-test2b-2-port to 47788
     34 commiting set of nids-test2b-3-port to 47789
     34 commiting set of nids-test2b-4-port to 47790
     34 commiting set of nids-test2b-5-port to 47791
     34 commiting set of nids-test2b-6-port to 47792
     34 commiting set of nids-test2b-7-port to 47793
     34 commiting set of nids-test2b-8-port to 47794
     34 commiting set of proxy-1-port to 47762
{code}

After the change it just does

{code}
      1 commiting set of configchksum to "8a94f1078550bc5bcfbb78228b673d75"
      1 commiting set of confignodechksum to "761eb1e973f7bdbc94120ae522b76db5"
     34 commiting set of manager-port to 47760
     34 commiting set of manager-port to 47761
{code}

> broctl sets the same state variables over and over
> --------------------------------------------------
>
>                 Key: BIT-1547
>                 URL: https://bro-tracker.atlassian.net/browse/BIT-1547
>             Project: Bro Issue Tracker
>          Issue Type: Problem
>          Components: BroControl
>    Affects Versions: git/master
>            Reporter: Justin Azoff
>            Assignee: Daniel Thayer
>
> I happened to notice broctl check on one of our test boxes was slow.  traced it to sqlite commits() being very slow.  Then noticed that broctl seems to call set_state() with the same key, val over and over again... once for each worker.. so a few thousand sets just to run broctl check.
> Changing set_state to
> {code}
>     # Set a dynamic state variable.
>     def set_state(self, key, val):
>         key = key.lower()
>         if self.state.get(key) == val:
>             return
>         self.state[key] = val
>         self.state_store.set(key, val)
> {code}
> Seemed to mostly fix it, aside from this:
> {code}
> Set manager-port to 47760
> Set manager-port to 47761
> Set manager-port to 47760
> Set manager-port to 47761
> Set manager-port to 47760
> Set manager-port to 47761
> Set manager-port to 47760
> Set manager-port to 47761
> Set manager-port to 47760
> Set manager-port to 47761
> Set manager-port to 47760
> Set manager-port to 47761
> {code}
> any idea why that is flipping around like that?
> We should possibly add a way for broctl to update state vars without calling commit where it knows it will be setting a large number of state vars in a loop.



--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-012#72000)