30 is good. If you had a retained DataContext somewhere you'd get the
same number as the number of stuck invocations.
1 million of Invocations is bad.
I guess this needs to be logged as a possible bug to investigate,
with as many details as possible on the JVM version, etc.
Andrus
On Feb 21, 2007, at 4:07 PM, Ayhan Kondoz wrote:
> Yes seems live there are DataContext instances.
>
> http://img73.imageshack.us/img73/3068/memta6.jpg
>
> I started the GC a few times before I took this screenshot.
> As you can see there are still 30 DataContext instances. No idea
> why thought...
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
>> Gesendet: Mittwoch, 21. Februar 2007 14:49
>> An: use..ayenne.apache.org
>> Betreff: Re: AW: possible bug / memory leak in DispatchQueue and
>> EventManager?
>>
>> Do you see DataContext instances in the memory profile? I wonder how
>> many of those do you have, as those are the only Cayenne objects that
>> have hard reference to the listeners.
>>
>> Andrus
>>
>>
>> On Feb 21, 2007, at 3:37 PM, Ayhan Kondoz wrote:
>>
>>> Hi,
>>>
>>> the JVM has 1 GB of memory and i already tried to run the GC
>>> manually but it doesn't make a difference. I can start the GC from
>>> the profiler tool but the number of instances does not change.
>>> I guess that there are some other references to this objects so
>>> that the weak ref. does not expire.
>>>
>>> And there are no EventManager exception in the log files.
>>>
>>>
>>> ayhan
>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Andrus Adamchik [mailto:andru..bjectstyle.org]
>>>> Gesendet: Mittwoch, 21. Februar 2007 14:17
>>>> An: use..ayenne.apache.org
>>>> Betreff: Re: possible bug / memory leak in DispatchQueue and
>>>> EventManager?
>>>>
>>>> Seems like your assessment of the EventManager leaking is correct.
>>>>
>>>> Now the cause is not that clear. A shot in the dark - this is
>>>> due to
>>>> a combination of lots of spare memory in JVM (so weak references
>>>> are
>>>> not collected fast enough) and slow custom 'equals' and 'hashCode'
>>>> methods in invocation.
>>>>
>>>> How much heap size do you have in your server? Also can you try to
>>>> add a request filter that would do 'System.gc()' on every few
>>>> thousands requests, and see if it makes any difference.
>>>>
>>>> Also - do you see any EventManager exceptions in the logs?
>>>>
>>>> Andrus
>>>>
>>>>
>>>> On Feb 21, 2007, at 10:55 AM, Ayhan Kondoz wrote:
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> i think i found a possible bug / memory leak within the
>>>>> DispatchQueue and EventManager.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> First a little bit about my setup:
>>>>>
>>>>>
>>>>>
>>>>> I have 3 servers. Each server runs an axis service. The service
>>>>> uses cayenne 1.2.1 to connect to a database. It reads customer and
>>>>> account information from the DB etc..
>>>>>
>>>>> The servers are using cayenne's shared caching with javagroups as
>>>>> the messaging service so that changes made from one server are
>>>>> dispatched to the other servers.
>>>>>
>>>>>
>>>>>
>>>>> The avarage connections per second is somewhere around 4-5.
>>>>>
>>>>>
>>>>>
>>>>> However I have a very strange problem with this setup so I startet
>>>>> to search for the reason. The problem is that with nearly constant
>>>>> and unchanging usage the load of each server increases over time.
>>>>> To further test this I created a test server with a similar setup.
>>>>> There I created a test program that creates totally constant
>>>>> usage.
>>>>> But even with the unchanging usage the load of the server is
>>>>> increasing until the cpu load is so high that the requests can not
>>>>> be processed anymore.
>>>>>
>>>>>
>>>>>
>>>>> I installed a java profiler to trying to pinpoint the location of
>>>>> this error and this is what I found out.
>>>>>
>>>>>
>>>>>
>>>>> I let the server run for 24 hours and then stopped the program
>>>>> which creates the test usage.
>>>>>
>>>>> But even while the server was idle there where still a lot of
>>>>> instances in the java heap after the GC run.
>>>>>
>>>>>
>>>>>
>>>>> http://img206.imageshack.us/img206/9769/memorysy5.jpg
>>>>>
>>>>>
>>>>>
>>>>> Please note the HashMap, WeakReference and Invocation counts. I
>>>>> pressume that the $ObjectProvider_*** is cayenne aswell but I am
>>>>> not sure.
>>>>>
>>>>>
>>>>>
>>>>> Now the following image shows the cpu profile with incoming
>>>>> connections.
>>>>>
>>>>>
>>>>>
>>>>> http://img339.imageshack.us/img339/2441/cpuci0.jpg
>>>>>
>>>>>
>>>>>
>>>>> As you can see 58% of the cpu time is used within HashSet.add().
>>>>>
>>>>>
>>>>>
>>>>> So when I consider the two facts i think that there might be a
>>>>> possible problem with the EventManager. The first table tells us
>>>>> that there are over 1 million instances of HashSet's and cayenne
>>>>> Invocations. So it seems like the set's within the DispatchQueue
>>>>> are not recycled properly so that the object count rises over time
>>>>> which result in extremly high processtime when trying to add new
>>>>> HashSet's.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Ayhan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Ayhan Kondoz
>>>>>
>>>>>
>>>>>
>>>>> Software-Entwicklung
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> --
>>>>> --
>>>>> ------------
>>>>>
>>>>> Telefon: +49 (0) 40 513 06 616
>>>>>
>>>>> Telefax: +49 (0) 40 513 06 998 616
>>>>>
>>>>> E-Mail: ayhan.kondo..reenet-ag.de
>>>>> <mailto:ayhan.kondo..reenet-
>>>>> ag.de>
>>>>>
>>>>> Website: http://www.freenet.de <http://www.freenet.de/> ; http://
>>>>> www.freenet-ag.de <http://www.freenet-ag.de/>
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> --
>>>>> --
>>>>> ------------
>>>>>
>>>>> freenet.de AG
>>>>>
>>>>> Deelbögenkamp 4c
>>>>>
>>>>> 22297 Hamburg
>>>>>
>>>>> ------------------------------------------------------------------
>>>>> --
>>>>> --
>>>>> ------------
>>>>>
>>>>> Vorsitzender des Aufsichtsrates: Prof. Dr. Helmut Thoma
>>>>>
>>>>> Vorstand: Eckhard Spoerr (Vors.),
>>>>> Axel Krieger, Stephan Esch, Eric Berger
>>>>>
>>>>> Amtsgericht Hamburg HRB 74048
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Diese Information ist ausschließlich für die adressierte Person
>>>>> oder Organisation bestimmt und könnte vertrauliches und/oder
>>>>> privilegiertes Material enthalten. Personen oder Organisationen,
>>>>> für die diese Information nicht bestimmt ist, ist es nicht
>>>>> gestattet, diese zu lesen, erneut zu übertragen, zu verbreiten,
>>>>> anderweitig zu verwenden oder sich durch sie veranlasst zu sehen,
>>>>> Maßnahmen irgendeiner Art zu ergreifen. Sollten Sie diese
>>>>> Nachricht
>>>>> irrtümlich erhalten haben, bitten wir Sie, sich mit dem
>>>>> Absender in
>>>>> Verbindung zu setzen und das Material von Ihrem Computer zu
>>>>> löschen.
>>>>>
>>>>>
>>>>>
>>>>> The information transmitted is intended only for the person or
>>>>> entity to which it is addressed and may contain confidential
>>>>> and/or
>>>>> privileged material. Any review, retransmission, dissemination or
>>>>> other use of, or taking of any action in reliance upon, this
>>>>> information by persons or entities other than the intended
>>>>> recipient is prohibited. If you received this in error, please
>>>>> contact the sender and delete the material from any computer.
>>>>>
>>>>>
>>>>>
>>>
>>>
>
>
This archive was generated by hypermail 2.0.0 : Wed Feb 21 2007 - 09:20:48 EST
