Author Topic: Global Cache issue  (Read 8226 times)

Frunple

  • Hero Member
  • *****
  • Posts: 774
    • View Profile
Global Cache issue
« on: August 09, 2010, 06:37:19 am »
Anyone else havig a problem with a GC-100 constantly disconnecting??

<rec><dt>2010-08-09 13:20:16.0619772Z</dt><s>Warning</s><msg>The connection to the Global Cache has been lost, the driver will keep trying to reconnect.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-09 13:20:16.0988735Z</dt><s>Debug</s><msg>Global Cache Driver successfully connected to &apos;10.12.8.23:4998&apos;.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-09 13:20:16.1596308Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getversion</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-09 13:20:16.8001265Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getdevices</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>


I tried sending a "light off" command to it last night and it took a full minute for the light to turn off. I actually gave up on it thinking it was disconnected again and then all of the sudden the light did turn off. I have noticed that after a couple of days I need to restart the device in the management console in order for the GC to receive commands.

On another note... where does the management console get it's time from?? As you can see from the log above it thinks it's 1:20 pm on the 9th and it reality I'm posting this message at 9:35am on the 9th.

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2851
    • View Profile
    • Codecore Technologies
Re: Global Cache issue
« Reply #1 on: August 09, 2010, 08:21:39 am »
Do you have something else connecting to it as well?  The GC100 only supports 1 tcp connection.

PS: The dates in the log are UTC time (hence the Z at the end per ISO).
John Hughes
Codecore Technologies

Frunple

  • Hero Member
  • *****
  • Posts: 774
    • View Profile
Re: Global Cache issue
« Reply #2 on: August 10, 2010, 05:02:25 am »
No, nothing else connecting to it. It may be related to that memory leak that was fixed... I don't know that it was fixed. My system becomes completely unusable still and the cpu and memory usage are still going way out of control. It looks to be the driverservice that is causing it.
Is anyone else seeing this still?

EDIT: I should also mention those log entries are constant, not just the one time but over and over constantly.
« Last Edit: August 10, 2010, 05:19:34 am by Frunple »

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2851
    • View Profile
    • Codecore Technologies
Re: Global Cache issue
« Reply #3 on: August 10, 2010, 07:09:13 am »
At what interval are the disconnects?

Are you running any other drivers which use a persistent tcp connection and are they disconnecting?

Can you give us some more information on the memory leak issue? How much memory is the driver service and master service using? Also what drivers are you using?
John Hughes
Codecore Technologies

Frunple

  • Hero Member
  • *****
  • Posts: 774
    • View Profile
Re: Global Cache issue
« Reply #4 on: August 11, 2010, 05:15:14 am »
At what interval are the disconnects?

I guess the log file doesn't store enough time for me to go back and check the errors that I posted and since I rebooted yesterday it seems to take a day or three fro the trouble to start so I will get back to you on an interval but it was quite frequent as I remember, maybe every minute or less.

Are you running any other drivers which use a persistent tcp connection and are they disconnecting?

HAI Omni, doesn't seem to be having any issues.

Can you give us some more information on the memory leak issue? How much memory is the driver service and master service using? Also what drivers are you using?

I'll have to get back to you on that when it becomes unresponsive again.

On another note, my last system allowed connecting the GC-IRE directly to a system serial port instead of the GC-100's only. It seemed to worked better that way. Any chance of that with Elve?

All Drivers I use are :

Date & Time
Global Cache (just added last week)
Global variables (not used but running)
HAI Omni Pro II
Nuvo Essentia E6D
Russound (David rules! He RULES!! Still lovin' the tuner control!!)
UPB
Weather
Zwave

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2851
    • View Profile
    • Codecore Technologies
Re: Global Cache issue
« Reply #5 on: August 11, 2010, 09:14:51 am »
I guess the log file doesn't store enough time for me to go back and check the errors that I posted and since I rebooted yesterday it seems to take a day or three fro the trouble to start so I will get back to you on an interval but it was quite frequent as I remember, maybe every minute or less.
The log file stores between 750KB to 1MB of data.  If you are not logging any debugging information this can be a couple weeks worth. If you are logging debug information then it can fill up quickly depending on what you are logging.

Quote
On another note, my last system allowed connecting the GC-IRE directly to a system serial port instead of the GC-100's only. It seemed to worked better that way. Any chance of that with Elve?
This has been discussed in several threads but I don't believe anyone has added a feature request for it yet. Please add a feature request so can see how much demand there is for it, but at the moment you would need to use a COM port redirector.

Quote
All Drivers I use are :

Date & Time
Global Cache (just added last week)
Global variables (not used but running)
HAI Omni Pro II
Nuvo Essentia E6D
Russound (David rules! He RULES!! Still lovin' the tuner control!!)
UPB
Weather
Zwave

I used a memory leak detection tool last night (with the initial installed drivers) and I didn't see anything. I'll take a closer look with your drivers as well.

I checked Elve server's log and there were no GC100 disconnects.  I restarted the services so I will keep an eye on it as well.
John Hughes
Codecore Technologies

Frunple

  • Hero Member
  • *****
  • Posts: 774
    • View Profile
Re: Global Cache issue
« Reply #6 on: August 12, 2010, 05:44:00 pm »
Heres the latest entries from my log. The disconnects shown are pretty consistant at around 30~ seconds.

<rec><dt>2010-08-12 06:19:27.7594694Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getdevices</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:20:02.8144432Z</dt><s>Warning</s><msg>The connection to the Global Cache has been lost, the driver will keep trying to reconnect.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:20:02.8508634Z</dt><s>Debug</s><msg>Global Cache Driver successfully connected to &apos;10.12.8.23:4998&apos;.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:20:02.8801468Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getversion</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:20:03.4053590Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getdevices</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:20:38.4674038Z</dt><s>Warning</s><msg>The connection to the Global Cache has been lost, the driver will keep trying to reconnect.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:20:38.5037108Z</dt><s>Debug</s><msg>Global Cache Driver successfully connected to &apos;10.12.8.23:4998&apos;.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:20:38.5325366Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getversion</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:20:39.0572655Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getdevices</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:21:14.7593578Z</dt><s>Warning</s><msg>The connection to the Global Cache has been lost, the driver will keep trying to reconnect.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:21:14.8656524Z</dt><s>Debug</s><msg>Global Cache Driver successfully connected to &apos;10.12.8.23:4998&apos;.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:21:14.9158619Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getversion</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:21:15.4473292Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getdevices</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:21:51.1593478Z</dt><s>Warning</s><msg>The connection to the Global Cache has been lost, the driver will keep trying to reconnect.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:21:51.1975316Z</dt><s>Debug</s><msg>Global Cache Driver successfully connected to &apos;10.12.8.23:4998&apos;.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:21:51.2262741Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getversion</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:21:51.7496126Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getdevices</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:22:05.5958383Z</dt><s>Fatal</s><msg>An unhandled exception occurred in the Elve Driver Service.

System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at ViziaRFLibrary.ViziaRF.processResponse(String response)
   at ViziaRFLibrary.ViziaRF.port_DataReceived(Object sender, SerialDataReceivedEventArgs e)
   at System.IO.Ports.SerialPort.CatchReceivedEvents(Object src, SerialDataReceivedEventArgs e)
   at System.IO.Ports.SerialStream.EventLoopRunner.CallReceiveEvents(Object state)
   at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
   at System.Threading.ExecutionContext.runTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>None</ct><cn></cn></rec>
<rec><dt>2010-08-12 06:22:26.7620762Z</dt><s>Warning</s><msg>The connection to the Global Cache has been lost, the driver will keep trying to reconnect.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:22:26.8016206Z</dt><s>Debug</s><msg>Global Cache Driver successfully connected to &apos;10.12.8.23:4998&apos;.</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:22:26.8322776Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getversion</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 06:22:27.3535734Z</dt><s>Debug</s><msg>Global Cache GC100 Driver sending: getdevices</msg><app>DriverService</app><ma>DIMENSION4550</ma><ct>Driver</ct><cn>Global Cache : Global Cache</cn></rec>
<rec><dt>2010-08-12 07:19:41.9996971Z</dt><s>SystemMessage</s><msg>Driver service is stopping on &apos;KITCHENPC&apos;.</msg><app>DriverService</app><ma>KITCHENPC</ma><ct>None</ct><cn></cn></rec>
<rec><dt>2010-08-12 07:19:58.6859038Z</dt><s>SystemMessage</s><msg>Driver service on &apos;KITCHENPC&apos; was unregistered from master server.</msg><app>MasterService</app><ma>DIMENSION4550</ma><ct>None</ct><cn></cn></rec>
<rec><dt>2010-08-12 07:21:53.1830648Z</dt><s>SystemMessage</s><msg>A compiled device drivers (.dll) directory did not exist: &apos;C:\ProgramData\Codecore Technologies\Elve\DeviceDrivers\Compiled&apos;.</msg><app>DriverService</app><ma>KITCHENPC</ma><ct>None</ct><cn></cn></rec>
<rec><dt>2010-08-12 07:22:15.8453159Z</dt><s>SystemMessage</s><msg>Driver service &apos;KITCHENPC&apos; registered with master server.</msg><app>MasterService</app><ma>DIMENSION4550</ma><ct>None</ct><cn></cn></rec>
<rec><dt>2010-08-12 07:22:02.9216041Z</dt><s>SystemMessage</s><msg>Driver service started on &apos;KITCHENPC&apos;.</msg><app>DriverService</app><ma>KITCHENPC</ma><ct>None</ct><cn></cn></rec>
<rec><dt>2010-08-13 00:38:14.9460882Z</dt><s>SystemMessage</s><msg>Authentication of username &apos;Admin&apos; resulted with: Success</msg><app>MasterService</app><ma>DIMENSION4550</ma><ct>Authentication</ct><cn>Admin</cn></rec>

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2851
    • View Profile
    • Codecore Technologies
Re: Global Cache issue
« Reply #7 on: August 12, 2010, 10:05:43 pm »
It sounds like it is fighting with something else that is also connecting to it.  I've been running mine since last night and it hasn't disconnected yet.

Try power cycling the GC100... I have seen it get confused before.
John Hughes
Codecore Technologies

Frunple

  • Hero Member
  • *****
  • Posts: 774
    • View Profile
Re: Global Cache issue
« Reply #8 on: August 13, 2010, 07:35:01 am »
It sounds like it is fighting with something else that is also connecting to it.  I've been running mine since last night and it hasn't disconnected yet.

Just ran a sniffer on my network for about 3 hrs... no traffic being generated by the GC (that's normal) and the only traffic with a destination of the GC is from my Elve pc.

Quote
Try power cycling the GC100... I have seen it get confused before.
It's been done but I'll try it again.

mustangcoupe

  • Full Member
  • ***
  • Posts: 108
    • View Profile
Re: Global Cache issue
« Reply #9 on: August 13, 2010, 09:29:38 am »
Can you give us some more information on the memory leak issue? How much memory is the driver service and master service using? Also what drivers are you using?

I agree with this John the memory leak is much better but I am still creeping up... I am about double what it is normally at, and starts up at... and about 1/2 what it was getting to before I had major problems without it fixed.  This is with just the driver service......  I am not on Version 0.25.2 yet if you made any changes in that one

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2851
    • View Profile
    • Codecore Technologies
Re: Global Cache issue
« Reply #10 on: August 13, 2010, 07:23:28 pm »
This is definately fishy.  I checked my GC100 and it is still connected (with no disconnects) and it still responds to commands.

My driver service is using 30MB and the master service is using 28MB and they have been running for a few days without interuption.

I am using the following drivers:
date & time
elk m1
elve media player
smtp
gc100
2 global var drivers
google contacts
http
image library
host monitor
music library
squeezebox
timers
weather channel
z-wave

I've also been monitoring another installation which is just running a few drivers (for about a week) and it doesn't appear to be leaking.

I'll install some of your drivers and see if I get any weird disconnections or leaks.
« Last Edit: August 14, 2010, 09:25:26 am by John Hughes »
John Hughes
Codecore Technologies

Frunple

  • Hero Member
  • *****
  • Posts: 774
    • View Profile
Re: Global Cache issue
« Reply #11 on: August 14, 2010, 05:14:53 am »
I have a feeling it's all related to that leak. Everytime I restart the system everything is fine for a "while" then it all goes haywire.

What OS are you running your tests on?
Win7 Pro 32bit here.

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2851
    • View Profile
    • Codecore Technologies
Re: Global Cache issue
« Reply #12 on: August 14, 2010, 07:37:10 am »
Vista 32
Win7 64
Server 2003

It sounds like the issues occur within about a day for you. If this is the case perhaps you could disable all of your drivers except the ones we are both using then restart the services and see if you still get the issue. This will tell us if it's one of the other drivers. I can run the opposite test as well with only your drivers but a don't have a couple if the devices to connect with.

Also v0.25.2 did fix a couple minor memory issues but they weren't really leaks.
John Hughes
Codecore Technologies

Frunple

  • Hero Member
  • *****
  • Posts: 774
    • View Profile
Re: Global Cache issue
« Reply #13 on: August 14, 2010, 08:40:01 am »
Didn't install it yet. Just got a chance to get back on here so I'll do it now.

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2851
    • View Profile
    • Codecore Technologies
Re: Global Cache issue
« Reply #14 on: August 14, 2010, 09:30:33 am »
I checked this morning and my private working set memory usages are:

Master Service: 24MB
Driver Service: 30MB
Touch Service: 9MB

There's no increase, but the Master Server is lower then before. They values do seem to fluctuate a little bit and they should not grow unless you are using a driver which keeps a growing amount of data in memory such as if you continually add music to your music library or your pop3 mailbox is getting bigger and bigger.

The GC100 is still connected and running fine.

What kind of memory usage are you guys seeing? Keep in mind that when the Driver service starts up it will not use much memory but as the drivers are loaded and started the memory usage will increase until all drivers are finished starting up.

Frunple, it looks like the drivers you are using that I am not are:

HAI Omni Pro II
Nuvo Essentia E6D
Russound (David rules! He RULES!! Still lovin' the tuner control!!)
UPB

I'll take a look at those drivers. Are you actively using the GC-100 or is it fairly unused?

I think the only overlap with mustangcoupe's drivers is the UPB driver.
« Last Edit: August 14, 2010, 09:47:31 am by John Hughes »
John Hughes
Codecore Technologies