Author Topic: Driver Service taking all available processing time.  (Read 3424 times)

Paul_Sharrock

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Driver Service taking all available processing time.
« on: September 21, 2011, 11:49:02 pm »
HI
I've come to my PC this morning to find my PC running extremely slowly.
On further investigation the Driver Service is using all the available CPU resource ranging from 85-99%.

On looking at the logs, its got about 100 errors since last night. An example of which are below:-

Example 1
------------

An error occurred while executing the script rule 'HAI - Zone Status Change'.
ScriptEvaluatorException: An unknown error occurred while evaluating the script.  SocketException: An existing connection was forcibly closed by the remote host
SocketException SocketErrorCode: ConnectionReset

CodecoreTechnologies.Elve.DriverFramework.Scripting.ScriptEvaluatorException: An unknown error occurred while evaluating the script. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

Server stack trace:
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Runtime.Remoting.Channels.SocketStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Runtime.Remoting.Channels.ChunkedMemoryStream.WriteTo(Stream stream)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.GetRequestStream(IMessage msg, Int32 contentLength, ITransportHeaders headers)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.SendRequest(IMessage msg, ITransportHeaders headers, Stream contentStream)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.SendRequestWithRetry(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream)
   at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at
  • :

   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at CodecoreTechnologies.Elve.DriverFramework.Scripting.IScriptObject.InvokeMethod(String methodName, IScriptObject[] parameters, IScriptObject& result)
   at Common.ScriptEvaluator.ScriptEvaluator.objectMemberAndArrayMember()
   at Common.ScriptEvaluator.ScriptEvaluator.postIncrementDecrement()
   at Common.ScriptEvaluator.ScriptEvaluator.factor()
   at Common.ScriptEvaluator.ScriptEvaluator.multiplicative()
   at Common.ScriptEvaluator.ScriptEvaluator.additive()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseShift()
   at Common.ScriptEvaluator.ScriptEvaluator.relational()
   at Common.ScriptEvaluator.ScriptEvaluator.equality()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseAnd()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseExclusiveOr()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseInclusiveOr()
   at Common.ScriptEvaluator.ScriptEvaluator.logicalAnd()
   at Common.ScriptEvaluator.ScriptEvaluator.logicalOr()
   at Common.ScriptEvaluator.ScriptEvaluator.factor()
   at Common.ScriptEvaluator.ScriptEvaluator.multiplicative()
   at Common.ScriptEvaluator.ScriptEvaluator.additive()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseShift()
   at Common.ScriptEvaluator.ScriptEvaluator.relational()
   at Common.ScriptEvaluator.ScriptEvaluator.equality()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseAnd()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseExclusiveOr()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseInclusiveOr()
   at Common.ScriptEvaluator.ScriptEvaluator.logicalAnd()
   at Common.ScriptEvaluator.ScriptEvaluator.logicalOr()
   at Common.ScriptEvaluator.ScriptEvaluator.variableAssignment()
   at Common.ScriptEvaluator.ScriptEvaluator.ifKeyword()
   at Common.ScriptEvaluator.ScriptEvaluator.tryKeyword()
   at Common.ScriptEvaluator.ScriptEvaluator.newKeyword()
   at Common.ScriptEvaluator.ScriptEvaluator.objectMemberAndArrayMember()
   at Common.ScriptEvaluator.ScriptEvaluator.postIncrementDecrement()
   at Common.ScriptEvaluator.ScriptEvaluator.factor()
   at Common.ScriptEvaluator.ScriptEvaluator.multiplicative()
   at Common.ScriptEvaluator.ScriptEvaluator.additive()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseShift()
   at Common.ScriptEvaluator.ScriptEvaluator.relational()
   at Common.ScriptEvaluator.ScriptEvaluator.equality()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseAnd()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseExclusiveOr()
   at Common.ScriptEvaluator.ScriptEvaluator.bitwiseInclusiveOr()
   at Common.ScriptEvaluator.ScriptEvaluator.logicalAnd()
   at Common.ScriptEvaluator.ScriptEvaluator.logicalOr()
   at Common.ScriptEvaluator.ScriptEvaluator.variableAssignment()
   at Common.ScriptEvaluator.ScriptEvaluator.evaluateBlock()
   at Common.ScriptEvaluator.ScriptEvaluator.evaluate(ScriptParser parser, ScriptVariableCollection variables, EvaluateFunctionDelegate evaluatefunction, ScriptObjectTypeInfoCollection objectInfos, ScriptFunctionInfoCollection scriptFunctionInfos)
   --- End of inner exception stack trace ---
   at Common.ScriptEvaluator.ScriptEvaluator.evaluate(ScriptParser parser, ScriptVariableCollection variables, EvaluateFunctionDelegate evaluatefunction, ScriptObjectTypeInfoCollection objectInfos, ScriptFunctionInfoCollection scriptFunctionInfos)
   at Common.ScriptEvaluator.ScriptEvaluator.Evaluate(ScriptParser parser, ScriptVariableCollection variables, EvaluateFunctionDelegate evaluatefunction, ScriptObjectTypeInfoCollection objectInfos, ScriptFunctionInfoCollection scriptFunctionInfos)
   at Common.ScriptEvaluator.ScriptEvaluator.Evaluate(String s, ScriptPosition position, ScriptVariableCollection variables, EvaluateFunctionDelegate evaluatefunction, ScriptObjectTypeInfoCollection objectInfos, ScriptFunctionInfoCollection scriptFunctionInfos)
   at Common.ScriptEvaluator.ScriptEvaluator.Evaluate(String s, ScriptVariableCollection variables, EvaluateFunctionDelegate evaluatefunction, ScriptObjectTypeInfoCollection objectInfos, ScriptFunctionInfoCollection scriptFunctionInfos)
   at Common.ScriptEvaluator.ScriptEvaluator.Evaluate(String s, ScriptVariableCollection variables, EvaluateFunctionDelegate evaluatefunction, ScriptObjectTypeInfoCollection objectInfos)
   at Common.ScriptEvaluator.ScriptEvaluator.EvaluateNow(String s, ScriptVariableCollection variables, EvaluateFunctionDelegate evaluatefunction, ScriptObjectTypeInfoCollection objectInfos)
   at MasterService.ScriptRunner.RunNow(String scriptName, String script, ScriptVariableCollection extraScriptVariables, Boolean throwError)
Location in script: line 1, column 36.
if (!vars.IsDefined( "smsusername" ))
                                   ^

Example 2
------------

An unexpected error occurred in the 'omni' device in HandleExtendedZoneStatusResponse

NullReferenceException: Object reference not set to an instance of an object.

System.NullReferenceException: Object reference not set to an instance of an object.
   at BuiltInDrivers.HaiOmniDriver.HandleExtendedZoneStatusResponse(clsOmniLinkMessageQueueItem M, Byte[] B, Boolean timeout)


Example 3
------------

Failed to retrieve weather information from NOAA.

SoapException: Program degrib returned no data.

System.Web.Services.Protocols.SoapException: Program degrib returned no data.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at BuiltInDrivers.NoaaWeather.ndfdXML.NDFDgen(Decimal latitude, Decimal longitude, String product, DateTime startTime, DateTime endTime, weatherParametersType weatherParameters)
   at BuiltInDrivers.NoaaWeatherDriver.getWeather()

All the remaining other errors (100+) are on the same lines as the above ones. I have seen example 3 before and dont think its related.

Using restart system within the diagnosis tab, doesn't stop the driver service using all the CPU.

Stopping the services manually stopped the running processes, however, the driver quickly ramped up to using 1600Mb of memory before settling back at 650Mb.

I'll update to the new beta, as the two issues that have been fixed will help me, but I doubt it will cure this problem.

Paul

 

Paul_Sharrock

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Re: Driver Service taking all available processing time.
« Reply #1 on: September 21, 2011, 11:56:23 pm »
On updating to the new version the driver service is only taking up 58Mb of memory.

Stability could be an issue here.

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2852
    • View Profile
    • Codecore Technologies
Re: Driver Service taking all available processing time.
« Reply #2 on: September 22, 2011, 07:49:36 am »
1. The 99% cpu could be caused by an infinite loop in a rule, be sure to check your rules to ensure this could not happen. If this happens again you can check the Elve Windows Performance monitor to determine if a rule is running endlessly.

It could also be caused by a bug in one of the drivers. If you see this consistently you may need to start disabling drivers to find the culprit.

The socket error is likely a symptom of the driver service being  at 99% but could be a network connection issue.

2. The HAI error occured while processing an extended zone status response. Elve uses this to capture zone temperatures and humidities. The next beta (.70) will log the received data so I can attempt to reproduce the issue if you see this again.

3. NOAA's web service is unreliable and can cause this error. This is not a bug in Elve. We recommend using the Weather channel driver.
« Last Edit: September 22, 2011, 08:03:47 am by John Hughes »
John Hughes
Codecore Technologies

Paul_Sharrock

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Re: Driver Service taking all available processing time.
« Reply #3 on: September 22, 2011, 08:00:46 am »
Lets see if it happens again.

Yes it could be script, but I don't think it will be, as there are very few areas in the scripts that loop, and I tested each one with extended logging to make sure it was working.

Paul

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2852
    • View Profile
    • Codecore Technologies
Re: Driver Service taking all available processing time.
« Reply #4 on: September 22, 2011, 08:11:03 am »
I updated my post above (just the #1 section).

I was hoping to update it before you ready the thread but you beat me to it. :)

PS: I posted a new beta which fixes the rule execution error.
John Hughes
Codecore Technologies

Paul_Sharrock

  • Jr. Member
  • **
  • Posts: 75
    • View Profile
Re: Driver Service taking all available processing time.
« Reply #5 on: September 22, 2011, 08:16:25 am »
Please see new thread on rule execution errors with new beta.

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2852
    • View Profile
    • Codecore Technologies
Re: Driver Service taking all available processing time.
« Reply #6 on: October 07, 2011, 02:37:49 pm »
I looked into the NOAA error more and found that they changed their web service interface which causes the driver to not work at all. Version 1.2 will be updated to use the latest NOAA web service. Hopefully their web service is more reliable than it used to be.
John Hughes
Codecore Technologies

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2852
    • View Profile
    • Codecore Technologies
Re: Driver Service taking all available processing time.
« Reply #7 on: February 19, 2012, 04:00:46 pm »
How are things going?

Is the driver service running smoothly for you now?
John Hughes
Codecore Technologies

John Hughes

  • Administrator
  • Hero Member
  • *****
  • Posts: 2852
    • View Profile
    • Codecore Technologies
Re: Driver Service taking all available processing time.
« Reply #8 on: May 09, 2012, 12:42:50 am »
I moved this topic to fixed since I have not heard otherwise.
John Hughes
Codecore Technologies