Company: Link Technologies
Case No: L13025. Project: 15.40: LinkSOFT Version 15.40
Logged By: Vineshwar Prasad (Edge Business Solutions) on behalf of Sanjay (Link Technologies) on 29 Apr 2024 12:24PM
Priority: High
Product: Payroll & HR
Group: Software Defect
Time Taken: 105.00 (Weight: 125.00)
Assigned To: Sanjay (Link Technologies)
Circulation: Sanjay, Vineshwar Prasad
Resolve By: Wednesday, 31 July 2024 12:15 AM [206 days since logged date]
Status: Closed
Subject: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Summary:    
1. Error Message (screenshot of the error)

ID: 23851 Date: Apr 28 2024 6:11PM User: LinkSOFT-FijianHoldings Module: System.Data Machine: LINKAU-AZURE Status: Error

Log Message: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.

Log Data: {"ClassName":"System.InvalidOperationException","Message":"Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":" at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\r\n at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\r\n at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\r\n at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)\r\n at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)\r\n at System.Data.SqlClient.SqlConnection.Open()\r\n at Microsoft.Practices.EnterpriseLibrary.Data.Database.GetNewOpenConnection()\r\n at Microsoft.Practices.EnterpriseLibrary.Data.Database.GetOpenConnection(Boolean disposeInnerConnection)\r\n at Microsoft.Practices.EnterpriseLibrary.Data.Database.ExecuteReader(DbCommand command)\r\n at LT.CORE.DAL.UtilityDAL.GetScheduleJobNames() in D:\\Source - LinkSOFT\\LinkTechnologiesCore - 2.0\\DAL\\UtilityDAL.cs:line 442","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":"8\nTryGetConnection\nSystem.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.Data.ProviderBase.DbConnectionFactory\nBoolean TryGetConnection(System.Data.Common.DbConnection, System.Threading.Tasks.TaskCompletionSource`1[System.Data.ProviderBase.DbConnectionInternal], System.Data.Common.DbConnectionOptions, System.Data.ProviderBase.DbConnectionInternal, System.Data.ProviderBase.DbConnectionInternal ByRef)","HResult":-2146233079,"Source":"System.Data","WatsonBuckets":null}
    Audit Notes:Edited by sanjay on 25/07/24 14:57. Edited by sanjay on 24/07/24 12:14. Edited by sanjay on 23/07/24 15:09. Edited by sanjay on 23/07/24 14:23. 
    10 May 202401:56PM Comment 1 by Sanjay (Link Technologies) This comment has been removed
    10 May 202403:18PM Comment 2 by Vineshwar Prasad (Edge Business Solutions) This comment has been removed
    15 May 202412:31PM Comment 3 by Sanjay (Link Technologies) Assigned To: Vineshwar Prasad (Edge Business Solutions) Followup Date: 15-05-2024 06:26 PM Time Taken: 5.00 Notes: ETC extended from: 11/05/2024 to 15/05/2024

    Thanks Vineshwar,

    I have added a "Purge Configuration" that removes FMIS transactions after a certain number of days. The default is 365, howerer, you can set this to "1" for companies not using FMIS integration.

    I have also removed the "Process Group" from FHL TNA process. If you need custom Groups, let me know so I can see how we can handle this in Azure.  This can be an issue on AZURE as the SQL pool is shared. 

    Can you monitor and see if this reduces the Timeout issue?


    16 May 202402:11PM Comment 4 by Vineshwar Prasad (Edge Business Solutions) Assigned To: Sanjay (Link Technologies) Followup Date: 16-05-2024 08:10 PM Notes: ETC extended from: 15/05/2024 to 16/05/2024
    Yes, I will make the changes and monitor the issue and advise if it happens again.

    16 May 202404:26PM Comment 5 by Sanjay (Link Technologies) Assigned To: Sanjay (Link Technologies) Followup Date: 16-05-2024 10:25 PM Time Taken: 1.00
    Thanks Vineshwar, we will close this case. Reopen if the issue is not resolved.

    19 Jul 202406:16PM Comment 6 by Sanjay (Link Technologies) Assigned To: Sanjay (Link Technologies) Followup Date: 23-07-2024 12:15 AM Time Taken: 1.00 Notes: Edited by sanjay on 19/07/24 18:17. ETC extended from: 16/05/2024 to 23/07/2024
    Reopened case as the issue is occurring again in the FHL environment.

    19 Jul 202406:18PM Comment 7 by Sanjay (Link Technologies) Assigned To: Sanjay (Link Technologies) Followup Date: 23-07-2024 12:16 AM Time Taken: 40.00

    Completed the following:

    1. Added logging to show hourly Process status
    2. Applied to FHL environment

    Next Step:

    1. Review logs on the next few days.

    19 Jul 202406:18PM Comment 8 by Sanjay (Link Technologies) Case L13025 added to project 15.40
    20 Jul 202404:26PM Comment 9 by Sanjay (Link Technologies) Assigned To: Sanjay (Link Technologies) Followup Date: 22-07-2024 10:25 PM Time Taken: 18.00

    Following changes have been made:

    1. Force closure and disposal of commonly high frequency procedures

    Next Step: Monitoring


    23 Jul 202402:25PM Comment 10 by Sanjay (Link Technologies) Assigned To: Sanjay (Link Technologies) Followup Date: 23-07-2024 08:24 PM Time Taken: 26.00

    Client environment monitoring over the last few days shows no sign of accumulating processes in SQL Server. 

    Connection Pool monitoring has been added to monitoring system.


    25 Jul 202408:43AM Comment 11 by Sanjay (Link Technologies) Assigned To: Sanjay (Link Technologies) Followup Date: 25-07-2024 02:33 PM Time Taken: 12.00

    Table 1 below shows Connection summary after applying update

    Table 2 shows Connection Summary before applying the update


    25 Jul 202402:57PM Comment 12 by Sanjay (Link Technologies) ETC was changed from 23/07/2024 to 31/07/2024
    25 Jul 202403:01PM Comment 13 by Sanjay (Link Technologies) Assigned To: Sanjay (Link Technologies) Followup Date: 25-07-2024 09:00 PM Time Taken: 2.00
    UAT Completed. Issue is resolved.

    If you have any queries regarding this support incident, please email admin@linktechnologies.com.au and include the Case No: L13025 in the subject line of all emails regarding this issue.

    Document size: 530.5 KB
    For call complaints, please contact the Managing Director of the company using this form