Ticket T934155
Visible to All Users

Audit - AuditTrailService is slow when there are many values ​​in the AuditDataItemPersistent table

created 5 years ago (modified 5 years ago)

Hello to all,
my application, use web project, XAF Framework 20.1.7.

I have unexpected behavior of the AUDIT module, especially the time it took to insert the record.

Currently I have this amount of records in the tables related to the AUDIT (AuditTrail)(abaut 5 milion record):

SQL
SELECT count(*) FROM [EAMS_OL_SL].[dbo].[AuditDataItemPersistent] -- 5.053.084 SELECT count(*) FROM [EAMS_OL_SL]. [dbo].[AuditedObjectWeakReference] -- 465.889 SELECT count(*) FROM [EAMS_OL_SL]. [dbo].[XPWeakReference]-- 2.954.143

I did this test (for one record):
A) In the case of the aforementioned record quantity: the time to insert this record is 3-4 seconds (for one record);

C#
using (UnitOfWork UW = Helpers.XpoHelperAudit.GetNewUnitOfWork()) { //AuditTrailService.Instance.BeginSessionAudit(UW, AuditTrailStrategy.OnObjectChanged); //******************************************************************************************* .......... ........ if (vIEPlantList != null) { XPQuery<IEValueList> qIEValueList = new XPQuery<IEValueList>(UW); int cont = qIEValueList.Where(w => w.IEPlantList.Oid == vIEPlantList.Oid && w.DataValueString == Datacsv).Count(); //&& w.DataValue > dataValore.AddMinutes(-1) && w.DataValue < dataValore.AddMinutes(1) ).Count(); if (cont == 0) /// se il valore non e' stato già inserito { //Console.WriteLine("file csv: Value of i:{0} - {1} {2}", riga, Datacsv, Valorecsv); Console.WriteLine(" file csv:{0} Riga:{1}, Data:{2}, Valore:{3}", PathFileName, riga, Datacsv, Valorecsv); /// carico la lista dei valori IEValueList myNewIEValueList = new IEValueList(UW); //myNewIEValueList.DataValue = dataValore; myNewIEValueList.DataValueString = Datacsv; myNewIEValueList.Value = Valorecsv; myNewIEValueList.IEPlantList = vIEPlantList; myNewIEValueList.DataAggiornamento = DateTime.Now; myNewIEValueList.Descrizione = string.Format("File Import: {0}", worksheet.Name); myNewIEValueList.Save(); UW.CommitChanges(); } else { Console.WriteLine("valore già inserito, file csv:{0} Riga:{1}, Data:{2}, Valore:{3}", PathFileName, riga, Datacsv, Valorecsv); myLogger.writeTestLog(string.Format("valore già inserito, file {0} riga {1}, Applicazione {2}", PathFileName, riga, pApplicazione)); } } //******************************************************************************************* Console.WriteLine(" file DateTime.Now):{0} ", DateTime.Now); //AuditTrailService.Instance.QueryCurrentUserName += OnAuditTrailServiceInstanceQueryCurrentUserName; //AuditTrailService.Instance.SaveAuditData(UW); //AuditTrailService.Instance.QueryCurrentUserName -= OnAuditTrailServiceInstanceQueryCurrentUserName; Console.WriteLine(" file DateTime.Now):{0}", DateTime.Now); }

B) In the case of the REDUCED record quantity, I have deleted 90% of the records and the time to insert this record is less than one second (same code as above);

it does not seem to me (from my knowledge) that the insertion time of the records in the table, from the point of view of the t-sql code, varies in this way according to the records count in the table.

so I assume that the "AuditTrail" module executes queries before / after inserting records on the "AuditDataItemPersistent" table.

if so, perhaps an index to be created is missing or the insertion procedure could be improved.

I await your clarification.

Show previous comments (2)
Dennis Garavsky (DevExpress) 5 years ago

    Hello Andrea,

    it does not seem to me (from my knowledge) that the insertion time of the records in the table, from the point of view of the t-sql code, varies in this way according to the records count in the table.

    This time may vary a lot based on record count without ongoing database maintenance. You may find the following articles helpful:

      Hello ,

      Adding XPWeakReference table Index helped a lot with speed, it works fast now.

      Thanks.

      Dennis Garavsky (DevExpress) 4 years ago

        @Murat YONAR: Thank you for sharing!

        Answers approved by DevExpress Support

        created 4 months ago

        Hello,

        In v24.2, we optimized indices used for AuditDataItemPersistent and other audit tables in the database. This change will substantially reduce audit-related save operations:

        Sample Usage Scenario v24.2 (s) v24.1 (s)
        Test #1 (10 runs) Small database (<1000 objects and audit records) 2.94 8.97
        Test #2 (10 runs) Large database (1 million objects, 3 million audit records). 4.33 172.37

        Best regards,
        Herman

          Other Answers

          created 5 years ago

          Hello
          after I have entered the proposed index,
          the intervention time has improved considerably.
          first an entry every 3 seconds.
          now 20 entries for one second.
          Thank you.
          this is the solution.

            Comments (3)
            Dennis Garavsky (DevExpress) 5 years ago

              Thank you for your confirmation, Andrea.

              DC DC
              Domenico Ciavarella 4 years ago

                Hi Dennis, could be useful store audit trail in db no sql (mongo db, cosmos db)

                Dennis Garavsky (DevExpress) 4 years ago

                  @Domenico Ciavarella: I've answered you in T995215: Store audit trail in NoSQL db.

                  Disclaimer: The information provided on DevExpress.com and affiliated web properties (including the DevExpress Support Center) is provided "as is" without warranty of any kind. Developer Express Inc disclaims all warranties, either express or implied, including the warranties of merchantability and fitness for a particular purpose. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.

                  Confidential Information: Developer Express Inc does not wish to receive, will not act to procure, nor will it solicit, confidential or proprietary materials and information from you through the DevExpress Support Center or its web properties. Any and all materials or information divulged during chats, email communications, online discussions, Support Center tickets, or made available to Developer Express Inc in any manner will be deemed NOT to be confidential by Developer Express Inc. Please refer to the DevExpress.com Website Terms of Use for more information in this regard.