How to cope with data growth

17.02.2016

Implementation is just where it begins. We have an initial data set at the beginning. Later on we start finding out that volume of data is growing and impacts application operation - we experience application slowdown. This is a usual scenario from the application life. Large database causes also problems during back up and data restore (everything takes longer time). Data growth has also financial implications because disk space is not for free no matter whether we speak about disk storage or back up library. Cause of data increase does not have to be only increased number of customers, contracts etc. Much larger impact has data growth in various logs or audit trail tables. It is true that log data can be stored outside the database. However, by this we would be loosing opportunity of an effective utilization of logs by the application. The most of logs are significant in analytical way only for a certain period of time. With time advancing after the event we do not need so detailed data any more. ObjectGears log maintenance solution comes out of this very principle.

You can set required time for which data shall be kept for particular logs and for particular audit trail tables. Older data will be automatically deleted. Let`s take Error log as an example. Exceptions that occur in the application need to be recorded for their furher analysis and correction. It is useful to have records for last days and months at disposal. However, we will not probably need to ge back into the past. Therefore, you can set to this log a regular deletion of old records. We can approach other system logs in a similar way.

Other logs are about application data. If there is audit trail defined in the class, all the changes are logged into a special archive table. Decision whether to use audit trail or not depends on need to trace who changed which data. The log maintenance system enables besides the decision yes or no also to set the time for which we need the audit trail. The older history of changes can be regularly deleted.