Data Retention in Microsoft Sentinel
When you deploy Microsoft Sentinel, one of the design decisions to make is how long data should be kept. This is part of the data retention configuration for the underlying Log Analytics workspace. The retention period is one of the most important settings for a workspace. It affects the efficiency of the SOC (Security Operations Center) and the cost of its SIEM, but configuring an appropriate retention period is an easy step to miss when setting up a new workspace.
By default, a Log Analytics workspace has a retention period of 30 days. Retention is calculated on the ingestion date for data, so if a workspace uses the default retention period, it means that Azure removes data from the workspace 30 days after its ingestion.
In this article, I review the different elements within the SOC that influence the choice of retention period. Choosing the retention period is highly organization-specific as it depends on factors such as the data, compliance and regulatory oversight, and other needs that the organization might have.
Before we can dive into retention, we should review the different logs within Microsoft Sentinel.
Exploring Log Types
Azure Log Analytics (the log resource on which Microsoft Sentinel uses) has three different log types:
- Analytics
- Basic
- Archive
Analytics logs are the default log type for Log Analytics and offer a good balance between features and price. If you are starting with Microsoft Sentinel, all your tables will probably be Analytics logs. Analytics logs can be retained for 730 days, but they are also the most expensive log type.
Basic Logs can be enabled on a per table level and are cheaper than analytics logs ($ 0.50 compared to $2.6 per GB), but they have three main limitations:
- Retention is limited to 8 days.
- This type of log doesn’t support all queries. For example, you cannot join the data with other tables.
- Queries on basic logs aren’t free and are billed per scanned GB.
The third log type is archive logs. Administrators can configure analytic and basic logs to send their logs to an archive after a certain number of days. Logs retained in archive tables are less expensive compared to analytic logs, but a major limitation exists; You cannot query archive logs interactively through KQL. You need to restore the tables or run search jobs, and both these actions are billed, as explained in this article about log analytics efficiency.
Choosing the Correct Log Type
Before picking an appropriate retention period, we need to choose the right log type for every table.
As mentioned earlier, analytic logs are the default log type and cover most use cases. Basic logs are cheaper but have fewer capabilities. There are specific use cases for basic logs. One of them is enrichment. Imagine ingesting your raw firewall or proxy logs; chances are you won’t create alerts from that data as you already ingested other endpoint logs (like those from Microsoft Defender for Endpoint). But this data might be a way to enrich information for incidents to help with investigations. Another reason is to ingest the logs into archive logs later; you might not need them now but want to retain them in an archive table. As the cost of ingestion for basic logs is lower than analytics, this is an excellent choice.
Archive logs are great when you need the data to be available for searching at some point in the future, but you don’t need to actively access the information. Finding the sweet spot for archive logs can be challenging, but it is important to understand. If you archive logs too quickly, you will query them all the time, and your restore cost will mount up. If you archive them too late, you will end up paying to retain data in the more expensive ‘analytics’ log state.
Workspace vs. Table Level Retention
You can configure the default analytics retention period or enable archive logs at the workspace and table level.
- Configuring it on a workspace level means you define the same retention for all tables in your Microsoft Sentinel workspace.
- Configuring on a table level means defining different retention for each table.
I prefer to mix and match the two options: I define default retention at the workspace level but use table level retention to increase or decrease retention for specific tables.
- Decreasing the default retention period with table level retention allows you to ensure that large tables with limited value move to the archive faster. A great example is the non-interactive sign-in logs table for Azure AD. The data can provide immense value, but it is also the largest Azure AD table, so moving this table to archive faster will decrease costs significantly.
- Increasing the retention of specific tables is a good approach for tables often used for queries that reflect a considerable period of time. An example is using the Azure AD audit logs to investigate who changed a specific setting 6 months ago.
How you set up retention depends on the log type:
- Defining analytics logs retention is done through the Azure Portal.
- Basic logs must be configured through an API, but Microsoft has a workbook available to help.
- Archive logs are also configured through the Azure Portal.
When to Retain Data
Organizations vary in their retention needs for all kinds of data, including Sentinel data. Major influences over the selection of a retention period include the type of business and any legal and regulatory requirements that must be satisfied. Most of the time, this is a decision where a CISO/CIO will need to provide their approval.
- Some organizations are legally obligated to keep logs for several years.
- Others might have a requirement from their security teams to keep the logs for longer periods of time. For instance, they might want to hunt for clues for incidents in retained data for up to six years.
When asked to make a recommendation, I give the following advice:
Follow your organization’s legal obligations. If your organization must retain data for 3 years, configure Microsoft Sentinel to meet this requirement. But most importantly, don’t overcomplicate it. Most organizations think they need data retention for 1 year, but most will never use the data during that time.
Retention is a form of insurance, and that makes choosing a retention period tricky. Will you need to access data from 6 months ago? Probably not, but if that use case happens, it will probably be for an important reason.
One thing is for sure; I recommend setting up the minimum analytics workspace retention to 90 days, as Microsoft Sentinel includes this for free. The billing only starts if you retain the data for longer than 90 days.
Most customers I know define 180-day retention for their analytics workspace retention and set archive retention to 90 days. This means the data is kept for 9 months. This setup provides a nice balance between retaining data for as long as possible and keeping costs low.
Putting it all Together
Microsoft Sentinel includes several options to retain data covering the type of retention (analytics, basic, and archive) and the scope of the retention (workspace and table level). Good implantation uses a combination of all configuration options, as they all have a specific use case. Finally, try to distinguish important from non-important logs to ensure you don’t pay to retain tables you will never use.
Source : Choosing the retention period for Microsoft Sentinel Workspaces (practical365.com)