IDS vs IPS vs SIEM: What You Should Know

The earliest days of the Internet were the time of a village with no locks on the doors. The users were academics who loved to give away information and trusted each other. There was the occasional prank, but people could live with it.

Then the public got on the Net, and the story became quite different. Viruses and worms appeared. Systems needed protection against hostile access. Necessity led to the development of intrusion detection systems (IDS) and intrusion prevention systems (IPS). They alerted administrators to hostile activity and stopped it from doing harm.

The early attacks were simple ones, taking advantage of glaring software weaknesses. The harm was usually easy to fix, even if it cost time. Firewalls did a pretty good job just by blocking hostile addresses and limiting port access. But as billion-dollar companies have moved onto the Web, the intrusions have evolved into high-stakes crime. People with professional skills put all their effort into devising ways to break into secure systems. Malware as a service has become a reality.

The creators of security measures have kept pace. One of the most valuable developments, going beyond IDS and IPS, is SIEM, Security Information and Event Management. Rather than just looking for specific indicators, it constructs a comprehensive picture of a network’s activity. It looks everywhere for suspicious behavior patterns. It finds threats that less sophisticated security systems miss.

The path to SIEM was a long one, with constant improvements. Threat protection will continue to evolve to meet new tactics.

The history of IDS, IPS, and SIEM

The first well-documented IDS came from Stanford Research Institute in 1983. It applied statistical methods to audit trails to identify anomalous activity. The Intrusion Detection Expert System (IDES) soon followed, using a rule-based approach to spot suspicious behavior. NIDES (Next-Generation Intrusion Detection Expert System) followed, looking for suspicious patterns of behavior. Its “resolver” component screened out false positives. These early systems used techniques that would later be adopted on a much larger scale.

For a while, the trend shifted toward signature-based detection. Malware can often be recognized by specific bit patterns, called signatures, in affected files or packets. Early behavior-based IDS had many false positives, while a bit pattern is unlikely to be a chance match. However, signature matching had its own problems, as the number of patterns to match grew into the thousands. It started to drag down system performance. Besides, it couldn’t catch previously unidentified threats.

The earliest intrusion detection systems were deployed on their host machines. As networks grew in size and complexity, people recognized the need to protect the whole network, not just individual machines. Networks deployed both host-based and network-based IDS for more complete coverage.

IDS vs IPS

An IDS by itself doesn’t fix problems. It detects and reports them. System administrators have to read the report, decide whether it indicates a real problem, and lay out a course of action. This takes time, and meanwhile the reported threat could be interfering with operations and doing additional damage. The next step was to move from mere detection to prevention — to IPS.

An IPS takes automated actions in response to a detected threat. It can close off an IP address, limit access, or block an account. This meant a quicker remedy, but the early efforts at IPS had a problem with false positives. Reporting legitimate activity as malicious is just inconvenient. Blocking it interferes with operations. It took until about 2005 for IPS software to become reliable enough for widespread use.

The terminology has grown blurry. What is called an IDS today often has some intrusion prevention capabilities. Conversely, a so-called IPS may have only a limited automatic response capability. An IPS can run in detection-only mode during its shakedown period so it can be configured to allow legitimate traffic. Always look at the features a system offers and its reputation, not just what it’s called.

The term “advanced persistent threat,” or APT, came into the vocabulary by 2011. These attacks come from well-funded organizations, often with nation-state connections. They lodge malware in target systems in a way that is hard to detect and remove. They siphon confidential information out of the infected systems over weeks or months without attracting notice. Data breaches involving APTs are often disastrously expensive and even destroy businesses. This made the development of powerful IPS technology more necessary than ever.

Threats grew more devious, becoming less susceptible to signature-based detection. They changed their bit patterns periodically. New threats appeared with increasing frequency. Signature detection is useless against zero-day threats. The changing threat environment increased the need for adaptive, behavior-based detection. Machine learning became an important part of detection and prevention.

In the early days, behavior-based detection usually meant spotting single anomalies, such as a ping packet larger than the standard size. Today it includes combinations of factors, any one of which might be doubtful by itself.

Machine learning requires a training period for the system to establish a baseline of normal behavior. A certain amount of deviation is acceptable, but serious shifts require investigation. The baseline will change along with normal network usage and business growth.

IDS and IPS can be either locally installed or cloud-based. When it’s cloud-based, all requests go through a remote location for examination and filtering and then are blocked or forwarded to the network. Local IPS is sometimes a specialized appliance that sits behind the Internet connection, or it can be integrated with a router.

From IPS and IDS to SIEM

Network security has been heading in a more holistic direction, getting information from multiple sources. The technology for correlating security information goes by the name SIEM, for Security Information and Event Management. (Surprisingly, it’s pronounced “sim,” not “seem.”)

An IPS focuses on incoming and outgoing Internet traffic. SIEM gathers information from many sources, correlating all the available information available. This lets it not only detect active threats but find hidden weaknesses and threats. Its inputs include system and application logs as well as live IDS and IPS data.

SIEM has been around since the end of the twentieth century, making it as old as IPS. However, the earliest systems by that name weren’t the comprehensive approach to security which we expect today. The early systems faced scalability limits, and their correlation of information to produce useful notifications was very limited.

It took several technical advances for SIEM to fulfill its promise. A major one was the adoption of large NoSQL databases for the efficient handling of huge amounts of log data. The addition of sophisticated analytics greatly increased the usefulness of its reports, letting analysts haul in usable information from a sea of data. Cloud-based SIEM offered almost unlimited scalability. Dashboards provided a single focus for all the information, allowing faster identification and resolution of problems.SIEM is not a replacement for IDS and IPS. Rather, it’s a more comprehensive approach that makes use of them. IDS and IPS still stand at the front door to screen visitors and keep trespassers out. SIEM uses the information from IDS, IPS, logs, and firewalls to construct a full picture of network security and take measures beyond the screening of hostile traffic. Logs provide the “information” part of the acronym; the others provide the “event” part.

Nor is SIEM a replacement for human expertise. It provides information for administrators and analysts to use. In straightforward cases, it can take automated action, but many apparent security incidents are judgment calls or don’t have a quick and obvious solution. People with the skills to assess a report’s seriousness and urgency have to make the decision in these cases.

The increasing complexity of networks and threats makes it necessary to bring together as much security-related information as possible. Not every network has the size and security needs to justify SIEM, but the rate of adoption is growing.

How a SIEM system works

Setting up a SIEM system is more complicated than installing an IDS or IPS. Companies that use it typically have large networks. It needs access to many information sources. It can be deployed either on-premises or as a cloud service; modern systems are usually cloud-based for greater scalability.

Planning 

Before touching the software, IT management should establish a list of objectives. What information needs the highest priority for protection? What policies should the system follow? Who should have access to the generated information? In brief, what results are expected?

The planning stage includes building a complete network inventory. Knowing all the places that have Internet access or hold important information is a requirement, or the system will have blind spots.

Configuration 

The planning stage leads to concrete configuration decisions. What information sources should SIEM have access to? How should alert thresholds be set? How should people be notified? What information should the dashboard present?

What needs reporting is a matter of policy and judgment. Compliance with privacy and security standards, such as HIPAA and GDPR, is very important when they apply. Rules can be set up to catch laxness in protecting information. Guarding against situations that can’t happen only overloads the system or triggers false alarms; for instance, attacks based on Drupal-specific vulnerabilities are of no concern if you never use Drupal.

A tricky question is how much should remediation be automated. An automated response can deal with a threat faster than people can. It reduces the damage and lets the staff focus on the more difficult issues. However, it might have unwanted effects that a nuanced human response could avoid. Disabling an account or quarantining a system in response to a false positive annoys the people affected and hurts productivity. Relying too much on automation could make IT staff complacent.

Setting up a SIEM environment involves many software components. Information sources need to be connected. Developers will have to create data conversion scripts. Access to SMS, email, or other channels for alerts needs setting up. If SIEM is used together with an IPS, they need to work together without conflict. When two systems compete to fix the same threat, they could have unexpected effects.

The inputs to SIEM come in different formats. It will need input agents and adapters, including custom scripts in some cases. Giving it access to a rich set of information without creating risks is a security challenge in itself. SIEM is not just another software installation, but a task in which security experts must take an active role.

SIEM should be integrated with a threat intelligence source. That way, it will receive regularly updated data on current threats and adapt its algorithms to catch them. New threats constantly appear, and old ones take new forms. Without regular threat intelligence updates, a network is vulnerable to the latest hostile innovations.

Working with a SIEM system

People need to be assigned roles. The best results come from having a security operations center (SOC) with analysts and engineers who have strong data security skills. The SOC can be outsourced, but it needs contacts in the IT department who can make informed decisions. Roles of responsibility need to be established so that everyone will know exactly what is expected.

The human element is essential to success. IDS, IPS, and SIEM need constant attention. The administrators who manage them will have to adjust the settings as they gain experience and the situation changes. New systems need to be integrated with the set of information sources. Alert thresholds may need raising or lowering.

The status reports that the system issues are as important as the alerts. They identify areas that need improvement and risks that should be corrected.

The relevance of SIEM reports depends on the quality of the data it receives. Logs will need adjustment over time to provide more relevant information and less noise. The configuration needs regular attention to produce good results.

The network will change over time. New machines and services need to be integrated into SIEM’s set of information sources. Log output may change with new application releases, requiring adjustments to their connections.

Many businesses find that using a managed security services provider (MSSP) is the best way to manage a SIEM system. It ensures that updates will happen regularly and alerts will get the attention they need. It can maintain a SOC more cost-effectively than most businesses can. However, the IT department needs to make sure that the MSSP stays abreast of changes in the network and that its practices match company policies.

For advanced protection

Reacting to threats isn’t enough. A secure network needs to stay ahead of them. It has to keep its protection strong and adjust it as the environment changes. SIEM offers more than just incident detection and prevention. It detects weaknesses and bad practices that could lead to trouble, as well as hostile activity before it becomes a full-scale attack.

Intrusion starts out as subtle probing. Hostile actors look for signs of vulnerability such as outdated software, lack of limits on login attempts, and open ports. They analyze the results before deciding where to focus their efforts.

Persistent malware tries to be subtle. It limits the rate that it infiltrates information so that it won’t trigger alarms. Sometimes its only effect is to steal processor cycles for tasks such as mining cryptocurrency. It may set systems up in a botnet that does nothing until the day it’s called up for a major DDoS attack. Discovering anomalies like this takes an in-depth analysis of network behavior, not just the detection of overtly hostile traffic.

Careless employee practices increase risks. Overuse of privileged accounts, visits to unsafe websites, and remaining logged in 24 hours a day are risky behaviors and may violate company policy. Setting up SIEM to detect these habits lets administrators know about them. They can tighten the technical restrictions or issue warnings.

SIEM tools can detect unusual patterns of account usage. If there is activity from a distant location outside of working hours, it might just be on-site work for a customer in another time zone. It could also mean the account has been compromised or an employee is using it for questionable purposes. Administrators should take a moment to find out which it is.

To adopt or not?

Every network is unique. It has a certain size, level of security needs, and budget. A small network that doesn’t manage confidential data has different needs from an enterprise-class network with sensitive personal information. Some organizations find that the combination of a firewall, an IDS, anti-malware software, and awareness training meets their needs.

However, underestimating security requirements can be a costly mistake. If there’s any doubt, the place to start is a comprehensive security assessment, determining the level of risk and the potential consequences of a serious incident. If the risk is significant and the network is complex enough, management should look into SIEM. The question to consider is how well it will improve security compared with other tools and how the cost compares. The study needs to look at the time to set it up, the personnel requirements, and both initial and ongoing costs.

BitLyft offers managed SIEM services as part of its AIR platform. If you’re looking into how SIEM can improve your organization’s security posture, talk with one of our experts. We will present a range of alternatives based on your requirements and budget. In today’s world, strong security is a requirement for every business.