Advanced Persistent Threats

Advanced Persistent Threats

Coming to a network near you, or maybe your network!

 

There are things that go bump in the night and that is all they do. But once in a while things not only go bump in the night, they can hurt you. Sometimes they make no bump at all! They hurt you before you even realize that you’re hurt. No, we are not talking about monsters under the bed or real home intruders; we are talking about Advanced Persistent Threats. This is a major trend that has been occurring at a terrifying pace across the globe. It targets not the typical servers in the DMZ or the Data Center, but the devices at the edge. More importantly, it targets the human at the interface. In short, the target is you.

Now I say ‘you’ to highlight the fact that it is you, the user who is the weakest link in the security chain. And like all chains, the security chain is only as good as its weakest link. I also want to emphasize that it is not you alone, but myself or anyone or any device for that matter that accesses the network and uses its resources. The edge is the focus of the APT. Don’t get me wrong, if they can get in elsewhere they will. They will use whatever avenue they find available. That is another point. The persistence part, they will not go away. They will keep at it until eventually they find a hole, however small and exploit it. Once inside however they will be as quiet as a mouse. Being unknown and undetected is the biggest asset to the APT.

How long this next phase goes is not determinable. It is very case specific. Many time’s its months if not years. The reason why is that is not about attacking, it’s about exfiltration of information from your network and its resources and/or totally compromising your systems and holding you hostage. This will obviously be specific to your line of business. In the last article we made it plain that regardless of the line of business there are some common rules and practices that can be applied regardless to the practice of data discovery. This article hopes to achieve the same goal. To not only edify you as to what the APT is but illustrate its various methods and of course provide advice for mitigation.

We will obviously speak to the strong benefits of SDN Fx and Fabric Connect to the overall security model. But as in the last article, it will take a second seat as it is the primary practices and use of technology regardless of its type, as well as the people, policies and practices that are mandated. In other words, a proper security practice is a holistic phenomenon that is transient and is only as good as the moment of space and time it is in. We will talk to our ability and perhaps soon the ability of artificial intelligence (AI) to think beyond the current threat landscape and even perhaps learn to better predict the next steps of the APT. This is how we will close. So, this will be an interesting ride. But its time you took it.

What is the Advanced Persistent Threat?

In the past we have dealt with all sorts of viruses, Trojans and worms. Many folks ask, what is different now? Well in a nutshell, in the past these things were largely automated software devices that were not really discerning on the actual target. In other words, if you were a worm meant for a particular OS or application and you found a target that was not updated with the appropriate protection you nested there. You installed and then looked to ‘pivot’ or ‘propagate’ within the infected domain. In other words in the past these malicious software were opportunistic and non-discretionary in the way they worked. The major difference with the APT is that they are targeted. They are also typically staffed and operated by a dark IT infrastructure. They will still use the tools, the viruses, the Trojans the worms. But they will do so with stealth and the intent is not to kill but to compromise, perform exfiltration and even establish control. They will often set up traps that once it is clear they have been discovered they will run a ransomware exploit as they leave the target. This gives them a lasting influence and extension of impact.

In short, this is a different type of threat. This is like moving from the moving columns of ancient roman armies to the fast and flexible mounted assaults of the steppe populations out of Asia. The two were not well suited for one another. In the open lands, the horseback was the optimal. But in the populated farm areas and particularly in the cities, the Roman method proved superior. This went on for centuries until history and biology decided the outcome. But afterwards there was a new morphing, the mounted knight. A method which took the best from both worlds and attempted to combine them and by that created a military system that lasted for almost a thousand years. So we have to say that it had a degree of success and staying power.

We face a similar dilemma. The players are different, as are the weapons, but the scenario is largely the same. The old is passing away and the new is the threat on the horizon. But I also want to emphasize that no one throughout the evolution of warfare probably threw a weapon away unless it was hopelessly broken. Folks still used swords and bows long after guns were invented. The point is that the APT will use all weapons, all methods of approach until they succeed. So how do you succeed versus them?

Well, this comes back to another dilemma. Most folk cannot account for what is on their networks. As a result they have no idea of what a normal baseline of behavior is. If you do not have any awareness of that how do you think you will catch and see the transient anomalies of the APT? This is the intention of this article. To get you think in a different mode.

The reality of it is that APT’s can come from anywhere. They can come from any country, even internally to your organization! They can be for any purpose, monetary, political, etc. They will also tend to source in the country where the target is and use the ambiguity of DNS mapping to trace ‘home’. This is what makes them advanced. They have very well educated and trained staffs who are mounting a series of strong phases of attack against your infrastructure. Their goal is to gain communications and control (C2) channels to either gain exfiltration of information of actual control of certain subsystems. They are not out to expose themselves by creating issues. As a curious parallel there has been a noted decrease in DOS and DDOS attacks on networks as the APT trend has evolved. It’s not that it isn’t used anymore; it’s just that it is now used in a very limited and targeted fashion. Which makes it far more dangerous. Often to cover up some other clandestine activity that the APT is executing and this would be a very last resort. For them being stealth is key to their long term success. So the decreases in these types of attacks make sense when looked at holistically. But note that a major IOT DDOS attack just occurred with home video surveillance equipment. Was it just an isolated DDOS or was it to get folks to turn their attentions to it? We may never know. These organizations may be nation states, political or terrorist groups, even corporations involved in industrial espionage. The APT has the potential to be anywhere and it could put its targets on anything, anywhere, at any time according to its directives. The reason why they are so dangerous is that they are actual people who are organized and who use their intelligence and planning against you. In short, if they know more about your network than you do… you lose. Pure and simple.

So what are the methods?

There has been a lot of research on the methods that APT’s will use. Due to the fact that this is largely driven by humans, the range can be very wide and dynamic. Basically it all gets down to extending the traditional kill chain. This concept was first devised by Lockheed Martin to footprint a typical cyber-attack. This is shown in the illustration below.

Figure 1. The traditional ‘kill chain’

The concept of infiltration needs to occur in certain fashion. An attacker can’t just willy-nilly their way into a network. Depending on the type of technology, the chain might be rather long. As an example, compare a simple WEP hacking example against a full grade Enterprise WPA implementation with strong micro-segmentation. There are many degrees of delta in the complexity of the two methods. Yet, many still run WEP. The APT will choose the easiest and most transparent method.

Reconnaissance

In the first initial phase of identifying a target a dark IT staff is called together as a team. This is known as the reconnaissance or information gathering phase. In the past, this was treated lightly at best by security solutions. Even now with highlighted interest in this area by security solutions, it tends to be the extended main avenue of knowledge acquisition. The reason for this is that much of this intelligence gathering can take place ‘off line’. There is no need to inject probes or pivots at this point. This is like shooting into a dark room and hoping you hit something. Instead the method is to gain as much intelligence about the targets as possible. This may go on for months or even years, as it continues as the next step and even the others occur. Note how I say ‘targets’. This notes that the target, when analyzed will result in a series of potential target systems. Now in the past these were typically servers, but now this may not be the case. The APT is more interested in the users or edge devices. These devices are typically more mobile with a wider degree of access media type. There is also another key thing on many of these devices. They have you or me at the interface.

Infiltration

Once the attacker feels that there is enough to move forward the next step is to try to establish a beach head into the target. In the past this was typically a server somewhere, but folks have been listening and following the advice of the security communities. They have been hardening their systems and keeping up to date and consistent with code releases. Score one for us.

There is the other side of the network though. This is more of a Wild West type of scenario. In the old west of the United States law was a tentative thing. If you were in a town out in the middle of nowhere and some dark character came into town, your safety was as good as the sheriff, which typically didn’t last the first night. Your defense was ‘thin’. Our end points are much the same way. As a result, truly persistent professional teams that are advanced in nature will target the edge, more specifically, the human at the edge. No one is immune. In the past a phishing attempt was easier to see. This has changed recently in that many times these attempts will be launched from a disguised email or other correspondence with an urgent request. The correspondence will appear very legitimate. Remember the APT has done their research. It appears to have the right format and headers; it is also from your manager. He is also referring to a project that you currently are working on with a link indicating that he needs to hear back immediately as he is in a board meeting. The link might be a spreadsheet; a word document… the list goes on. Many people would click on this well devised phish. Many have. There are also many other ways, some that in the right circumstances does not even require the user to click.

There are also methods to create ‘watering holes’ which is basically an Infiltration of websites that are known to be popular or required with the target. Cross site scripting is a very common set of methods to make this jump. Once visited the proper scripts are run and the infiltration then begins. A nice note is that this has fallen off due to improvements in the JRE.

There are also physical means. USB ‘jump sticks’. These devices can carry malware that can literally jump into any designed system interface. There is no need to log on to the computer. Only access to the USB port is necessary and even then only momentarily. In the right circumstances a visitor could wreak a huge amount of damage. In the past this would have been felt immediately. Now you might not feel anything at all. But it is now inside your network. It is wreaking no damage. It remains invisible.

Exploitation (now the truth of the matter is that it’s complicated)

When the APT does what it does if it is successful you will not know it. The exploit will occur and if undiscovered continue on. It is a scary point to note that most APT infiltrations are only pointed out after the fact to the target by a third party such as a service provider or the law enforcement. This is sad. It means that both the infiltration and exploitation capabilities of the APT are very high. The question is how does this get accomplished? The reality of it is that each phase in the chain will yield information and the need to make decisions as to the next best steps in the attack. Well, the realization is that this is the next step in the tree. This is shown in the figure below there are multiple possible exploits and further infiltrations that could be leveraged off of the initial vector. It is in reality a series of decisions that will take the intruder closer and closer to its target.

Figure 2. The Attack Tree

Depending upon what the APT finds as it moves forward its strategy will change and optimize over time. In reality it will morph to your environment in a very specific and targeted way. So while many folks think that exploitation is it. It’s really not. In the past it was visible. Now it’s not. The exploitation phase is used to further implant into the network.

 

Execution or Weaponization

In this step there is some method established to the final phase which is either data exfiltration or complete command and control (C2). Note that again, these steps may be linked and traced back. This is important as we shall see shortly. Note that execution is a process that will have a multitude of methods ranging from complete encryption (ransomware) to simple probes or port and keyboard mappers to gain yet further intelligence. Nothing is done to expose its presence. Ideally, it will gain access to the right information and then begin the next phase.

 

Exfiltration

This is one of the options. The other is command and control (C2) which to some degree is required for exfiltration anyways. So APT’s will do both. Hey, why not? Seeing as you are already into the belly of the beast why are you not leveraging all avenues available to you? It turns out that both require a common trait; an outbound traffic requirement. At this point if the APT wants to pull the desired data out of the target it must establish an outbound communication. This is also referred to as a ‘phone home’ or ‘call back’. These channels are often very stealthy and they also are typically encrypted and mixed within the profile of the normal data flow. Remember, while there are well known ports assigned that we all should comply to, an individual with even limited skills can generate a payload with ‘counterfeit’ port mappings. DNS, ICMP and SMTP are three very common protocols for this type of behavior. It’s key to look for anomalies in behavior at these levels. The reality of it is that you need some sort of normalized baseline before you can judge whether there is an anomaly. This makes total sense.

If you bring me to edge of a river and say “Ed, tell me the high and low levels”, I could not reliably provide you with that information given what I am seeing. I would need to monitor the river for a length of time. To ‘normalize’ it, in order to tell you the highs and the lows. Even then with the possibility of extreme outliers. This is very much the same with security. We need to normalize our environments in order to see anomalies. If we can see these odd outbound behaviors early, then we can cut the intruder off and prevent the exploit from completing.

The APT needs systems to communicate in order for the tools to work for them. This means that they need to leave some sort of ‘footprint’ as they look to establish outbound channels. They will often use encryption to maintain a cloak of darkness for the transport of the data.

Remember, unlike the typical traditional threat which you probably are well prepared for. The APT will look to establish a ‘permanent’ outbound channel. The reason I use quotes around permanent is that these channels may often jump sessions, port behaviors or even whole transit nodes if the APT has built enough supporting malicious infrastructure into your network. Looking at the figure below, if the APT has compromised a series of systems; it has a choice on how to establish outbound behaviors.

Figure 3. Established exfiltration channels

The larger the footprint the APT has the better it can adjust and randomize its outbound behaviors, which makes it much more difficult to tease out. So catching the APT early is very key. Otherwise it’s much like trying to stamp out a fire that is growing out of control.

 

Command and Control (C2)

This is the second option. Sometimes the APT wants more than just data from you. Sometimes they want to establish C2 channels. This can be for multiple purposes. As in the case above, it might be to establish a stealth outbound channel network to support exfiltration of data. On the other side of the spectrum this might be complete (C2). Think power grids, high security military, intelligent traffic management systems, automated manufacturing, subways, trains, airlines. The list goes on and on.

The reality of it is that once the APT is inside most networks it can move laterally. This could be through the network directly but it might also be through social venues that might traverse normal segment boundaries. So the lateral movement could be at the user account level, the device, or completely random based on a set of rules. Also, let’s not forget the old list of viruses, web bots and worms that the APT can use internally within the target and on a very focused basis. It has the vectors for transport and execution. Note how I do not say outright propagation, in this case it is much more controlled. As noted above once the APT has established a presence at multiple toeholds it’s very tough to knock it out of the network. A truly comprehensive approach is required to mitigate these outbound behaviors. It starts at the beginning, the infiltration. Ideally we need to catch it there. But the reality is that in some instances this will not be the case. I have written about this in the past. With the offense there is the nature of surprise. The APT can come up with a novel method that has not been seen before by us. So we are always vulnerable to infiltration to some degree. But if not cutting it off before it enters we can work to prevent the exploit and later phases of attack. While not perfect, this has merit. If we can make the infiltration limited and transient in its nature the later steps become much more difficult to accomplish. We will speak to this later as it is a very key defense tactic that if done properly is very difficult to penetrate past. Clearly these outbound behaviors are not the time to finally detect something, particularly if you pick it out of weeks of logs. The APT has already established its infrastructure, you are in reaction mode.

The overall pattern (hint – its data centric)

By now hopefully you are seeing a strong pattern. It is still nebulous and quite frankly it always will be. The offense still has a lot of flexibility. For us to think that the APT will not evolve is foolish. So we need to figure out a way to somehow co-exist with its constant and impinging presence. Due to its advanced and persistent nature (hence the APT acronym) the threat cannot be absolutely eliminated. To do so would make systems totally isolated. And while this might be desired to a certain level for certain systems as we will cover later, we have to expose some systems to the external Internet if we wish to have any public presence.

Perhaps this is another realization. We should strongly limit our public systems and strongly segment with no confidential data access. When you get down to it, the APT is not about doing a DDOS attack on your point of sales. It’s not even about it absconding credit card data on a one time hit. None of these are good for you obviously. But the establishment of a persistent dark covert channel out of your network is one of the worst scenarios that could evolve. By this time you should be seeing a pattern. It’s all about the data. They are not after general communications or other such data unless they are doing further reconnaissance. They are about moving specific forms of information out or executing C2 on specific systems within the environment. Once we recognize this we see that the intent of the APT is long term residence and preferably totally stealth. The figure below shows a totally different way to view these decision trees.

Figure 4. A set of scoped and defined decision trees

Each layer from outer to center represents different phases in the extended kill chain. As can be seen they move from external (access), to internal (pivot compromise) and target compromise kill chains. You can also see that the external points are exposed vulnerabilities that the APT could leverage. These might be targeted and tailored email phishing or extensive water holing. There may also be explicit attacks against service points discovered. The goal is to establish a network of pivot points that can allow for a better exposure of the target. The series of decision trees all fall inward towards the target and if the APT gets its way and goes undiscovered, this will be the footprint of its web within the target. It is always looking to expand and extend it but not at the cost of losing secrecy. Its major strength lies in its invisibility.

So the concept of a linear flow to the attack has to go out the window. Again, this is the key term to persistence. This is very cyclic is the way it evolves over time. The OODA loop comes to mind which is typically taught to military pilots and quick response forces is – Orient, Observe, Decide, Access. The logic that the APT uses is very similar. This is because it is raw constructive logic. Trying to break down OODA any further becomes counterproductive, believe me many have tried. So you can see that the OODA principle is well established by the APT. Remain stealth, morph and move. But common to all of this is the target. Note how everything revolves around that center set of goals. If you are starting to see a strategy of mitigation and you haven’t read my previous article then my hat is off to you. If you have read my article and see the strategy then my hat is off to you as well. If you have not read my article and are puzzled – hang on. If you have read my last article and you are still puzzled I need to say emphatically. It’s all about the data!!!

We also should start to see and understand another pattern. This is shown in simpler terms in the diagram above; there is an inbound, a lateral and an outbound movement to the APT. This is the signature of the APT. While it looks simple, the mesh of pivots that the APT establishes can be quite sophisticated. But from this we can begin to discern that if we have enough knowledge of how our network normally behaves we can perhaps tease out these anomalies, which obviously did not exist before the APT gained residence. Note the statement I just made. Normalization means normalization against a known secure environment. A good time to establish this might be after compliance testing for example. You want to see the network as it should be.

Once you have that, you should with the right technologies and due diligence be able to see any anomalies. We will talk later about these in detail, but it can range from odd DNS behavior to random encrypted outbound channels. We will speak to methods of mitigation, detection as well as provide a strategic roadmap on goals against the APT realizing that we have limited resources available in our IT budgets.

So is this the end of IT Security as we know it?

Given all of the trends that we have seen in the industry one is tempted to throw up their arms and give up. Firewalls have been shown to have shortcomings and compromises; encryption has been abused as a normal mode of operation by the APT. What good is anti-virus in any of this? Many senior executives are questioning the value of the investment that they have made into security infrastructure, particularly if you are an executive of an organization that has been recently compromised.

After all, encryption is now being used by the bad guys, as are many other ‘security’ approaches. The target has shifted from the server to the edge. Does this mean that we jettison all of what we have built because it is no longer up to the challenge? Absolutely not! It does however indicate that we need to rethink how we are using these technologies and how they can be used with newer technologies that are coming into existence. Basically, the concept of what a perimeter is needs to change and we will discuss this in detail later on, but additionally we need to start thinking more aggressively in our security practice. We can no longer be sheep sitting behind our fences. We must learn to be more like the wolves. This may sound parochial but take a look at the recent news on the tracking and isolation of several APT groups not only down to the country of origin but the actual organization and in some instances even the site! This is starting to change the rules on the attackers.

But this is the stuff of advanced nation state cyber-warfare, what can the ‘normal’ IT practitioner do to combat this rising threat? Well, it turns out there is quite a bit. And it turns out that aside from launching your own attacks (which you shouldn’t do obviously), there is not much that the nation states can do that you can’t do. So let’s put on some different hats for this article. Let’s make them not black, but very nice dark gray. The reason why I say this is that in order to be really effective in security today you need to think like the attacker. You need to do research; you should attempt penetration and exploitation yourself (in a nice safe ISOLATED lab of course!). In short, you need to know them better than they know you because in the end it’s all about information. We will return to this very shortly. But we also need to realize that we need to create a security practice that is ‘data centric’. It needs to place the most investment in the protection of critical data assets that are often tiered in importance. Gone are the days of the hard static perimeter and the soft gooey core. We need to carry micro-segmentation to the nth degree. The microsegments need to not only strongly but exactly correspond to the tiers of the risk assets mentioned earlier. Assets with more risk should be ‘deeper’ and ‘darker’ and should require stronger authentication and much more granular monitoring and scrutinizing. All of this makes sense but it only makes sense if you have your data in order and have knowledge as to its usage, movement and residence. This gets back to the subject of my previous article and it sets the stage well for this next conversation. If you have not read it, I strongly urge you to do so before you continue.

 

Information and warfare

This is a relationship that is very ancient, as ancient as warfare itself. The basic premise is three fold. First, aggressors (hence weapon technology to a large part) has had the advantage in the theory of basic conflict. After all, it’s difficult to design defenses against weapons that you do not know about yet. But it doesn’t mean the defense lacks the ability to innovate either. As a matter of fact with a little ingenuity almost anything used in offense can be used for defense as well. So we need to think aggressively in defense. We cannot be passive sheep. Second, victory is about expectation. Expectation on a plan, on a strategy of some sort to achieve an end goal; in essence very few aggressive conflicts have no rationale. There is always a reason and a goal. Third, information is king. It to a very large degree will dictate the winners and the losers in any conflict, whether its Neolithic or modern day cyber-space. If the attacker knows more that you do, then you are likely to lose.

OK Ed! You might be saying wow! We are talking spears and swords here! Well, the point is that it’s not much has changed since the inception of conflict itself. Spying and espionage goes back as far as history, perhaps further. Let us not forget that it was espionage, according to legend that was the downfall of the Spartan 300. I can give you dozens (and dozens) of examples of espionage throughout history right up to modern times. Clandestine practice is certainly nothing new. But there may be a lot of things that we as security folk have forgotten along the way. Things that that the attackers might still remember; in today’s world if the APT knows more about your network and applications than you do; if they know more about your data than you do. You are going to lose.

Here you may be startled at the comment. How dare I. But if the question is extended to “Do you have a comprehensive data inventory? Is it mapped to relevant systems and validated? Do you know where its residence is? Who has access?” Many cannot answer these questions. The problem is that that APT can. They know where your data is and they know how it moves through your network, or at least they are in constant effort to understand that. They also understand where they can do exfiltration of the data as well. If they know and you don’t, they could be pulling information for quite a long time and you will not know. Do you think I am kidding? Well consider this. About 90% of the information compromises that occur are not discovered by internal IT security staff, they are notified of them by third parties such as their service providers or law enforcement agencies. Here is another sobering fact, the APT on average had residence in the victims network for 256 days.

So clearly things are changing. The ground as it were, is shifting underneath our feet. The traditional methods of security are somehow falling short. Or perhaps they always were and we just didn’t realize it until the rules changed. In any event, the old ‘keep ‘em out’ strategy is no longer sufficient. We need to realize that our networks will at some point be compromised. We will talk a little later as to some of the methods. Because of this, we need to shift our focus to detection. We need to identify the foreign entity and hopefully remove it before it does to much damage or gains to much knowledge. So IT security as we know it will not go away. We still require firewalls and DMZ’s, we will still require encryption and strong identity policy management as well as intrusion detection technologies. We will just need to learn to use them differently then we have in the past. We also have to utilize new technologies and concepts to create discrete visibility into the secure data environments. New architectures and practices will evolve over time to address these imminent demands. This article is intended to provide baseline insight into these issues and how they can be addressed.

 

It’s all about the user (and I’m not talking about IT quality of experience!)

Whenever you see a movie about hacking you always see someone standing in front of several consoles, cracking into various servers and doing their mischief. It’s fast moving and very intense. I always laugh because this is most definitely not the case. Slow and steady is always best and the server is most definitely not the place to start. It’s you. You are the starting point.

Think about it, you move around. You have multiple devices. You probably have less stringent security practices than the IT staff that maintains the server. You are also human. You are the weakest link in the security chain. Now I’ve spoken about this before but it has always been from the perspective of IT professionals who are not as diligent as they should be in the security practice of their roles. Here we are talking about the normal user, who may not be very technically savvy at all. Also, let’s consider that as humans we are all different. Some are more impulsive. Some who are more trusting. Some who simply don’t care. This is the major avenue or rather set of avenues that an attacker could use to gain compromised access into the network. Let’s look at a couple.

Deep Sea Phishing –

Many folks are aware of the typical ‘phishing’ email that says ‘Hey, you’ve won a prize! Click on this URL below!’ Hopefully, most folks now know not to click on the URL. But the problem is that this has moved into new dimensions with whole orders of magnitude in the increase of intelligence behind these types of attacks. As I indicated earlier, much of the reconnaissance that an APT does is totally off of your network. They use publicly posted information. News updates, social media, blog post (yikes – I’m writing one now!). They will not stop there either. There is a lot of financial data and profiling as well as the tagging of individuals to certain organizational chains and projects. Once the right chain is identified the phishing attack is launched. The target user receives a rather normal looking email from his or her boss. The email is about a project that they are currently working on and that they need to hear back on some new numbers that are being crunched. Could they take a look and get back to them by the end of the day. Time is of the essence as we are coming to the end of the quarter. They need to hear back by end of day. Many would open the spreadsheet and understandably so. HTML enabled email makes it even worse in that the SMTP service chain is obscured making it difficult to see the odd chain. And even then, many users wouldn’t even notice that. Many data breaches have occurred in just such a scenario. Once the url is clicked or the document is opened, the malicious code goes to work and establishes two things. The first is command and control back to the attacker, the second is evasion and resilience. From that point of presence the attacker will usually privilege escalate the local machine and then utilize it as a launching point to gain access to other systems.

The Poisoned Watering Hole or Hot Spot –

We all go out on the web and we all probably have sties that we hit regularly. We all go out to lunch and most probably go to our favorite places regularly. This is another thing that attackers can leverage the concept that we are creatures of habit. So let’s change the scenario. Let’s say that the attacker gets a good profile of the targets web behavior. They also learn where the target goes for lunch. But they don’t even need to know that. Typically they will select a place that is popular with multiple individuals in the target organization. That way the probability of will provide greater hits. Then they will emulate the local hot spot with aggressive parameters to force the targets to associate with it. Once that occurs the targets would gain internet access as always but now the attacker is in the middle. As the targets go about using the web they can be re-directed to poisoned sites. Once the hit occurs the attacker shuts down the rogue hot spot and then waits for the malicious code that is now resident on the targets to dial back. From the target users perspective the WLAN dropped and they simply re-associate to the real hot spot. Once the users go back to work, they log on and as a part of it they establish an outbound encrypted TCP connection to the APT. These will not be full standing sessions however, but intermittent. This makes the behavior seem more innocuous. The last thing that the APT wants is to stand out. From there the scenario proceeds much like before.

In both of the scenarios the user is the target. There are dozens of other examples that could be given but I think the two suffice. The human behavior dimension is just too wide to expect technology to fulfill the role, at least at this point. Until then we need firm clear policies that are well understood by everyone in the organization. There also needs to be firm enforcement of the policies in order for them to be effective. This is all in the human domain, not in the technology domain. But technology can help.

 

It’s all about having a goal as well

When an advanced persistent threat organization first starts to put you in their sites, they usually have a pretty good idea of what they are looking for or what they want to do. Only amateurs gain compromised access and then rummage or blunder about. It’s not that an APT wouldn’t take information that it comes across if it found it useful, but they usually have a solid goal and corresponding target data set. What that is depends on what the target does. Credit Card data is often a candidate, but it could be patient record data, confidential financial or research information, the list can be endless. We discussed this in my previous article on data discovery and micro-segmentation practices. It is critical that the critical data gets identified and accounted for. Because you can bet that the APT has.

This means that there is deliberate action on behalf of the APT. Again, only amateurs are going to bungle about. The other thing is that time is, unlike in the movies, not of essence! The average residency number that I quoted earlier illustrates this. In short, they are highly intelligent to their targets, they are very persistent and will wait many months until the right opportunity to move and they are very quiet.

This means that you need to get your house in order on the critical data that you need to protect. You need to know how it moves through your organization and you need to establish a solid idea of what normal is within those data flows. Then you need to move to fight to protect it.

The Internet – The ultimate steel cage

When you think about it, you are in the ultimate steel cage. You have to have a network. You have to have an Internet presence of some sort. You need to use it. You cannot go away. If you do you will go out of business. You are always there and so is the APT. The APT also will not go away. It will try and wait and wait and try and go on and on until it succeeds in compromising access. This paradigm means that you cannot win. No matter what you as a security professional does in your practice, the war can never be won. But the APT can win. It can win big. It can win to the point on putting you out of business. This creates a very interesting set of gaming rules if you are interested in that sort of thing. In a normal zero sum game, there is a set of pieces or tokens that can be won. Two players can sit down and with some sort of rules and maybe some random devices such as dice play the game. The winner is established by the first player to win all of the tokens. But if we remove the dice we have a game more like chess where the players’ lose or win pieces based on skill. This is much more akin to the type of ‘game’ that like to think that we play in information security. Most security architects I know do not use dice in their practice. Now in a normal game of chess, each player is more or less equal with the only real delta being skill. But remember you are sitting at the board with the APT. So here are the new rules. You cannot win all of his or her pieces. You may win some but even if you come down to the last one, you need to give it back. What’s more, there will not be just one. There will be ‘some number’ of pieces that you cannot win. Let’s say that it’s a quarter or maybe even half of the pieces are ‘unwinnable’. Well, it is pretty clear that you are in a losing proposition. You cannot win. The best you can do is stay at the board for as long as you can. Then also consider that the APT’s skill and resources may be just as great if not greater than yours. Does that help put things in perspective?

So the scenario is stark, but it is not hopeless. The game can actually go on for quite some time if you are smart in the way you play. Remember, I said ‘some number’ of pieces that you cannot win I did not say which types. If you look at a chess board you will note that the power pieces and the pawns are exactly half the count. This means that you could win all or most of the power pieces and leave the opponent with a far minimized ability to do damage to you as long as you aren’t stupid. So mathematically the scenario is not hopeless, but it is not bright either. While you can never win you can establish a position of strength that allows you to stand indefinitely.

Realize that the perimeter is now everywhere

Again, an old notion is that we can somehow draw a line our network and systems is becoming antiquated. The trends in BYOD, mobility, virtualization and cloud have forever changed what a security perimeter is. We have to realize that we are in a world of extreme mobility. Users crop up everywhere demanding access from almost anywhere, with almost any consumer device. These devices are also of consumer grade with little or no thought to systems security. As a result these devices, if not handled correctly with the appropriate security practices become a very attractive vector for malicious behavior.

This means that the traditional idea of a network perimeter that can be protected is no longer sufficient. We need to realize that there are many perimeters and these can be dynamic due to the demands of wireless mobility. This doesn’t mean that firewalls and security demarcations are no longer of any use; it just means that we need to relook at the way we use them and compare them with new technologies that can vastly empower them.

It is becoming more and more accepted is that micro-segmentation is one of the best strategies for a comprehensive security practice and to make it as difficult as possible for the APT. But this can’t be a simple set of segments off of a single firewall but multiple tiered segments with traffic inspection points that can view the isolated data sets within. The segmentation provides for two things. First, it creates a series of hurdles for the attacker, both on the way in and on the way out as they seek the exfiltration of data. Second and perhaps less obvious, segmentation provides for isolated traffic patterns with very narrow application profiles as well as interacting systems. In short, these isolated segments are much easier to ‘normalize’ from a security perspective. Why is this important? It is important because in the current environment 100% prevention is not a realistic proposition. If an APT has targeted you, they will get in. You are dealing with a very different beast here. The new motto you need to learn is that “Prevention is an ideal, but detection is a MUST!”

In order to detect you need to know what is normal. In order to make this clear let’s use a mundane example of a shoplifter in a store. The shoplifter wants to look like any other normal shopper, they will browse and try on various items like anyone else. In other words they strongly desire to blend into the normal behavior of the rest of the shoppers in the store. An APT is no different. They want to blend into the user community and appear like any other user in the network. As a matter of fact they will often commandeer normal users machines by the methods discussed earlier. They will learn the normal patterns of behavior and try as much as possible to match them. But at some point, in order to shoplift the shopper needs to diverge from the normal behavior. They need to use some sort of method to take items out of the store undetected. In order to do this, they need to avoid video surveillance direct views and allow for a time where they can ‘lift’ the items. But regardless of the technique, there needs to be delta. Point A, product… point B, no product. The question is will it be noticed. This is what detection is all about. In a retail environment it is also accepted that a certain amount of loss needs to be ‘accepted’ as the normal business risk for operations. The reasons being for this is that there is a cost point where further expense in the areas of prevention and detection do not make any fiscal sense.

It is very much the same thing with APT’s. You simply cannot seal off your borders. They will get in. The question is how far they penetrate and how much they are able to discover about you and what information they are able to pull out. There is common joke in the security industry, it goes like this. “If you want a totally secure computer, unplug all network connections. Seal it off physically with thick walls, including all and any RF with no entrance. Then take several armed guards and an equivalent number of very large attack dogs and place around the perimeter 24 x 7. Also you need to be sure that you have total independence of power, which means you need a totally separate micro grid that in turn cannot be compromised by using the above methods.” Like all tech sector jokes, the humor is dry at best and serves to show the irony of a thought process. Such a perfectly secure computer would be perfectly useless! We like the shop owner need to assume and accept a certain amount of risk and exposure to be on line. It is simply the reality of the situation, hence the steel cage analogy I used earlier. So detection is of absolute key importance to the overall security model.

How to catch a thief

So the next question is how do you detect an APT is in your network? Additionally how do you do it as early as possible taking into consideration that time is on the attackers’ side – not yours. Once again, it serves to revisit the analogy of the shoplifter. Retail outfits usually have store detectives. These individuals are specialists in retail security. They know the patterns of behavior and inflections of movement that will cause a highlight around a certain individual. Many of these individuals have a background in psychology and have been specifically trained to watch for telltale signs. Note that such indicators cannot cause arrest or even ejection from the store. They can only serve to highlight that additional attention is needed on a certain individual. Going further, there are often methods to get into dressing rooms and the counting of items before entry and upon exit. This could be viewed both as a preventative as well as a detective measure. There are also usually RF tags that will flag an alarm if the item is removed from the premises. Often these tags are ink loaded so that they will despoil the product if removal is attempted without the correct tool. All of this can be more or less replicated in the cyber environment. The key is what to look for and how to spot it.

A compromised system

This is the obvious thing to look for as it generally all starts here. But the problem is that APT’s are pretty good at hiding and staying under cover until the right time. So the key is to look for patterns of behavior that are unusual from a historical standpoint. This gets back to the concept of normalization. In order to know that a user’s behavior is abnormal, it is important to have a good idea on what the normal behavior profile is. Some things to look for are unusual patterns of session activity. Lots of peer to peer activity where in the past there was little or none. Port scanning and the use of discovery methods should be monitored as well. Look for unusual TCP connections, particularly peer to peer or outbound encrypted connections.

Remember that there is a theory to all types of intrusion. First, an attacker needs to compromise the perimeter to gain access to the network. Unless the attacker is very lucky, they will not be where they need or want to be. This means that a series of lateral and northbound moves will be required in order to establish a foothold and command and control. This is why it is not always a good idea to take a suspicious or malicious node off of the network. You can gain quite a bit by watching it. As an example, if a newly compromised system begins to implement a series of scans and no other behavior then it is probably an isolated or early compromise. If the same behavior is accompanied by a series of encrypted TCP sessions then there is a good probability that the attacker has an established footprint and is working to expand their presence.

Malicious or suspicious activities

Once again normalization is required in order to flag unusual activities on the network. If you can set up a lab to provide an idealized ‘clean’ runtime environment, a known good pattern and corresponding signature can be developed. This idealized implementation provides a clean reference that is normalized by its very nature. After all, you don’t want to normalize an environment with an APT in it now do you? Once this clean template is created, it is easy to spot deltas and unusual patterns of behavior. These should be investigated immediately. Systems should be located and identified with the corresponding user if appropriate. There may or may not be the confiscation of equipment. As pointed out earlier, sometimes it is desirable to monitor their activities in a controlled fashion with the option of quarantine at any point.

 

Exfiltration & C2  There must be some kind of way out of here                                  (Said the joker to the thief)

In order for any information to leave your organization there has to be an outbound exfiltration channel that is set up prior. Obviously, this is something that the APT has been working to accomplish in the initial phases of compromise. Again, going back to the analogy of the shoplifter, this is another area where the APT has to diverge from the normal behavior of a user. The APT needs to establish a series of outbound channels to move the data out of the organization. In the earlier days, a single outbound TCP encrypted channel would be established to move data as quickly as possible. But now that most threat protection systems are privy to this, they tend to establish networks that can utilize a series of shorter lived outbound sessions, moving only smaller portions of the data so as to blend in to the normal activities of the network. But even with this improvement in technique, they still have to diverge from the normal user pattern. If you are watching close enough you will catch it. But you have to watch close and you have to watch 24 by 7.

Here is a list of things that you want to look for,

1). Logon activity

Logon’s to new or unusual systems can be a flag of malicious behavior. New or unusual session types are also an important flag to watch for, particularly new or unusual out bound encrypted session. Other flags are unusual time of day or location. Watch also for jumps in activity or velocity as well as shared account usage or privileged accounts.

2). Program execution

Look for new or unusual program executions or the execution of the programs at unusual times of the day or from new or unusual locations. Or the executing of the program from privileged account status rather than a normal user account.

3). File access

You want to catch data acquisition attempts before they succeed with access, but if you can’t, you at least want to catch the data as it attempts to leave the network. Look for unusual high volume access to files servers or unusual file access patterns. Also be sure to monitor cloud based sharing uploads as these are a very good way to hide in the flurry of other activity.

4). Network activity

New IP addresses or secondary addresses can be a flag. Unusual DNS queries should be looked into, particularly those with a bad or no reputation. Look for the correlation between the above points and new or unusual network connection activity. Also look for unusual or suspicious application behaviors. These could be dark outbound connections that may use lateral movement internally. Many C2 channels are established in this fashion.

5). Database access

Most users do not have to access the database directly. This is an obvious flag, but also look for manipulated applications calls that doing sensitive table access, modifications or deletions. Also be sure to lock down the database environment by disabling many of the added options that most modern databases provide. Be aware that many of them are enable by default. Be sure to be aware of what services are exposed out of the database environment. An application proxy service should be implemented to prevent direct access in a general fashion.

6). Data Loss Prevention methods

Always monitor sensitive data movement. As pointed out in the last blog, if you have performed your segmentation design correctly according to the confidential data footprint then you should already have isolated communities of interest that you can monitor very tightly, particularly at the ingress and egress to the microsegments. Always monitor FTP usage as well as mentioned earlier cloud services.

Analysis, but avoid the paralysis

The goal is to arrive at a risk score based on the aggregate of the above. This involves the session serialization of hosts as they access resources. As an example a new secondary IP address is created and an outbound encrypted session is established to a cloud service, but earlier in the day or perhaps during the wee hours that same system accessed several sensitive file servers with the administrator profile. Now this is a very obvious set of flags, these can and will be increasingly more subtle and difficult to tease out. This is where security analytics enters the picture. There are many vendors out there who can provide products and solutions in this space. There are several firms and consortiums that can provide ratings for these various vendors so we will not even attempt to replicate here. The goal of this section is on how to use it.

The problem with us humans is that if we are barraged with tons of data and forced to do the picking out of significant data, we are woefully inefficient. First of all, we have a very large capacity for missing certain data sets. How often have you heard the saying, “Another set of eyes”? It’s true, though we don’t like to admit it, when faced with large data sets we can miss certain patterns that others will see and visa-versa. This brings two lessons two lessons. First never manually analyze data alone, always have another set of eyes go over it. Second, perhaps we are not the best choice for this type of activity. There is another reason to look at though. It’s called bias. We are emotional beings. While we like to think we are always intellectual in our decisions this has been proven not to be the case. As a matter of fact, many neurologist researchers are saying that without emotions, we really can’t make a decision. At its root decision making for us is an emotional endeavor.

So enter computers and the science of data analytics. Computers and algorithms do not exhibit the same shortcomings as us humans. But they exhibit others. They are extremely good at sifting through large sets of data and identifying patterns then analyzing them against certain rules such as those noted above. They are also extremely fast in these tasks when compared to us. What they offer will be unadulterated and pure without bias, IF and only if the algorithms are written correctly and do not induce any bias in their design. This whole subject warrants another blog article sometime, but for now let be safe to say that algorithms and theories of operation as well as application design are all done by us. So the real fact of the matter is that there will be biases that are embedded into any solution. But there is one thing that computers do not do well yet. They can’t look at patterns and emotionally ‘suspect’ an activity ‘knowing’ the normal behavior of a user. As an example, to say to itself, “Fred just wouldn’t do this type of thing. Perhaps his machine has been compromised. I think I should give him a call before I escalate this. We can confiscate the machine if this is true, get him a replacement and get the compromised unit into forensics.” Note that I say for now. Artificial intelligence is moving forwards at rapid pace, but what is to say that AI will eventually roadblock on bias just like we have! Many cognitive researches are now coming to this conclusion. So it is clear that we and computers will be co-dependent for the foreseeable future, each side keeping the other from invoking bias. The real fact is that there will always be false negatives and false positives. The cyber-security universe simply moves too fast to assume otherwise. So the concept of setting and forgetting is not valid here. These systems will need assistance from humans, particularly once a system has been identified as ‘suspect’.

Automation and Security

At Avaya we have developed a shortest path bridging networking fabric we refer to as SDN Fx that is based on three basic self-complimentary security principles.

Hyper-segmentation

This is a new term that we have coined to indicate the primary deltas of this new approach to traditional network micro-segmentation. First, hyper-segments are extremely dynamic and lend themselves well to automation and dynamic service chaining as is often required with software defined networks. Second, they are not based on IP routing and therefore do not require traditional route policies or access control lists to constrict access to the micro-segment. These two traits create a service that is well suited to security automation.

 

Stealth

We have spoken to this many times in the past. Due to the fact that SDN Fx is not based on IP, it is dark from an IP discovery perspective. Many of the topological aspects to the network, which are of key importance to an APT simply cannot be discovered by traditional port scanning and discovery techniques. So the hyper-segment holds the user or intruder into a narrow and dark community which has little or no communications capability with the outside world except through well-defined security analytic inspection points.

Elasticity

This refers to the dynamic component. Due to the fact that we are not dependent on IP routing to establish service paths, we can extend or retract certain secure hyper-segments based on authentication and proper authorization. Just as easily however, SDN FX can retract a hyper-segment, perhaps based on an alert from security analytics that something is amiss with the suspect system. But as we recall, we may not want to simply cut the intruder off but place them into a forensic environment where we can watch their behavior and perhaps gain insight into methods used. There may even be the desire to redirect them into Honey pot environments where whole network can be replicated in SDN Fx for little or no cost from a networking perspective.

Welcome to my web (It’s coated with honey! Yum!)

If we take the concept of the honey pot and extend it with SDN Fx, we can create a situation where the APT no longer has complete confidence of where they at and whether they are looking at real systems. Recall that the APT relies on shifting techniques that evolve over time, even during a single attack scenario. There is no reason why you could not so the same. Modern virtualization of servers and storage along with the dynamic attributes of SDN Fx create an environment where we can keep the APT guessing and ALWAYS without a total scope of knowledge about the network. Using SDN Fx we can automate paths within the fabric to redirect suspect or known malicious systems to whatever type of forensic or honey pot service required.

Avaya has been very active in building out the security ecosystem in an open system approach with a networking fabric based on IEEE standards. The concept of closed loop security now becomes a reality. But we need to take it further. Humans still need to communicate and interact about these threats on a real time basis. The ability to alert staff for threats and even set up automated conferencing where staf can compare data and decide on the next best course of action are now possible as such services can be rendered in only a couple of minutes in an automated fashion.

Figure 6. Hyper-segmentation, Stealth and Elasticity to create the ‘Everywhere Perimeter’

All of this places the APT in a much more difficult position. As the illustration above shows, hyper-segmentation creates a series of hurdles that need to be compromised before access to a given resource is possible. Then it becomes necessary to create out bound channels for the exfiltration of data across the various hyper-segment boundaries and associated security inspection points. Also note that as the figure above illustrates, you can create hyper-segments where there simply is no connectivity to the outside world. For all intents and purposes they are totally and completely orthogonal. The only way to gain access is to actually log into the segment. This creates even more difficultly for the APT as exfiltration becomes more difficult and if you are watching, easier to catch.

In summary

One could say and most probably should say that this was occurrence that was bound and destined. While I don’t like the term ‘destined’, I must admit that it is particularly true here. As our ability to communicate and compute has increased it has created a new avenue for illegal and illegitimate usage. The lesson here is that the Internet does not make us better people. It only makes us better at being what we already are. It can provide immense transformative power to convert folks to perform unspeakable acts and it can in a few hours’ notice take a global enterprise to its knees.

But it can also be a force for a very powerful good. As an example, I am proud to be involved in the effort on behalf of colleagues such as Mark Fletcher and Avaya in the wider sense to support Kari’s law for the consistent behavior of 9-1-1 emergency services. Mark is also actively engaged abroad in the subject of emergency response as I am for security. The two go hand in hand in many respects because the next thing the APT will attempt is to take out our ability to respond. The battle is not over. Far from it.