Security 101 for Cloud – building it right

For those of you who’ve known me or my work for the last decade or more you’ll appreciate that one of my main call to arms is security and in particular enforcement of security enforcing technologies at the gateway and application level, my little hobby (developing publishing and supporting a firewall technology which with variants based on the code) reached millions of homes, offices and enterprises across the globe and allowed me to make a career out of security.

So it’s often a question I get asked at conferences and when speaking about security in Cloud and security enforcement and responsibility in the Cloud and virtualisation arena. Fortunately at Red Hat we take security incredibly seriously and have contributed technologies such as SELinux and sVirt into our architectures and supported versions of our releases, as well as employing the mainstays in the SELinux world on our payroll to ensure that we have continuity and those folk are rewarded for their efforts.

However, to put it bluntly most architects and network  guys turn SELinux off when building out platforms and virtualised instances which is quite short sighted. When I do pose the question why a lot of responses are aligned to the fact that SELinux can sometimes due to configuration issues and past experiences where stuff broke and was hard to diagnose so easier to just turn off.

Let’s be blunt, it’s there to help you, it’s a free secure template based technology so turning it off if you haven’t got a full toolkit of other security hardening in your build schema or your platform is at best shortsighted. Did I say it was free ? In this current credit crunch culture can you justify not looking at using it ?

If you’re concerned or you struggle then enable it in permissive mode in the first instance making sure you make relevant mods to /etc/sysconfig/selinux to make it persistent on reboot. Simple boolean logic is the best way (and easiest way) to start experimenting with the functionality you want to add. Then if you want to know more then search for the audit2allow function and remember if you’re concerned with restrictive AVC denials breaking stuff then a quick search through auditd in /var/log/audit/audit.log then aureport is your friend. There are loads of howto’s available or if you’re thinking about large scale SELinux use in anger Red Hat even have a course to upgrade your RHCE to give you a complete comfort blanket in your own capabilities. It’s part of the assurance and certification mode we bring to the whole Linux piece. Belt and braces if you will.

Now this article really isn’t a security masterclass or SELinux howto, I’m actually more interested in getting to grips with culture change and trying to pass on my thoughts of how we need to get traction in influencing how protecting your assets, your data and your reputation in Cloud can take shape.

Over the last three years I’ve been using what I would describe as an almost military approach to building out legacy platforms be they physical or virtual. In days of old people might remember Jay Beale and his Bastille Linux hardening script, which was a great starting point when building simple Linux stacks. I remember vividly when he posted it to newsgroups and Slashdot picked up on it. It represented for the first time really in the Linux Open Source community someone who took a simple exercise but made it mainstream towards security as a standard rather than a retrofit. It enabled many of us to not only run it but get under the hood to find out “how” it worked. What is it they say “a little bit of knowledge is a dangerous thing ?”.

So as we move into provisioning our Cloud environments across one or multiple hypervisor types, or moving applications into hybrid or public Cloud having that “accreditation” process or controls breakdown is invaluable. Mine runs over about 5 tabs of a spreadsheet and would make most auditor feel out of a job. However maybe my way of having a moving spreadsheet of controls that I’ve built up over time for all the certifications / governances that I’ve had to deploy to (including in NATO battlefield accredited above classified environments) probably is going a bit far for standard run of the mill server environments.

So its fortunate that my friends and fellow members of the Cloud Security Alliance started many moons ago to put together an authoritative set of controls to allow you to get to work now building out your platforms or engaging with a Cloud provider regardless of the tenacity or the aggressive nature of your certification or audit model. The controls are designed to get you out the blocks building Cloud platformst that need to meet the regulations around ISO 27001/27002, ISACA COBIT, PCI, NIST, Jericho Forum and NERC CIP. Let’s not mention SAS 70. I still, do not, and believe me I’ve tried, understand why an accounting standard has ANY place in Cloud service provision. CCM will help you there and you can also take a look at the CSA STAR programme while you’re there.

I’ve mentioned the Cloud Security Alliance before here numerous times (lets call them the CSA from now on). The CSA are one of the most critical building blocks of the Cloud community and Jim Reavis and the steering members of the CSA have made the education and communication of security best practices to the community their ethos and commitment since they were founded. Red Hat support the CSA and if you’ve heard me talk you’ll hear me mention them proudly on a regular basis. I am continually mentioning them.

Shortly I am recording an often re-arranged podcast with Jim Reavis of the CSA and we’ll get that out to to you as fast as I can mix it in the coming days and weeks.

Whether you’re playing with Cloud in your dev/test sandpit or migrating to a hybrid  cloud understanding what part reputation protection of your app dev environment and your underlying transportation of data is critical. Reputations are lost in minutes as are share prices when a company is seen as damaged by data loss. Simple breaches of major household name organisations are often met with lax fines and investigation by sovereign territory governments and information commissioners, however the risk factors involved are enormous. At the back end of the application architecture – in the trenches – are the technical guys who have to turn the dreams and aspirations of sales people and marketing types into the portals and customer facing Cloud hosted environments that will generate the revenue. If we arm you to do your job better and to do it in a way that allows generic controlled growth of your platforms and your Cloud aspirations then thats a good thing right ?

Do visit the CCM matrixes today and learn how they help you go to work in ways that will make your auditor despair. It’s kinda cool actually because auditing Cloud and typically follow the sun type datacentre clouds has always been a dark art. By following this article and my advice you can actually have a retort to this argument. Cut a huge percentage out your auditors workload (and their resulting invoice) by owning the moral upper ground and in the process maybe think about turning SELinux back on. Blended use of SELinux, sVirt, supported certifed Red Hat subscriptions and technology such as CloudForms gives you everything you need from an IaaS perspective today to go to work. If PaaS security is your thing then listen out soon to another podcast I’m going to record with Tim Kramer of the OpenShift team (in fact if you haven’t already read it go visit Tim’s great security post here).

Also I’m promised a security podcast with Mark Cox at some point in the coming month so if security is your thing you’re going to be kept busy listening to me warble down your earbuds about everything related to CloudSec. If you think that more people could benefit from a primer in Cloud security deployment and the need to think out the box then share this article – I appreciate every Twitter mention I get if it helps educate another Linux user as to how to do things better.

Then get to the CSA website and join. It costs nothing and you’ll learn a lot if you are an active participant. Tell them I sent you.

Technical accumen beats a Crystal Ball in Cloud

Last week while I was on vacation, before I got waylaid in preparing for todays Cloud Computing World Forum in London and next weeks Open.CH Cloud event in Switzerland I promised my snapshot on Gartner’s release a fortnight ago now on EMEA Cloud activity being a pale comparison of the US’s activity.You can read it here, in fact reading it before digesting this article might be a great start.

So before we start let’s be very clear, I’m not remotely out to bash Gartner, they have a well earnt position in the pantheon of analysts and are a valued member of the technical analyst community working hard to help a lot of customers across verticals globally make comprehensive strategies. The report itself lists four specific inhibitors for adjudging that Cloud growth in the EU region as a whole will fall behind the North American marketplace.

Inhibitor 1. Diverse (and Changing) Data Privacy Regulations

Gartner make a good job of outlining the concerns many companies have over data regulation and privacy.They do so without actually going into any concise clarified detail but do at least admit that a lot of the privacy issues are communicated and understood badly by organisations, which is a positive. Certainly the Cloud community as a whole has a duty of care to ensure that we make it easier for companies and institutions to understand that issues such as ENISA and EU guidelines at the provider level and your enshrined responsibilities as a data processor are actually quite simple to quantify. That issues such as the Patriot Act and Safe Harbor that apparently scare many companies off hosting in North America are not actually as realistic as painted. It’s an unwritten rule that even in the EU the liason between intelligence services is acknowledged as making local EU sovereign data privacy controls and the Patriot Act immaterial therefore nullifying the concerns in the first instance. If you read the authoritative report by Hogan Lovells on behalf of the OpenForum Academy published last month you’d understand even more that it should be the Cloud community and providers working harder to communicate this as a non risk to customers regardless of geographical location, that actually if you architecture your public key encryption properly it actually disappears as a risk.

Inhibitor 2. Complex B2B Multienterprise Integration and Processes

In the EU we have a better understanding than most other global territories around working across boundaries. It’s a fact many of the boundaries between organisations in multiple EU territories where data transmission storage and processing occur daily have evolved their own processes based around international standards such as COBIT, ITIL, ISO, BASEL as mandatory controls in business nullifying actual risks to growth. So this inhibitor seems to be badly defined and badly understood as a doorstop to Cloud. EU businesses as a whole adopting Cloud are better positioned than many organisations outside Europe given that we have had corporate governance in place that dwarfs SOX, SAS 70 and less capable non EU derived process controls.

Inhibitor 3. The Slowness and Undesired Effects of Some EU Policies

Gartner do a good job of outlining where they think sovereign mandated process and policy can potentially act as a roadblock to inertia in Cloud. In four years of Cloud specific activity up to and including EU government ENISA guided Cloud architecture I’m yet to identify one actual identifiable deployment slowed down by this “inhibitor”. Gartner then give an example of the European Multi Stakeholder Forums e-invoicing guidelines published in March which are at best a steering piece designed to help and assist organisations rather than slow them down, although it has taken almost five years to get to it’s findings it’s still comforting to know that it exists.

Inhibitor 4. The Investment Hold Caused by the Euro Crisis

I can’t argue with this point, there is a critical crisis of confidence in the euro and the financial markets, this is a technical blog not a financial one. You’d have to have had your head in the sand to have not noticed the major slowdown in IT spend across all areas of technology not just Cloud. It’s an added benefit to the marketplace that Red Hat is positioned to actually allow customers in that position to actually achieve a lot more with a huge amount less and the Open nature of Red Hat cloud technologies and our continued work with emerging technologies to prosper growth during a time of economic and financial instability. In fact Red Hat is growing continually even during a downturn as our customers enjoy so much more capability based on our subscription and Cloud access model for their workloads. This then increases when they see how CloudForms and OpenShift start reducing workload costs and reduce complex associated ownership and process costs.

I’m very surprised that nobody from Gartner read the synopsis of the Cloud Security Alliance’s 2011 study into EU Cloud growth and factors which gave more clarified detail and credible guidance to the very readers that digest Gartner articles as verbatim. I’ve uploaded my copy of their slides here as it returns a more authoritative piece to you towards doing your own clarified research.

So my message here is one of balance. Read the Gartner article, it’s a balanced and authoritative viewpoint from a global leader. Once you’re done then go read the links below:

PC World Report on Data Concerns over Patriot Act
Business Software Alliance report on Cloud in Europe (downloadable pdf)
Jipitec EU Cloud Computing Synopsis

My last words on this article from Gartner is that they missed a trick by forgetting that the same people who read their reports are the same architects and technically capable thought leaders who use open architectures and enjoy more competitive and open economies of scale from using Open Cloud.

If you use an Open Cloud, if you think about your architecture planning and build that portability and security of process and control into your Cloud using tools such as CloudForms then I reckon 80% of the actual inhibitors outlined in the Gartner report become actual reasons to go Open and to speed up Cloud adoption.

Fluffy Clouds = Fluffy Auditors ?

If you’ve sat across a table from me when I am poked gently I’ll often bring up the topic of the need to behave in a way that doesn’t detract from the nature of your responsibilities in computing. What I mean by that is that designing any platform or architecture should have the effect on a developer / architect / sysadmin to generally earn their keep and to stand up to be counted. It’s the one time in a project where the program manager or project lead can honestly do nothing else other than hope and trust that the right people are in situ towards ensuring that the processes and controls, both paper and network/software based are equal in measure to his/her task of getting to deployment.

There is a need now, not later, to think about how Cloud impacts on your ability to satisfy the needs of audit processes. Luckily, companies tending to work with networks and deployments utilising Open Source components, especially supported Red Hat stack components have a general understanding of the need of how platforms need to be constructed and managed. You take your average Linux sysadmin / architect and you put him in a technological boxing ring with his Wintel adversary and my money isn’t on the guy wearing the colour co-ordinating shorts and vest and the Technet subscription being declared victor. I often find, and this isn’t a new phenomenon that your average Linux professional has a tacit understanding of not just his or her platform but also everything that impacts on it. I’ve seen sysadmins wince from across a table in project meetings when the network guy starts talking about upstream network problems because they’re already thinking logfile size and CPU utilisation before the guy has even finished talking. There is a palpable sense of pride and responsibility for most Open Source architectured platforms due to the rite of passage to get them there in the first place when faced with adversarial opposition and the “nobody ever got sacked for buying Microsoft” mantra. Not that theres anything remotely wrong with Microsoft Windows as an environment, it’s just a very closed mindset towards engineering that drives the development and evolution of the platform that makes little to no sense to many of us in Cloud or datacentre centric operations. Bit like being given a burrito – told to eat it and not being allowed to see what’s inside it. Personally I’m a bit fussy when it comes to stuff I consume hence I make a living out of Open Source.

So when I start talking about audit in Cloud, theres an automatic assumption that I’m going to bang on about SAS 70 and the fact many providers in the Public space use it as a get out of jail free card when it comes to their approach to audit response and service level based liability. I’m not going there, there are 101 articles you can grab and reports from WebCPA to Gartner and all points west that will give you balanced industry views from seasoned commentators.

My focus is on specific things that we can do as individuals in Open Cloud to make the impact of audit less of a drain on company resources and management cycles whilst also allowing it to proceed smoother and to learn from both sides of the fence as to what you can sometimes do better. I’ve talked already in previous blog articles here about the Cloud Security Alliance (CSA) Cloud Control Matrixes (CCM). If you didn’t look you should, it’s free and it will give you a grounding at the developer and platform provisioning level of controls you should be using and documenting – or if you’re already installed, retrofitting. I’m a big supporter of CloudAudit.org which last year published it’s API specification which in a nutshell means a public Cloud provider to be able to publish detailed information to a defined namespace to allow proper queries to assist with audit and GRC (Governance Risk and Control) requirements. The fact that they went away as a body and also built a range of rock sold compliance packs which target specific governance types, e.g HIPAA, PCI-DSS, ISO27002 etc and by reflection upping the ante with the marketplace by highlighting differences between vendors. So theres a gap here, a gap that can be filled by the roadwarrior journeyman with his RHEL / RHEV installation building out his cloud to be able to document, based on his or hers tacit understanding. An understanding or realisation of both the global build out of your cloudy endeavours to have time to build attestation reporting and flexible approaches to security and audit as part of 24/7 ops not waiting for your CFO or FC to send you an email to tell you that next week you’re under audit. If you are smart, an early OpenShift or JBoss exponent then you’ll not disagree with me when I say we give you the tools to ensure audit is a 24/7/365 normality not a tickbox in time to satisfy governance.

The great news for you, if at this point you’re getting concerned is that the Red Hat University is running courses on Cloud and also security and virtualisation security that will prepare you to be that man/woman of steel.

Do not get into the liablity argument with project managers. Liability being thrown out as a buzzword by some analysts is hardly something thats new to what we do as an industry. It probably matters more in a hybrid / public world where you’re having to accept an Amazon / Verizon / Rackspace or whoever is your partner to maintain and build out as well as support locally from an app dev environment – each having their own level of documented controls and risk appetites / Service Level Agreements.

Anytime you outsource any business process be it payroll, be it estates management etc you introduce an element of risk. The same risks you can argue is just as pertinent to hand cranked platforms you build internally to support BAU activities. Just at least in your datacentre you can go and pull ethernet / fibre out and hide in the restroom till everyone goes home before you venture out with an iso image nervously to resolve the problem. In the Cloud there is no virtual restroom 🙂

Understanding your risks, setting a risk appetite, using proper automated tools and platform technologies such as OpenShift, JBoss or the soon to be released CloudForms and working out your application blueprints, environments then your business is going to enjoy a more relaxed and shorter audit by virtue of you owning your own destiny and understanding application and platform lifecycle in Cloud. If you outsource to a Cloud provider, especially one where a follow the sun plan takes centre stage to their operations it makes auditing harder and also your descriptive synopsis of the related costs and problem to your CEO and CFO that much more of a bind to deliver. Cloud could be argued to bring in the concept of a chain of liability where a Cloud provider may have other third party vendors in situ to suit your stack – do you have visibility of that – or do you have the right to even know that ?

Is working more openly sounding more attractive by the second ? Outsourcing by its very nature is tricky especially when negotiating liablity and access to audit required functions. I can see you pointing at the screen and saying “This was the same in managed hosting” and I’d agree to a point the only difference was you had more ability to sit down with your provider and work out terms of reference and access for audit. In Cloud with the majority of leading vendors you actually stand to release them from that responsibility unless you understand how to engage or the ecosystems that you will rely on.

Red Hat are your technology partner. The clue is in the word partner – not provider – partner. We have built Cloud Foundation Pathways to help you get through this minefield, backed up by training and support and also the ability to work with us towards making those steps to audit friendly Cloud deployment that doesn’t stop you going to work or shackle you from innovation.

Engage with us, Experience why we’re the number one at what we do in the number one fastest growing industry on the planet. Building the next house of Red Hat on solid foundations and being the compute and node building blocks that are the bedrock of Cloud. Open Cloud.

How to avoid Aasholes

Those of you who have been reading my stuff for almost a decade or using the security stuff I’ve been writing and bringing to the market for more than that length of time will know that I have a passion for security as a business as usual accepted practice. That extends from perimeter security through to application level security and the chagrin of being intelligent about your management and change controls around every aspect of your deployment be it on-premise or in a third party hosted datacentre or hybrid/public Cloud.

One of the reasons for finally joining Red Hat is here is a company that has grown in every aspect of it’s operation that is relied upon by the largest brands and the institutions we all rely upon to handle our financial transactions, our well being and the processing of our needs as consumers. I can be picky who I work for, I do this for the love, not remotely for the money and whoever I work with has to be able to add to what I bring to the table around the whole security value add. Never more so is that intrinsic to what we do as an industry as in Cloud. There is literally nowhere to hide. Security through obscurity is not a practical approach and a zero day exploit or a badly coded application or a drop in escalation of a privilege level can be the difference between a Cloud environment succeeding or failing and a platform collapsing like a pack of cards.

A conversation I often have with friends in the Security space is one of understanding risks. Mark Cox who runs the Security Response team at Red Hat is someone I’ve known for over a decade and who I talk to very regularly. He runs a blog outside Red Hat which is crammed full of illustrations around the maturity of security controls in the Red Hat release and engineering space (see this report from December around the vulnerabilities and advisories and our responses as a vendor for RHEL). Mark’s team work very closely with the engineering teams in Westford and globally to ensure that our appetite for risk (given we’re the platform people rely on to go to work) is entirely focused around visible responses in lightning fast windows.

So why is the title of this article talking about Aasholes, what is an aashole ?

For starters I’d have loved to have coined the description, to be the one adding this to the Cloud vernacular but unfortunately I can’t take the praise for it. Fred Pinkett the popular blogger came up with it and it’s the perfect word to describe a potential or actual security hole in a PaaS, SaaS or IaaS environment. I point you with genuine admiration to his article from June 2011 as a primer on the very basic needs and structures as you build your own Aashole Protection System (let’s just refer to it going forward as an APS).

An APS can take many formats but one thing that I start to try and get across to people, and those of you who have sat and listened to me at conferences or across a table will hear me bang on about controls and mindset to deployment and beyond. I have long been a major fanbois for the Cloud Security Alliance and I work closely with their founder Jim Reavis (watch for an upcoming announcement soon from the CSA about working with Red Hat). Since 2009 I’ve been responsible for signing off and accrediting some of the largest Linux deployments in the most dangerous and critical parts of national and international infrastructure and in the defence sector (or defense for the majority of you reading this article appreciating you already think I spelt datacentre wrong earlier in this article). I would not have been able to do so without being able to take often badly written and badly managed higher level design documents and to cross reference them against the freely developed and distributed Cloud Security Alliance control matrixes or CCM’s. I cannot stress heavily enough or place enough emphasis on why these are uber critical towards getting on your personal radars if you don’t already know what I am talking about.

Here are some pointers why you should already be aware or using them !

1) These controls are free !!! If you haven’t got a copy – get a copy.
2) If you read them and you build and deploy with them in mind you are going to have a very boring life but you’ll be able to rely on your own deployed controls to avoid an Aashole incident.
3) They are a living, breathing document that changes over time – make sure you check for updates as the CSA community have more strength in depth than any blue chip consultancy security company / pen testing organisation / managed services organisation.
4) Working with them when designing your Cloud and working out which apps you can and can’t move to a Cloudy environment and how you fit into legislative governance requirements and audit needs (PCI-DSS/ISO 27001/2/SAS 70/HIPAA etc) will save your organisations tens of thousands of dollars.
5) Using the CSA CCM matrixes alongside proven segregation controls such as sVirt and SELinux templates in RHEL / RHEV deployments will give you the strongest industry controls that you can find. Belt and braces.

So you have the Cloud Security Alliance freely propogating and educating more than any other body in the world around standards adoption and building security as a cornerstone of your application and provisioning environment and you have a healthy fear of a pink slip / P45 / being able to work again because you’re an Aashat (I am claiming this one Fred – sorry) and more than anything you take a pride in what you do as an individual in your team or as a solo warrior in your Cloud efforts within your organisation.

Now if you didn’t read Tim Kramer’s article I posted last week on Security in the Cloud please go read it now.  We’re all about playing safe and being sensible. Nobody wants to be the Aashat who didn’t go the extra mile.

Last but not least we hope to have an interview in Podcast form with Jim Reavis from the CSA that we’ve been trying to get in the can for three weeks but we keep missing diaries / travel schedules. If you’re in Germany and you want to go and hear him speak he’s at the CSA conference in Frankfurt next week, details here.

You can also listen to a podcast I recently recorded with Gordon Haff and Ellen Newlands when I was in Boston around the whole Cloud Security piece in MP3 and OGG formats by following those links.

The Red Hat Security Knowledgebase

Mark Cox has asked me to point out that we have a Security Knowledgebase that is now for the first time publically available from access.redhat.com containing a depth of information that aligned with the CSA controls give you as a practitioner / administrator security in depth and able to work with us to move to Cloud even more securely. Alongside the cookbooks that are available on request (please feel free to ask me for more info) we hope that you find these massively useful.

Just in case anyone reading this has a sight impairment and uses a text to speech / Festival type converter I hope you didn’t have a heart attack listening to the transcription of this article. Sometimes to get a very serious critical point across you have to bow to the influence of others and Fred Pinkett wrote the book on this.