Over the last few years, we have witnessed the start of a true revolution
in how middleware operations are carried out. Starting with the release
of the IBM WebSphere® CloudBurst® Appliance, and then with
the subsequent releases of IBM Workload Deployer and IBM
PureApplication System, the introduction of pattern-based deployment
approaches have helped clients achieve a fundamental change in the way
in which they plan, deploy, and manage IBM middleware. What we have
seen is that this approach has changed the landscape of system
operations, and has also had significant influence on the relationship
between development and operations in those companies that have adopted
it. The combination of these pattern-based approaches with the full
system integration provided in PureApplication System has revolutionized
the way in which the IT organizations that have adopted them function.
However, what we have also found is that in order for the
full benefit of these changes to be realized, these companies need to
also adopt a set of organizational and operational best practices that
maximize the potential of the PureApplication System. This article
documents several of these best practices, and provides a rationale for
why you can realize a major positive effect on the efficiency of your
organization and the total cost of ownership from adopting
PureApplication System.
Before explaining these best practices, we have to begin
with a brief discussion of the type of assets that exist in
PureApplication System, and introduce a classification scheme that will
help you understand the different management techniques we suggest be
put in place for these assets in order to realize the maximum benefits
that you can gain from PureApplication System.
At a very, very high level there are two types of assets in PureApplication
system; for the purposes of this article, let's call those shared assets
and transient assets.
Shared assets are those assets that
are
long-lived resources that are used by multiple teams or
individuals.
These include the physical cloud assets (ITE hardware,
storage, and
networking switches) as well as virtual cloud assets
(environment profiles, cloud groups, IP groups). This category also
includes software assets like images and script packages, and
importantly, virtual system patterns. We classify
virtual system patterns as shared assets because they should be built in
such a way as to maximize their reuse potential. This bears directly on
the organizational structures that are most appropriate for creating
and managing the lifecycle of these assets.
If shared assets are long-lived resources, then by contrast, transient
assets are those that relatively short-lived. Transient assets
include virtual system pattern instances and volume allocations. Another
key type of transient asset is a virtual application pattern.
Virtual application patterns in general (and web app patterns
specifically) contain specific deployable resources that are included as
part of the pattern when it is built. In these "cloud-centric" types of
applications, the topology is a transient asset; the actual topology
can change at run time based on the execution of policies, such as
scalability policies. The definition of the pattern is also a transient
asset; that is, because everything you are specifying in a virtual
application pattern is application-specific. This includes the required
resources (application servers, databases, and so on), as well as the
policies that describe how those resources should be used. In this
case, the EAR file or DDL is simply another application-specific asset
that has the same relative lifetime as all the other aspects of that
particular application.
Each asset in PureApplication System, regardless of
whether it is a shared or transient asset, will have its own lifecycle
and specific monitoring points within that lifecycle. In order to
maximize the utility of PureApplication System, the IT organization
needs to consider setting up an organization responsible for the
following general aspects of asset management:
- Maintaining the catalog or list of assets that are available.
- Setting in place policies around managing the lifecycles of assets.
- Tracking assets through their individual lifecycles and moving them along as appropriate.
While we won't specify an exhaustive list of the types of
governance policies that you will need to put in place, here is a
representative list that covers some of the most common scenarios.
Some of the common shared resource policies would include:
- How long do you keep virtual system pattern versions available for use?
- How long do you support previous or deprecated pattern versions?
- What are the naming conventions for virtual system versions?
Some of the more common transient resource policies would include:
- How much of each resource (storage, CPU, network) do you allow per pattern instance?
- How long do you leave pattern instances (virtual system and virtual application) running if not in use?
- How long do you keep pattern instances available if not currently in use?
When we consider the policies around the management of
physical and virtual
cloud assets (not just patterns), the list grows even
longer. For
instance, if a client owns more than one PureApplication
System, then the
management of pattern catalogs across two instances
becomes an interesting
and important consideration. What patterns are defined
in the software
catalog of each rack and what patterns are deployed onto
each
PureApplication System rack needs to be determined by a
combination of factors, such as the need to support HA and DR
configurations of specific patterns, the usage of each pattern (for
example, a pattern with batch behavior like MDM might have high
utilization only at certain times of the day), and the need to balance
development needs like load testing against production needs. In
general, it is possible to use racks both for production and
development/test, as long as you set up appropriate divisions using cloud groups and environment profiles for isolation.
This section outlines the major best practices that we
have identified for effectively adopting PureApplication System. We
begin by discussing who creates the assets for which the policies are
enforced, and what automation is needed to enforce the asset policies
described.
IBM PureApplication System ships with a number of
existing virtual
application system patterns that represent standard
configurations of IBM
WebSphere Application Server and other middleware, such
as IBM DB2®. However, in most environments you will need to customize
those patterns in one way or another, or you will need to build your own
patterns that combine middleware from IBM and other vendors. Rather
than have every project create its own patterns from scratch, a better
approach is usually to have a common content team create a few standard
patterns that represent the common uses in your organization as a whole,
which are then slightly customized as needed by each application team.
Therefore, you’ll need a few people in your organization with a wide
view of all the different projects in your organization so that they can
do that topology abstraction and determine, first, which patterns need
to be created, and then build them.
Likewise, IBM ships an even higher-level abstraction called virtual
applications that provide developers with a way to specify policies
that describe a desired outcome, and lets the system dynamically determine
how to produce the correct middleware topology to meet those outcomes. Many business
partners also provide virtual application patterns for their own
applications through the IBM PureSystems Centre. However, advanced clients might also want to
build their own virtual application pattern components customized to their
own needs. For that purpose, IBM provides the Plugin Development Kit that enables developers to build
these components. This requires some amount of thought and planning
in order to figure out how to best meet your organization’s needs, so
again, this is usually something that will be done by a centralized
content team.
What we have found is that the alternative to this best
practice (letting each development team create its own patterns) leads
to less reuse than if forethought is put into developing a few reusable
patterns from the start. It also becomes more difficult to manage the
large and poorly organized pattern catalog that results from letting
each individual developer or project team build its own patterns. When
implemented properly, this foundational best practice creates a vehicle
for implementing many of the best practices that follow.
As discussed earlier, PureApplication System contains two
broad classes of assets, transient assets and shared assets. Transient
assets, in particular virtual system pattern instances, have a
particular lifecycle that is managed either by an operations team (in
production) or by a development team (in development), or by a combined
dev/ops team in both environments. What is worth pointing out is that
most of the steps in this lifecycle can be automated. Let's consider
the simple illustrative lifecycle of a virtual system pattern, shown in
Figure 1.
In this simple case (which does not consider the
possibility of patching a
running virtual system instance), a newly provisioned
pattern instance
(let's say in a dev/test environment) begins in the
Started state. At any
time, an administrator can stop the virtual system
instance, which does
not release the reserved resources in the cloud, but
does terminate the virtual machines comprising the virtual system. They
could then Store the virtual system, which does release those
resources, but keeps the virtual system instance available for
redeployment at any time. All of these functions are available through
the Instances > Virtual Systems menu in the workload console.
As discussed earlier, an IT organization should set up
policies regarding how long an instance should reside in each of these
states, and under what conditions they should move from one state to
another. So, if an instance is not used for a period of time (which
could be verified, for example, by referring to the HTTP server's web
logs) then it should be stopped. It could then remain in that state for
a fixed period of time before the owner restarts it, or it should be
moved to the stored state. Finally, if an instance is in stored state
and has not been started for some time, it can be deleted.
Once these policies are set in place and have been
validated over a period of time, then our best practice recommendation
is that they should be automated. This guarantees that the policies will
be enforced consistently and that the shared resources of the
PureApplication System will be utilized most efficiently. The
PureApplication System command line interface and REST API contain
functions that let you query the state of a virtual system pattern
instance, as well as start, stop, store, and delete that instance. You
can either write scripts to automate these functions and implement the
defined policies, or you can write programs that manage this through the
REST interfaces.
Even though we did not cover it in the previous example,
the CLI and REST interfaces also contain functions that enable you to
automate the patching and fix application processes for virtual systems
as well. That is a more complicated case, and should be handled with
care — in many cases, it might be better to simply replace the instance
with another one at the newer patch level.
We have seen several different approaches to automating
these lifecycle steps with different clients, utilizing a number of
different technologies. In some cases, the automation is tied into
existing system automation tools already in use in the datacenter; this
is especially true if the organization is very mature in its use of
virtualization and already had policies in place for managing virtual
images as transient assets. However, in other situations where the
organization is not as mature and does not have these policies and tools
in place, we have seen companies be very successful in building net new
automation with a combination of custom web portals and batch programs
written within WebSphere Application Server.
As a final note, an emerging approach that we believe
holds the most
promise for hosting this kind of automation is IBM
SmartCloud® Orchestrator. You can think of the implementation of these
policies as being different workflows built using SmartCloud
Orchestrator. While it is still early in the adoption of SmartCloud
Orchestrator integrated with PureApplication System, we can easily
envision standard libraries of different PureApplication System asset
lifecycle policy implementations being built using SmartCloud
Orchestrator.
Essentially, when you build a virtual system pattern you
are describing a
general topology onto which you can suitably deploy
transient application assets. If you build a clustered WebSphere
Application Server pattern that connects to a standalone instance of IBM
DB2 Enterprise Edition, it could be potentially used for many different
applications that require that same topology. What differs for each
particular application is detailed application configuration
information, such as JVM configuration (heap size, and so on) and,
critically, application assets such as EAR files and DDL. In a large
organization with many applications, if each application built its own
pattern you would end up with a proliferation of nearly identical
patterns in the library. From simply an asset management perspective, it
would become challenging to manage, maintain, and track all of these
patterns. When we have witnessed this approach of using IBM Workload
Deployer and PureApplication System, the result has been a pattern
catalog that quickly becomes unmanageable.
What we encourage instead as a best practice is that
virtual system patterns should not include purely application-centric
information in the scripts that they execute. Instead, that kind of
specific information should be read in by the scripts that run in the
deployment process as configuration from a shared repository of
information, or from instance-specific locations that are specified as
part of the deployment process. Another possibility is to use an
external deployment automation tool to deploy these application-specific
configuration items into a pattern instance when that tool is triggered
by an external event such as the completion of a new build. This
encourages reuse of each virtual system pattern, and also reduces the
total number of virtual system patterns that must be managed. This also
fits with the tooling and policies that most development and operations
teams already have in place for handling application build and
automated deployment. It then becomes a matter of taking those tools
that the IT department is already productive with and integrating those
into the PureApplication System structure.
As a detailed example, consider the case of script
packages within PureApplication System. In following this best
practice, you would not include either direct references to
application-dependent information in the script package (such as JDBC
datasources, JMS resources, or JVM properties) or include specific
application JAR or EAR files in your script package. Instead, reference
these from an external location. This would enable a decoupling of the
application lifecycle from the lifecycle of the script package and
enable you to deploy the same pattern with the same script package into
multiple environments, such as development, test, and production — even
where the details of the JDBC datasources, JMS resources, or JVM
properties differed in each environment.
IBM provides tooling inside PureApplication System (the
Advanced
Middleware Configuration tool) that can be used to separate
configuration information from the specific deployments themselves.
Likewise, IBM provides a tool called
IBM
SmartCloud Continuous Delivery (which fully
integrates with PureApplication System) that helps development teams
deploy and test software in a production-like environment in a
straightforward, repeatable way. Another possibility is to use IBM
Urban{code} uDeploy to automate artifact deployment into your pattern
instances.
A common question we receive is how big a pattern should
be and what the
scope of applications should be (for example, should
applications all be
contained within a single pattern, or should an
application spread across
patterns). In general, there's no single answer for
this question because
the term "application" means different things to
different people and
organizations. To some organizations, a single
application could comprise
only a single EAR file running on an application server
instance. In
others, a single application can spread over multiple
runtimes, including database runtimes, application server runtimes,
portal server runtimes, and ESB runtimes. Rather, we should rephrase
the question to "How do you determine the right scope of a virtual system pattern?" and "How do you determine the right scope of a virtual application pattern?" For those more tightly-scoped questions, there are some basic principles we can apply.
Essentially, we can phrase the recommendation as follows:
A single pattern should contain only components that are directly
related to each other and should not contain components that are shared
by any components outside the pattern unless all those components are
shared. Let's describe this by example.
Suppose you have two applications, a Web Banking application and a
Wealth Management application. They
each have EAR files that are deployed onto application servers. They
both share some common information in a shared database (the Customer
Accounts database). However, the Wealth Management application also
relies on information from a database that is not used by the Web
Banking application (a Portfolios database). In this case, you should
best divide these applications into three patterns along the following
lines:
- Pattern 1: Web Banking application servers and web servers
- Pattern 2: Wealth Management application servers, web servers and Portfolios database
- Pattern 3: Customer Accounts database
Now, any changes that happen to the individual
applications are localized to the specific patterns that are affected by
those changes. If a change needs to be made to the Web Banking
application, then the Wealth Management application will continue to
function unhindered, and vice versa.
As discussed earlier, one of the approaches that enables
you to gain maximum reuse from virtual applications is separating any
application-specific information in them from the topology-related
information that is general across an entire class of applications. The
application-specific information would then be read from an external
source at deployment time, most likely from an external source code
repository like Git or Subversion. This same principle can and should be
extended to the shared assets in general, such as the definitions of
your virtual system patterns, your script packages, and your virtual
images.
PureApplication System provides an extensive ability to
perform import and export of these shared assets between racks, and from
a rack to external storage. This is useful, for instance, in the case
where one PureApplication System rack might be used for all dev/test
environments, while another is used for all production environments.
However, what you would really want to do is tie together the
application-independent assets together with the application-dependent
parts of your assets into a common external repository so that they can
be tracked as a single combined asset. This discipline will ensure that
if anything untoward happens during a change (say, that an upgrade
fails due to a poorly written script, or that a major application bug is
found that necessitates a rollback to an earlier state) that both the
application topology (represented by the virtual system pattern) and its
images and script packages and the application-specific information
(such as the JDBC datasources and a specific version of the application
EAR file) are available in a form that had been previously tested and
shown to be stable.
Even besides being a generally good practice to follow,
this particular best practice helps address some specific issues in the
current version of PureApplication System. For example, with regard to
script packages, IBM Workload Deployer and IBM PureApplication System do
not (as of version 1.1.0.0) provide versioning. Likewise, the naming
of script packages is arbitrary, nothing is enforced; while it is a best
practice to indicate version numbers through a consistent, enforced
naming policy, only keeping all versions of the script packages under
source code control in an external repository will ensure that they can
be fully tracked and managed appropriately.
This then leads to two additional approaches in how to
deal with the fact that versioning is not currently fully supported.
Those two approaches are:
- Version everything in the name
This is the most common approach. The names of all
patterns and script packages should have a version included in it. To
facilitate continuous integration scenarios the build system then can
use some form of a token in the cbscript.json and pattern.json files,
and then the resulting pattern can be imported into the cloud provider
as such. This approach works well but can result in an expanding
catalog; therefore, it is recommended that the build system also has a
pruning method built into it (as described in this article) or some
other form of a regular purge of the catalog and instances.
- Layer the pattern such that the fast moving pieces are handled by an
external system that understand builds and versioningAs
described earlier, teams should build a standard set of "base patterns"
that would then change relatively infrequently. Script packages can
then be layered on top of the base pattern. The job of a script package
is to apply a specific version of a package on top of the base pattern.
The script package would typically take the version of the package as an
argument. The script package can then retrieve that content dynamically
from a media library, build system, or repository. This is in line with
our earlier recommendation to separate topology from configuration in
virtual applications. This is a fantastic way to accomplish other
objectives such as integrating with existing systems for build and
automation, as well as maintaining automation or deployment efforts that
are not limited to cloud. The downside of this approach is that the
pattern is no longer a complete package, but requires the external build
system or repository to function. However, we believe this tradeoff is
worth the effort.
This second approach is productized within Smart Cloud
Continuous Delivery,
though it can be accomplished using many external
automation solutions
with version control and programmatic interfaces. For
example, a script
package can take a few parameters and call out to
external systems such as
BuildForge Projects, Advanced Middleware Configuration
Projects,or IBM
Urban{code} uDeploy. There are also a number of teams
that have
implemented the “version everything” approach simply
using IBM Rational® Team Concert and JazzBuild with Ant. The important
pieces to keep in mind is that re-using mature automation systems is a
good choice if continued investment in those systems is desired, and
making sure that input into patterns (such as version information) be
kept as simple as possible for the user. As an example, if your pattern
takes in a build number as input, then the pattern should be updated to
have a good default (such as the last committed) or should take values
such as {latest | committed | release}; a team member deploying a
pattern should not need to spend a lot of effort to look up or find out a
valid value for any deployment parameter to the pattern.
Many clients have WebSphere Application Server cells and
other system environments whose lifetime is very long — it usually takes
a major version upgrade or something of similar magnitude to motivate a
team to decommission an environment and create a new one. These are
often long and complex processes in conventional environments. However,
PureApplication System makes the creation of a new environment a fast
and painless process. This leads to a major change in the way that
environments are viewed, based upon the amount of effort required to
create or re-create them.
If all of the important configuration information is
stored in a pattern, as opposed to only residing in the configuration
files of the cell, then it’s easier to simply recreate the cell on any
significant change to the configuration – even on adopting a new point
release of WebSphere Application Server or other middleware. In IBM
PureApplication System, the two cells could even run side-by-side for a
few days or weeks until it is assured that the new cell is functioning
normally, at which point it can be deleted with the assurance that it
could be recreated at any time if necessary. Another change we have
seen is that, again, clients were building few large cells due to the
complexity of creating the cells; when this can be done simply from a
pattern, it becomes possible for each project to be contained within its
own cell – resulting in smaller cells, and more of them.
The major change to the way in which operations teams
work when they adopt this pattern is that they must move all
administrative aspects into scripts. While you might perform an
administrative action like setting a tuning parameter, or installing an
application in the console for that particular middleware (like the
WebSphere Application Server console), you should also, at a minimum,
create a script that also performs that same action and attach that to
the virtual system pattern definition. This enables the recreation of
the environment exactly as it stands at any point in time and guarantees
that you can quickly adapt to both planned and unplanned failures.
This article demonstrated a few of the major best practices dealing with
integration of IBM PureApplication System into your IT governance policies
and data center management strategies. There are many, many more best
practices in other specific areas, such as application migration, not
covered here that we will look at in follow-on articles. Likewise, in
our discussion of assets we have hinted at the types of organizational
responsibilities and changes that might be necessary for you to best
leverage PureApplication System. The article Aligning organizations to achieve integrated system benefits with IBM PureApplication System begins the process of describing those changes, but we will cover this
subject in significant additional depth in another follow-on article.
By developerWorks (IBM)
