Deploying Blueprints

Locations are the environments to which Brooklyn deploys applications. Most commonly these are cloud services such as AWS, GCE, and IBM Softlayer. Brooklyn also supports deploying to a pre-provisioned network or to localhost (primarily useful for testing blueprints).

See also:

• The Locations yaml guide
• Use within an entity of the configuration option provisioning.properties
• How to add location definitions to the Catalog; and
• How to use Externalized Configuration.

The requirements for how a provisioned machine should behave will depend on the entites subsequently deployed there.

Below are a set of common assumptions, made by many entity implementations, which could cause subsequent errors if they do not hold. These relate to the machine's configuration, rather than additional networking or security that a given Cloud might offer.

Also see the Troubleshooting docs.

Remote Access

SSH or WinRM Access

Many entities require ssh'ing (or using WinRM for Windows), to install and configure the software.

An example of disabling all ssh'ing is shown below:

location:
  aws-ec2:us-east-1:
    identity: XXXXXXXX
    credential: XXXXXXXX
    waitForSshable: false
    pollForFirstReachableAddress: false
services:
- type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess
  brooklyn.config:
    onbox.base.dir.skipResolution: true
    sshMonitoring.enabled: false

Parsing SSH stdout: No Extra Lines

For entities that execute ssh commands, these sometimes parse the resulting stdout.

It is strongly recommended that VMs are configured so that no additional stdout is written when executing remote ssh (or WinRM) commands. Such stdout risks interfering with the response parsing in some blueprints.

For example, if configuring the VM to write out "Last login" information, this should be done for only "interactive" shells (see Stackoverflow for more details).

Passwordless Sudo

Does passwordless sudo work?

Try executing:

sudo whoami

See Passwordless Sudo.

Advertised Addresses

Hostname Resolves Locally

Does the hostname known at the box resolve at the box?

Try executing:

ping $(hostname)

if not, consider setting generate.hostname: true in the location config, for jclouds-based locations.

IP Resolves Locally

For the IP address advertised in Brooklyn using the sensor host.addresses.private (or host.subnet.address), can the machine reach that IP?

Get the sensor value, and then try executing:

ping ${PRIVATE_IP}

Is there a public IP (advertised using the sensor host.addresses.public, or host.address), and can the machine reach it?

Get the sensor value, and then try executing:

ping ${PUBLIC_IP}

Networking

Public Internet Access

Can the machine reach the public internet, and does DNS resolve?

Try executing:

ping www.example.org

Machine's Hostname in DNS

Is the machine hostname well-known? If ones does a DNS lookup, e.g. from the Brooklyn server, does it resolve and does it return the expected IP (e.g. the same IP as the host.addresses.public sensor)? Try using the hostname that the machine reports when you execute hostname.

Many blueprints do not require this, instead using IP addresses directly. Some blueprints may include registration with an appropriate DNS server. Some clouds do this automatically.

Reachability

When provisioning two machines, can these two machines reach each other on the expected IP(s) and hostname(s)?

Try using ping from one machine to another using the public or subnet ip or hostname. However, note that ping requires access over ICMP, which may be disabled. Alternatively, try connecting to a specific TCP port using telnet <address> <port>.

Firewalls

What firewall(s) are running on the machine, and are the required ports open? On Linux, check things like iptables, firewalld, ufw or other commercial firewalls. On Windows, check the settings of the Windows Firewall.

Consider using openIptables: true, or even stopIptables: true.

Sufficient Entropy for /dev/random

Is there sufficient entropy on the machine, for /dev/random to respond quickly?

Try executing:

{ cat /dev/random > /tmp/x & } ; sleep 10 ; kill %1 ; { cat /dev/random > /tmp/x & } ; sleep 1 ; kill %1 ; wc /tmp/x | awk '{print $3}'

The result should be more than 1M.

If not, consider setting installDevUrandom: true for jclouds-based locations.

See instructions to Increase Entropy.

File System

Permissions of /tmp

Is /tmp writable?

Try executing:

touch /tmp/amp-test-file ; rm /tmp/amp-test-file

Are files in /tmp executable (e.g. some places it has been mounted NO_EXECUTE)?

Try executing:

echo date > /tmp/brooklyn-test.sh && chmod +x /tmp/brooklyn-test.sh && /tmp/brooklyn-test.sh && rm /tmp/brooklyn-test.sh

Clouds

For most cloud provisioning tasks, Brooklyn uses Apache jclouds. The identifiers for some of the most commonly used jclouds-supported clouds are (or see the full list):

• jclouds:aws-ec2:<region>: Amazon EC2, where :<region> might be us-east-1 or eu-west-1 (or omitted)
• jclouds:softlayer:<region>: IBM Softlayer, where :<region> might be dal05 or ams01 (or omitted)
• jclouds:google-compute-engine: Google Compute Engine
• jclouds:openstack-nova:<endpoint>: OpenStack, where :<endpoint> is the access URL (required)
• jclouds:cloudstack:<endpoint>: Apache CloudStack, where :<endpoint> is the access URL (required)

For any of these, of course, Brooklyn needs to be configured with an identity and a credential:

location:
  jclouds:aws-ec2:
    identity: ABCDEFGHIJKLMNOPQRST
    credential: s3cr3tsq1rr3ls3cr3tsq1rr3ls3cr3tsq1rr3l

The above YAML can be embedded directly in blueprints, either at the root or on individual services. If you prefer to keep the credentials separate, you can instead store them as a catalog entry or set them in brooklyn.properties in the jclouds.<provider> namespace:

brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST  
brooklyn.location.jclouds.aws-ec2.credential=s3cr3tsq1rr3ls3cr3tsq1rr3ls3cr3tsq1rr3l

And in this case you can reference the location in YAML with location: jclouds:aws-ec2.

Alternatively, you can use the location wizard tool available within the web console to create any cloud location supported by Apache jclouds. This location will be saved as a catalog entry for easy reusability.

Brooklyn irons out many of the differences between clouds so that blueprints run similarly in a wide range of locations, including setting up access and configuring images and machine specs. The configuration options are described in more detail below.

In some cases, cloud providers have special features or unusual requirements. These are outlined in More Details for Specific Clouds.

OS Initial Login and Setup

Once a machine is provisioned, Brooklyn will normally attempt to log in via SSH and configure the machine sensibly.

The credentials for the initial OS log on are typically discovered from the cloud, but in some environments this is not possible. The keys loginUser and either loginUser.password or loginUser.privateKeyFile can be used to force Brooklyn to use specific credentials for the initial login to a cloud-provisioned machine.

(This custom login is particularly useful when using a custom image templates where the cloud-side account management logic is not enabled. For example, a vCloud (vCD) template can have guest customization that will change the root password. This setting tells Apache Brooklyn to only use the given password, rather than the initial randomly generated password that vCD returns. Without this property, there is a race for such templates: does Brooklyn manage to create the admin user before the guest customization changes the login and reboots, or is the password reset first (the latter means Brooklyn can never ssh to the VM). With this property, Brooklyn will always wait for guest customization to complete before it is able to ssh at all. In such cases, it is also recommended to use useJcloudsSshInit=false.)

Following a successful logon, Brooklyn performs the following steps to configure the machine:

1. creates a new user with the same name as the user brooklyn is running as locally (this can be overridden with user, below).

2. install the local user's ~/.ssh/id_rsa.pub as an authorized_keys on the new machine, to make it easy for the operator to ssh in (override with privateKeyFile; or if there is no id_{r,d}sa{,.pub} an ad hoc keypair will be generated for the regular Brooklyn user; if there is a passphrase on the key, this must be supplied)

3. give sudo access to the newly created user (override with grantUserSudo: false)

4. disable direct root login to the machine

These steps can be skipped or customized as described below.

jclouds Config Keys

The following is a subset of the most commonly used configuration keys used to customize cloud provisioning. For more keys and more detail on the keys below, see JcloudsLocationConfig.

VM Creation

• Most providers require exactly one of either region (e.g. us-east-1) or endpoint (the URL, usually for private cloud deployments)

• Hardware requirements can be specified, including minRam, minCores, minDisk and os64Bit; or as a specific hardwareId

• VM image constraints can be set using osFamily (e.g. Ubuntu, CentOS, Debian, RHEL) and osVersionRegex, or specific VM images can be specified using imageId or imageNameRegex

• Specific VM images can be specified using imageId or imageNameRegex

• Specific Security Groups can be specified using securityGroups, as a list of strings (the existing security group names), or inboundPorts can be set, as a list of numeric ports (selected clouds only)

• Where a key pair is registered with a target cloud for logging in to machines, Brooklyn can be configured to request this when provisioning VMs by setting keyPair (selected clouds only). Note that if this keyPair does not correspond your default ~/.ssh/id_rsa, you must typically also specify the corresponding loginUser.privateKeyFile as a file or URL accessible from Brooklyn.

• A specific VM name (often the hostname) base to be used can be specified by setting groupId. By default, this name is constructed based on the entity which is creating it, including the ID of the app and of the entity. (As many cloud portals let you filter views, this can help find a specific entity or all machines for a given application.) For more sophisticated control over host naming, you can supply a custom CloudMachineNamer, for example cloudMachineNamer: CustomMachineNamer. CustomMachineNamer will use the entity's name or following a template you supply. On many clouds, a random suffix will be appended to help guarantee uniqueness; this can be removed by setting vmNameSaltLength: 0 (selected clouds only).

• A DNS domain name where this host should be placed can be specified with domainName (in selected clouds only)

• User metadata can be attached using the syntax userMetadata: { key: value, key2: "value 2" } (or userMetadata=key=value,key2="value 2" in a properties file)

• By default, several pieces of user metadata are set to correlate VMs with Brooklyn entities, prefixed with brooklyn-. This user metadata can be omitted by setting includeBrooklynUserMetadata: false.

• You can specify the number of attempts Brooklyn should make to create machines with machineCreateAttempts (jclouds only). This is useful as an efficient low-level fix for those occasions when cloud providers give machines that are dead on arrival. You can of course also resolve it at a higher level with a policy such as ServiceRestarter.

• If you want to investigate failures, set destroyOnFailure: false to keep failed VM's around. (You'll have to manually clean them up.) The default is false: if a VM fails to start, or is never ssh'able, then the VM will be terminated.

• You can set useMachinePublicAddressAsPrivateAddress to true to overwrite the VMs private IP with its public IP. This is useful as it can be difficult to get VMs communicating via the private IPs they are assigned in some clouds. Using this config, blueprints which use private IPs can still be deployed to these clouds.

OS Setup

• user and password can be used to configure the operating user created on cloud-provisioned machines

• The loginUser config key (and subkeys) control the initial user to log in as, in cases where this cannot be discovered from the cloud provider

• Private keys can be specified using privateKeyFile; these are not copied to provisioned machines, but are required if using a local public key or a pre-defined authorized_keys on the server. (For more information on SSH keys, see here.)

• If there is a passphrase on the key file being used, you must supply it to Brooklyn for it to work, of course! privateKeyPassphrase does the trick (as in brooklyn.location.jclouds.privateKeyPassphrase, or other places where privateKeyFile is valid). If you don't like keys, you can just use a plain old password.

• Public keys can be specified using publicKeyFile, although these can usually be omitted if they follow the common pattern of being the private key file with the suffix .pub appended. (It is useful in the case of loginUser.publicKeyFile, where you shouldn't need, or might not even have, the private key of the root user when you log in.)

• Provide a list of URLs to public keys in extraSshPublicKeyUrls, or the data of one key in extraSshPublicKeyData, to have additional public keys added to the authorized_keys file for logging in. (This is supported in most but not all locations.)

• Use dontCreateUser to have Brooklyn run as the initial loginUser (usually root), without creating any other user.

• A post-provisioning setup.script can be specified to run an additional script, before making the Location available to entities. This may take the form of a URL of a script or a data URI. Note that if using a data URI it is usually a good idea to base64 this string to escape problem characters in more complex scripts. The base64 encoded script should then be prefixed with data:text/plain;base64, to denote this. For example if you wanted to disable a yum repository called reponame prior to using the machine, you could use the following command:

  sudo yum-config-manager --disable reponame

  Base64 encoding can be done with a with a tool such as this or a Linux command such as:

  echo "sudo yum-config-manager --disable reponame" | base64

  With the base64 prefix this would then look like this:

  setup.script: data:text/plain;base64,c3VkbyB5dW0tY29uZmlnLW1hbmFnZXIgLS1kaXNhYmxlIHJlcG9uYW1l

  The setup.script can also take FreeMarker variables in a setup.script.vars property. Variables are set in the format key1:value1,key2:value2 and used in the form ${key1}. So for the above example:

  setup.script.vars: repository:reponame

  then

  setup.script: data:sudo yum-config-manager --disable ${repository}

  or encoded in base64:

  setup.script: data:text/plain;base64,c3VkbyB5dW0tY29uZmlnLW1hbmFnZXIgLS1kaXNhYmxlICR7cmVwb3NpdG9yeX0=

  This enables the name of the repository to be passed in to the script.

• Use openIptables: true to automatically configure iptables, to open the TCP ports required by the software process. One can alternatively use stopIptables: true to entirely stop the iptables service.

• Use Entity configuration flag effector.add.openInboundPorts: true to add an effector for opening ports in a cloud Security Group. The config is supported for all SoftwareProcessImpl implementations.

• Use installDevUrandom: true to fall back to using /dev/urandom rather than /dev/random. This setting is useful for cloud VMs where there is not enough random entropy, which can cause /dev/random to be extremely slow (causing ssh to be extremely slow to respond).

• Use useJcloudsSshInit: false to disable the use of the native jclouds support for initial commands executed on the VM (e.g. for creating new users, setting root passwords, etc.). Instead, Brooklyn's ssh support will be used. Timeouts and retries are more configurable within Brooklyn itself. Therefore this option is particularly recommended when the VM startup is unusual (for example, if guest customizations will cause reboots and/or will change login credentials).

• Use brooklyn.ssh.config.noDeleteAfterExec: true to keep scripts on the server after execution. The contents of the scripts and the stdout/stderr of their execution are available in the Brooklyn web console, but sometimes it can also be useful to have them on the box. This setting prevents scripts executed on the VMs from being deleted on completion. Note that some scripts run periodically so this can eventually fill a disk; it should only be used for dev/test.

Custom Template Options

jclouds supports many additional options for configuring how a virtual machine is created and deployed, many of which are for cloud-specific features and enhancements. Brooklyn supports some of these, but if what you are looking for is not supported directly by Brooklyn, we instead offer a mechanism to set any parameter that is supported by the jclouds template options for your cloud.

Part of the process for creating a virtual machine is the creation of a jclouds TemplateOptions object. jclouds providers extends this with extra options for each cloud - so when using the AWS provider, the object will be of type AWSEC2TemplateOptions. By examining the source code, you can see all of the options available to you.

The templateOptions config key takes a map. The keys to the map are method names, and Brooklyn will find the method on the TemplateOptions instance; it then invokes the method with arguments taken from the map value. If a method takes a single parameter, then simply give the argument as the value of the key; if the method takes multiple parameters, the value of the key should be an array, containing the argument for each parameter.

For example, here is a complete blueprint that sets some AWS EC2 specific options:

location: AWS_eu-west-1
services:
- type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess
  provisioning.properties:
    templateOptions:
      subnetId: subnet-041c8373
      mapNewVolumeToDeviceName: ["/dev/sda1", 100, true]
      securityGroupIds: ['sg-4db68928']

Here you can see that we set three template options:

• subnetId is an example of a single parameter method. Brooklyn will effectively try to run the statement templateOptions.subnetId("subnet-041c88373");
• mapNewVolumeToDeviceName is an example of a multiple parameter method, so the value of the key is an array. Brooklyn will effectively true to run the statement templateOptions.mapNewVolumeToDeviceName("/dev/sda1", 100, true);
• securityGroupIds demonstrates an ambiguity between the two types; Brooklyn will first try to parse the value as a multiple parameter method, but there is no method that matches this parameter. In this case, Brooklyn will next try to parse the value as a single parameter method which takes a parameter of type List; such a method does exist so the operation will succeed.

If the method call cannot be matched to the template options available - for example if you are trying to set an AWS EC2 specific option but your location is an OpenStack cloud - then a warning is logged and the option is ignored.

Cloud Machine Naming

The name that Apache Brooklyn generates for your virtual machine will, by default, be based on your Apache Brooklyn server name and the IDs of the entities involved. This is the name you see in places such as the AWS console and will look something like:

brooklyn-o8jql4-machinename-rkix-tomcat-wi-nca6-14b

If you have created a lot of virtual machines, this kind of naming may not be helpful. This can be changed using the following YAML in your location's brooklyn.config:

cloudMachineNamer: org.apache.brooklyn.core.location.cloud.names.CustomMachineNamer
custom.machine.namer.machine: My-Custom-Name-${entity.displayName}

A FreeMarker format is used in custom.machine.namer.machine which can take values from places such as the launching entity or location.

The above example will create a name such as:

My-Custom-Name-Tomcat

Allowing you to more easily identify your virtual machines.

More Details on Specific Clouds

Clouds vary in the format of the identity, credential, endpoint, and region. Some also have their own idiosyncracies. More details for configuring some common clouds is included below. You may also find these sources helpful:

• The template brooklyn.properties file in the Getting Started guide contains numerous examples of configuring specific clouds, including the format of credentials and options for sometimes-fiddly private clouds.
• The jclouds guides describes low-level configuration sometimes required for various clouds.

Amazon Web Services (AWS)

Credentials

AWS has an "access key" and a "secret key", which correspond to Brooklyn's identity and credential respectively.

These keys are the way for any programmatic mechanism to access the AWS API.

To generate an access key and a secret key, see jclouds instructions and AWS IAM instructions.

An example of the expected format is shown below:

location:
  jclouds:aws-ec2:
    region: us-east-1
    identity: ABCDEFGHIJKLMNOPQRST
    credential: abcdefghijklmnopqrstu+vwxyzabcdefghijklm

Users are strongly recommended to use externalized configuration for better credential management, for example using Vault.

Common Configuration Options

Below are examples of configuration options that use values specific to AWS EC2:

• The region is the AWS region code. For example, region: us-east-1. You can in-line the region name using the following format: jclouds:aws-ec2:us-east-1. A specific availability zone within the region can be specified by including its letter identifier as a suffix. For example, region: us-east-1a.

• The hardwareId is the instance type. For example, hardwareId: m4.large.

• The imageId is the region-specific AMI id. For example, imageId: us-east-1/ami-05ebd06c.

• The securityGroups option takes one or more names of pre-existing security groups. For example, securityGroups: mygroup1 or securityGroups: [ mygroup1, mygroup2 ].

EBS boot volume size

The size of the EBS boot volume for a VM can be specified using mapNewVolumeToDeviceName in templateOptions as shown below:

location:
  jclouds:aws-ec2:
    region: ...
    templateOptions:
      mapNewVolumeToDeviceName:
       - /dev/sda1
       - 123
       - true

Where /dev/sda1 is the device name of the boot volume on the selected AMI, 123 is the size of the volume in gigabytes and true is a boolean indicating the volume should be deleted on VM termination.

Using a Registered Key Pair

You can specify a keyPair to use for initial provisioning as a configuration option. If this is omitted, Brooklyn will use jclouds to create a new ad hoc key pair at AWS for that machine, and it will delete it afterwards. This is usually seamless and occurs behind the scenes, with the post-provision user set up and configured as normal for all locations. However using AWS heavily or optimizing creation, using a known key pairs can
make some images more reliable and speed things up.

First, in the AWS Console, open the EC2 service in the region you are interested in, then click "Key Pairs" at the left. For us-east-1, the link is here. Click "Create Key Pair" (or "Import Key Pair" if you want to provide a public key) and follow the instructions.

Then define your location as follows for aws-us-east-1. Make sure to replace the XXXX sections with the key-pair name defined above and the corresponding private key data.

brooklyn.catalog:
  version: "1.0"
  itemType: location
  items:
  - id: aws-base
    item:
      type: jclouds:aws-ec2
      brooklyn.config:
        identity: XXXXXXXXXXXXXXXX
        credential: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
  - id: aws-us-east-1
    item:
      type: aws-base
      brooklyn.config:
        region: us-east-1
        keyPair: XXXXXXXXX
        loginUser.privateKeyData: |
          -----BEGIN RSA PRIVATE KEY-----
          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
          XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
          -----END RSA PRIVATE KEY-----

Using Subnets and Security Groups

Apache Brooklyn can run with AWS VPC and both public and private subnets. Simply provide the subnet-a1b2c3d4 as the networkName when deploying:

location:
  jclouds:aws-ec2:
    region: us-west-1
    networkName: subnet-a1b2c3d4   # use your subnet ID

Subnets are typically used in conjunction with security groups. Brooklyn does not attempt to open additional ports when private subnets or security groups are supplied, so the subnet and ports must be configured appropriately for the blueprints being deployed. You can configure a default security group with appropriate (or all) ports opened for access from the appropriate (or all) CIDRs and security groups, or you can define specific securityGroups on the location or as provisioning.properties on the entities.

Make sure that Brooklyn has access to the machines under management. This includes SSH, which might be done with a public IP created with inbound access on port 22 permitted for a CIDR range including the IP from which Brooklyn contacts it. Alternatively you can run Brooklyn on a machine in that same subnet, or set up a VPN or jumphost which Brooklyn will use.

EC2 "Classic" Problems with VPC-only Hardware Instance Types

If you have a pre-2014 Amazon account, it is likely configured in some regions to run in "EC2 Classic" mode by default, instead of the more modern "VPC" default mode. This can cause failures when requesting certain hardware configurations because many of the more recent hardware "instance types" only run in "VPC" mode. For instance when requesting an instance with minRam: 8gb, Brooklyn may opt for an m4.large, which is a VPC-only instance type. If you are in a region configured to use "EC2 Classic" mode, you may see a message such as this:

400 VPCResourceNotSpecified: The specified instance type can only be used in a VPC.
A subnet ID or network interface ID is required to carry out the request.

This is a limitation of "legacy" accounts. The easiest fixes are either:

• specify an instance type which is supported in classic, such as m3.xlarge (see below)
• move to a different region where VPC is the default (eu-central-1 should work as it only offers VPC mode, irrespective of the age of your AWS account)
• get a new AWS account -- "VPC" will be the default mode (Amazon recommend this and if you want to migrate existing deployments they provide detailed instructions)

To understand the situation, the following resources may be useful:

• Background on VPC vs Classic: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html
• Good succinct answers to FAQs: http://aws.amazon.com/vpc/faqs/#Default_VPCs
• Check if a region in your account is "VPC" or "Classic": http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html#default-vpc-availability
• Regarding instance types:
  • All instance types: https://aws.amazon.com/ec2/instance-types
  • Those which require VPC: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html#vpc-only-instance-types

If you want to solve this problem with your existing account, you can create a VPC and instruct Brooklyn to use it:

1. Use the "Start VPC Wizard" option in the VPC dashboard, making sure it is for the right region, and selecting a "Single Public Subnet". (More information is in these AWS instructions.)
2. Once the VPC is created, open the "Subnets" view and modify the "Public subnet" so that it will "Auto-assign Public IP".
3. Next click on the "Security Groups" and find the default security group for that VPC. Modify its "Inbound Rules" to allow "All traffic" from "Anywhere". (Or for more secure options, see the instructions in the previous section, "Using Subnets".)
4. Finally make a note of the subnet ID (e.g. subnet-a1b2c3d4) for use in Brooklyn.

You can then deploy blueprints to the subnet, allowing VPC hardware instance types, by specifying the subnet ID as the networkName in your YAML blueprint. This is covered in the previous section, "Using Subnets".

Tidying up after jclouds

Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group from being deleted until there are no VMs using it. However, there is eventual consistency for recording which VMs still reference those security groups: after deleting the VM, it can sometimes take several minutes before the security group can be deleted. jclouds retries for 3 seconds, but does not block for longer.

Whilst there is eventual consistency for recording which VMs still reference security groups, after deleting a VM, it can sometimes take several minutes before a security group can be deleted

There is utility written by Cloudsoft for deleting these unused resources: http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after-jclouds.html.

Azure Compute ARM

Azure Resource Manager (ARM) is a framework for deploying and managing applications across resources and managing groups of resources as single logical units on the Microsoft Azure cloud computing platform.

Setup the Azure credentials

Azure CLI 2.0

Firstly, install and configure Azure CLI following these steps.

You will need to obtain your subscription ID and tenant ID from Azure. To do this using the CLI, first, log in:

az login

Or, if you are already logged in, request an account listing:

az account list

In either case, this will return a subscription listing, similar to that shown below.

[
  {
    "cloudName": "AzureCloud",
    "id": "012e832d-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "isDefault": true,
    "name": "QA Team",
    "state": "Enabled",
    "tenantId": "ba85e8cd-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "user": {
      "name": "qa@example.com",
      "type": "user"
    }
  },
  {
    "cloudName": "AzureCloud",
    "id": "341751b0-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "isDefault": false,
    "name": "Developer Team",
    "state": "Enabled",
    "tenantId": "ba85e8cd-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
    "user": {
      "name": "dev@example.com",
      "type": "user"
    }
  }
]

Choose one of the subscriptions and make a note of its id - henceforth the subscription ID - and the tenantId.

Next we need to create an application and a service principle, and grant permissions to the service principle. Use these commands:

# Create an AAD application with your information.
az ad app create --display-name <name> --password <Password> --homepage <home-page> --identifier-uris <identifier-uris>

# For example: az ad app create --display-name "myappname"  --password abcd --homepage "https://myappwebsite" --identifier-uris "https://myappwebsite"

Take a note of the appId shown.

# Create a Service Principal
az ad sp create --id <Application-id>

Take a note of the objectId shown - this will be the service principal object ID. (Note that any of the servicePrincipalNames can also be used in place of the object ID.)

# Assign roles for this service principal. The "principal" can be the "objectId" or any one of the "servicePrincipalNames" from the previous step
az role assignment create --assignee <Service-Principal> --role Contributor --scope /subscriptions/<Subscription-ID>/

By this stage you should have the following information:

• A subscription ID
• A tenant ID
• An application ID
• A service principle (either by its object ID, or by any one of its names)

We can now verify this information that this information can be used to log in to Azure:

az login --service-principal -u <Application-ID> --password abcd --tenant <Tenant-ID>

Should you need to change the password, you can use the following command:

az ad app update --id <Application-ID> --password MyNewPassword

Azure CLI 1.0

Firstly, install and configure Azure CLI following these steps.

Using the Azure CLI, run the following commands to create a service principal

# Set mode to ARM
azure config mode arm

# Enter your Microsoft account credentials when prompted
azure login

# Set current subscription to create a service principal
azure account set <Subscription-id>

# Create an AAD application with your information.
azure ad app create --name <name> --password <Password> --home-page <home-page> --identifier-uris <identifier-uris>

# For example: azure ad app create --name "myappname"  --password abcd --home-page "https://myappwebsite" --identifier-uris "https://myappwebsite"

# Output will include a value for `Application Id`, which will be used for the live tests

# Create a Service Principal
azure ad sp create --applicationId <Application-id>

# Output will include a value for `Object Id`, to be used in the next step 

Run the following commands to assign roles to the service principal

# Assign roles for this service principal
azure role assignment create --objectId <Object-id> -o Contributor -c /subscriptions/<Subscription-id>/

Look up the the tenant Id

azure account show -s <Subscription-id> --json

# output will be a JSON which will include the `Tenant id`

Verify service principal

azure login -u <Application-id> -p <Password> --service-principal --tenant <Tenant-id>

Using the Azure ARM Location

Below is an example Azure ARM location in YAML which will launch a Ubuntu instance in south east asia:

brooklyn.catalog:
  id: my-azure-arm-location
  name: "My Azure ARM location"
  itemType: location
  item:
    type: jclouds:azurecompute-arm
    brooklyn.config:
      identity: <Application-id>
      credential: <Password>
      endpoint: https://management.azure.com/subscriptions/<Subscription-id>
      oauth.endpoint: https://login.microsoftonline.com/<Tenant-id>/oauth2/token

      jclouds.azurecompute.arm.publishers: OpenLogic
      region: southeastasia
      loginUser: brooklyn
      templateOptions:
        overrideAuthenticateSudo: true 

Fill the values <Application-id>, <Password>, <Subscription-id> and <Tenant-id> in from the values generated when setting up your credentials. In addition; several keys, not required in other locations need to be specified in order to use the Azure Compute ARM location. These are:

jclouds.azurecompute.arm.publishers: OpenLogic

The publishers is any item from the list available here: https://docs.microsoft.com/en-us/azure/virtual-machines/virtual-machines-linux-cli-ps-findimage

region: southeastasia    

The region is any region from the list available here: https://azure.microsoft.com/en-us/regions/

loginUser: brooklyn

The loginUser can be anything, as long as it's specified.

templateOptions:
    overrideAuthenticateSudo: true

The overrideAuthenticateSudo: true key tells Apache Brooklyn that default on Azure images do not have passwordless sudo configured by default.

Useful configuration options for Azure ARM

You can add these options directly under the brooklyn.config element in the example above:

• jclouds.compute.resourcename-prefix and jclouds.compute.resourcename-delimiter - defaults to jclouds and - respectively. If jclouds is choosing the name for a resource (for example, a virtual machine), these properties will alter the way the resource is named.

You can add these options into the templateOptions element inside the brooklyn.config element in the example above:

• resourceGroup - select a Resource Group to deploy resources into. If not given, jclouds will generate a new resource group with a partly-random name.

Using Windows on Azure ARM

This section contains material how to create a Windows location on Azure ARM. Some of the used parameters are explained in the section above.

Windows on Azure ARM requires manually created Azure KeyVault Azure KeyVaults can be created via Azure CLI or Azure portal UI. KeyVault's secret is a key stored in protected .PFX file. It needs to be prepared upfront or created with the Add-AzureKeyVaultKey cmdlet.

• osFamily: windows tells Apache Brooklyn to consider it as a Windows machine

• useJcloudsSshInit: false tells jclouds to not try to connect to the VM

• vmNameMaxLength: 15 tells the cloud client to strip the VM name to maximum 15 characters. This is the maximum size supported by Azure Windows VMs.

• winrm.useHttps tells Apache Brooklyn to configure the WinRM client to use HTTPS.

• secrets Specifies the KeyVault configuration

  sourceVault Resource id of the KeyVault

  vaultCertificates certificateStore has to use My as a value. KeyVault's certificateUrl. An URI to the Secret Identifier

• windowsConfiguration

  provisionVMAgent whether Azure to install an agent on the VM. It must be set to true

  winRM It defines the listeners section. If listeners is https then certificateUrl needs to be set. Its value must match the one of secrets's certificateUrl.

• additionalUnattendContent Additional content. Normally it can be defined as null

• enableAutomaticUpdates whether to enable the automatic Windows updates. It can be set to false, if automatic updates are not desired

Sample Windows Blueprint

Placeholders surrounded with <> have to be replcaced with their respective values.

brooklyn.catalog:
  id: my-azure-arm-location
  name: "My Azure ARM location"
  itemType: location
  item:
    type: jclouds:azurecompute-arm
    brooklyn.config:
      identity: <Application-id>
      credential: <Password>
      endpoint: https://management.azure.com/subscriptions/<Subscription-id>
      oauth.endpoint: https://login.microsoftonline.com/<Tenant-id>/oauth2/token
      jclouds.azurecompute.arm.publishers: MicrosoftWindowsServer
      jclouds.azurecompute.operation.timeout: 120000

      winrm.useHttps: true
      osFamily: windows
      imageId: <Azure_location>/MicrosoftWindowsServer/WindowsServer/2012-R2-Datacenter
      region: <Azure_location>
      vmNameMaxLength: 15
      useJcloudsSshInit: false
      destroyOnFailure: false

      templateOptions:
        overrideLoginUser: brooklyn
        overrideLoginPassword: "secretPass1!"
        secrets:
        - sourceVault:
            id: "/subscriptions/<Subscription-id>/resourceGroups/<ResourceGroup>/providers/Microsoft.KeyVault/vaults/<KeyVault-name>"
          vaultCertificates:
          - certificateUrl: "<KeyVault-uri>"
            certificateStore: My
        windowsConfiguration:
          provisionVMAgent: true
          winRM:
            listeners:
            - protocol: https
              certificateUrl: "<KeyVault-uri>"
          additionalUnattendContent: null
          enableAutomaticUpdates: true

Known issues

There are currently two known issues with Azure ARM:

• It can take a long time for VMs to be provisioned
• The Azure ARM APIs appear to have some fairly strict rate limiting that can result in AzureComputeRateLimitExceededException

Azure Compute Classic

Azure is a cloud computing platform and infrastructure created by Microsoft. Apache Brooklyn includes support for both Azure Classic and Azure ARM, as one of the Apache jclouds supported clouds Microsoft Azure Compute.

The two modes of using Azure are the "classic deployment" model and the newer "Azure Resource Manager" (ARM) model. See https://azure.microsoft.com/en-gb/documentation/articles/resource-manager-deployment-model/ for details.

Setup the Azure credentials

Microsoft Azure requests are signed by SSL certificate. You need to upload one into your account in order to use an Azure location.

# create the certificate request
mkdir -m 700 $HOME/.brooklyn
openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout $HOME/.brooklyn/azure.pem -out $HOME/.brooklyn/azure.pem
# create the p12 file, and note your export password. This will be your test credentials.
openssl pkcs12 -export -out $HOME/.brooklyn/azure.p12 -in $HOME/.brooklyn/azure.pem -name "brooklyn :: $USER"
# create a cer file
openssl x509 -inform pem -in $HOME/.brooklyn/azure.pem -outform der -out $HOME/.brooklyn/azure.cer

Finally, upload .cer file to the management console at https://manage.windowsazure.com/@myId#Workspaces/AdminTasks/ListManagementCertificates to authorize this certificate.

Please note, you can find the "myId" value for this link by looking at the URL when logged into the Azure management portal.

Note, you will need to use .p12 format in the brooklyn.properties.

How to configure Apache Brooklyn to use Azure Compute

First, in your brooklyn.properties define a location as follows:

brooklyn.location.jclouds.azurecompute.identity=$HOME/.brooklyn/azure.p12
brooklyn.location.jclouds.azurecompute.credential=<P12_EXPORT_PASSWORD>
brooklyn.location.jclouds.azurecompute.endpoint=https://management.core.windows.net/<YOUR_SUBSCRIPTION_ID>
brooklyn.location.jclouds.azurecompute.vmNameMaxLength=45
brooklyn.location.jclouds.azurecompute.jclouds.azurecompute.operation.timeout=120000
brooklyn.location.jclouds.azurecompute.user=<USER_NAME>
brooklyn.location.jclouds.azurecompute.password=<PASSWORD>

During the VM provisioning, Azure will set up the account with <USER_NAME> and <PASSWORD> automatically. Notice, <PASSWORD> must be a minimum of 8 characters and must contain 3 of the following: a lowercase character, an uppercase character, a number, a special character.

To force Apache Brooklyn to use a particular image in Azure, say Ubuntu 14.04.1 64bit, one can add:

brooklyn.location.jclouds.azurecompute.imageId=b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_1-LTS-amd64-server-20150123-en-us-30GB

From $BROOKLYN_HOME, you can list the image IDs available using the following command:

./bin/client "list-images --location azure-west-europe"

To force Brooklyn to use a particular hardwareSpec in Azure, one can add something like:

brooklyn.location.jclouds.azurecompute.hardwareId=BASIC_A2

From $BROOKLYN_HOME, you can list the hardware profile IDs available using the following command:

./bin/client "list-hardware-profiles --location azure-west-europe"

At the time of writing, the classic deployment model has the possible values shown below. See https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-size-specs/ for further details, though that description focuses on the new "resource manager deployment" rather than "classic".

• Basic_A0 to Basic_A4
• Standard_D1 to Standard_D4
• Standard_G1 to Standard_G5
• ExtraSmall, Small, Medium, Large, ExtraLarge

Named location

For convenience, you can define a named location, like:

brooklyn.location.named.azure-west-europe=jclouds:azurecompute:West Europe
brooklyn.location.named.azure-west-europe.displayName=Azure West Europe
brooklyn.location.named.azure-west-europe.imageId=b39f27a8b8c64d52b05eac6a62ebad85__Ubuntu-14_04_1-LTS-amd64-server-20150123-en-us-30GB
brooklyn.location.named.azure-west-europe.hardwareId=BASIC_A2
brooklyn.location.named.azure-west-europe.user=test
brooklyn.location.named.azure-west-europe.password=MyPassword1!

This will create a location named azure-west-europe. It will inherit all the configuration defined on brooklyn.location.jclouds.azurecompute. It will also augment and override this configuration (e.g. setting the display name, image id and hardware id).

On Linux VMs, The user and password will create a user with that name and set its password, disabling the normal login user and password defined on the azurecompute location.

Windows VMs on Azure

The following configuration options are important for provisioning Windows VMs in Azure:

• osFamily: windows tells Apache Brooklyn to consider it as a Windows machine

• useJcloudsSshInit: false tells jclouds to not try to connect to the VM

• vmNameMaxLength: 15 tells the cloud client to strip the VM name to maximum 15 characters. This is the maximum size supported by Azure Windows VMs.

• winrm.useHttps tells Apache Brooklyn to configure the WinRM client to use HTTPS.

  This is currently not supported in the default configuration for other clouds, where Apache Brooklyn is deploying Windows VMs.

  If the parameter value is false the default WinRM port is 5985; if true the default port for WinRM will be 5986. Use of default ports is stongly recommended.

• winrm.useNtlm tells Apache Brooklyn to configure the WinRM client to use NTLM protocol.

  For Azure, this is mandatory.

  For other clouds, this value is used in the cloud init script to configure WinRM on the VM.
  If the value is true then Basic Authentication will be disabled and the WinRM client will only use Negotiate plus NTLM.
  If the value is false then Basic Authentication will be enabled and the WinRM client will use Basic Authentication.

  NTLM is the default Authentication Protocol.

  The format of this configuration option is subject to change: WinRM supports several authentication mechanisms, so this may be changed to a prioritised list so as to provide fallback options.

• user tells Apache Brooklyn which user to login as. The value should match that supplied in the overrideLoginUser of the templateOptions.

• password: tells Apache Brooklyn the password to use when connecting. The value should match that supplied in the overrideLoginPassword of the templateOptions.

• templateOptions: { overrideLoginUser: adminuser, overrideLoginPassword: Pa55w0rd! }
  tells the Azure Cloud to provision a VM with the given admin username and password. Note that no "Administrator" user will be created.

  If this config is not set then the VM will have a default user named "jclouds" with password "Azur3Compute!". It is Strongly Recommended that these template options are set.

  Notice: one cannot use Administrator as the user in Azure.

  This configuration is subject to change in future releases.

Sample Windows Blueprint

Below is an example for provisioning a Windows-based entity on Azure. Note the placeholder values for the identity, credential and password.

name: Windows Test @ Azure
location:
  jclouds:azurecompute:West Europe:
    identity: /home/users/brooklyn/.brooklyn/azure.p12
    credential: xxxxxxxp12
    endpoint: https://management.core.windows.net/12345678-1234-1234-1234-123456789abc
    imageId: 3a50f22b388a4ff7ab41029918570fa6__Windows-Server-2012-Essentials-20141204-enus
    hardwareId: BASIC_A2
    osFamily: windows
    useJcloudsSshInit: false
    vmNameMaxLength: 15
    winrm.useHttps: true
    user: brooklyn
    password: secretPass1!
    templateOptions:
      overrideLoginUser: brooklyn
      overrideLoginPassword: secretPass1!
services:
- type: org.apache.brooklyn.entity.software.base.VanillaWindowsProcess
  brooklyn.config:
    install.command: echo install phase
    launch.command: echo launch phase
    checkRunning.command: echo launch phase

Below is an example named location for Azure, configured in brooklyn.properties. Note the placeholder values for the identity, credential and password.

brooklyn.location.named.myazure=jclouds:azurecompute:West Europe
brooklyn.location.named.myazure.displayName=Azure West Europe (Windows)
brooklyn.location.named.myazure.identity=$HOME/.brooklyn/azure.p12
brooklyn.location.named.myazure.credential=<P12_EXPORT_PASSWORD>
brooklyn.location.named.myazure.endpoint=https://management.core.windows.net/<YOUR_SUBSCRIPTION_ID>
brooklyn.location.named.myazure.vmNameMaxLength=15
brooklyn.location.named.myazure.jclouds.azurecompute.operation.timeout=120000
brooklyn.location.named.myazure.imageId=3a50f22b388a4ff7ab41029918570fa6__Windows-Server-2012-Essentials-20141204-enus
brooklyn.location.named.myazure.hardwareId=BASIC_A2
brooklyn.location.named.myazure.osFamily=windows
brooklyn.location.named.myazure.useJcloudsSshInit=false
brooklyn.location.named.myazure.winrm.useHttps=true
brooklyn.location.named.myazure.user=brooklyn
brooklyn.location.named.myazure.password=secretPass1!
brooklyn.location.named.myazure.templateOptions={ overrideLoginUser: amp, overrideLoginPassword: secretPass1! }

User and Password Configuration

As described under the configuration options, the username and password must be explicitly supplied in the configuration.

This is passed to the Azure Cloud during provisioning, to create the required user. These values correspond to the options AdminUsername and AdminPassword in the Azure API.

If a hard-coded password is not desired, then within Java code a random password could be auto-generated and passed into the call to location.obtain(Map<?,?>) to override these values.

This approach differs from the behaviour of clouds like AWS, where the password is auto-generated by the cloud provider and is then retrieved via the cloud provider's API after provisioning the VM.

WinRM Configuration

The WinRM initialization in Azure is achieved through configuration options in the VM provisioning request. The required configuration is to enabled HTTPS (if Azure is told to use HTTP, the VM comes pre-configured with WinRM encrypted over HTTP). The default is then to support NTLM protocol.

The setup of Windows VMs on Azure differs from that on other clouds, such as AWS. In contrast, on AWS an init script is passed to the cloud API to configure WinRM appropriately.

Windows initialization scripts in Azure are unfortunately not supported in "classic deployment"
model, but are available in the newer "resource manager deployment" model as an "Azure VM Extension".

Apache CloudStack

Connection Details

The endpoint URI will normally have the suffix /client/api/.

The identity is the "API key" and the credential is the "secret key". These can be generated in the CloudStack GUI: under accounts, select "view users", then "generate key".

location:
  jclouds:cloudstack:
    endpoint: https://cloud.acme.com/client/api
    identity: abcdefghijklmnopqrstuvwxyz01234567890-abcdefghijklmnopqrstuvwxyz01234567890-abcdefghij
    credential: mycred-abcdefghijklmnopqrstuvwxyz01234567890-abcdefghijklmnopqrstuvwxyz01234567890-abc

Users are strongly recommended to use externalized configuration for better credential management, for example using Vault.

Common Configuration Options

Below are examples of configuration options that use values specific to CloudStack environments:

• The imageId is the template id. For example, imageId: db0bcce3-9e9e-4a87-a953-2f46b603498f.

• The region is CloudStack zone id. For example region: 84539b9c-078e-458a-ae26-c3ffc5bb1ec9..

• networkName is the network id (within the given zone) to be used. For example, networkName: 961c03d4-9828-4037-9f4d-3dd597f60c4f.

For further configuration options, consult jclouds CloudStack template options. These can be used with the templateOptions configuration option.

Using a Pre-existing Key Pair

The configuration below uses a pre-existing key pair:

location:
  jclouds:cloudstack:
    ...
    loginUser: root
    loginUser.privateKeyFile: /path/to/keypair.pem
    keyPair: my-keypair

Using Pre-existing Security Groups

To specify existing security groups, their IDs must be used rather than their names (note this differs from the configuration on other clouds!).

The configuration below uses a pre-existing security group:

location:
  jclouds:cloudstack:
    ...
    templateOptions:
      generateSecurityGroup: false
      securityGroupIds:
      - 12345678-90ab-def0-1234-567890abcdef

Using Static NAT

Assigning a public IP to a VM at provision-time is referred to as "static NAT" in CloudStack parlance. To give some consistency across different clouds, the configuration option is named autoAssignFloatingIp. For example, autoAssignFloatingIp: false.

CloudMonkey CLI

The CloudStack CloudMonkey CLI is a very useful tool. It gives is an easy way to validate that credentials are correct, and to query
the API to find the correct zone IDs etc.

Useful commands include:

# for finding the ids of the zones:
cloudmonkey api listZones

# for finding the ids of the networks.
cloudmonkey api listNetworks | grep -E "id =|name =|========="

CloudStack Troubleshooting

These troubleshooting tips are more geared towards problems encountered in old test/dev CloudStack environment.

Resource Garbage Collection Issues

The environment may run out of resources, due to GC issues, preventing the user from creating new VMs or allocating IP addresses (May respond with this error message: errorCode=INTERNAL_ERROR, errorText=Job failed due to exception Unable to create a deployment for VM). There are two options worth checking it to enforce clearing up the zombie resources:

• Go to the Accounts tab in the webconsole and tap on the Update Resource Count button.
• Restart the VPC in question from the Network tab.

Releasing Allocated Public IP Addresses

Releasing an allocated Public IP from the web console did not free up the resources. Instead CloudMonkey can be used to dissociate IPs and expunge VMs.

Here is a CloudMonkey script to dissociate any zombie IPs:

cloudmonkey set display json;
cloudmonkey api listPublicIpAddresses | grep '"id":' > ips.txt; 
sed -i -e s/'      "id": "'/''/g ips.txt;
sed -i -e s/'",'/''/g ips.txt
for line in $(cat ips.txt); do cloudmonkey api disassociateIpAddress id="$line"; done
rm ips.txt;
cloudmonkey set display default;

Restarting VPCs

Errors have been encountered when a zone failed to provision new VMs, with messages like:

Job failed due to exception Resource [Host:15] is unreachable: Host 15: Unable to start instance due to null

The workaround was to restart the VPC networks:

• Log into the CloudStack web-console.
• Go to Network -> VPC (from the "select view")
• For each of the VPCs, click on the "+" in the "quickview" column, and invoke "restart VPC".

Other symptoms of this issue were that: 1) an administrator could still provision VMs using the admin account, which used a different network; and 2) the host number was very low, so it was likely to be a system host/VM that was faulty.

Google Compute Engine (GCE)

Credentials

GCE uses a service account e-mail address for the identity and a private key as the credential.

To obtain credentials for GCE, use the GCE web page's "APIs & auth -> Credentials" page, creating a "Service Account" of type JSON, then extracting the client_email as the identity and private_key as the credential. For more information, see the jclouds instructions.

An example of the expected format is shown below. Note that when supplying the credential in a properties file, it can either be one long line with \n representing the new line characters, or in YAML it can be split over multiple lines as below:

location:
  jclouds:google-compute-engine:
    region: us-central1-a
    identity: 1234567890-somet1mesArand0mU1Dhere@developer.gserviceaccount.com
    credential: |
      -----BEGIN RSA PRIVATE KEY-----
      abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz
      0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmn
      opqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+ab
      cdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz01
      23456789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnop
      qrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcd
      efghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz0123
      456789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqr
      stuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdef
      ghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz012345
      6789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrst
      uvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefgh
      ijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz01234567
      89/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuv
      wxyz0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghij
      klmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz0123456789
      /+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwx
      yz0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijkl
      mnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+
      abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz
      0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+abcdefghijklmn
      opqrstuvwxyz0123456789/+abcdefghijklmnopqrstuvwxyz0123456789/+ab
      cdefghijklmnopqrstuvwxyz
      -----END RSA PRIVATE KEY-----

It is also possible to have the credential be the path of a local file that contains the key. However, this can make it harder to setup and manage multiple Brooklyn servers (particularly when using high availability mode).

Users are strongly recommended to use externalized configuration for better credential management, for example using Vault.

Quotas

GCE accounts can have low default quotas.

It is easy to request a quota increase by submitting a quota increase form.

Networks

GCE accounts often have a limit to the number of networks that can be created. One work around is to manually create a network with the required open ports, and to refer to that named network in Brooklyn's location configuration.

To create a network, see GCE network instructions.

For example, for dev/demo purposes an "everything" network could be created that opens all ports.

|| Name || everything | || Description || opens all tcp ports | || Source IP Ranges || 0.0.0.0/0 | || Allowed protocols and ports || tcp:0-65535 and udp:0-65535 |

To configure the location to use this, you can include a location configuration option like:

templateOptions:
  network: https://www.googleapis.com/compute/v1/projects/<project name>/global/networks/everything

IBM SoftLayer

Credentials

Credentials can be obtained from the Softlayer API, under "administrative -> user administration -> api-access".

For example:

location:
  jclouds:softlayer:
    region: ams01
    identity: my-user-name
    credential: 1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef

Users are strongly recommended to use externalized configuration for better credential management, for example using Vault.

Common Configuration Options

Below are examples of configuration options that use values specific to Softlayer:

• The region is the Softlayer datacenter. For example, region: dal05.

• The hardwareId is an auto-generated combination of the hardware configuration options. This is because there is no concept of hardwareId or hardware profile names in Softlayer. An example value is hardwareId: "cpu=1,memory=1024,disk=25,type=LOCAL".

• The imageId is the Image template. For example, imageId: CENTOS_6_64.

VLAN Selection

SoftLayer may provision VMs in different VLANs, even within the same region. Some applications require VMs to be on the same internal subnet; blueprints for these can specify this behaviour in SoftLayer in one of two ways.

The VLAN ID can be set explicitly using the fields primaryNetworkComponentNetworkVlanId and primaryBackendNetworkComponentNetworkVlanId of SoftLayerTemplateOptions when specifying the location being used in the blueprint, as follows:

location:
  jclouds:softlayer:
    region: ams01
    templateOptions:
      # Enter your preferred network IDs
      primaryNetworkComponentNetworkVlanId: 1153481
      primaryBackendNetworkComponentNetworkVlanId: 1153483

This method requires that a VM already exist and you look up the IDs of its VLANs, for example in the SoftLayer console UI, and that subsequently at least one VM in that VLAN is kept around. If all VMs on a VLAN are destroyed SoftLayer may destroy the VLAN. Creating VLANs directly and then specifying them as IDs here may not work. Add a line note

The second method tells Brooklyn to discover VLAN information automatically: it will provision one VM first, and use the VLAN information from it when provisioning subsequent machines. This ensures that all VMs are on the same subnet without requiring any manual VLAN referencing, making it very easy for end-users.

To use this method, we tell brooklyn to use SoftLayerSameVlanLocationCustomizer as a location customizer. This can be done on a location as follows:

location:
  jclouds:softlayer:
    region: lon02
    customizers:
    - $brooklyn:object:
        type: org.apache.brooklyn.location.jclouds.softlayer.SoftLayerSameVlanLocationCustomizer
    softlayer.vlan.scopeUid: "my-custom-scope"
    softlayer.vlan.timeout: 10m

Usually you will want the scope to be unique to a single application, but if you need multiple applications to share the same VLAN, simply configure them with the same scope identifier.

It is also possible with many blueprints to specify this as one of the provisioning.properties on an application:

services:
- type: org.apache.brooklyn.entity.stock.BasicApplication
  id: same-vlan-application
  brooklyn.config:
    provisioning.properties:
      customizers:
      - $brooklyn:object:
          type: org.apache.brooklyn.location.jclouds.softlayer.SoftLayerSameVlanLocationCustomizer
    softlayer.vlan.scopeUid: "my-custom-scope"
    softlayer.vlan.timeout: 10m

If you are writing an entity in Java, you can also use the helper method forScope(String) to create the customizer. Configure the provisioning flags as follows:

JcloudsLocationCustomizer vlans = SoftLayerSameVlanLocationCustomizer.forScope("my-custom-scope");
flags.put(JcloudsLocationConfig.JCLOUDS_LOCATION_CUSTOMIZERS.getName(), ImmutableList.of(vlans));

Configuration Options

The allowed configuration keys for the SoftLayerSameVlanLocationCustomizer are:

• softlayer.vlan.scopeUid The scope identifier for locations whose VMs will have the same VLAN.

• softlayer.vlan.timeout The amount of time to wait for a VM to be configured before timing out without setting the VLAN ids.

• softlayer.vlan.publicId A specific public VLAN ID to use for the specified scope.

• softlayer.vlan.privateId A specific private VLAN ID to use for the specified scope.

An entity being deployed to a customized location will have the VLAN ids set as sensors, with the same names as the last two configuration keys.

NOTE If the SoftLayer location is already configured with specific VLANs then this customizer will have no effect.

OpenStack

Brooklyn requires the following API to deploy an application onto an OpenStack-based provider

• Identity API (a.k.a. OpenStack Keystone): Identity API v2 and Identity API v3 The Identity API provides an authentication and authorization service for other OpenStack services. It also provides a catalog of endpoints for all OpenStack services, like the Compute service, a.k.a OpenStack Nova.

• Compute API (a.k.a. OpenStack Nova): Compute API manages the lifecycle of compute instances in an OpenStack environment. Responsibilities include spawning, scheduling and decommissioning of machines on demand by talking to different hypervisors. In some old OpenStack installations, Nova SecurityGroupAPI extension is generally used to manage Security Groups, when required.

• Networking API (a.k.a OpenStack Neutron): Network API v2.0 enables network connectivity as a service for other OpenStack services, such as OpenStack Compute. It provides an API for users to define networks and the attachments into them. It has a pluggable architecture that supports many popular networking vendors and technologies.

Apache jclouds may optionally need to talk to the OpenStack Neutron API to manage networks, subnets, security groups and ports, by linking Neutron Context to Nova Context.

In some cases it may be required to leverage OpenStack Neutron Extensions such as LoadBalancer-aaS, the FloatingIp API, the Router API or the Firewall-aaS API when available.

Brooklyn offers supports up to OpenStack Pike release via Apache jclouds. For more information, see their guide here.

Connection Details

The endpoint URI is that of keystone (normally on port 5000), notice v2 or v3 suffix may be required.

The identity normally consists of a colon-separated tenant and username. The credential is the password. For example:

location:
  jclouds:openstack-nova
    brooklyn.config:
      endpoint: http://x.x.x.x:5000/v3
      identity: "your-tenant:your-username"
      credential: password
      jclouds.keystone.version: 3 # default is `2`
      # Authorization Scopes - optional
      # Project scoped authorization (can use the project name or the ID) 
      # jclouds.keystone.scope: 'project:project-id' # or 'project:project-name' 
      # Domain scoped authorization (can use the domain name or the ID)
      # jclouds.keystone.scope: 'domain:domain-id' # or 'domain:domain-name'

Notice, by default, openstack-nova location tries to manage network concepts required for the deployment in this order:

1. Use OpenStack Networking API to manage security groups and floating IPs, if linked.
2. Use OpenStack Nova SecurityGroupApi and FloatingIpApi extensions as fallback strategies, if available.

To link Openstack Neutron context use the following YAML schema:

location:
  jclouds:openstack-nova
    brooklyn.config:
      endpoint: http://x.x.x.x:5000/v3
      identity: "your-tenant:your-username"
      credential: password
      jclouds.keystone.version: 3 # default is `2`
      # Authorization Scopes - optional
      # Project scoped authorization (can use the project name or the ID) 
      # jclouds.keystone.scope: 'project:project-id' # or 'project:project-name' 
      # Domain scoped authorization (can use the domain name or the ID)
      # jclouds.keystone.scope: 'domain:domain-id' # or 'domain:domain-name'
      # neutron
      jclouds.linkContext:
        - $brooklyn:object:
          type: org.apache.brooklyn.location.jclouds.domain.JcloudsContext
          object.fields:
            providerOrApi: openstack-neutron
            # Additional credentials for OpenStack Neutron - optional
            # identity: identity # by default, it's OpenStack Nova identity
            # credential: credential # by default, it's OpenStack Nova credential

OpenStack Nova access information can be downloaded from the openstack web interface, for example as an openrc.sh file. It is usually available from API Access tab in "Access & Security" section. This file will normally contain the identity and credential.

Users are strongly recommended to use externalized configuration for better credential management, for example using Vault.

Common Configuration Options

Below are examples of configuration options that use values specific to OpenStack environments:

• The imageId is the id of an image. For example, imageId: RegionOne/08086159-8b0b-4970-b332-a7a929ee601f. These ids can be found from the the CLI or the web-console, for example in IBM Blue Box London, the URL is https://tenant-region.openstack.blueboxgrid.com/project/images/.

• The hardwareId is the flavor id. For example hardwareId: RegionOne/1. These ids can be found from the the CLI or the web-console, for example in IBM Blue Box, the URL is https://tenant-region.openstack.blueboxgrid.com/admin/flavors/.

The default flavors are shown below (though the set of flavors can be managed by the admin):

+-----+-----------+-----------+------+
| ID  | Name      | Memory_MB | Disk |
+-----+-----------+-----------+------+
| 1   | m1.tiny   | 512       | 1    |
| 2   | m1.small  | 2048      | 20   |
| 3   | m1.medium | 4096      | 40   |
| 4   | m1.large  | 8192      | 80   |
| 5   | m1.xlarge | 16384     | 160  |
+-----+-----------+-----------+------+

For further configuration options, consult jclouds Nova template options. These can be used with the templateOptions configuration option.

Networks

When multiple networks are available you should indicate which ones machines should join. Do this by setting the desired values id as an option in the templateOptions configuration:

location:
  jclouds:openstack-nova:
    ...
    templateOptions:
      # Assign the node to all networks in the list.
      networks:
      - network-one-id
      - network-two-id
      - ...

Floating IPs

The autoAssignFloatingIp option means that a floating ip will be assigned to the VM at provision-time.

A floating IP pool name can also be specified. If not set, a floating IP from any available pool will be chosen. This is set using the template option. For example:

location:
  jclouds:openstack-nova:
    ...
    autoAssignFloatingIp: true
    templateOptions:
      # Pool names to use when allocating a floating IP
      floatingIpPoolNames:
      - "pool name"

Basic Location Structure

This is a basic inline YAML template for an OpenStack location:

location:
    jclouds:openstack-nova:
        endpoint: http://x.x.x.x:5000/v2.0/
        identity: "your-tenant:your-username"
        credential: your-password

        # imageId, hardwareId, and loginUser* are optional
        imageId: your-region-name/your-image-id
        hardwareId: your-region-name/your-flavor-id
        loginUser: 'ubuntu'
        loginUser.privateKeyFile: /path/to/your/privatekey

        jclouds.openstack-nova.auto-generate-keypairs: false
        jclouds.openstack-nova.auto-create-floating-ips: true

        templateOptions:
            networks: [ "your-network-id" ]
            floatingIpPoolNames: [ "your-floatingIp-pool" ]
            securityGroups: ['your-security-group']

            # Optional if 'jclouds.openstack-nova.auto-generate-keypairs' is assigned to 'true'
            keyPairName: "your-keypair"

This is the same OpenStack location in a format that can be added to your brooklyn.properties file:

brooklyn.location.named.My\ OpenStack=jclouds:openstack-nova:http://x.x.x.x:5000/v2.0/
brooklyn.location.named.My\ OpenStack.identity=your-tenant:your-username
brooklyn.location.named.My\ OpenStack.credential=your-password
brooklyn.location.named.My\ OpenStack.endpoint=http://x.x.x.x:5000/v2.0/

brooklyn.location.named.My\ OpenStack.imageId=your-region-name/your-image-id
brooklyn.location.named.My\ OpenStack.hardwareId=your-region-name/your-flavor-id
brooklyn.location.named.My\ OpenStack.loginUser=ubuntu
brooklyn.location.named.My\ OpenStack.loginUser.privateKeyFile=/path/to/your/privatekey
brooklyn.location.named.My\ OpenStack.openstack-nova.auto-generate-keypairs=false
brooklyn.location.named.My\ OpenStack.openstack-nova.auto-create-floating-ips=true

brooklyn.location.named.My\ OpenStack.networks=your-network-id
brooklyn.location.named.My\ OpenStack.floatingIpPoolNames=your-floatingIp-pool
brooklyn.location.named.My\ OpenStack.securityGroups=your-security-group
brooklyn.location.named.My\ OpenStack.keyPair=your-keypair

Troubleshooting

Cloud Credentials Failing

If the cloud API calls return 401 Unauthorized (e.g. in a org.jclouds.rest.AuthorizationException), then this could be because the credentials are incorrect.

A good way to check this is to try the same credentials with the OpenStack nova command line client.

Unable to SSH: Wrong User

If SSH authentication fails, it could be that the login user is incorrect. For most clouds, this is inferred from the image metadata, but if no (or the wrong) login user is specified then it will
default to root. The correct login user can be specified using the configuration option loginUser. For example, loginUser: ubuntu.

The use of the wrong login user can also result in the obscure message, caused by an unexpected response saying to use a different user. For more technical information, see this sshj github issue. The message is:

Received message too long 1349281121

I Want to Use My Own KeyPair

By default, jclouds will auto-generate a new key pair for the VM. This key pair will be deleted automatically when the VM is deleted.

Alternatively, you can use a pre-existing key pair. If so, you must also specify the corresponding private key (pem file, or data) to be used for the initial login. The name used in the keyPair configuration must match the name of a key pair that has already been added in OpenStack. For example:

location:
  jclouds:clouds:openstack-nova:
    ...
    jclouds.openstack-nova.auto-generate-keypairs: false
    keyPair: "my-keypair"
    loginUser: ubuntu
    loginUser.privateKeyFile: /path/to/my/privatekey.pem

Error "doesn't contain ... -----BEGIN"

If using loginUser.privateKeyFile (or loginUser.privateKeyData), this is expected to be a .pem file. If a different format is used (e.g. a .ppk file), it will give an error like that below:

Error invoking start at EmptySoftwareProcessImpl{id=TrmhitVc}: chars
PuTTY-User-Key-File-2: ssh-rsa
...
doesn't contain % line [-----BEGIN ]

Warning Message: "Ignoring request to set..."

If you see a warning log message like that below:

2016-06-23 06:05:12,297 WARN  o.a.b.l.j.JcloudsLocation [brooklyn-execmanager-XlwkWB3k-312]: 
Ignoring request to set template option loginUser because this is not supported by 
org.jclouds.openstack.nova.v2_0.compute.options.NovaTemplateOptions

It can mean that the location configuration option is in the wrong place. The configuration under templateOptions must correspond to those options on the jclouds Nova template options. However, template options such as loginUser are top-level configuration options that should not be inside the templateOptions section.

HttpResponseException Accessing Compute Endpoint

The Keystone endpoint is first queried to get the API access endpoints for the appropriate services.

Some private OpenStack installs are (mis)configured such that the returned addresses are not always directly accessible. It could be that the service is behind a VPN, or that they rely on hostnames that are only in a private DNS.

You can find the service endpoints in OpenStack, either using the CLI or the web-console. For example, in Blue Box the URL is https://tenant-region.openstack.blueboxgrid.com/project/access_and_security/. You can then check if the Compute service endpoint is directly reachable.

VM Failing to Provision

It can be useful to drop down to the OpenStack nova CLI, or to jclouds, to confirm that VM provisioning is working and to check which options are required.

For example, try following these jclouds instructions.

jclouds Namespace Issue

A change to Nova's API (in the Mitaka release) resulted in all extensions having the same "fake" namespace which the current version of jclouds does not yet support.

If you are having problems deploying to OpenStack, consult your Brooklyn debug log and look for the following:

"namespace": "http://docs.openstack.org/compute/ext/fake_xml"

If you already have jclouds:openstack-mitaka-nova, then try using this instead of the vanilla jclouds:openstack-nova. For example:

location:
    jclouds:openstack-mitaka-nova:
        endpoint: http://x.x.x.x:5000/v2.0/
        identity: "your-tenant:your-username"
        credential: your-password
        templateOptions:
            networks: [ "your-network-id" ]
            floatingIpPoolNames: [ "your-floatingIp-pool" ]

Note that the following values will be set by default when omitted above:

jclouds.keystone.credential-type=passwordCredentials
jclouds.openstack-nova.auto-generate-keypairs: true
jclouds.openstack-nova.auto-create-floating-ips: true

Inheritance and Named Locations

Named locations can be defined for commonly used groups of properties, with the syntax brooklyn.location.named.your-group-name. followed by the relevant properties. These can be accessed at runtime using the syntax named:your-group-name as the deployment location.

Some illustrative examples using named locations and showing the syntax and properties above are as follows:

# Production pool of machines for my application (deploy to named:prod1)
brooklyn.location.named.prod1=byon:(hosts="10.9.1.1,10.9.1.2,produser2@10.9.2.{10,11,20-29}")
brooklyn.location.named.prod1.user=produser1
brooklyn.location.named.prod1.privateKeyFile=~/.ssh/produser_id_rsa
brooklyn.location.named.prod1.privateKeyPassphrase=s3cr3tCOMPANYpassphrase

# AWS using my company's credentials and image standard, then labelling images so others know they're mine
brooklyn.location.named.company-jungle=jclouds:aws-ec2:us-west-1
brooklyn.location.named.company-jungle.identity=BCDEFGHIJKLMNOPQRSTU  
brooklyn.location.named.company-jungle.privateKeyFile=~/.ssh/public_clouds/company_aws_id_rsa
brooklyn.location.named.company-jungle.imageId=ami-12345
brooklyn.location.named.company-jungle.minRam=2048
brooklyn.location.named.company-jungle.userMetadata=application=my-jungle-app,owner="Bob Johnson"
brooklyn.location.named.company-jungle.machineCreateAttempts=2

brooklyn.location.named.AWS\ Virginia\ Large\ Centos = jclouds:aws-ec2
brooklyn.location.named.AWS\ Virginia\ Large\ Centos.region = us-east-1
brooklyn.location.named.AWS\ Virginia\ Large\ Centos.imageId=us-east-1/ami-7d7bfc14
brooklyn.location.named.AWS\ Virginia\ Large\ Centos.user=root
brooklyn.location.named.AWS\ Virginia\ Large\ Centos.minRam=4096

Named locations can refer to other named locations using named:xxx as their value. These will inherit the configuration and can override selected keys. Properties set in the namespace of the provider (e.g. b.l.jclouds.aws-ec2.KEY=VALUE) will be inherited by everything which extends AWS Sub-prefix strings are also inherited up to brooklyn.location.*, except that they are filtered for single-word and other known keys (so that we exclude provider-scoped properties when looking at sub-prefix keys). The precedence for configuration defined at different levels is that the value defined in the most specific context will apply.

This is rather straightforward and powerful to use, although it sounds rather more complicated than it is! The examples below should make it clear. You could use the following to install a public key on all provisioned machines, an additional public key in all AWS machines, and no extra public key in prod1:

brooklyn.location.extraSshPublicKeyUrls=http://me.com/public_key
brooklyn.location.jclouds.aws-ec2.extraSshPublicKeyUrls="[ \"http://me.com/public_key\", \"http://me.com/aws_public_key\" ]"
brooklyn.location.named.prod1.extraSshPublicKeyUrls=

And in the example below, a config key is repeatedly overridden. Deploying location: named:my-extended-aws will result in an aws-ec2 machine in us-west-1 (by inheritance) with VAL6 for KEY:

brooklyn.location.KEY=VAL1
brooklyn.location.jclouds.KEY=VAL2
brooklyn.location.jclouds.aws-ec2.KEY=VAL3
brooklyn.location.jclouds.aws-ec2@us-west-1.KEY=VAL4
brooklyn.location.named.my-aws=jclouds:aws-ec2:us-west-1
brooklyn.location.named.my-aws.KEY=VAL5
brooklyn.location.named.my-extended-aws=named:my-aws
brooklyn.location.named.my-extended-aws.KEY=VAL6

BYON

"Bring-your-own-nodes" mode is useful in production, where machines have been provisioned by someone else, and during testing, to cut down provisioning time.

Your nodes must meet the following prerequisites:

• A suitable OS must have been installed on all nodes
• The node must be running sshd (or similar)
• the brooklyn user must be able to ssh to each node as root or as a user with passwordless sudo permission. (For more information on SSH keys, see here.)

To deploy to machines with known IP's in a blueprint, use the following syntax:

location:
  byon:
    user: brooklyn
    privateKeyFile: ~/.ssh/brooklyn.pem
    hosts:
    - 192.168.0.18
    - 192.168.0.19

Some of the login properties as described above for jclouds are supported, but not loginUser (as no users are created), and not any of the VM creation parameters such as minRam and imageId. (These clearly do not apply in the same way, and they are not by default treated as constraints, although an entity can confirm these where needed.) As before, if the brooklyn user and its default key are authorized for the hosts, those fields can be omitted.

Named locations can also be configured in your brooklyn.properties, using the format byon:(key=value,key2=value2). For convenience, for hosts wildcard globs are supported.

brooklyn.location.named.On-Prem\ Iron\ Example=byon:(hosts="10.9.1.1,10.9.1.2,produser2@10.9.2.{10,11,20-29}")
brooklyn.location.named.On-Prem\ Iron\ Example.user=produser1
brooklyn.location.named.On-Prem\ Iron\ Example.privateKeyFile=~/.ssh/produser_id_rsa
brooklyn.location.named.On-Prem\ Iron\ Example.privateKeyPassphrase=s3cr3tpassphrase

Alternatively, you can create a specific BYON location through the location wizard tool available within the web console. This location will be saved as a catalog entry for easy reusability.

For more complex host configuration, one can define custom config values per machine. In the example below, there will be two machines. The first will be a machine reachable on ssh -i ~/.ssh/brooklyn.pem -p 8022 myuser@50.51.52.53. The second is a Windows machine, reachable over WinRM. Each machine has also has a private address (e.g. for within a private network).

location:
  byon:
    hosts:
    - ssh: 50.51.52.53:8022
      privateAddresses: [10.0.0.1]
      privateKeyFile: ~/.ssh/brooklyn.pem
      user: myuser
    - winrm: 50.51.52.54:8985
      privateAddresses: [10.0.0.2]
      password: mypassword
      user: myuser
      osFamily: windows

The BYON location also supports a machine chooser, using the config key byon.machineChooser. This allows one to plugin logic to choose from the set of available machines in the pool. For example, additional config could be supplied for each machine. This could be used (during the call to location.obtain()) to find the config that matches the requirements of the entity being provisioned. See FixedListMachineProvisioningLocation.MACHINE_CHOOSER.

SSH Keys

SSH keys are one of the simplest and most secure ways to access remote servers. They consist of two parts:

• A private key (e.g. id_rsa) which is known only to one party or group

• A public key (e.g. id_rsa.pub) which can be given to anyone and everyone, and which can be used to confirm that a party has a private key (or has signed a communication with the private key)

In this way, someone -- such as you -- can have a private key, and can install a public key on a remote machine (in an authorized_keys file) for secure automated access. Commands such as ssh (and Brooklyn) can log in without revealing the private key to the remote machine, the remote machine can confirm it is you accessing it (if no one else has the private key), and no one snooping on the network can decrypt of any of the traffic.

Creating an SSH Key

If you don't have an SSH key, create one with:

$ ssh-keygen -t rsa -N "" -m PEM -f ~/.ssh/id_rsa

Note: For previous versions of OpenSSH, the -m PEM flag is not necessary. However, for newer versions of OpenSSH, if the -m PEM flag is omitted, the generated key will be in RFC4716 format, not PEM format. Check the first line of the generated secret key file, which should be as follows:

$ head -n 1 ~/.ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----
$

Localhost Setup

If you want to deploy to localhost, ensure that you have a public and private key, and that your key is authorized for ssh access:

# _Appends_ id_rsa.pub to authorized_keys. Other keys are unaffected.
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Now verify that your setup by running the command: ssh localhost echo hello world

If your setup is correct, you should see hello world printed back at you.

On the first connection, you may see a message similar to this:

The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 7b:e3:8e:c6:5b:2a:05:a1:7c:8a:cf:d1:6a:83:c2:ad.
Are you sure you want to continue connecting (yes/no)?

Simply answer 'yes' and then repeat the command again.

If this isn't the case, see below.

Potential Problems

• MacOS user? In addition to the above, enable "Remote Login" in "System Preferences > Sharing".

• Got a passphrase? Set brooklyn.location.localhost.privateKeyPassphrase as described here. If you're not sure, or you don't know what a passphrase is, you can test this by executing ssh-keygen -y. If it does not ask for a passphrase, then your key has no passphrase. If your key does have a passphrase, you can remove it by running ssh-keygen -p.

• Check that you have an ~/.ssh/id_rsa file (or id_dsa) and a corresponding public key with a .pub extension; if not, create one as described above

• ~/.ssh/ or files in that directory may have permissions they shouldn't: they should be visible only to the user (apart from public keys), both on the source machine and the target machine. You can verify this with ls -l ~/.ssh/: lines should start with -rw------- or -r-------- (or -rwx------ for directories). If it does not, execute chmod go-rwx ~/.ssh ~/.ssh/*.

• Sometimes machines are configured with different sets of support SSL/TLS versions and ciphers; if command-line ssh and scp work, but Brooklyn/java does not, check the versions enabled in Java and on both servers.

• Missing entropy: creating and using ssh keys requires randomness available on the servers, usually in /dev/random; see here for more information

Localhost

If passwordless ssh login to localhost and passwordless sudo is enabled on your machine, you should be able to deploy some blueprints with no special configuration, just by specifying location: localhost in YAML.

For example:

location: localhost
services:
- type: org.apache.brooklyn.entity.database.mysql.MySqlNode

If you use a passphrase or prefer a different key, these can be configured as follows:

location:
  localhost:
    privateKeyFile=~/.ssh/brooklyn_key
    privateKeyPassphrase=s3cr3tPASSPHRASE

Alternatively, you can create a specific localhost location through the location wizard tool available within the web console. This location will be saved as a catalog entry for easy reusability.

Passwordless Sudo

If you encounter issues or for more information, see SSH Keys Localhost Setup.

For some blueprints, passwordless sudo is required. (Try executing sudo whoami to see if it prompts for a password. To enable passwordless sudo for your account, a line must be added to the system /etc/sudoers file.
To edit the file, use the visudo command:

sudo visudo

Add this line at the bottom of the file, replacing username with your own user:

username ALL=(ALL) NOPASSWD: ALL

If executing the following command does not ask for your password, then sudo has been setup correctly:

sudo whoami

Location Customizers

Apache Brooklyn supports a number of ways to configure and customize locations. These include the JcloudsLocationCustomizer, which is for advanced customization of VM provisioning through jclouds. There is also a MachineLocationCustomizer, which allows customization of machines being obtained from any kind of location (including Bring Your Own Nodes).

Usage Guidelines

Clearly there is an overlap for where things can be done. This section describes the recommended
separation of responsibilities.

These are guidelines only - users are obviously free to make alternative usage decisions based on their particular use-cases.

Responsibilities of Entity versus Location

From an entity's perspective, it calls location.obtain(options) and gets back a usable MachineLocation that has a standard base operating system that gives remote access (e.g. for Linux it expects credentials for a user with sudo rights, and ssh access).

However, there are special cases - for example the location.obtain(options) could return a Docker container with the software pre-installed, and no remote access (see the Clocker project for more information on use of Docker with Brooklyn).

The entity is then responsible for configuring that machine according to the needs of the software to be installed.

For example, the entity may install software packages, upload/update configuration files, launch processes, etc.

The entity may also configure iptables. This is also possible through the JcloudsLocation configuration. However, it is preferable to do this in the entity because it is part of configuring the machine in the way required for the given software component.

The entity may also perform custom OS setup, such as installing security patches. However, whether this is appropriate depends on the nature of the security patch: if the security patch is specific to the entity type, then it should be done within the entity; but if it is to harden the base OS to make it comply with an organisation's standards (e.g. to overcome shortcomings of the base image, or to install security patches) then a MachineLocationCustomizer is more appropriate.

Location Configuration Options

This refers to standard location configuration: explicit config keys, and explicit jclouds template configuration that can be passed through.

This kind of configuration is simplest to use. It is the favoured mechanism when it comes to VM provisioning, and should be used wherever possible.

Note that a jclouds TemplateBuilder and cloud-specific TemplateOptions are the generic mechanisms within jclouds for specifying the details of the compute resource to be provisioned.

Jclouds Location Customizer

A JcloudsLocationCustomizer has customization hooks to execute code at the various points of building up the jclouds template and provisioning the machine. Where jclouds is being used and where the required use of jclouds goes beyond simple configuration, this is an appropriate solution.

For example, there is a org.apache.brooklyn.location.jclouds.networking.JcloudsLocationSecurityGroupCustomizer which gives more advanced support for setting up security groups (e.g. in AWS-EC2).

Machine Customizer

The MachineLocationCustomizer allows customization of machines being obtained from any kind of location. For example, this includes for jclouds and for Bring Your Own Nodes (BYON).

It provides customization hooks for when the machine has been provisioned (before it is returned by the location) and when the machine is about to be released by the location.

An example use would be to register (and de-register) the machine in a CMDB.

Jclouds Location Customizers

Warning: additional methods (i.e. customization hooks) may be added to the JcloudsLocationCustomizer interface in future releases. Users are therefore strongly encouraged to sub-class BasicJcloudsLocationCustomizer, rather than implementing JcloudsLocationCustomizer directly.

The JcloudsLocationCustomizer provides customization hooks at various points of the Brooklyn's use of jclouds. These can be used to adjust the configuration, to do additional setup, to do custom logging, etc.

• Customize the org.jclouds.compute.domain.TemplateBuilder, before it is used to build the template. This is used to influence the choice of VM image, hardware profile, etc. This hook is not normally required as the location configuration options can be used in instead.

• Customize the org.jclouds.compute.domain.Template, to be used when creating the machine. This
  hook is most often used for performing custom actions - for example to create or modify a security group or volume, and to update the template's options to use that.

• Customize the org.jclouds.compute.options.TemplateOptions to be used when creating the machine. The TemplateOptions could be cast to a cloud-specific sub-type (if this does not have to work across different clouds). Where the use-case is to just set simple configuration on the TemplateOptions, consider instead using the config key templateOptions, which takes a map of type String to Object - the strings should match the method names in the TemplateOptions.

• Customize the org.apache.brooklyn.location.jclouds.JcloudsMachineLocation that has been created. For Linux-based VMs, if the config waitForSshable was not false, then this machine is guaranteed to be ssh'able. Similarly for WinRM access to Windows machines, if waitForWinRmAvailable was not false.

• Pre-release of the machine. If the actions required are specific to jclouds (e.g. using jclouds to make calls to the cloud provider) then this should be used; otherwise one should use the more generic MachineLocationCustomizer.

• Post-release of the machine (i.e. after asking jclouds to destroying the machine).

To register a JcloudsLocationCustomizer in YAML, the config key customizers can be used to provide a list of instances. Each instance can be defined using $brooklyn:object to indicate the type and its configuration. For example:

location:
  jclouds:aws-ec2:us-east-1:
    customizers:
    - $brooklyn:object:
        type: com.acme.brooklyn.MyJcloudsLocationCustomizer

To register JcloudsLocationCustomizer instances programmatically, set the config key JcloudsLocationConfig.JCLOUDS_LOCATION_CUSTOMIZERS on the location, or pass this config option when calling location.obtain(options).

The SharedLocationSecurityGroupCustomizer configures a shared security group on Jclouds locations. It only works on AWS and Azure ARM.

To register a SharedLocationSecurityGroupCustomizer in YAML, you can use the config key customizers and configure it with $brooklyn:object and object.fields. For example:

location:
  jclouds:aws-ec2:us-east-1:
    customizers:
    - $brooklyn:object:
        type: org.apache.brooklyn.location.jclouds.networking.SharedLocationSecurityGroupCustomizer
        object.fields: {locationName: "myloc", tcpPortRanges: ["22", "8080", "9443"], udpPortRanges: ["2001", "4013"], cidr: "82.40.153.101/24"}

where cidr can be optionally set to restrict the range that the ports that are to be opened can be accessed from.

Machine Location Customizers

Warning: additional methods (i.e. customization hooks) may be added to the MachineLocationCustomizer interface in future releases. Users are therefore strongly encouraged to sub-class BasicMachineLocationCustomizer, rather than implementing MachineLocationCustomizer directly.

The MachineLocationCustomizer provides customization hooks for when a machine is obtained/released from a MachineProvisioningLocation. The following hooks are supported:

• After the machine has been provisioned/allocated, but before it has been returned.

• When the machine is about to be released, but prior to actually destroying/unallocating the machine.

To register a MachineLocationCustomizer in YAML, the config key machineCustomizers can be used
to provide a list of instances. Each instance can be defined using $brooklyn:object to indicate the type and its configuration. For example:

location:
  jclouds:aws-ec2:us-east-1:
    machineCustomizers:
    - $brooklyn:object:
        type: com.acme.brooklyn.MyMachineLocationCustomizer

To register MachineLocationCustomizer instances programmatically, set the config key CloudLocationConfig.MACHINE_LOCATION_CUSTOMIZERS on the location, or pass this config option when calling location.obtain(options).

Hostname Customizer

org.apache.brooklyn.entity.machine.SetHostnameCustomizer Sets the hostname on an ssh'able machine. Currently only CentOS and RHEL are supported. The customizer can be configured with a hard-coded hostname, or with a freemarker template whose value (after substitutions) will be used for the hostname.

Customizing Cloud Security Groups

Before using SharedLocationSecurityGroupCustomizer, please first refer to Port Inferencing.

A security group is a named collection of network access rules that are use to limit the types of traffic that have access to instances.
Security group is the standard way to set firewall restrictions on the AWS-EC2 environment. docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_SecurityGroups.html

When deploying to AWS EC2 target, by default Apache Brooklyn creates security group attached to the VM. It is easy to add additional rules to the initial security group using org.apache.brooklyn SharedLocationSecurityGroupCustomizer.

YAML Example:

name: ports @ AWS
location: jclouds:aws-ec2:us-west-2
services:
- type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess
  brooklyn.config:
    provisioning.properties:
      customizers:
      - $brooklyn:object:
          type: org.apache.brooklyn.location.jclouds.networking.SharedLocationSecurityGroupCustomizer
          object.fields: {tcpPortRanges: ["900-910", "915", "22"], udpPortRanges: ["100","200-300"], cidr: "82.40.153.101/24"}

Make sure that you have rule which makes port 22 accessible from Apache Brooklyn.

Opening ports during runtime.

Apache Brooklyn exposes the SharedLocationSecurityGroupCustomizer functionality after entity is deployed
just by supplying effector.add.openInboundPorts: true "brooklyn.config". Example configuration in effector

location: jclouds:aws-ec2:us-west-2
services:
- type: org.apache.brooklyn.entity.software.base.EmptySoftwareProcess
  brooklyn.config:
    effector.add.openInboundPorts: true

Known limitations

Not all cloud providers support Security Group abstraction. SharedLocationSecurityGroupCustomizer is known to work well with Amazon EC2.
Other clouds which support Security Groups:

• Openstack
• Azure - jclouds-labs azurecompute implementation uses endpoints rules when creating a VM instance. jclouds:azurecompute based location do not have security groups so SharedLocationSecurityGroupCustomizer is used it will fail to find a security group.

Specialized Locations

Some additional location types are supported for specialized situations:

Single Host

The spec host, taking a string argument (the address) or a map (host, user, password, etc.), provides a convenient syntax when specifying a single host. For example:

location: host:(192.168.0.1)
services:
- type: org.apache.brooklyn.entity.webapp.jboss.JBoss7Server

Or, in brooklyn.properties, set brooklyn.location.named.host1=host:(192.168.0.1).

The Multi Location

The spec multi allows multiple locations, specified as targets, to be combined and treated as one location.

Sequential Consumption

In its simplest form, this will use the first target location where possible, and will then switch to the second and subsequent locations when there are no machines available.

In the example below, it provisions the first node to 192.168.0.1, then it provisions into AWS us-east-1 region (because the bring-your-own-nodes region will have run out of nodes).

location:
  multi:
    targets:
    - byon:(hosts=192.168.0.1)
    - jclouds:aws-ec2:us-east-1
services:
- type: org.apache.brooklyn.entity.group.DynamicCluster
  brooklyn.config:
    cluster.initial.size: 3
    dynamiccluster.memberspec:
      $brooklyn:entitySpec:
        type: org.apache.brooklyn.entity.machine.MachineEntity

Round-Robin Consumption and Availability Zones for Clustered Applications

A DynamicCluster can be configured to cycle through its deployment targets round-robin when provided with a location that supports the AvailabilityZoneExtension -- the multi location supports this extension.

The configuration option dynamiccluster.zone.enable on DynamicCluster tells it to query the given location for AvailabilityZoneExtension support. If the location supports it, then the cluster will query for the list of availability zones (which in this case is simply the list of targets) and deploy to them round-robin.

In the example below, the cluster will request VMs round-robin across three different locations (in this case, the locations were already added to the catalog, or defined in brooklyn.properties).

location:
  multi:
    targets:
    - my-location-1
    - my-location-2
    - my-location-3
services:
- type: org.apache.brooklyn.entity.group.DynamicCluster
  brooklyn.config:
    dynamiccluster.zone.enable: true
    cluster.initial.size: 3
    dynamiccluster.memberspec:
      $brooklyn:entitySpec:
        type: org.apache.brooklyn.entity.machine.MachineEntity

Of course, clusters can also be deployed round-robin to real availability zones offered by cloud providers, as long as their locations support AvailabilityZoneExtension. Currently, only AWS EC2 locations support this feature.

In the example below, the cluster will request VMs round-robin across the availability zones provided by AWS EC2 in the "us-east-1" region.

location: jclouds:aws-ec2:us-east-1
services:
- type: org.apache.brooklyn.entity.group.DynamicCluster
  brooklyn.config:
    dynamiccluster.zone.enable: true
    cluster.initial.size: 3
    dynamiccluster.memberspec:
      $brooklyn:entitySpec:
        type: org.apache.brooklyn.entity.machine.MachineEntity

For more information about AWS EC2 availability zones, see this guide.

Custom alternatives to round-robin are also possible using the configuration option dynamiccluster.zone.placementStrategy on DynamicCluster.

The Server Pool

The ServerPool entity type allows defining an entity which becomes available as a location.