Create AMI image as part of a cloudformation stack

Question

I want to create an EC2 cloudformation stack which basically can be described in the following steps:

1.- Launch instance

2.- Provision the instance

Answer 1

For what it's worth, here's Python variant of wjordan's AMIFunction definition in the original answer. All other resources in the original yaml remain unchanged:

AMIFunction:
  Type: AWS::Lambda::Function
  Properties:
    Handler: index.handler
    Role: !GetAtt LambdaExecutionRole.Arn
    Code:
      ZipFile: !Sub |
        import logging
        import cfnresponse
        import json
        import boto3
        from threading import Timer
        from botocore.exceptions import WaiterError

        logger = logging.getLogger()
        logger.setLevel(logging.INFO)

        def handler(event, context):

          ec2 = boto3.resource('ec2')
          physicalId = event['PhysicalResourceId'] if 'PhysicalResourceId' in event else None

          def success(data={}):
            cfnresponse.send(event, context, cfnresponse.SUCCESS, data, physicalId)

          def failed(e):
            cfnresponse.send(event, context, cfnresponse.FAILED, str(e), physicalId)

          logger.info('Request received: %s\n' % json.dumps(event))

          try:
            instanceId = event['ResourceProperties']['InstanceId']
            if (not instanceId):
              raise 'InstanceID required'

            if not 'RequestType' in event:
              success({'Data': 'Unhandled request type'})
              return

            if event['RequestType'] == 'Delete':
              if (not physicalId.startswith('ami-')):
                raise 'Unknown PhysicalId: %s' % physicalId

              ec2client = boto3.client('ec2')
              images = ec2client.describe_images(ImageIds=[physicalId])
              for image in images['Images']:
                ec2.Image(image['ImageId']).deregister()
                snapshots = ([bdm['Ebs']['SnapshotId'] 
                              for bdm in image['BlockDeviceMappings'] 
                              if 'Ebs' in bdm and 'SnapshotId' in bdm['Ebs']])
                for snapshot in snapshots:
                  ec2.Snapshot(snapshot).delete()

              success({'Data': 'OK'})
            elif event['RequestType'] in set(['Create', 'Update']):
              if not physicalId:  # AMI creation has not been requested yet
                instance = ec2.Instance(instanceId)
                instance.wait_until_stopped()

                image = instance.create_image(Name="Automatic from CloudFormation stack ${AWS::StackName}")

                physicalId = image.image_id
              else:
                logger.info('Continuing in awaiting image available: %s\n' % physicalId)

              ec2client = boto3.client('ec2')
              waiter = ec2client.get_waiter('image_available')

              try:
                waiter.wait(ImageIds=[physicalId], WaiterConfig={'Delay': 30, 'MaxAttempts': 6})
              except WaiterError as e:
                # Request the same event but set PhysicalResourceId so that the AMI is not created again
                event['PhysicalResourceId'] = physicalId
                logger.info('Timeout reached, continuing function: %s\n' % json.dumps(event))
                lambda_client = boto3.client('lambda')
                lambda_client.invoke(FunctionName=context.invoked_function_arn, 
                                      InvocationType='Event',
                                      Payload=json.dumps(event))
                return

              success({'Data': 'OK'})
            else:
              success({'Data': 'OK'})
          except Exception as e:
            failed(e)
    Runtime: python2.7
    Timeout: 300

Answer 2

Yes, you can create an AMI from an EC2 instance within a CloudFormation template by implementing a Custom Resource that calls the CreateImage API on create (and calls the DeregisterImage and DeleteSnapshot APIs on delete).

Since AMIs can sometimes take a long time to create, a Lambda-backed Custom Resource will need to re-invoke itself if the wait has not completed before the Lambda function times out.

Here's a complete example:

Description: Create an AMI from an EC2 instance.
Parameters:
  ImageId:
    Description: Image ID for base EC2 instance.
    Type: AWS::EC2::Image::Id
    # amzn-ami-hvm-2016.09.1.20161221-x86_64-gp2
    Default: ami-9be6f38c
  InstanceType:
    Description: Instance type to launch EC2 instances.
    Type: String
    Default: m3.medium
    AllowedValues: [ m3.medium, m3.large, m3.xlarge, m3.2xlarge ]
Resources:
  # Completes when the instance is fully provisioned and ready for AMI creation.
  AMICreate:
    Type: AWS::CloudFormation::WaitCondition
    CreationPolicy:
      ResourceSignal:
        Timeout: PT10M
  Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      UserData:
        "Fn::Base64": !Sub |
          #!/bin/bash -x
          yum -y install mysql # provisioning example
          /opt/aws/bin/cfn-signal \
            -e $? \
            --stack ${AWS::StackName} \
            --region ${AWS::Region} \
            --resource AMICreate
          shutdown -h now
  AMI:
    Type: Custom::AMI
    DependsOn: AMICreate
    Properties:
      ServiceToken: !GetAtt AMIFunction.Arn
      InstanceId: !Ref Instance
  AMIFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: !Sub |
          var response = require('cfn-response');
          var AWS = require('aws-sdk');
          exports.handler = function(event, context) {
            console.log("Request received:\n", JSON.stringify(event));
            var physicalId = event.PhysicalResourceId;
            function success(data) {
              return response.send(event, context, response.SUCCESS, data, physicalId);
            }
            function failed(e) {
              return response.send(event, context, response.FAILED, e, physicalId);
            }
            // Call ec2.waitFor, continuing if not finished before Lambda function timeout.
            function wait(waiter) {
              console.log("Waiting: ", JSON.stringify(waiter));
              event.waiter = waiter;
              event.PhysicalResourceId = physicalId;
              var request = ec2.waitFor(waiter.state, waiter.params);
              setTimeout(()=>{
                request.abort();
                console.log("Timeout reached, continuing function. Params:\n", JSON.stringify(event));
                var lambda = new AWS.Lambda();
                lambda.invoke({
                  FunctionName: context.invokedFunctionArn,
                  InvocationType: 'Event',
                  Payload: JSON.stringify(event)
                }).promise().then((data)=>context.done()).catch((err)=>context.fail(err));
              }, context.getRemainingTimeInMillis() - 5000);
              return request.promise().catch((err)=>
                (err.code == 'RequestAbortedError') ?
                  new Promise(()=>context.done()) :
                  Promise.reject(err)
              );
            }
            var ec2 = new AWS.EC2(),
                instanceId = event.ResourceProperties.InstanceId;
            if (event.waiter) {
              wait(event.waiter).then((data)=>success({})).catch((err)=>failed(err));
            } else if (event.RequestType == 'Create' || event.RequestType == 'Update') {
              if (!instanceId) { failed('InstanceID required'); }
              ec2.waitFor('instanceStopped', {InstanceIds: [instanceId]}).promise()
              .then((data)=>
                ec2.createImage({
                  InstanceId: instanceId,
                  Name: event.RequestId
                }).promise()
              ).then((data)=>
                wait({
                  state: 'imageAvailable',
                  params: {ImageIds: [physicalId = data.ImageId]}
                })
              ).then((data)=>success({})).catch((err)=>failed(err));
            } else if (event.RequestType == 'Delete') {
              if (physicalId.indexOf('ami-') !== 0) { return success({});}
              ec2.describeImages({ImageIds: [physicalId]}).promise()
              .then((data)=>
                (data.Images.length == 0) ? success({}) :
                ec2.deregisterImage({ImageId: physicalId}).promise()
              ).then((data)=>
                ec2.describeSnapshots({Filters: [{
                  Name: 'description',
                  Values: ["*" + physicalId + "*"]
                }]}).promise()
              ).then((data)=>
                (data.Snapshots.length === 0) ? success({}) :
                ec2.deleteSnapshot({SnapshotId: data.Snapshots[0].SnapshotId}).promise()
              ).then((data)=>success({})).catch((err)=>failed(err));
            }
          };
      Runtime: nodejs4.3
      Timeout: 300
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal: {Service: [lambda.amazonaws.com]}
          Action: ['sts:AssumeRole']
      Path: /
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      - arn:aws:iam::aws:policy/service-role/AWSLambdaRole
      Policies:
      - PolicyName: EC2Policy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
              - 'ec2:DescribeInstances'
              - 'ec2:DescribeImages'
              - 'ec2:CreateImage'
              - 'ec2:DeregisterImage'
              - 'ec2:DescribeSnapshots'
              - 'ec2:DeleteSnapshot'
              Resource: ['*']
Outputs:
  AMI:
    Value: !Ref AMI

Answer 3

1. No.
2. I suppose Yes. Once the stack you can use the "Update Stack" operation. You need to provide the full JSON template of the initial stack + your changes in that same file (Changed AMI) I would run this in a test environment first (not production), as I'm not really sure what the operation does to the existing instances.

Why not create an AMI initially outside cloudformation and then use that AMI in your final cloudformation template ?

Another option is to write some automation to create two cloudformation stacks and you can delete the first one once the AMI that you've created is finalized.

Answer 4

While @wjdordan's solution is good for simple use cases, updating the User Data will not update the AMI.

(DISCLAIMER: I am the original author) cloudformation-ami aims at allowing you to declare AMIs in CloudFormation that can be reliably created, updated and deleted. Using cloudformation-ami You can declare custom AMIs like this:

MyAMI:
  Type: Custom::AMI
  Properties:
    ServiceToken: !ImportValue AMILambdaFunctionArn
    Image:
      Name: my-image
      Description: some description for the image
    TemplateInstance:
      ImageId: ami-467ca739
      IamInstanceProfile:
        Arn: arn:aws:iam::1234567890:instance-profile/MyProfile-ASDNSDLKJ
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash -x
          yum -y install mysql # provisioning example
          # Signal that the instance is ready
          INSTANCE_ID=`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`
          aws ec2 create-tags --resources $INSTANCE_ID --tags Key=UserDataFinished,Value=true --region ${AWS::Region}
      KeyName: my-key
      InstanceType: t2.nano
      SecurityGroupIds:
      - sg-d7bf78b0
      SubnetId: subnet-ba03aa91
      BlockDeviceMappings:
      - DeviceName: "/dev/xvda"
        Ebs:
          VolumeSize: '10'
          VolumeType: gp2