AWS CloudFormation Adventures: Part 2 — S3 Bucket Creation

11 min readJul 14, 2021

After having created our VPC in Part 1, in this article we are going to setup an S3 bucket. I will cover the very basic operations for bucket creation. The intent is to have a bucket that we can use in our next part to set up and configure EC2 instances.

Our objective is to create a simple AWS CloudFormation Stack that we can use to customise the creation of buckets in terms of:

Bucket Name
Bucket Versioning
Bucket Access Control

In order to make the template useful when used in concert with other stacks we also want to generate some output parameters, such as:

Bucket Name
Bucket Domain Name
Bucket Regional Domain Name
Bucket ARN

Let’s dive into it!

Prerequisites

In this series of articles we will also be using the AWS CLI. You can download it here. And finally, you will need an AWS Account that with an API KEY that you can use for the CLI.

Also, you would need to have a general understanding of the AWS CloudFormation and its use from the AWS CLI. You can easily explore the operations available via the CLI by running:

aws cloudformation help

You may want to have a look at the most essential operations required to use CloudFormation in Part 1 of this series.

Amazon S3 Resources in CloudFormation

AWS CloudFormation provides four main resources that can be used to create and configure a bucket:

AWS::S3::AccessPoint: this resource is used to define an access point that can be used from within a VPC to access the bucket without exiting the Amazon network.
AWS::S3::Bucket: this is the main resource for the creation and the configuration of the bucket.
AWS::S3::BucketPolicy: this resource is used to create bucket policies, which can be attached to the bucket to control the access to it.
AWS::S3::StorageLens: this resources is used to create and configure an instance of the Storage Lens service.

In this article we will only focus on the use of the AWS::S3::Bucket resource as it is sufficient for what we need to do. This resource comes with a considerable number of properties that allow managing the following aspects of a bucket:

replication (see: ReplicationConfiguration)
access control (see: AccessControl)
encryption (see: BucketEncryption)
lifecycle (see: LifecycleConfiguration)
inventory (see: InventoryConfiguration)
versioning (see: VersioningConfiguration)
CORS setup (see: CorsConfiguration)
notifications (see: NotificationConfiguration)
metrics (see: MetricsConfiguration)
logging (see: LoggingConfiguration)
object locking (see: ObjectLockEnabled and ObjectLockConfiguration)
website setup (see: WebsiteConfiguration)
intelligent tiering (see: IntelligentTieringConfiguration)
transfer acceleration settings (see: AccelerateConfiguration)
bucket analytics (see AnalyticsConfiguration).

To accomplish what we need to do we will be only be using AccessControl and VersioningConfiguration. The rest of the parameters can be omitted as most of them are optional or have defaults.

Step 1 — Creating a Simple Bucket

The snippet below provides a simple CloudFormation template that can be used to create a bucket.

AWSTemplateFormatVersion: 2010-09-09Description: >
  This templates demonstrates the creation of a simple S3. 
  bucket. It allows for parameterising the bucket name and
  a few other attributes.Parameters:
  BucketName: 
    Type: String
    Description: The name of the bucket.Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref BucketName
      Tags:
      - Key: Name
        Value: !Join ['', [!Ref "AWS::StackName", "-Bucket"]]

The creation of the bucket is very simple. All that we need to do is to specify the name of the bucket and the rest is taken care of by the default settings. The name is specified via an input parameter that will allow us to reuse the template for the creation of multiple buckets. The listing includes also an additional tag that is added to the bucket to record it as part of the current stack. This is not needed, but it simplifies cataloguing AWS resources.

We can now test our template by running the command:

> aws cloudformation create-stack \
    --stack-name <stack-name> \
    --template-body file://<s3-template-file-path> \
    --parameters ParameterKey=BucketName,ParameterValue=<name-of-the-bucket>

The placeholders <stack-name>, <s3-template-file-path> and <name-of-the-bucket> refer to the name of the stack, the path to the file where the YAML template is saved and the desired name of the bucket.

If the operation completes successfully CloudFormation will output the ARN of the associated stack that can be looked up either via the APIs or via the CloudFormation console, as shown in the figure below.

Figure 1 — Successful Creation of the S3 Bucket.

NOTE: This operation creates a Bucket with storage class S3 Standard. I have not found in the CloudFormation documentation any parameter that enables creating the buckets with a different storage class, but through lifecycle rules it is possible to define transitions to other storage classes (see: LifecycleConfiguration).

Step 2 —Configuring Simple Access Control

Now that our bucket has been created successfully, we want to modify our template to manage the access control. By default the bucket is created as private but there isn’t any explicit permission that blocks public access as shown in Figure 2.

Figure 2 — S3 Bucket Default Permissions

We want to provide a minimum degree of customisation via the template and allow the user to control the access by setting the parameter AccessControl. This parameter can have the following values:

Private
PublicRead
PublicReadWrite
AuthenticatedRead
LogDeliveryWrite
BucketOwnerRead
BucketOwnerFullControl
AwsExecRead

These are the enumeration values of the corresponding Canned ACLs. These are a set of pre-configured permissions that allow for quickly setup the most common access control configurations. The default value if omitted is Private.

In order to enable access control settings in our template we will define a parameter that declares the allowed value and map the parameter value to the corresponding AccessControl property in the bucket resource.

AWSTemplateFormatVersion: 2010-09-09Description: >
  This templates demonstrates the creation of a simple S3. 
  bucket. It allows for parameterising the bucket name and
  a few other attributes.Parameters:
  BucketName: 
    Type: String
    Description: The name of the bucket.
  BucketAccessControl:
    Type: String
    Default: Private
    AllowedValues:
    - Private
    - PublicRead
    - PublicReadWrite
    - AuthenticatedRead
    - LogDeliveryWrite
    - BucketOwnerRead
    - BucketOwnerFullControl
    - AwsExecRead
    Description: Access control configuration for the bucket (default is Private).Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref BucketName
      AccessControl: !Ref BucketAccessControl      
      Tags:
      - Key: Name
        Value: !Join ['', [!Ref "AWS::StackName", "-Bucket"]]

In creating the parameter we want to make sure that the default behaviour is the same as the default behaviour of the AWS::S3::Bucket resource when the parameter is omitted. Therefore we set the default value to Private.

We can now update the stack with the following command:

> aws cloudformation deploy \
      --stack-name <stack-name> \
      --template-file <s3-template-file-path> \
      --parameter-overrides BucketName=<bucket-name> BucketAccessControl=PublicRead

The command will change the access permissions of the bucket from Private (the original default setting) to PublicRead, thus allowing public read access. This setting might be useful if we store a static website in the bucket. Once the stack has been updated, we can see the changes to the access control list associated to the bucket in “Access control list (ACL)” section of the bucket details page, which should look like as shown in Figure 3.

Figure 3 — Update Bucket Access Control List.

NOTE: it is important to notice the difference between the parameter specification of the create-stack and the deploy commands. The former uses the —-parameters flag and requires the parameters to be specified in the form: ParameterKey=<Param-Name>,ParameterValue=<Param-Value> while the latter uses the --parameter-overrides flag and requires the parameters to be specified in the form: <Param-Name>=<Param-Value> . Moreover, the deploy command will use the existing values of the parameters if no default values are specified, while the create-stack command will fail if parameters that have no defaults are omitted. This is because the deploy command operates on an existing stack which already has values for each parameters when used for updates.

Step 3 — Adding Versioning Details

The final step required for customising the bucket creation is configuring the versioning. This parameter controls the behaviour that S3 has in relation to updates and deletions of objects in bucket. By default the versioning is not enabled and this means that objects in the bucket will only be retained with their latest changes. When versioning is enabled every object update will create a new version of the object and all copies will be retained with default access to the latest version. Object deletions will add a marker that signals the object as deleted. Once enabled versioning cannot be removed, but can only be suspended. This process entails retaining the history of changes originated since versioning was active, and making direct updates to the latest version of each of the objects in the bucket.

If we move to the “Properties” tab of the bucket the versioning is currently disabled and we should see the setup shown in Figure 4.

Figure 4 — Bucket Versioning (Disabled).

In order to control the versioning of the bucket from the CloudFormation template we will introduce the VersioningConfiguration block, which only contains one property: Status. This can either be Suspended or Enabled, with the former being the default value. We will then modify our template to include an additional parameter allowing the specification of the status of the versioning as shown in the following listing.

AWSTemplateFormatVersion: 2010-09-09Description: >
  This templates demonstrates the creation of a simple S3. 
  bucket. It allows for parameterising the bucket name and
  a few other attributes.Parameters:
  BucketName: 
    Type: String
    Description: The name of the bucket.
  BucketAccessControl:
    Type: String
    Default: Private
    AllowedValues:
    - Private
    - PublicRead
    - PublicReadWrite
    - AuthenticatedRead
    - LogDeliveryWrite
    - BucketOwnerRead
    - BucketOwnerFullControl
    - AwsExecRead
    Description: Access control configuration for the bucket (default is Private).
  BucketVersioning:
    Type: String
    Default: Suspended
    AllowedValues:
    - Suspended
    - Enabled
    Description: Versioning configuration for the bucket.Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref BucketName
      AccessControl: !Ref BucketAccessControl 
      VersioningConfiguration:
        Status: !Ref BucketVersioning    
      Tags:
      - Key: Name
        Value: !Join ['', [!Ref "AWS::StackName", "-Bucket"]]

As we did for the AccessControl property, we make sure to set the default value for the parameter to Suspended to ensure a consistent behaviour of the template with the default behaviour of the AWS::S3::Bucket resource.

We can now deploy the changes and enable versioning by issuing the following command:

> aws cloudformation deploy \
      --stack-name <stack-name> \
      --template-file <s3-template-file-path> \
      --parameter-overrides BucketVersioning=Enabled

As discussed previously, we do not need to specify all the parameter values because we already have an existing stack and therefore we only specify what we want to change. If the operation is successful, we should be able to see the updated versioning settings in the details of the Bucket on the AWS console as shown in Figure 5.

Step 4 — Exposing Bucket Parameters

The final step to completed our bucket template is to externalise some useful parameters that the AWS::S3::Bucket resource exposes via Output Parameters.

What we want to expose are the following attributes of the resource:

Bucket name (resolved as Fn::GetAtt Bucket.BucketName )
Bucket domain name (resolved as Fn::GetAtt Bucket.DomainName )
Bucket regional domain name (resolved as GetAtt Bucket.RegionalDomainName )
Bucket ARN (resolved as Fn::GetAtt Bucket.Arn )

In addition, it is also possible to expose the Website URL (i.e. and the Dual-stack domain name (i.e. IPv6 domain name of the bucket). The following listing demonstrates how to externalise these attributes for the newly created bucket.

AWSTemplateFormatVersion: 2010-09-09Description: >
  This templates demonstrates the creation of a simple S3. 
  bucket. It allows for parameterising the bucket name and
  a few other attributes.Parameters:
  BucketName: 
    Type: String
    Description: The name of the bucket.
  BucketAccessControl:
    Type: String
    Default: Private
    AllowedValues:
    - Private
    - PublicRead
    - PublicReadWrite
    - AuthenticatedRead
    - LogDeliveryWrite
    - BucketOwnerRead
    - BucketOwnerFullControl
    - AwsExecRead
    Description: Access control configuration for the bucket (default is Private).
  BucketVersioning:
    Type: String
    Default: Suspended
    AllowedValues:
    - Suspended
    - Enabled
    Description: Versioning configuration for the bucket.Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref BucketName
      AccessControl: !Ref BucketAccessControl 
      VersioningConfiguration:
        Status: !Ref BucketVersioning    
      Tags:
      - Key: Name
        Value: !Join ['', [!Ref "AWS::StackName", "-Bucket"]]Outputs:
  BucketName:
    Value: !Ref Bucket
    Description: The name of the bucket.
  BucketDomainName:
    Value: !GetAtt Bucket.DomainName
    Description: The DNS domain name of the bucket.
  BucketRegionalDomainName:
    Value: !GetAtt Bucket.RegionalDomainName
    Description: The domain name including the AWS Region where the bucket is located.
  BucketArn: 
    Value: !GetAtt Bucket.Arn
    Description: The Amazon Resource Name (ARN) of the bucket.

We can now update the stack by using the deploy command. This time we don’t need to specify any parameter because we are ok with the previous settings, we only want to have additional output parameters.

> aws cloudformation deploy 
      --stack-name <stack-name> \
      --template-file <s3-template-file-path>

If the operation is successful, we can check the externalised output parameters in the AWS CloudFormation Console as shown in Figure 6.

An Important Consideration About State

CloudFormation provides the ability to create, update, and delete stacks, but sometimes the effects of these operations may not be as expected. Surprises may occur when the resources that are created have an associated state. This is the case of S3 buckets, which are create to contain objects. These are not part of the entities that CloudFormation manipulates and tracks and therefore disconnected from the template management activity.

This is particularly, important to when we make updates that require the replacement of the S3 resource created, such as changing the name of a bucket or deleting by deleting the stack.

For instance if we try to delete the stack from the AWS CloudFormation console the operation will fail as shown in Figure 7. The reason of the failure is because the bucket was not empty and this prevented the deletion of the Bucket resource.

Figure 7 — CloudFormation Stack Delete Failed.

This operation will cause the stack to be in the DELETE_FAILED status thus preventing any other operation. To resolve this condition, we need to log in into the AWS CloudFormation Console, select the stack and click “Delete” again. This time the user interface will prompt the user with all the dependent resources that failed deletion and ask the user whether he/she wants to retain such resources. By retaining these resources, these will no longer be managed through the stack that can now be deleted.

The same will happen if we try to change the bucket name. This is an operation that does require the deletion of the bucket and its recreation under a different name. Because the bucket is not empty the operation will fail again.

> aws cloudformation deploy 
      --stack-name <stack-name> \
      --parameter-overrides BucketName=<new-name>

We set in this case <new-name> to a name different from the previous name and the result can be seen in the AWS CloudFormation Console as shown in Figure 8.

Figure 8 — CloudFormation Stack Update Failed.

If you need the ability to update and delete S3 buckets that are not empty you can use the CDK, which provide advanced capabilities in relation to this matter.

Conclusions

In this article we have demonstrated how to quickly create an S3 bucket with CloudFormation and configure with a minimum set of parameters that allow managing the access control through Canned ACLs and versioning. The configuration capabilities exposed by the AWS::S3::Bucket resource are quite extensive and in this article we have simple shown a basic use case. We have also discussed the implication of state in relation to updates and deletions of S3 resources via CloudFormation.

The entire CloudFormation template creating the S3 bucket can be found here in Github.

In the next article we will put the S3 bucket to use with the creation of templates for the configuration of EC2 instances. Stay tuned!