Setting it all up and how to avoid some stupid pitfalls..
I created a rails app to server users with the ability for them to upload images and files.
To improve the performance of the site, I serve all app assets (JavaScript, Images, CSS files, etc.) from a CloudFront distribution. I then decided to also add a CloudFront in front of the bucket I use for user uploaded files.
Using carrier wave gem is quite straightforward and even adding the CloudFront CDN was quite easy. Just in case you have no experience, here are the basics:
1. add the carrierwave and carrierwave-aws gems - this is much better than using fog which bloats your app with multiple unneeded gems. It also supports more of the AWS API.
2. add a field to your relevant table to store the uploaded image or asset, like:
add_column :users, :avatar, :string
3. In your model class, mount an uploader for this field:
mount_uploader :avatar, AvatarUploader
4. Implement an uploader class (AvatarUploader in this exmple):
class AvatarUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
storage :aws
# Override the directory where uploaded files will be stored.
# This is a sensible default for uploaders that are meant to be mounted:
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
# Create different versions of your uploaded files:
# version :thumb do
# process :resize_to_fit => [50, 50]
# end
# Add a white list of extensions which are allowed to be uploaded.
# For images you might use something like this:
def extension_white_list
%w(jpg jpeg gif png)
end
def fix_exif_rotation #this is my attempted solution
manipulate! do |img|
img.tap(&:auto_orient)
end
end
process :fix_exif_rotation
process resize_to_limit: [200, 200]
end
In the above code, I am including MiniMagick to manipulate the uploaded image (resize and fix orientation - the orientation was to fix issue with landscape oriented images uploaded from iPhone).
If you want to use MiniMagick, you will have to add the mini_magick gem.
5. You will then need to configure carrierwave with AWS credentials, bucket name and region like that:
CarrierWave.configure do |config|
config.storage = :aws
config.aws_bucket = ENV['S3_BUCKET'] || 'default-bucket'
config.aws_acl = 'public-read'
config.aws_attributes = {
expires: 1.week.from_now.httpdate,
cache_control: 'max-age=604800'
}
config.aws_credentials = {
access_key_id: ENV['S3_ACCESS_KEY'] || 'your-key',
secret_access_key: ENV['S3_SECRET_KEY'] || 'your-secret',
region: 'eu-west-1' # Required
}
end
BTW, you could use local file storage for development by changing both carrier_wave initializer to use storage of :file as well as your uploader class.
6. To serve app assets from AWS CloudFront (for much improved performance and less load on your poor web server...), you can add an assets CDN host in your config/environments/production.rb. That's what I have:
if ENV['APP_CDN_HOST'].present?
config.action_controller.asset_host = ENV['APP_CDN_HOST']
end
7. To server those app assets from CloudFront, you will need to create a distribution pointing to your domain as origin. To do that, go to your AWS console, select the CloudFront service, and click "Create Distribution" at the top. In the following screen, click "Get Started" under the "Web" section ("RTMP" is used for streaming media). In the Origin Domain Name field, type in your domain (where your application is served from). If you want to support both http and https request you should pick "Match Viewer".
Once created, set the APP_CDN_HOST to point to the CloudFront distribution URL.
8. To place CloudFront in front of your user uploaded files, you need to add the following to your carrier_wave configuration file:
config.asset_host = ENV['CDN_HOST'] || 'http://.cloudfront.net'
9. Now all is smooth and performant. There is one small issue (which may or may not be relevant to you): anyone with a link to a user uploaded file, can share that link and so others can access the files. If you care about privacy and security of those files, you may want to protect them with a CloudFront signed URLs. Signed URLs are signed with a trusted private key, and can encode an expiration time for the link. This means that anyone trying to access a cloud front signed URL after the expiration time, will get an access denied result. To add this extra security step, you should follow the following steps...
10. Create an IAM user identity that will be used as a trusted signer. This can be the user creating the bucket, or another user. I recommend NOT using your root identity for either of those operations. What I did, is I created a user for both creating the bucket, as well as the cloud front distribution and gave that user the trusted signer role. I automated all those actions in a ruby script (assuming the aws_boto user is an admin user I have with the given credentials):
require 'aws-sdk'
# get admin credentials from ENV variables:
abk = `echo $aws_boto_key`.strip
abs = `echo $aws_boto_secret`.strip
Aws.config.update(
{
region: 'eu-west-1',
credentials: Aws::Credentials.new(abk, abs),
}
)
# create bucket if does not exist
s3 = Aws::S3::Resource.new
sd = `echo $subdomain`.strip
bucket_name = "#{sd}-mydomain"
bucket = s3.bucket(bucket_name)
if bucket.exists?
puts "bucket already exists, no need to worry..."
else
puts "no such bucket, create it now"
bucket.create
# create IAM user and get the access key and secret
iam = Aws::IAM::Resource.new
user_name = "#{sd}-mydomain"
user = iam.user(user_name)
if user.exists?
puts "user already exists, no need to worry..."
else
puts "no such user, create it now"
user.create
accesskeypair = user.create_access_key_pair
File.open("userkey.cfg", 'w') { |file| file.write("#{accesskeypair.access_key_id}") }
File.open("usersecret.cfg", 'w') { |file| file.write("#{accesskeypair.secret}") }
# build a custom policy
policy_doc = '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1444035127000",
"Effect": "Allow",
"Action": [
"s3:Delete*",
"s3:Get*",
"s3:List*",
"s3:Put*"
],
"Resource": [
"arn:aws:s3:::'+"#{bucket_name}"+'/*"
]
}
]
}'
user.create_policy({
policy_name: "S3_MyDomain_Access", # required
policy_document: policy_doc, # required
})
end
# create cloud front distribution for app assets (origin = app root directory)
cloudfront = Aws::CloudFront::Client.new()
app_dist = cloudfront.create_distribution(
{
distribution_config: {
# required
caller_reference: "app-dist-caller-#{Time.now.to_i}", # required - unique string for the request
aliases: {
quantity: 0
},
default_root_object: "",
origins: {
# required
quantity: 1, # required
items: [
{
id: "1", # required - unique within distribution
domain_name: "#{sd}.mydomain.com", # required
origin_path: "",
custom_origin_config: {# use only for custom origin, not for bucket
http_port: 80, # required
https_port: 443, # required
origin_protocol_policy: "match-viewer", # required, accepts http-only, match-viewer
},
},
],
},
default_cache_behavior: {
# required
target_origin_id: "1", # required
forwarded_values: {# required
query_string: true, # required
cookies: {# required
forward: "none", # required, accepts none, whitelist, all
whitelisted_names: {
quantity: 0
},
},
headers: {
quantity: 0
},
},
trusted_signers: {# required
enabled: false, # required
quantity: 0
},
viewer_protocol_policy: "allow-all", # required, accepts allow-all, https-only, redirect-to-https
min_ttl: 600, # required
allowed_methods: {
quantity: 2, # required
items: ["GET", "HEAD"], # required, accepts GET, HEAD, POST, PUT, PATCH, OPTIONS, DELETE
cached_methods: {
quantity: 2, # required
items: ["GET", "HEAD"], # required, accepts GET, HEAD, POST, PUT, PATCH, OPTIONS, DELETE
},
},
smooth_streaming: false,
default_ttl: 86400,
max_ttl: 2592000,
},
cache_behaviors: {
quantity: 0
},
custom_error_responses: {
quantity: 0
},
comment: "created automatically by SroolTheKnife", # required
logging: {
enabled: false, # required
include_cookies: false, # required
bucket: "", # required
prefix: "", # required
},
price_class: "PriceClass_100", # accepts PriceClass_100 (US and Europe), PriceClass_200, PriceClass_All
enabled: true, # required
viewer_certificate: {
cloud_front_default_certificate: true,
minimum_protocol_version: "TLSv1", # accepts SSLv3, TLSv1
},
restrictions: {
geo_restriction: {# required
restriction_type: "none", # required, accepts blacklist, whitelist, none
quantity: 0
},
},
},
})
File.open("appdist.cfg", 'w') { |file| file.write("#{app_dist.distribution.domain_name}") }
# create cloud front distribution for uploaded assets (origin = above bucket)
cloudfront = Aws::CloudFront::Client.new()
bucket_dist = cloudfront.create_distribution(
{
distribution_config: {
# required
caller_reference: "bucket-dist-caller-#{Time.now.to_i}", # required - unique string for the request
aliases: {
quantity: 0
},
default_root_object: "",
origins: {
# required
quantity: 1, # required
items: [
{
id: "1", # required - unique within distribution
domain_name: "#{bucket_name}.s3.amazonaws.com", # required
origin_path: "",
s3_origin_config: {# use only for bucket
origin_access_identity: "", # required
},
},
],
},
default_cache_behavior: {
# required
target_origin_id: "1", # required
forwarded_values: {# required
query_string: true, # required
cookies: {# required
forward: "none", # required, accepts none, whitelist, all
whitelisted_names: {
quantity: 0
},
},
headers: {
quantity: 0
},
},
trusted_signers: {
# required
enabled: true, # required
quantity: 1,
items: ['self'] # same user creating the bucket is the trusted signer
},
viewer_protocol_policy: "allow-all", # required, accepts allow-all, https-only, redirect-to-https
min_ttl: 600, # required
allowed_methods: {
quantity: 2, # required
items: ["GET", "HEAD"], # required, accepts GET, HEAD, POST, PUT, PATCH, OPTIONS, DELETE
cached_methods: {
quantity: 2, # required
items: ["GET", "HEAD"], # required, accepts GET, HEAD, POST, PUT, PATCH, OPTIONS, DELETE
},
},
smooth_streaming: false,
default_ttl: 86400,
max_ttl: 2592000,
},
cache_behaviors: {
quantity: 0
},
custom_error_responses: {
quantity: 0
},
comment: "created automatically by SroolTheKnife", # required
logging: {
enabled: false, # required
include_cookies: false, # required
bucket: "", # required
prefix: "", # required
},
price_class: "PriceClass_100", # accepts PriceClass_100 (US and Europe), PriceClass_200, PriceClass_All
enabled: true, # required
viewer_certificate: {
cloud_front_default_certificate: true,
minimum_protocol_version: "TLSv1", # accepts SSLv3, TLSv1
},
restrictions: {
geo_restriction: {# required
restriction_type: "none", # required, accepts blacklist, whitelist, none
quantity: 0
},
},
},
})
File.open("bucketdist.cfg", 'w') { |file| file.write("#{bucket_dist.distribution.domain_name}") }
end
11. Once you have that, you can add the following to carrier_wave initializer, to add code signing:
config.aws_signer = -> (unsigned_url, options) { Aws::CF::Signer.sign_url unsigned_url, options }
12. You should add the cloudfront-signer gem, and initializers/cloudfront-signer.rb with the following code:
Aws::CF::Signer.configure do |config|
config.key = ENV["CLOUDFRONT_KEY"] || '-----BEGIN RSA PRIVATE KEY-----
paste your private key here if you like...
-----END RSA PRIVATE KEY-----'
config.key_pair_id = ENV["CLOUDFRONT_KEY_PAIR_ID"] || 'paste key pair'
config.default_expires = 600
end
That key and key pair ID are required to be set in the AWS console as explained here: Specifying Trusted Signers.
One last issue that caused me a lot of confusion and few hours of wasted time is that if you find that some of the uploaded assets "disappear" - for example, an uploaded image shows as a broken image. The reason might be (and was on my end) the rails cache kicking in, and serving the fragment (including the image URL) beyond the signed URL expiration time. You better not cache such fragments or else they will be broken once the signed URL expiration time arrives.
And a message from our advertisers:
Toptal provides remote engineers and designers of high quality. I recommend them. Follow this link (full disclosure: this is my affiliate link):
https://www.toptal.com/#engage-honest-computer-engineers-today
Perfect guide, pleasure to read!
ReplyDelete