Notes on Using Amazon AWS PowerShell Cmdlets (or CLI Commands) to Maintain a Static S3 Website
Table of Contents
- 1. Overview
- 2. Simplest possible basics
- 3. DONE Figure out how to exclude
.gitsubdirectory - 4. DONE Figure out how to only upload what's changed
(HTML generated from org-mode text file.)
1 Overview
Some dumb notes on maintaining a static website as an S3 bucket. To be refined as I figure out more than just the absolute basics.
Note that "CLI" here means the AWS CLI, which is implemented as a set of Python scripts.
1.1 Documentation
https://docs.aws.amazon.com/cli/index.html (includes both Python and PowerShell CLIs).
1.2 Installation
Use pip3 for Python 3 (as opposed to pip, which might just use Python 2).
Note on Linux Mint 19.1, there were a couple of problems with PyYAML (if I recall correctly).
One package complained that setuptools was not available. I solved this with (a) a few reboots
and (b) an explicit call to install setuptools:
pip3 install setuptools
2 Simplest possible basics
2.1 Set up credentials
2.1.1 Download access keys from AWS (if you haven't already)
You only get one chance to do this, but you can always create new keys if you need to.
In the AWS (web) console:
Go to IAM | Users | specific user | Security credentials | Create access key (button)
After you've done this, you'll have the opportunity to download the keys you just created.
2.1.2 Store in AWS credential store
Set-AWSCredential -AccessKey AKIAIOSFODNN7EXAMPLE -SecretKey wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -StoreAs MyProfileName
2.1.2.1 How to check if you've already done this
2.1.2.1.1 TODO PowerShell
2.1.2.1.2 CLI (Python)
The file ~/.aws/credentials is a plain-text file that has your credentials (hopefully no one else can read it).
2.1.2.2 CLI
aws configure --profile JohnS3
You will be prompted for access key and secret access key. The other parameters can be defaulted.
(Note: on Linux, credentials are in text file: ~/.aws/credentials.)
2.1.3 Choose a credential for the current session (will expire, so you'll have to do it again the next day)
Set-AWSCredential -ProfileName JohnS3
2.1.3.1 CLI
For the CLI, set an environment variable:
export AWS_PROFILE=JohnS3
(On Linux, cat ~/.aws/credentials will tell you what credentials are available on your system,
in case you forgot (like I do, all the time).)
(On Windows (PowerShell), try cat ~\.aws\credentials)
2.1.3.1.1 DONE Test
- CLOSING NOTE
aws s3 ls s3://tarheel-nc
(Should result in a directory listing, obviously.)
2.2 Download a bucket
cd into the local directory you want to work from.
Read-S3Object tarheel-nc -KeyPrefix / -Folder $(pwd) # 'tarheel-nc' being the bucket name.
THIS WILL PROBABLY OVERWRITE YOUR CURRENT DIRECTORY CONTENTS so be sure you know what you're doing.
2.2.1 CLI
aws cp s3://tarheel-nc . --recursive
(UNTESTED) – This is probably how it works. By this point, I'd put the source files into Github and found a scheme to maintain the generated files from the sources in git (org-mode publish, see org-mode "publish" operation).
2.3 Upload a bucket
(Assuming you set up with the download above.)
Write-S3Object tarheel-nc -keyp / $(pwd) -rec -PublicReadOnly
WILL PROBABLY OVERWRITE THE DESTINATION BUCKET
2.3.1 CLI (will not overwrite destination)
(Assumes you've already set your profile via environment variable export, as above, but, if not,
add the option --profile JohnS3.)
This only copies new and updated files.
ls -rec *~ | rm # Cleanup, because "publish" generates these aws s3 sync . s3://tarheel-nc --acl public-read
Or
find . -iname '*~' | xargs rm -v aws s3 sync . s3://tarheel-nc --acl public-read
2.3.1.1 CLI to totally re-upload a bucket
If you bollix up the timestamps and want to just start from scratch, do this in the root of the directory you want to publish:
ls -rec *~ | rm # Cleanup, because "publish" generates these aws s3 rm s3://tarheel-nc --recursive # Blows away the CONTENTS of the bucket aws s3 cp . s3://tarheel-nc --recursive --acl public-read # Don't forget the ACL!
3 DONE Figure out how to exclude .git subdirectory
- CLOSING NOTE
Don't need this if publishing from a source directory to a local destination, and then syncing
from that directory to S3, but you can probably do it with some variation of an --exclude option
to a command.
4 DONE Figure out how to only upload what's changed
- CLOSING NOTE
Based on date?
Based on what git says is uncommitted? (Means upload before committing.)
Answer: sync CLI action (CLI (will not overwrite destination)).