Ephemeral storage

1. *NOT Persistent!!* Files saved will be gone after reboot of EC2 instance.
2. Physically attachable to EC2 instance, so does behave like a virtual hard disk. Often mounted as /media/ephemeral0.
3. It is free, but only comes with the larger instances, of increasing size.
4. better performance than EBS.
5. Ideal as the root drive of an HPC cluster node, where no storage is needed (AMI is copied to ephemeral disk on boot?).
6. Ephemeral storage is considered instance storage. But to use this as boot device, need to do so before the host is created, by using a device mapping such as /dev/sdc=ephemeral0.

Direct-attached storage

1. Not to be confused with physically-attached storage, which is what EBS is.
2. It is native to a specific EC2 instance. Likely hard drive on the same physical server hosting the EC2 instance. As such, it is not mountable to a different EC2 instance!
3. Not shared, so Potentially/Likely better performance than EBS, and less variance in performance.
4. Offered on beefier EC2 only. Maybe more worthwhile to use than paying for PIOPS EBS.
5. There *is* SPOF in direct-attached storage.
6. Persistent data (should be, double check).

EBS

1. EBS emulates a virtual hard drive, so it is mounted by a specific EC2 instance for use, but it lives independent of any given EC2 instance. ie, it is NOT instance storage, and persist after an EC2 is terminated (deleted).
2. Only mountable in the same availability region. But then it has no replication delay.
3. Has two tiers. Standard IOPS, and Provisioned IOPS (PIOPS). The latter allow extra payment to get dedicated performance. It is not necessary faster, but will be more predictable (less variable, less likely to have bad performance because other instance is sharing the hardware and hitting it hard).
4. gp2 (general purpose ssd storage) = 0.10 per GB-month of provisioned storage - ie $102/TB-month
5. gp3 (general purpose ssd storage) = 0.08 per GB-month of provisioned storage - ie $ 82/TB-month + IOPS cost. gp3 + 9000 IOPS = same perf but cheaper than gp2. so use gp3!

S3

1. Web-centric way to access files. Main use case is programmer coding app to access files/objects using AWS S3 API.
2. It does not emulate a virtual hard drive as EBS does.
3. Files in S3 is accessible from any AWS Region. It also can be replicated for availability and performance.
4. Works on "eventually consistent" model. Has replication delay.

Glacier

1. Glacier is like S3, much slower, and much cheaper. but has an upload/download charge structure.
2. intended for archival. may take hours for files to be fetched from (likely tape) before it is usable.

Cloud "VM"

				aws		gcp		azure
				---------------	---------------	----------------
service name			ec2
def username			ec2-user
billing inc			1 sec
billing min			10 sec ?

Cloud Container

				aws		gcp		azure
				---------------	---------------	----------------
service name			eks
def username			
billing 

AWS Region

us-east-1 = Virginia     ###
us-east-2 = Ohio
us-west-1 = California
us-west-2 = Oregon       #

• AMI-id. ie, TF script that create EC2 instance must use ami-nnnnn from the same region, or get vague "resource unavailable" error. Permission are also not copied when AMI is copied across region
• EC2 key pair. These ssh key to login to machine (needed during instance creation) are listed under EC2, not under user security.
• Security Groups. But TF create them and will track them.
• EC2 instance. the i-nnnn is region specific.

• S3
• IAM user

Using S3 to serve static-content web site

• Create a bucket. eg tin6150.
• Use standard storage, not "infrequenst access storage" or "glacier storage", as acces surchage on the latter are expensive!
• Upload files to the bucket, set upload details, permissions of "Make everything public". This just means the file's properties will say Grantee: Everyone to open/download, but not edit. View Permissions apparently is not needed.
• Set bucket property to enable web hosting. This will generate an website end point based on the region the bucket is, eg: http://tin6150.s3-website-us-west-1.amazonaws.com
• Upload will overwrite old files w/o warning. it will set new permission as per latest upload.
• S3 Pricing details
• Storage cost isn't bad.
• GET request are substantially cheaper than POST and PUT request
• Upload to AWS (xfer IN) is free
• xfer OUT (ie, visitor retrieving files to see the web site) has a per GB data xfer OUT fee. This is in ADDITION to the GET or POST request fee. Like the Hotel California song, you can check out but you can never leave! :-P
• Upload a folder works, no need to pre-create a folder.
• There are options to set a DNS domain to point to the S3 web site, see AWS Route 53 or even a DNS CNAME to the S3 endpoint, eg tiny.cc/TIN.

S3 commands

# S3 buckets are accessible globally, so while hosted in a region, I/O commands can work w/o specifying any region.

aws s3 ls					# list buckets
aws s3 mb s3://sapsg				# make bucket (name has to be GLOBALLY unique)
aws s3 rb ...					# remove bucket 
aws s3 ls sn-s3-bucket-oregon-webhosting	# list content of bucket named "sn-s3-bucket-oregon-webhosting"		## cat@grumpy
aws s3 ls sn-s3-bucket-oregon-webhosting/fig/	# it is more like "ls -ld", add tailing / to see content inside a dir
aws s3 ls s3://sn-s3-bucket-oregon-webhosting	# prefixing bucket name with s3:// is req with older awscli

aws s3 ls s3://sapsg				# t6@g 
aws s3 ls s3://tin6150				# t6@g
aws s3 ls s3://ask-margo			# t6@g
aws s3 ls s3://nibr				# cat@grump

aws s3 sync . s3://tin6150  --acl public-read	# sync is like rsync, skip files already in destination
aws s3 sync . s3://sapsg    --acl public-read	# xfer-in is free, so okay to test upload to s3 like this :)

aws s3 sync conf     s3://nibr/conf   --acl public-read		# conf is name of a dir in this eg
aws s3 sync conf     s3://ask-margo   --acl public-read		# the dirname must be stated in the destination too, or all files in the script/* dir will be placed at one level higher!
aws s3 sync conf/    s3://nibr        --acl public-read		# whether / is added to explicity state src is a dir.  This was likely aws cli 1.0 behavior

aws s3 sync my_data_dir s3://bild-aq-tin6150-data/my_data_dir          # aws cli 2.0 need to specify a destination dir 
aws s3 sync my_data_dir s3://bild-aq-tin6150-data/two-folder/deep-ok/  # and s3 will automagically do the "mkdir -p" for any necessary folder path


aws s3 sync .   s3://tin6150     --acl public-read  --exclude ".git/*"  # exclude .git DB files
aws s3 cp   fig s3://tin6150/    --acl public-read  --recursive         # cp -R  fig folder copied and folder created in dest bucket

# note, it is possible to cp/sync between two s3 buckets

# remove all files in bucket before removing the bucket.  It cannot be renamed nor change region.
 aws s3 rm --recursive s3://old-junk
 aws s3 rb             s3://old-junk

Security Stuff

aws ec2 describe-key-pairs
aws ec2 describe-key-pairs --key-names tin@aws2208blactam.withPass

aws iam list-ssh-public-keys

HPC in EC2

MIT StarCluster

• The AMI is based on Ubuntu 13.04 (as of 2015.12).
• Utilize Open Grid Scheduler (OGS, fork of SGE), Condor workload management.
• Programming environment include, SciPy, NumPy, IPython, CUDA, PyCuda, PyOpenCL, OpenBLAS...
• Provides OpenMPI, Hadoop,
• A cluster-wide NFS mounted FS. (An additional EBS volume need to be defined in the config, mounted by the master node)
• IAM "EC2 Full Access" should be granted to the user that need to create nodes that form the starcluster. ref: http://star.mit.edu/cluster/mlarchives/2112.html

StarCluster setup

pip install starcluster
starcluster help
starcluster --region us-west-2 listpublic 	# list avail AMI
starcluster createkey -o ~/.ssh/mycluster.rsa  mycluseter
	# the public key is not returned by the above command
	# alt, can use -i option to import pre-generated ssh keys.
	# that key has to be imported into AWS IAM key pairs, or else get strange error about key does not exist in region us-east-1

# generate starcluster configfile, 
# edit ~/.starcluster/config with new key
# AWS info, NODE_IMAGE_ID with ami id in the desired region, NODE_INSTANCE_TYPE.

starcluster start -s 2 mycluster	# create and start a new cluster named "mycluster".  config read from ~/.startcluster/config
					# the default config, master node is also an sge exec host
					# they use a single NIC/IP, no distiction b/w private and public network.  
					# EC2 allocate a public IP for each node by default.

starcluster listclusters

starcluster restart    mycluster	# reboot all nodes.
starcluster terminate  mycluster	# terminate AMI, stop paying for it.  #EBS remains?
starcluster stop       mycluster	# only poweroff node, preserving EBS image (/mnt ephemeral storage will still be lost, of course!)
starcluster start -x   mycluster	# restart stopped cluster, all nodes will come back.


starcluster sshmaster  mycluster		# login to master node as root
starcluster sshmaster  mycluster -u sgeadmin	# login to master node as sgeadmin, can issue typical qconf commands from there.

1. Admin mag article
2. AWS Marketplace page

AWS Batch

• AWS Batch is modeled after the HPC batch job running. Perhaps more like HTC than HPC.
  
• No additional cost or special pricing, just need to pay for the EC2 instance needed to run the job.
• Does not need to setup server to run the batch scheduler (so don't need to pay for an extra EC2 server to host such management (?))
• Jobs that it run are docker container job. So, maybe a kubernetes thing...
• Batch has a manager to help bid for spot instances.
• Could scale job wide for fewer hours rather than in-house HPC that is static size and run for days.
• Resources are scaled up automatically to satisfy jobs, and scaled down when runnable jobs decreases. One still have to manage min,desired,max vCPU.
• ECS Agent is used to run containerized jobs.

• AWS Batch FAQ
• AWS Batch Getting Started

Virtualization Tech

Performance

..

Database

GCP - Google Cloud

GCP 101

ssh access

ssh keys - grumpy old sys admin way

ssh keys - google way

(this seems to have been linked to some IAM , which might be pre-requisite)
1. Assign OS Login IAM roles - Project level
2. Assign OS Login IAM roles - Instance level

ecdsa-sha2-nistp256 
AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEbcqIe/lbynFItQAZpmwxr5mpecBeunOziOJgPiN18wEHx6jPZsS6ov+cjhpwiSxV6pMliZrfY3MgewtY5i6U0= 
google-ssh {
"userName":"tin@lbl.gov",
"expireOn":"2023-08-29T21:27:33+0000"
}

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCU6pjFRh+yAJ84AcKfD8xEUEyPzQoQhenPl47x9PL7bTOXK5qSdUyMtWk23T5AI+yFzdquWF8LyamH1qfFn2KKKIOeduSfgV+QmaBdIyjELbZ5SmxuiZ4k5PsXop9MxY65OULaJ1/LWXN4+/WURVvjTM0IrhtZayjpkarlItBvt/RxdBZSitI7STYLJCmCkQwJu5BRJM0fimxvAxFusER9JCwXZeVtiyJxne03+UP0vdy7K09ZQwbcS8nYWVQJuvMypAUsPECguDHgZnHmCbJ3O50Jt8LIjGbJJdId9j4UqOdqIwoofp5Fna1N1b3cbB8WF++w6/5RROO/VfEAngLt google-ssh {"userName":"tin@lbl.gov","expireOn":"2023-08-29T21:27:48+0000"}

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDmvqObMYMqJ0zuzxb60xjtZOObA8nZCrlvx1oR6p6+F tin@wombat_bangwu.clave

sudoers

GCP Getting Started

gcloud compute instances list					# list all instances
gcloud compute instances describe atlas-tin2023  | less		# get metadata of instance

Terraform

terraform init
terraform fmt		# format all .tf files (for indentation), in place, so consider backup before running this.
			# returned filenames are files that it changed.  none if no changes.

terraform validate
terraform plan
terraform apply		# this is DESTRUCTIVE.  re-apply ami-id to the instance, previous data is erased!  even security group seems recreated.
terraform destroy	# will terminate instance, remove any created security group.

# terraform has no way to restart an instance, nor to just stop or start it.
# use the aws cli tool for that

aws ec2 stop-instances --region us-east-2 --instance-ids i-0123456789abcdef

Reference, see also...

• AWS CLI
• AWS concepts
• http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html
• Docker, linux container
• puppet
• cfengine

Useful links

• vantage.sh AWS EC2 pricing and instance comparison
• instance-pricing.com useful, but not as friendly as vantage.sh above.

Random tidbits to be sorted

finding pricing info on ec2

aws pricing get-products --service-code AmazonEC2 --filters "Type=TERM_MATCH,Field=instanceType,Value=c5ad.16xlarge" "Type=TERM_MATCH,Field=location,Value=US East (N. Virginia)" --region us-east-1 | jq -rc '.PriceList[]' | jq -r '[ .product.attributes.servicecode, .product.attributes.location, .product.attributes.instancesku?, .product.attributes.instanceType, .product.attributes.usagetype, .product.attributes.operatingSystem, .product.attributes.memory, .product.attributes.physicalProcessor, .product.attributes.processorArchitecture, .product.attributes.vcpu, .product.attributes.currentGeneration, .terms.OnDemand[].priceDimensions[].unit, .terms.OnDemand[].priceDimensions[].pricePerUnit.USD, .terms.OnDemand[].priceDimensions[].description] ' | less

Copyright info about this work

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike2.5 License. Pocket Sys Admin Survival Guide: for content that I wrote, (CC) some rights reserved. 2005,2012 Tin Ho [ tin6150 (at) gmail.com ]
Some contents are "cached" here for easy reference. Sources include man pages, vendor documents, online references, discussion groups, etc. Copyright of those are obviously those of the vendor and original authors. I am merely caching them here for quick reference and avoid broken URL problems.

Where is PSG hosted these days?