S3 Bucket Policy to Restrict Access by Referrer, Yet Allow Direct Access to File(s)

Recently Amazon rolled out S3 Bucket Policies (see Access Policy Language) to more finely control access to S3 buckets or resources in buckets, than with just ACL's alone.  This was very timely as I had a need arise to use a bucket policy just after it came out.  Basically I needed to block access of a single file, let's call it xyz.htm, from certain referrers, yet allow all others.  After a little research and some trial-and-error I was able to define a policy which did just this:

{
"Version":"2008-10-17",
"Id":"mydomain-widgettest",
"Statement":[{
"Sid":"1",
"Effect":"Deny",
"Principal":{
"AWS":"*"
},
"Action":"s3:GetObject",
"Resource":"arn:aws:s3:::widgettest.mydomain.com/xyz.htm",
"Condition":{
"StringLike":{
"aws:Referer":[
" http://blockedreferer1.com/*",
" http://blockedreferer2.net/*",
]}}},
{
"Sid":"2",
"Effect":"Allow",
"Principal":{
"AWS":"*"
},
"Action":"s3:GetObject",
"Resource":"arn:aws:s3:::widgettest.mydomain.com/xyz.htm",
"Condition":{
"StringLike":{
"aws:Referer":[
"*",
" http://widgettest.mydomain.com/*"
]}}}]}

However, this had the undesired effect of blocking direct access to the file, i.e. http://widgettest.mydomain.com/xyz.htm, where there is no referrer, or the referrer is null.  This one took me a little longer to figure out, and a key piece of it was found in the Amazon developer forums.  I was then able to write a bucket policy which behaves as desired:

{
"Version":"2008-10-17",
"Id":"mydomain-widgettest",
"Statement":[{
"Sid":"1- Allow direct access to xyz.htm - i.e. no referrer.",
"Effect":"Allow",
"Principal":{
"AWS":"*"
},
"Action":"s3:GetObject",
"Resource":"arn:aws:s3:::widgettest.mydomain.com/xyz.htm",
"Condition":{
"Null":{
"aws:Referer":true
}}},
{
"Sid":"2- Allow all referrers to xyz.htm except those listed.",
"Effect":"Allow",
"Principal":{
"AWS":"*"
},
"Action":"s3:GetObject",
"Resource":"arn:aws:s3:::widgettest.mydomain.com/xyz.htm",
"Condition":{
"StringNotLike":{
"aws:Referer":[
" http://blockedreferer1.com/*",
" http://blockedreferer2.net/*"
]}}}]}

This policy effectively allows direct access to xyz.htm (null or "no" referrer), and allows access to all referrers except those explicitly listed in the Sid:2 section.  One important note is that "public" read access must not be set in the ACL for this file as it will allow anyone access, effectively bypassing this policy.

NOTES:

• Amazon S3 bucket policies use JSON.  If you aren't familiar with JSON as I wasn't you can read more here.
• I found a handy JSON Formatter and Validator - to do just that. . .
• Since Amazon doesn't provide an easy method for us non-programmers to apply bucket policies I found CloudBerry S3 Bucket Explorer Pro essential and simple to use to apply bucket policies.
• Sometimes as I applied a policy to test I would receive the message "invalid aspen elements," which basically mean something is wrong, usually one of the required elements was either missing or incorrect, and, interestingly no results were found using Google.

See Also:

• S3 Bucket Policy to Restrict Access by Referrer, Yet Allow Direct Access to File(s) on Experts Exchange.
  

Get Yesterday's date in MS DOS Batch file

A while back while I was trying to figure out the best way to gather some log files from Amazon S3 buckets and some web servers I run.  These resources are currently generating around 10-15GB of uncompressed log files daily.  Besides being fairly large in size the S3 (and CloudFront) log files are numerous.  Any given bucket can easily generate 1,000 or more log files per day - that's a whole other story. . .

Anyway, I wanted to be able to run a process sometime after midnight that would gather and zip the previous day's files and stash the zipped files in another location for archival.  It's pretty easy to calculate the previous day's date if it's in the middle of the month, but what if it's the first of the month, first of the year, and what about leap year, etc., etc. . . ?  So I searched around the web a bit and came across a great solution to this issue on Experts Exchange (Get Yesterday date in MS DOS Batch file).  Thanks to SteveGTR for this one.

I have modified the original script a bit to suite my needs.  Most notably at the end of the script I create two variables, IISDT and AWSDT, to match IIS and Amazon Web Services (S3 and CloudFront) log formats, respectively.  I use this in a simple batch file which is executed like, "gather_log_files.bat 1."  The number "1" is passed into the script which calculates the date of "1" day before the current date.  Of course you could pass any number in there to generate a date x days in the past.  It's very slick. NOTE: If you don't specify a number after the batch file "1" is assumed.

So, without further ado, here's the script.

@echo off

set yyyy=

set $tok=1-3
for /f "tokens=1 delims=.:/-, " %%u in ('date /t') do set $d1=%%u
if "%$d1:~0,1%" GTR "9" set $tok=2-4
for /f "tokens=%$tok% delims=.:/-, " %%u in ('date /t') do (
for /f "skip=1 tokens=2-4 delims=/-,()." %%x in ('echo.^|date') do (
set %%x=%%u
set %%y=%%v
set %%z=%%w
set $d1=
set $tok=))

if "%yyyy%"=="" set yyyy=%yy%
if /I %yyyy% LSS 100 set /A yyyy=2000 + 1%yyyy% - 100

set CurDate=%mm%/%dd%/%yyyy%
set dayCnt=%1

if "%dayCnt%"=="" set dayCnt=1

REM Substract your days here
set /A dd=1%dd% - 100 - %dayCnt%
set /A mm=1%mm% - 100

:CHKDAY
if /I %dd% GTR 0 goto DONE
set /A mm=%mm% - 1
if /I %mm% GTR 0 goto ADJUSTDAY
set /A mm=12
set /A yyyy=%yyyy% - 1

:ADJUSTDAY
if %mm%==1 goto SET31
if %mm%==2 goto LEAPCHK
if %mm%==3 goto SET31
if %mm%==4 goto SET30
if %mm%==5 goto SET31
if %mm%==6 goto SET30
if %mm%==7 goto SET31
if %mm%==8 goto SET31
if %mm%==9 goto SET30
if %mm%==10 goto SET31
if %mm%==11 goto SET30
REM ** Month 12 falls through

:SET31
set /A dd=31 + %dd%
goto CHKDAY

:SET30
set /A dd=30 + %dd%
goto CHKDAY

:LEAPCHK
set /A tt=%yyyy% %% 4
if not %tt%==0 goto SET28
set /A tt=%yyyy% %% 100
if not %tt%==0 goto SET29
set /A tt=%yyyy% %% 400
if %tt%==0 goto SET29

:SET28
set /A dd=28 + %dd%
goto CHKDAY

:SET29
set /A dd=29 + %dd%
goto CHKDAY

:DONE
if /I %mm% LSS 10 set mm=0%mm%
if /I %dd% LSS 10 set dd=0%dd%

REM Set IIS and AWS date variables
set IISDT=%yyyy:~2,2%%mm%%dd%
set AWSDT=%yyyy%-%mm%-%dd%

The results would look like:

IIS Date: 20100727

AWS Date: 2010-07-27

Legal Guide for Bloggers

Whether you're a newly minted blogger or a relative old-timer, you've been seeing more and more stories pop up every day about bloggers getting in trouble for what they post.

Like all journalists and publishers, bloggers sometimes publish information that other people don't want published. You might, for example, publish something that someone considers defamatory, republish an AP news story that's under copyright, or write a lengthy piece detailing the alleged crimes of a candidate for public office.

The difference between you and the reporter at your local newspaper is that in many cases, you may not have the benefit of training or resources to help you determine whether what you're doing is legal. And on top of that, sometimes knowing the law doesn't help - in many cases it was written for traditional journalists, and the courts haven't yet decided how it applies to bloggers.

But here's the important part: None of this should stop you from blogging. Freedom of speech is the foundation of a functioning democracy, and Internet bullies shouldn't use the law to stifle legitimate free expression. That's why EFF created this guide, compiling a number of FAQs designed to help you understand your rights and, if necessary, defend your freedom.

To be clear, this guide isn't a substitute for, nor does it constitute, legal advice. Only an attorney who knows the details of your particular situation can provide the kind of advice you need if you're being threatened with a lawsuit. The goal here is to give you a basic roadmap to the legal issues you may confront as a blogger, to let you know you have rights, and to encourage you to blog freely with the knowledge that your legitimate speech is protected.

Read entire article at EFF (Electronic Frontier Foundation).

Recover From 120 Day Terminal Services Eval Time Bomb in Windows Servers on EC2

I've always been frustrated by Windows messages like, "please see your administrator. . ."  I AM the administrator, I don't need to see myself, I need useful information to lead me in the right direction to troubleshoot and correct a problem.

Here's a new one that really frustrated me this week.  I have several Amazon EC2 servers.  Most of which run Windows 2003 or Windows 2008.  Often when I start a server for our development team I will install Terminal Services (120 day eval) so more than two developers can connect at a time with RDP.  Usually those servers are in use for a few weeks to a couple months.  Every so often they are used over four months.  Well, as that time approaches Windows kindly displays reminders as to how many days remain in the trial.  We see this so often it just gets ignored.

Well, after the 120 days we can't login to the box any longer, which sucks in and of itself, especially since we can only use RDP (unless we installed something else) to connect to these servers and cannot log on to the console.  The first few times this happened I had to scrap the server and start a new instance.  I have since figured out a work-around. . .

It used to be that after the 120 days was up a nice, informative message was displayed (don't remember the exact wording) that basically said, "time is up you cheap bastard.  You cannot log in to this server any longer and must pay the mighty Micro$oft."  Or something like that.

Now for some reason I'm getting the message, "To log on to this remote computer, you must have Terminal Server user Access permissions on this computer.  by default, members of the Remote Desktop Users group have these permissions.  If you are not a member of the Remote Desktop users group or another group that has these permissions, or if the Remote Desktop User group does not have these permissions, you must be granted these permissions manually."

When I first saw this new message it scared me.  Recently we had some employees leave under less-than-ideal circumstances.  And while I was careful to disable their accounts on our production servers I missed a couple of the dev servers.  My first thought was that one of these guys removed my account and all others from the Administrators group.  After all, that's what the message indicated.  I was able to connect to the crippled server from another EC2 server with Computer Management where I reviewed the security event logs and found nothing afoul.  I also checked the date on NTUSER.DAT for all users.  Again, no smoking gun.

When the same thing happened to another server last night I began to get more worried.  What in the world was happening?  After some crack investigation on my part I was also not able to find anything on this other server which would lead me to the culprit.

I did discover though that both servers had been started initially about four months ago.  This really got me thinking that perhaps the 120 day terminal services time bomb might be the problem.

As I mentioned earlier I discovered how to reset this 120 days on a Windows server running on EC2 - image the machine.  While imaging an EC2 server has a couple of annoying side-effects, like resetting the timezone to Pacific time and creating a new certificate, it does whatever is required by Windows to reset the time to 0 for things like 120 day eval of TS.

Not trying to cheat the system here, just pointing out a way I found to logon to a server I thought was toast.  We are actually done with both of these servers and can terminate them now anyway.

WALK NAKED IN AMERICA DAY

As you may already know, it is a sin for a Muslim male to see any woman other than his wife naked, and if he does, he must commit suicide.  So next Saturday at 10 AM Pacific/1 PM Eastern Time, all American women are asked to walk out of their house completely naked to help weed out any neighborhood terrorists.  Circling your block for one hour is recommended for this anti-terrorist effort.

All patriotic men are to position themselves in lawn chairs in front of their houses to demonstrate their support for the women and to prove that they are not Muslim terrorist sympathizers.  Since Islam also does not approve of alcohol, a cold 6-pack at your side is further proof of your patriotism.

The American government appreciates your efforts to root out terrorists and applauds your participation in this anti-terrorist activity.

God bless America!

Famous Failures

What do Michael Jordan, Thomas Edison, Walt Disney, Abraham Lincoln, Lucille Ball, Ulysses S. Grant, and The Beatles have in common?

Copying ElasticFox Tags from One Browser to Another

The ElasticFox Firefox extension allows you to tag EC2 instances, EBS volumes, EBS snapshots, Elastic IPs, and AMIs. ElasticFox’s tags are stored locally within Firefox, so if you use ElasticFox from more than one browser your tags from one browser are not visible in any other browser.  Also, if your browser crashes you may lose your tags - so back them up.

Manually Copy ElasticFox Tags
Use this method to manually copy ElasticFox tags for backup or copy to another machine.

1. Open about:config (enter about:config in address field of Firefox.
2. In the filter field enter the word, "tags."
3. Copy entries with data in the value field such as:
1. ec2ui.eiptags...
2. ec2ui.instancetags...
3. ec2ui.volumetags...
4. ec2ui.snapshotTags...

See full article, which covers:

• How to Export ElasticFox Settings
• How to Import ElasticFox SettingsCopying ElasticFox Tags to Another Browser Manually
• Copying ElasticFox Tags to Another Browser with OPIE
• Copying ElasticFox Tags to Another Browser with Shell Scripts

"Tech Star" - Parody of Nickelback's "Rock Star"

Tech Star Lyrics

I'm through with standing in line for a bullshit promotion,
Working for a luddite who ain't got a notion.
This career hasn't turned out quite the way I want it to be.

(So tell me what you want?)

I want to join a start-up in the digital space,
And build a rock star business just like Steve Case,
Or better Sergey and Larry or Mark or Evan Jack and Biz.

(So what you need?)

I need some VC funding from a quality firm,
Have to watch the dilution and the other terms,
Don't want to give away too much precious equity.

(Been there, done that.)

I want to build a business with lots of scale,
That levereges real time / local / long tail,
Then sell it for $100 mil seems fine to me.

(So how you gonna do it?)

We'll build an iPhone app, distribute it for free,
Then monetize with ads that target to a tee.

'Cause we all just want to be big tech stars,
Second time entrepreneurs with exits and scars.
The backing comes easy and the money comes cheap,
We'll sell it before we get in too deep.
And I'll tweet my drivel like a digital player,
And check in everywhere so I can be Mayor.
Spend all my time on conference panels,
Which will all be covered in the digital annals.
Hey, hey, I wanna be a tech star.
Hey, hey, I wanna be a tech star.

Now digital innovation's back in spades,
The traditional players are digging their graves.
They claim pay walls will be their savior.

(But you can't change ingrained consumer behavior.)

We'll use cloud computing and open source,
And Facebook for free distribution of course.
Gonna do it all without a need for a series B.

(So how you gonna do it?)

We'll build an iPhone app, distribute it for free,
Then monetize with ads that target to a tee.

'Cause we all just want to be big tech stars,
Second time entrepreneurs with exits and scars.
The backing comes easy and the money comes cheap,
We'll sell it before we get in too deep.
And I'll tweet my drivel like a digital player,
And check in everywhere so I can be Mayor.
Spend all my time on conference panels,
Which will all be covered in the digital annals.

And the markets and deals are back for sure,
Thank God the doomsday call was premature.
The big companies are chock full of cash,
And all I'm lookin' for is my own little stash, well. . .

Hey, hey, I wanna be a tech star.

I'll hire the staffers and the engineers,
And pay them with options, pizza and beers.
I'll get laid off bankers to do all my deals,
Maybe a smoking hot salesperson in high heels.

'Cause we all just want to be big tech stars,
Second time entrepreneurs with exits and scars.
The backing comes easy and the money comes cheap,
We'll sell it before we get in too deep.
And I'll tweet my drivel like a digital player,
And check in everywhere so I can be Mayor.
Spend all my time on conference panels,
Which will all be covered in the digital annals.

And to ensure we get a sky-high price,
We'll hire the best bank to give is advice.
Getting the right team is very worthwhile,
Everybody should have these guys on speed dial.

Hey, hey, I wanna be a tech star.
Hey, hey, I wanna be a tech star.

Clonezilla

You're probably familiar with the popular proprietary commercial package Norton Ghost®. The problem with these kind of software packages is that it takes a lot of time to massively clone systems to many computers. You've probably also heard of Symantec's solution to this problem, Symantec Ghost Corporate Edition® with multicasting. Well, now there is an OpenSource clone system (OCS) solution called Clonezilla with unicasting and multicasting!

Clonezilla, based on DRBL, Partclone and udpcast, allows you to do bare metal backup and recovery. Two types of Clonezilla are available, Clonezilla live and Clonezilla SE (server edition). Clonezilla live is suitable for single machine backup and restore. While Clonezilla SE is for massive deployment, it can clone many (40 plus!) computers simultaneously. Clonezilla saves and restores only used blocks in the harddisk. This increases the clone efficiency. At the NCHC's Classroom C, Clonezilla SE was used to clone 41 computers simultaneously. It took only about 10 minutes to clone a 5.6 GBytes system image to all 41 computers via multicasting!

Features of Clonezilla

• Free (GPL) Software.
• Filesystem supported: (1) ext2, ext3, ext4, reiserfs, reiser4, xfs, jfs of GNU/Linux, (2) FAT, NTFS of MS Windows, (3) HFS+ of Mac OS, (4) UFS of FreeBSD, NetBSD, and OpenBSD, and (5) VMFS of VMWare ESX. Therefore you can clone GNU/Linux, MS windows, Intel-based Mac OS, and FreeBSD, NetBSD, and OpenBSD, no matter it's 32-bit (x86) or 64-bit (x86-64) OS. For these file systems, only used blocks in partition are saved and restored. For unsupported file system, sector-to-sector copy is done by dd in Clonezilla.
• LVM2 (LVM version 1 is not) under GNU/Linux is supported.
• Grub (version 1 and version 2) is supported.
• Multicast is supported in Clonezilla SE, which is suitable for massively clone. You can also remotely use it to save or restore a bunch of computers if PXE and Wake-on-LAN are supported in your clients.
• Based on Partclone (default), Partimage (optional), ntfsclone (optional), or dd to image or clone a partition. However, Clonezilla, containing some other programs, can save and restore not only partitions, but also a whole disk.
• By using another free software drbl-winroll, which is also developed by us, the hostname, group, and SID of cloned MS windows machine can be automatically changed.

Learn more and download Clonezilla.

Clonezilla Kicks Ghost's Ass!

Back in '93 I found a great little utility called Ghost, which could quickly and easily clone DOS or Windows (3.1, 95, NT) computers.  I used to teach week-long IT training classes and had to set a dozen or so computers back to default settings in order for each class to start fresh.  Ghost.exe was awesome & did the job easily and quickly.  I wish I could remember the name of the developer - he was a nice fellow from down under, New Zealand if I remember correctly. I spoke with him a few times about using Ghost & gave him some feedback from my perspective.  I used it for several years and it was great!

Then along came Symantec who bought Ghost (I think Norton actually bought Ghost, then Symantec bought them).  No matter.  They screwed it up royally.  Just like they do with about everything they buy.  Oh, they don't have the corner on screwing things up - Microsoft, CA and myriad others do the same thing.

Anyway, Clonezilla is a fresh new product that gets back to the basics of cloning.  Here's a little from the Clonezilla site:

Clonezilla, based on DRBL, Partclone and udpcast, allows you to do bare metal backup and recovery. Two types of Clonezilla are available, Clonezilla live and Clonezilla SE (server edition). Clonezilla live is suitable for single machine backup and restore. While Clonezilla SE is for massive deployment, it can clone many (40 plus!) computers simultaneously. Clonezilla saves and restores only used blocks in the harddisk. This increases the clone efficiency. At the NCHC's Classroom C, Clonezilla SE was used to clone 41 computers simultaneously. It took only about 10 minutes to clone a 5.6 GBytes system image to all 41 computers via multicasting!

Features of Clonezilla

• Free (GPL) Software.
• Filesystem supported: (1) ext2, ext3, ext4, reiserfs, reiser4, xfs, jfs of GNU/Linux, (2) FAT, NTFS of MS Windows, (3) HFS+ of Mac OS, (4) UFS of FreeBSD, NetBSD, and OpenBSD, and (5) VMFS of VMWare ESX. Therefore you can clone GNU/Linux, MS windows, Intel-based Mac OS, and FreeBSD, NetBSD, and OpenBSD, no matter it's 32-bit (x86) or 64-bit (x86-64) OS. For these file systems, only used blocks in partition are saved and restored. For unsupported file system, sector-to-sector copy is done by dd in Clonezilla.
• LVM2 (LVM version 1 is not) under GNU/Linux is supported.
• Grub (version 1 and version 2) is supported.
• Multicast is supported in Clonezilla SE, which is suitable for massively clone. You can also remotely use it to save or restore a bunch of computers if PXE and Wake-on-LAN are supported in your clients.
• Based on Partclone (default), Partimage (optional), ntfsclone (optional), or dd to image or clone a partition. However, Clonezilla, containing some other programs, can save and restore not only partitions, but also a whole disk.
• By using another free software drbl-winroll, which is also developed by us, the hostname, group, and SID of cloned MS windows machine can be automatically changed. 

Read more about Clonezilla, download it and check it out.