In Brisbane

Relaxing

Amazon S3 outage

Recently I had moved all my resources over to Amazon's S3. This morning, I woke up to a system wide outage that has beeing going on since at least 6:30am EST.

I was able to change all the asset code back to locally hosted by changing the "config.action_controller.asset_host" configuration setting and now all is working just fine.

Last year I lost a few Amazon EC2 instances and vowed to always have a backup plan that involved my colocated rack we could drive to. Although I have seen outages there, they have never lasted more than an hour. They also involved something that I could control and make reduntant going forward. With S3, there is not much you can except wait or have a backup plan.

Backup plans are fine for things like hosted assets but would be much harder where you have integrated customer data. Imagine if all your hosted attachments were in S3, there would be no way around an outage unless you mirrored the attachments in real time locally. That would involve paying twice for bandwidth and thus obviate all the savings with the S3 system.

I predict this will be big news and make a lot of people who relied n S3 think some serious thoughts about how they will structure their data going forward.

update 11am: There is talk the issue is resolved but it seems slow and it looks like there still might be some issues. To play it safe, I will keep the backup plan in place for another day to see what happens.

Pubget now available to Harvard/Partners

The fastest way to search science! If you are at Harvard or one of its affiliated hospitals, you can sign up now at pubget.com.

Shopping List

Grains
Make sure any whole-wheat products you buy are labeled 100% whole wheat.
Brown rice
Steel-cut oatmeal
Whole-grain or oat breakfast cereal (Cheerios, Kashi cereals, Grape Nuts)
Whole-grain pizza dough/crust
Whole-wheat or whole-grain bread
Whole-wheat pasta
Whole-wheat pitas or tortillas

Canned/Jarred Items
Black beans
Olives
Sun-dried tomatoes (not in oil)
Tomato sauce (no added sugars)
Tomatoes: whole, crushed, or diced
Unsweetened fruit
Vegetable or chicken stock/broth (low-salt)
White beans

Dried Fruits and Nuts
Nuts should be raw, rather than roasted or salted.

Almonds
Dried cranberries and apricots
Pistachios, chopped
Raisins
Walnuts and hazelnuts

Condiments and Spices
Balsamic vinegar
Canola oil, regular and spray-on
Chocolate, dark (not milk) with at least 70% cocoa
Cinnamon and nutmeg
Extra-virgin olive oil
Honey
Low-sodium soy sauce
Mustard
Real maple syrup
Red pepper flakes
Turmeric or curry powder
Wine vinegar

Refrigerated Items
Eggs
Feta cheese, low-fat
Milk, skim or low-fat soy
Orange or grapefruit juice (100%) with pulp
Part-skim mozzarella cheese
Yogurt with active cultures (probiotic), low-fat
Sour cream, low-fat

Poultry/Fish
Chicken breast halves, skinless and boneless
Chicken thighs, skinless
Deli meat, sliced and skinless (not processed cold cuts)
Salmon fillets, skinless
Whole fish or fillets: trout, tilapia, snapper, or sea bass

Frozen Food
Blueberries and raspberries, frozen and unsweetened
Fruit sorbet
Vanilla frozen yogurt, nonfat or low-fat

Health Foods
Chia seed
Flaxseed
Soy protein powder

Fruits and Vegetables
Stock up on plenty of fresh fruits and veggies from each color group, but don’t buy more than you’ll be able to eat in a week. Fruits and vegetables lose their nutrient goodness when they sit around.

Blue/Purple:
Blueberries, blackberries, plums, eggplant

Orange/Yellow:
Carrots, sweet potatoes, squash, mangoes, pineapple

Red:
Tomatoes, cherries, cranberries, red peppers, red apples

Yellow/Green:
Avocados, broccoli, spinach, kiwifruit, lemons, limes

White/Green:
Garlic, onions, bananas, mushrooms

My Default CentOS Setup

This script will probably only be valid for a week, but I thought I would share my ideas on a good CentOS 5 (64bit) install with Ruby on Rails, MySQL and Apache that will work with a capistrano deployment from subversion:

yum -y update
yum -y install ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs ruby-devel mysql mysql-devel mysql-server mysql-admin subversion httpd svn

togglesebool httpd_can_network_connect

cd /tmp
wget http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
tar -xvf rubygems-1.0.1.tgz
cd rubygems-1.0.1
ruby setup.rb

gem install rails json rfacebook acts_as_ferret capistrano mongrel mongrel_cluster pdf-toolkit actionmailer actionpack actionwebservice activerecord activeresource activesupport acts_as_taggable acts_as_versioned ferret google4r calendar mysql sources vpim mime-type --include-dependencies

#Fedora 8 also wants
yum -y install rubygem-mongrel gcc ruby-mysql
yum install gd gd-devel zlib-devel openssl-devel
gem install gem_plugin daemons capistrano --include-dependencies
gem install mongrel mongrel_cluster railsmachine --include-dependencies
gem install --version=2.7 mysql -- --with-mysql-config=/usr/bin/mysql_config

or

yum -y install ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs ruby-devel mysql mysql-devel mysql-server mysql-admin subversion httpd svn rubygem-mongrel gcc ruby-mysql gd gd-devel zlib-devel openssl-devel partimage
gem install -y rails -v=2.0.2
gem install json rfacebook acts_as_ferret capistrano mongrel mongrel_cluster pdf-toolkit actionmailer actionpack actionwebservice activerecord activeresource activesupport acts_as_taggable acts_as_versioned ferret google4r calendar mysql sources vpim mime-type ferret mime-types solr-ruby ruby-openid mechanize pdf-toolkit mongrel mongrel_cluster railsmachine bio --include-dependencies


I tried to make it as silent and complete as possible so that I could let the computer do all the work.

Mysql Master to Master Replication

This will setup two Amazon EC2 CentOS 5 machines to cluster replicate all their MySQL databases.

Server 1: /etc/my.sql


#replication
server-id=1
log-bin=mysql-bin
binlog-ignore-db=mysql
binlog-ignore-db=test

#information for becoming slave.
master-host =
master-user = replication
master-password = slave
master-port = 3306



Server 2: /etc/my.sql

#replication
server-id=2
log-bin=mysql-bin
binlog-ignore-db=mysql
binlog-ignore-db=test

#information for becoming slave.
master-host =
master-user = replication
master-password = slave
master-port = 3306


Restart /etc/init.d/mysql restart on both servers.

Then run the following SQL on...
server 1:

STOP SLAVE;
CHANGE MASTER TO MASTER_HOST='10.253.15.15',MASTER_USER='replication',MASTER_PASSWORD='slave';
START SLAVE;
GRANT ALL PRIVILEGES ON *.* TO 'replication'@'' identified by 'slave';

server 2:

MASTER_HOST='10.253.65.221',MASTER_USER='replication',MASTER_PASSWORD='slave';
STOP SLAVE;
show slave status\G;
GRANT ALL PRIVILEGES ON *.* TO 'replication'@'' identified by 'slave';


You should then be able to create and remove databases on either servers and it will replicate to the other.

I tried to use "load data from master" but it only worked on very simple data. On large sets, it was better to use mysqlhotcopy (perl script that came with mysql on centos)

On CentOS, you first need to export the mysql directory in the /etc/exports file like this:
var/lib/mysql 10.253.15.15(rw,no_root_squash)
and then run:
exportfs -r

then on the machine with the tables, you run:
mkdir /tmp/mounted_mysql_directory
mount -o vers-3 10.253.65.221:/var/lib/mysql /tmp/mounted_mysql_directory

Now that you have the directory exported and mounted, you can hot copy the files from server 1 to server 2:

mysqlhotcopy --addtodest --resetmaster --resetslave -u root -p yourdatabasename_development /tmp/mounted_mysql_directory

The resetmaster and resetslave are important here. MySQL will keep an index in the replication log where it thinks its master is and where it is as a slave. After I copy the database over, I want them to start fresh. So, from here you can look at the master status on server 1 and master status on server 2 to get these indexes. then you can run CHANGE MASTER TO with an extra parameter:
MASTER_LOG_POS=98;
if the index was 98. This will then get them both in sync and ready to accept further transactions to apply.

You can also skip a given number of replication transactions using:
SET GLOBAL SQL_SLAVE_SKIP_COUNTER = N;
just be careful what you skip.

Coming from a Notes/Domino background, this idea of replicating transactions was a little new. Notes, will keep a sequence number for each record and keep the data in sync. It does not matter how the data got into its current state, it will just do its best at keeping the data replicated (using sequence numbers and time stamps as needed).

MySQL is more transactional based. The documents talk about the my.cnf parameters like "binlog-ignore-db" will act as a filter for how transactions get written to the replication log. To a Notes person, this seems a little inefficient. It means to keep the data in sync, you have to run the exact same transactions on each server (even if the operations are repetitive and act on the same records). In Notes, it will be able to skip ahead to the final state rather than going through each transaction to get to the final state. MySQL has no such ability.

As a result, MySQL cannot just sync itself if you run into problems. This is also true when you add a member to the cluster or replace a failed one. To get things in sync again, you really have to think it through and make sure you lock tables as needed, get the data the same with copy tools like mysqlhotcopy and let the transactions start again on the cluster when you know it is ready.

I would also expect this would have a different scalability path than Notes - if your bottleneck is the amount of transactions you are getting, then adding additional replicas may not help in MySQL where it would in Notes.

Upgrading to rails 2.0.2

There were just a few too many cool features in 2.0 that I wanted so I have taken the plunge in a project. I did wait for 2.0.2, so it is not really bleeding edge - but I feel nice and up to date.
The process was fairly straight forward even though I knew I used a lot of plugins and deprecated methods. I had to perform the following to get it working:

• gem install rails
• script/plugin install svn://errtheblog.com/svn/plugins/classic_pagination
  
• script/plugin install http://svn.rubyonrails.org/rails/plugins/in_place_editing/

I also referenced:

• http://mislav.caboo.se/rails/rails-2-0-taking-the-plunge/
• http://localtone.blogspot.com/2008/01/upgrade-to-rails-20.html

After restarting the server, users will get a warning about cookies. However, this is a small price to pay for cookie based sessions.

bellasante of boston

I tried to make a booking for my wife and a friend's wife yesterday for a spa treatment. So, I called the concierge to book it in for friday afternoon. The problem was they seemed very busy and I didn't know what to order or which location would be best. I didn't mind either Lexington or Wellesley but the concierge seemed in a rush so made me pick one right away. Then I was told after picking Wellesley there was nothing on the day I wanted.
I really felt like they were too busy to deal with me and decided the best way was to get off the phone as quickly as possible and try again with another concierge or spa. After replying to a few "contact us" forms for a site I am working on, I thought I should at least let them know the feedback to see if indeed they are too busy or if it was just a bad time and I got the wrong concierge. In any case, I filled out their web form only to find it was a "post" action to "#" and didn't actually go anywhere.
They seem to have a lot of "best of boston" awards so the spa part might be nice - it just seems that half the challenge is to get to that part.

Jack Cards on the WSJ.

One of the startup web sites we have been working on and hosting at ProjectLounge, was reviewed by the Wall Street Journal. I am looking forward to taking the site to the next level - but a review like this is certainly a good start for the site. It is full of good suggestions and a positive conclusion.
"I think its system would be welcomed by many people looking for good-quality, well-designed cards that catch the eye. It saves users the hassle of last-minute trips to the card store, and still lets greeting cards keep their personal touch, which is what makes them such a permanent fixture in our lives."

Fire Truck Photos

This morning we saw a Fire Truck visit our street - no fire or anything exciting. However, I did capture some photos.

Commando Crawl

This our friends commando Teddy.

Everyone loves cats

Mount ssh/sftp drives into OSX

Dealing with a server that is hosted requires a stronger level of security. This typically means no ftp or file sharing is allowed. However, ssh is secure and you can even prevent passwords and use rsa keys.
However, all this security does not mean you cannot mount a drive using OSX. Using the MacFuse at google, you can mount a remote ssh/sftp drive with a hostname, username, password (or rsa key) and path.
Steps:
1. Download and install MacFuse from the link
2. Restart your OSX (it does some low level magic that makes writing mounted disks easier)
3. Download and copy the sshfs into your Applications from the link.
Now that you have it download, you can run the sshfs application and connect to your disk.
If you do use an RSA key or the like, you will need to first connect via the command line so that the host gets to be trusted and then it will work fine from there on.

DIIOP NotesException: Server access denied

The R5 settings in the server document still play a part in how DIIOP handles permissions in 7.0.2. The getFirstDatabase function in DbDirectory, will return "Server access denied" if the "Allow HTTP clients to browse databases" is not set to "Yes" on the R5 Basics tab under HTTP Internet Protocols in your Domino server document.

Customized iPhone ringtone

One the the missing features of the iPhone is not being able to select our own ringtone. Well, thanks to this web site, I was able to upload my own mp3 file as a ringtone and it worked.
It is not as easy as other phones but still possible with a little instructions.

Outlook Permissions for ICS files

A user recently got the error: "Can't create file: calendarevent.ics. Right-click the folder you want to create the file in, and then click Properties on the shortcut menu to check your permissions for the folder."

After some serching, it is not really a permission problem but rather when you install certain Microsoft Small Business products, the OutlookSecureTempFolder key in the registry is changed from "..\Temp\..." to "...\Temporary Internet Files\...".

To fix this, have your system administrator, go into the registry and change the HKEY_CURRENT_USER\Software\Microsoft\Office\11.0\Outlook\Security registry key OutlookSecureTempFolder value from "...\Temporary Internet Files\..." to "...\Temp\..." and you should be able to open ICS events from QuickPlace or any other service again.

Christmas come early?

Quickr has shipped early and after installing it and the latest fixes from "fix central", I have started the process of picking through the new features.
I have tested our existing themes and they all seem to work just fine so far and with a little more testing, I think we will be safe to upgrade some servers.
I have to say well done to Satwik and his team for getting QuickPlace team buzzing like it was 1999.

It is all happening on the 29th

On the 29th of June, there will be so many things to do, I won't know what to do with myself.
1. Quickr 8 Launches;
2. iPhone is released; and
3. Sicko comes out.
I am sure there will be lots more happening that day, but these are things I have actually been waiting for and anticipating.
This week, I have been very busy sorting out some Windows Vista issues with QuickPlace 7 as well as a ruby on rails development project for jackcards. I am also in the process of installing a NAS for domino which so far indicates 2 to 4 times increase in our current IO performance.
So, if I have been a little quiet lately, I apologize and will be a little more sociable soon (even if it is just to show off an iPhone if I get the courage to buy one).

Rational Climate Change Argument

Rational Climate Change Argument

This is basically the same as Pascal's Wager with some key differences. These are that Pascal was based on religion where this is actually based on science. However, it is a very good use of a truth table

Ruby on Rails Domino Active Record

I have been using this class for a while that allows some ActiveRecord like features for IBM Lotus Domino databases.

An example use would be:


class Contact < DominoRecord
attr_accessor :email_address, :first_name, :last_name

validates_presence_of :first_name, :last_name, :email_address

validates_format_of :email_address, :with =>

/^\s*(?:(?:[^,@\s]+)@(?:(?:[-a-z0-9]+\.)+[a-z]{2,}\s*(,\s*|\z)))+$/i,

:message => "must be a valid email address",

:on => :create

@@database_name = "contacts.nsf" #your database here

@@server_name = 'www/projectlounge' #your server here

end


It currently uses the OLE/COM bridge to get to domino so only will work on windows at this stage. The next step would be to compile it with the NotesAPI to allow it to work on all platforms.

The model source code is here (I do not maintain this anymore but happy for others to take it on under LGPL):


require 'win32ole'

class DominoRecord



attr_accessor :form



def save

@document = @@database.CreateDocument

form = self.class.to_s.downcase

@document.ReplaceItemValue "form", form

self.instance_variables.each do |inst|

field_name = inst.sub(/[@]/, '')

field_value = self.instance_variable_get(inst)

@document.ReplaceItemValue field_name, field_value.to_s

end

@document.Save(true, false)

end



def save!;

save

end

def update_attribute; end

def new_record?; end

include ActiveRecord::Validations



@@session = nil

@@database = nil

@@database_name = nil

@@server_name = nil



def [](key)

instance_variable_get(key)

end



def self.find(*args)

options = extract_options_from_args!(args)

validate_find_options(options)



case args.first

when :first then

@document = find_initial(options)

when :all then find_every(options)

else

@document = find_from_ids(args, options)

end



record = self.new

@document.Items.each do |item|

record.send(item.Name.downcase + "=", item.Text) unless ["document", "errors"].include?(item.Name)

end

record

end



def DominoRecord.human_attribute_name(attribute_key_name)

ActiveRecord::Base.human_attribute_name(attribute_key_name)

end



def initialize(attributes=nil)

unless @@session

@@session = WIN32OLE.new('Lotus.NotesSession')

@@session.Initialize

end

@@database = @@session.GetDatabase(@@server_name,@@database_name) unless @@database



unless form == nil

fields = @@database.GetForm(form).Fields

else

fields = @@database.GetForm(self.class.to_s).Fields

end



fields.each { |field| self.class.__send__(:attr_accessor, field) }



return if attributes.nil?

attributes.each do |k, v|

send(k + "=", v)

end

end



def finalize

@@session.Finalize

end



def session

@@session

end



def database

@@database

end



def document

@document

end



def attributes=(attributes)

return if attributes.nil?

attributes.stringify_keys!

multi_parameter_attributes = []

attributes.each do |k, v|

send(k + "=", v)

end

end



def self.method_missing(method_id, *arguments)



unless @@session

@@session = WIN32OLE.new('Lotus.NotesSession')

@@session.Initialize

end

@@database = @@session.GetDatabase(@@server_name,@@database_name) unless @@database



if match = /find_by_([_a-zA-Z]\w*)/.match(method_id.to_s)



@view = @@database.GetView($1)

puts arguments

@document = @view.GetDocumentByKey(arguments[0])



record = self.new

@document.Items.each do |item|

record.send(item.Name.downcase + "=", item.Text) unless ["document", "errors"].include?(item.Name)

end

record

else

super

end

end



private

def self.find_initial(options)

#not working

end



def self.find_every(options)

#not working

end



def self.find_from_ids(ids, options)

#not working

end



def self.find_one(id, options)



if result = @@database.GetDocumentByUNID( id )

result

else

raise RecordNotFound, "Couldn't find #{name} with ID=#{id}#{conditions}"

end

end



def self.find_some(ids, options)



result = find_every(options)



if result.size == ids.size

result

else

raise RecordNotFound, "Couldn't find all #{name.pluralize} with IDs (#{ids_list})#{conditions}"

end

end





def self.extract_options_from_args!(args) #:nodoc:

args.last.is_a?(Hash) ? args.pop : {}

end



VALID_FIND_OPTIONS = [ :conditions, :include, :joins, :limit, :offset,

:order, :select, :readonly, :group, :from ]



def self.validate_find_options(options) #:nodoc:

options.assert_valid_keys(VALID_FIND_OPTIONS)

end

end