Monday, November 03, 2008

Mysql and Ruby on Rails with OSX Leopard

To get mysql working on leopard. Run:


sudo env ARCHFLAGS="-arch i386" gem install mysql -- \
--with-mysql-dir=/usr/local/mysql --with-mysql-lib=/usr/local/mysql/lib \
--with-mysql-include=/usr/local/mysql/include

then

cd /usr/local/mysql/lib
sudo mkdir mysql
sudo cp libmysqlclient.15.dylib mysql/libmysqlclient.15.dylib

It is a hack, but got it to compile for me.

Tuesday, July 22, 2008

Fishing on the weekend - loads of fun

I have just synced my camera so can post some of the photos:
This is the first day's catch.

and this was from the second day.

It was a nice weekend break and Kim even did a little relaxing on the beach (without getting sun burn).

Sunday, July 20, 2008

Amazon S3 down again

Here is the thread on their forum:

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=23285&start=0&tstart=0

It is making some of my sites look bad as I chose to put all the css and images on S3 so they would be severed up in a distributed fast manner. However, that speed is not being appreciated right now as it is not working.

My task this week will be to design a backup plan or move the css and images.

UPDATE: I have a backup plan now - which is nice. Amazon is also back up after 7 hours of downtime so I have turned off the backup service. One positive thing is that I now have a backup plan that can be turned in within minutes.

Monday, April 28, 2008

Simple email alert error page.

Mongrel, Apache, mod_proxy and Rails do go very well together. However, there are times when it can break down. While you can rely to some degree on monitoring tools, if a user is given an error page, it is nice to know as soon as it happens.
One good way to do this, have the error page, trigger an email alert letting you know some basic information so that you can debug and figure out what has gone wrong.
When mod proxy cannot connect to the mongrel cluster, it will return a 503 error page. Apache lets you specify this to be a cgi script so that you can bounce them to a cgi error page that packages up the environment and emails it to an administrator.

Here are 4 steps to follow to use this technique:

Step 1: Write a cgi error document

#!/usr/bin/perl
use strict;
use CGI;


my $query = new CGI;

my $sendmail = "/usr/sbin/sendmail -t";
my $reply_to = "Reply-to: REPLYEAMAIL\@REPLYDOMAIN.com\n";
my $subject = "Subject: Apache Error Page\n";

my $content = "SERVER_SOFTWARE = " . $ENV{'SERVER_SOFTWARE'} . "\n";
$content = $content . "SERVER_SOFTWARE = " . $ENV{'SERVER_SOFTWARE'} . "\n";
$content = $content . "SERVER_NAME = " . $ENV{'SERVER_NAME'} . "\n";
$content = $content . "GATEWAY_INTERFACE = " . $ENV{'GATEWAY_INTERFACE'} . "\n";
$content = $content . "SERVER_PROTOCOL = " . $ENV{'SERVER_PROTOCOL'} . "\n";
$content = $content . "SERVER_PORT = " . $ENV{'SERVER_PORT'} . "\n";
$content = $content . "REQUEST_METHOD = " . $ENV{'REQUEST_METHOD'} . "\n";
$content = $content . "HTTP_ACCEPT = '" . $ENV{'HTTP_ACCEPT'} . "\n";
$content = $content . "PATH_INFO = " . $ENV{'PATH_INFO'} . "\n";
$content = $content . "PATH_TRANSLATED = " . $ENV{'PATH_TRANSLATED'} . "\n";
$content = $content . "SCRIPT_NAME = " . $ENV{'SCRIPT_NAME'} . "\n";
$content = $content . "QUERY_STRING = " . $ENV{'QUERY_STRING'} . "\n";
$content = $content . "REMOTE_HOST = " . $ENV{'REMOTE_HOST'} . "\n";
$content = $content . "REMOTE_ADDR = " . $ENV{'REMOTE_ADDR'} . "\n";
$content = $content . "REMOTE_USER = " . $ENV{'REMOTE_USER'} . "\n";
$content = $content . "CONTENT_TYPE = " . $ENV{'CONTENT_TYPE'} . "\n";
$content = $content . "CONTENT_LENGTH = " . $ENV{'CONTENT_LENGTH'} . "\n";

my $to = "To: YOUREMAIL\@YOURDOMAIN.com\n";

open(SENDMAIL, "|$sendmail") or die "Cannot open $sendmail: $!";
print SENDMAIL $reply_to;
print SENDMAIL $subject;
print SENDMAIL $to;
print SENDMAIL "Content-type: text/plain\n\n";
print SENDMAIL $content;
close(SENDMAIL);


print "Content-type: text/html\n\n";
print <

Step 2: Place it in your script alias path and set it as the error document for 503 errors

...edit your httpd.conf file...
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
ErrorDocument 503 /cgi-bin/error.pl

Step 3: Allow it to run (chmod a+x error.pl)
Step 4: Tell modd proxy to leave /cgi-bin scripts alone

...insert before your RewriteRule that proxies requests...
RewriteCond %{REQUEST_URI} !^/cgi-bin

Tuesday, April 22, 2008

Pubget is on the iPhone


For those Doctors, that love their iPhone, you can now run your Pubmed and Pubget [latest] searches and still get to the PDF right away.
You can read more about it on the pubget blog, or visit pubget.com/mobile on your iphone.

Friday, April 04, 2008

Pubget Blog Launched

I have launched a new blog on Pubget.
http://pubget.blogspot.com
I will be keeping this updated with the latest features, events and news for Pubget. If you are in medical/biological research or practice and you use Pubmed - this new service will save you time. Check it out, it is currently working for open access articles as well as licensed ones for MGH, MIT or Harvard users.

Monday, March 10, 2008

Lily Bean


Grand opening

Wednesday, March 05, 2008

Walk along the river


Beautiful sunny Brisbane morning.

Thursday, February 28, 2008

Friday, February 15, 2008

Amazon S3 outage

Recently I had moved all my resources over to Amazon's S3. This morning, I woke up to a system wide outage that has beeing going on since at least 6:30am EST.

I was able to change all the asset code back to locally hosted by changing the "config.action_controller.asset_host" configuration setting and now all is working just fine.

Last year I lost a few Amazon EC2 instances and vowed to always have a backup plan that involved my colocated rack we could drive to. Although I have seen outages there, they have never lasted more than an hour. They also involved something that I could control and make reduntant going forward. With S3, there is not much you can except wait or have a backup plan.

Backup plans are fine for things like hosted assets but would be much harder where you have integrated customer data. Imagine if all your hosted attachments were in S3, there would be no way around an outage unless you mirrored the attachments in real time locally. That would involve paying twice for bandwidth and thus obviate all the savings with the S3 system.

I predict this will be big news and make a lot of people who relied n S3 think some serious thoughts about how they will structure their data going forward.

update 11am: There is talk the issue is resolved but it seems slow and it looks like there still might be some issues. To play it safe, I will keep the backup plan in place for another day to see what happens.

Monday, February 11, 2008

Pubget now available to Harvard/Partners


The fastest way to search science! If you are at Harvard or one of its affiliated hospitals, you can sign up now at pubget.com.

Shopping List

Grains
Make sure any whole-wheat products you buy are labeled 100% whole wheat.
Brown rice
Steel-cut oatmeal
Whole-grain or oat breakfast cereal (Cheerios, Kashi cereals, Grape Nuts)
Whole-grain pizza dough/crust
Whole-wheat or whole-grain bread
Whole-wheat pasta
Whole-wheat pitas or tortillas

Canned/Jarred Items
Black beans
Olives
Sun-dried tomatoes (not in oil)
Tomato sauce (no added sugars)
Tomatoes: whole, crushed, or diced
Unsweetened fruit
Vegetable or chicken stock/broth (low-salt)
White beans

Dried Fruits and Nuts
Nuts should be raw, rather than roasted or salted.

Almonds
Dried cranberries and apricots
Pistachios, chopped
Raisins
Walnuts and hazelnuts

Condiments and Spices
Balsamic vinegar
Canola oil, regular and spray-on
Chocolate, dark (not milk) with at least 70% cocoa
Cinnamon and nutmeg
Extra-virgin olive oil
Honey
Low-sodium soy sauce
Mustard
Real maple syrup
Red pepper flakes
Turmeric or curry powder
Wine vinegar

Refrigerated Items
Eggs
Feta cheese, low-fat
Milk, skim or low-fat soy
Orange or grapefruit juice (100%) with pulp
Part-skim mozzarella cheese
Yogurt with active cultures (probiotic), low-fat
Sour cream, low-fat

Poultry/Fish
Chicken breast halves, skinless and boneless
Chicken thighs, skinless
Deli meat, sliced and skinless (not processed cold cuts)
Salmon fillets, skinless
Whole fish or fillets: trout, tilapia, snapper, or sea bass

Frozen Food
Blueberries and raspberries, frozen and unsweetened
Fruit sorbet
Vanilla frozen yogurt, nonfat or low-fat

Health Foods
Chia seed
Flaxseed
Soy protein powder

Fruits and Vegetables
Stock up on plenty of fresh fruits and veggies from each color group, but don’t buy more than you’ll be able to eat in a week. Fruits and vegetables lose their nutrient goodness when they sit around.

Blue/Purple:
Blueberries, blackberries, plums, eggplant

Orange/Yellow:
Carrots, sweet potatoes, squash, mangoes, pineapple

Red:
Tomatoes, cherries, cranberries, red peppers, red apples

Yellow/Green:
Avocados, broccoli, spinach, kiwifruit, lemons, limes

White/Green:
Garlic, onions, bananas, mushrooms

Monday, January 28, 2008

My Default CentOS Setup

This script will probably only be valid for a week, but I thought I would share my ideas on a good CentOS 5 (64bit) install with Ruby on Rails, MySQL and Apache that will work with a capistrano deployment from subversion:

yum -y update
yum -y install ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs ruby-devel mysql mysql-devel mysql-server mysql-admin subversion httpd svn

togglesebool httpd_can_network_connect

cd /tmp
wget http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
tar -xvf rubygems-1.0.1.tgz
cd rubygems-1.0.1
ruby setup.rb

gem install rails json rfacebook acts_as_ferret capistrano mongrel mongrel_cluster pdf-toolkit actionmailer actionpack actionwebservice activerecord activeresource activesupport acts_as_taggable acts_as_versioned ferret google4r calendar mysql sources vpim mime-type --include-dependencies

#Fedora 8 also wants
yum -y install rubygem-mongrel gcc ruby-mysql
yum install gd gd-devel zlib-devel openssl-devel
gem install gem_plugin daemons capistrano --include-dependencies
gem install mongrel mongrel_cluster railsmachine --include-dependencies
gem install --version=2.7 mysql -- --with-mysql-config=/usr/bin/mysql_config

or

yum -y install ruby ruby-libs ruby-mode ruby-rdoc ruby-irb ruby-ri ruby-docs ruby-devel mysql mysql-devel mysql-server mysql-admin subversion httpd svn rubygem-mongrel gcc ruby-mysql gd gd-devel zlib-devel openssl-devel partimage
gem install -y rails -v=2.0.2
gem install json rfacebook acts_as_ferret capistrano mongrel mongrel_cluster pdf-toolkit actionmailer actionpack actionwebservice activerecord activeresource activesupport acts_as_taggable acts_as_versioned ferret google4r calendar mysql sources vpim mime-type ferret mime-types solr-ruby ruby-openid mechanize pdf-toolkit mongrel mongrel_cluster railsmachine bio --include-dependencies


I tried to make it as silent and complete as possible so that I could let the computer do all the work.

Thursday, January 24, 2008

Mysql Master to Master Replication

This will setup two Amazon EC2 CentOS 5 machines to cluster replicate all their MySQL databases.

Server 1: /etc/my.sql


#replication
server-id=1
log-bin=mysql-bin
binlog-ignore-db=mysql
binlog-ignore-db=test

#information for becoming slave.
master-host =
master-user = replication
master-password = slave
master-port = 3306



Server 2: /etc/my.sql

#replication
server-id=2
log-bin=mysql-bin
binlog-ignore-db=mysql
binlog-ignore-db=test

#information for becoming slave.
master-host =
master-user = replication
master-password = slave
master-port = 3306


Restart /etc/init.d/mysql restart on both servers.

Then run the following SQL on...
server 1:

STOP SLAVE;
CHANGE MASTER TO MASTER_HOST='10.253.15.15',MASTER_USER='replication',MASTER_PASSWORD='slave';
START SLAVE;
GRANT ALL PRIVILEGES ON *.* TO 'replication'@'' identified by 'slave';

server 2:

MASTER_HOST='10.253.65.221',MASTER_USER='replication',MASTER_PASSWORD='slave';
STOP SLAVE;
show slave status\G;
GRANT ALL PRIVILEGES ON *.* TO 'replication'@'' identified by 'slave';


You should then be able to create and remove databases on either servers and it will replicate to the other.

I tried to use "load data from master" but it only worked on very simple data. On large sets, it was better to use mysqlhotcopy (perl script that came with mysql on centos)

On CentOS, you first need to export the mysql directory in the /etc/exports file like this:
var/lib/mysql 10.253.15.15(rw,no_root_squash)
and then run:
exportfs -r

then on the machine with the tables, you run:
mkdir /tmp/mounted_mysql_directory
mount -o vers-3 10.253.65.221:/var/lib/mysql /tmp/mounted_mysql_directory

Now that you have the directory exported and mounted, you can hot copy the files from server 1 to server 2:

mysqlhotcopy --addtodest --resetmaster --resetslave -u root -p yourdatabasename_development /tmp/mounted_mysql_directory

The resetmaster and resetslave are important here. MySQL will keep an index in the replication log where it thinks its master is and where it is as a slave. After I copy the database over, I want them to start fresh. So, from here you can look at the master status on server 1 and master status on server 2 to get these indexes. then you can run CHANGE MASTER TO with an extra parameter:
MASTER_LOG_POS=98;
if the index was 98. This will then get them both in sync and ready to accept further transactions to apply.

You can also skip a given number of replication transactions using:
SET GLOBAL SQL_SLAVE_SKIP_COUNTER = N;
just be careful what you skip.

Coming from a Notes/Domino background, this idea of replicating transactions was a little new. Notes, will keep a sequence number for each record and keep the data in sync. It does not matter how the data got into its current state, it will just do its best at keeping the data replicated (using sequence numbers and time stamps as needed).

MySQL is more transactional based. The documents talk about the my.cnf parameters like "binlog-ignore-db" will act as a filter for how transactions get written to the replication log. To a Notes person, this seems a little inefficient. It means to keep the data in sync, you have to run the exact same transactions on each server (even if the operations are repetitive and act on the same records). In Notes, it will be able to skip ahead to the final state rather than going through each transaction to get to the final state. MySQL has no such ability.

As a result, MySQL cannot just sync itself if you run into problems. This is also true when you add a member to the cluster or replace a failed one. To get things in sync again, you really have to think it through and make sure you lock tables as needed, get the data the same with copy tools like mysqlhotcopy and let the transactions start again on the cluster when you know it is ready.

I would also expect this would have a different scalability path than Notes - if your bottleneck is the amount of transactions you are getting, then adding additional replicas may not help in MySQL where it would in Notes.

Wednesday, January 16, 2008

Upgrading to rails 2.0.2

There were just a few too many cool features in 2.0 that I wanted so I have taken the plunge in a project. I did wait for 2.0.2, so it is not really bleeding edge - but I feel nice and up to date.
The process was fairly straight forward even though I knew I used a lot of plugins and deprecated methods. I had to perform the following to get it working:
  • gem install rails
  • script/plugin install svn://errtheblog.com/svn/plugins/classic_pagination
  • script/plugin install http://svn.rubyonrails.org/rails/plugins/in_place_editing/
I also referenced:
After restarting the server, users will get a warning about cookies. However, this is a small price to pay for cookie based sessions.

Thursday, December 27, 2007

bellasante of boston


I tried to make a booking for my wife and a friend's wife yesterday for a spa treatment. So, I called the concierge to book it in for friday afternoon. The problem was they seemed very busy and I didn't know what to order or which location would be best. I didn't mind either Lexington or Wellesley but the concierge seemed in a rush so made me pick one right away. Then I was told after picking Wellesley there was nothing on the day I wanted.
I really felt like they were too busy to deal with me and decided the best way was to get off the phone as quickly as possible and try again with another concierge or spa. After replying to a few "contact us" forms for a site I am working on, I thought I should at least let them know the feedback to see if indeed they are too busy or if it was just a bad time and I got the wrong concierge. In any case, I filled out their web form only to find it was a "post" action to "#" and didn't actually go anywhere.
They seem to have a lot of "best of boston" awards so the spa part might be nice - it just seems that half the challenge is to get to that part.

Wednesday, December 05, 2007

Jack Cards on the WSJ.

One of the startup web sites we have been working on and hosting at ProjectLounge, was reviewed by the Wall Street Journal. I am looking forward to taking the site to the next level - but a review like this is certainly a good start for the site. It is full of good suggestions and a positive conclusion.
"I think its system would be welcomed by many people looking for good-quality, well-designed cards that catch the eye. It saves users the hassle of last-minute trips to the card store, and still lets greeting cards keep their personal touch, which is what makes them such a permanent fixture in our lives."

Monday, November 19, 2007

Fire Truck Photos

This morning we saw a Fire Truck visit our street - no fire or anything exciting. However, I did capture some photos.

Tuesday, August 28, 2007

Commando Crawl


This our friends commando Teddy.

Friday, August 24, 2007