Programming gone awry

Thoughts and tutorials on programming

Wednesday, September 23, 2009

Pictures wrong sometimes?

Anybody else notice that pictures are sometimes...just wrong?

ping!

in reference to: Google News (view on Google Sidewiki)

Benefits of Firefox

The benefits of firefox:

the adblock plus plugin [plus myriad other plugins]
Reasonably speedy, well supported.

Drawbacks:

It takes up more RAM than Chrome (i.e. slower for older machines), renders, for me, slightly slower than Safari/Chrome [in windows, at least].

Written only to be able to say I was the first to sidewiki FF :)

in reference to: Firefox Browser | Free ways to customize your Internet (view on Google Sidewiki)

Beautiful

Google Earth is a beautiful thing--the stand alone app is smoother than the google maps web version.

[written just to be able to say I wrote the first sidewiki for google earth]

in reference to: Google Earth (view on Google Sidewiki)

Wednesday, August 05, 2009

how to compile ruby with increased garbage size

You may have noticed when you're running long-ish rails processes in windows that it sometimes pauses briefly, then continues on its merry way.

I have, at least.

The problem can be attributed to the garbage collector.
Ex: with one operation large_table.find(:all).each{||...}

I would receive something like 10 0.10 second pauses before it finished--increasing my overall time by at least 25%

Tweaking the malloc limit [the amount it will grow before doing a GC limit] changed that to something like 2 0.25 second pauses--which caused things to speedup considerably.


How:
install msysgit
c:> git://github.com/oneclick/rubyinstaller.git
cd rubyinstaller
set CCFLAGS=-DGC_MALLOC_LIMIT=60000000
rake # build ruby 1.8 in sandbox/ruby18_mingw
rake ruby19 # build ruby 1.9 in sandbox/ruby19_mingw

Installers hopefully coming soon.

Another option for speeding things up is building with GCC 4.4 instead of the 3.4.5 currently used.
Here's a download that does that for the adventurous:

http://all.faithpromotingstories.org/ruby_distros/ruby19_mingw_44.tgz

Anyway, watch this area as I'll post various ruby executables with different (faster) compiler options, and experimental like using tcmalloc, etc.
Thanks and good luck to all your windows+ruby users (like myself) out there!

Also much thanks to Luis and the one click group for making it possible to use these cool tools.

=r

Monday, August 03, 2009

how to proxy to a port when your firewall blocks that port [using mod_proxy instead]

For me, it was getting some external server to forward all requests on to that port.

I realize you can use ssh -Rport:host:port as well (through some other server), however ssh tunnelling was at times interrupted and didn't have a retry or restart, it seemed, at least that I knew of.

Anyway the final forward was:

1) create new subdomain that will forward it on.

2) edit apache's conf [apache.conf in my example] to forward things on

Order deny,allow
Allow from all
ServerName audio-mp3.ibiblio.org.8000.doachristianturndaily.info
ProxyPass / http://audio-mp3.ibiblio.org:8000/
DocumentRoot /home/rdp/www/faithpromotingstories.org

Order deny,allow

Allow from all

ServerName audio-mp3.ibiblio.org.8000.doachristianturndaily.info

ProxyPass / http://audio-mp3.ibiblio.org:8000/

3) install forwarders:

sudo a2enmod proxy

sudo a2enmod proxy_http

4) restart apache

5) enjoy the tunes [in my case].

Saturday, August 01, 2009

how to setup a remote git repository using ssh

Finding the normal instructions slightly lacking on how to create a brand-new, empty git repository via SSH for a central location, here they are:


$ ssh myserver.com
Welcome to myserver.com!
$ mkdir /var/git/myapp.git && cd /var/git/myapp.git
$ git --bare init
Initialized empty Git repository in /var/git/myapp.git
$ exit
Bye!

Add the remote repository to your existing local git repo and push:

$ cd ~/Sites
$ mkdir myapp
$ git init
$ git remote add origin ssh://myserver.com/var/git/myapp.git
$ touch README
$ git add README
$ git commit -m "initial commit"
$ git push origin master

Thursday, July 23, 2009

How to compile using visual studio for ruby

Extension building in ruby on windows is a little complicated, because the "mainstream" one click installer uses mingw for its build, which is based on a rather outdated msvcrt.dll [version 6 I believe--the one used by Visual Studio 6, no longer in production].

It is, however, according to Charlie Savage, possible to mix code built from newer compilers into it [1].


Mixing runtime libraries
========================

The most obvious criticism of this plan is that it will lead to mixing of microsoft runtime c libraries. From my experience this works as long as extension developers follow the rules described here:

http://msdn.microsoft.com/en-us/library/ms235460(VS.80).aspx

To be more concrete, this boils down to two simple rules:

* If you call ALLOC or ALLOC_N, use xfree and not free
* Don't call sprintf or printf in an extension, instead use rb_f_sprintf/rb_vsprintf/rb_io_printf

If an extension violate these two rules then its obvious, a segmentation fault happens. Thus these bugs are easy to find and easy to fix.

Since VC6 is thankfully no longer available, supporting msvc absolutely, positively requires mixing c runtime libraries and therefore extension writers must follow these two simple rules.

(Note that if you use rb_x_malloc and its siblings, they will use the right free call for you).

i.e. (summarizing) "If it uses ALLOC, xfree, doesn't use STDIN/STDOUT, it can link against any runtime it wants" [though nobu did note there may be a problem with errno's differing, probably not a problem.]

That being said it would appear that you need to also avoid the /MD compiler directive in MS compilers, as well [2].

Also, users may need to download a redistributable for the version of C compiler you used.

However, it should work.
Note that for distributing gems, you can use Luis Lavena's "rake compiler" to build your binaries for you [3].
Good luck!
[1] http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/22727
[2] http://groups.google.com/group/thin-ruby/browse_thread/thread/c15b2472eb97c2ba/ea7c5127035d193b
[3] http://github.com/luislavena/rake-compiler/tree/master ex: ext.cross_platform = ['i386-mswin32', 'i386-mingw32']

Tuesday, June 23, 2009

rdocs for all gems from rubyforge

I am pleased to announce the release of my site that has the rdocs for [almost] all existing rubygems.

Why?

Because it's convenient to have all rdocs installed in a single known place to be able to browse/search them. Because it's a central repository, too, it eliminates the need to install local rdocs for gems, which (when you turn it off) means you install local gems *much* more quickly. It makes me happy every time I do a "gem install" :)
It also eliminates the need for running a local gem server.

These gems' rdocs are all in the hanna theme, which provides for method search and an easy on the eyes layout. Though darkfish is also quite pretty, it isn't as easy to read because of font contrast.

Check it out!

http://allgems.faithpromotingstories.org/gems

core docs: http://coredocs.faithpromotingstories.org/

Feedback welcome.

Note also that they're using a temporarily subdomain url. If anybody is interested and could help me with a subdomain of a more ruby related url that would be cool. I hate to fork over that $10 a year for another domain you know me :)

Enjoy.
=r

Thursday, June 18, 2009

How to save MUCH RAM when running rails (linode/slicehost) and mod_rails passenger

If you're using mod_rails on a VPS with little RAM [ex: linode, slicehost], then there are a few things you'll want to do to save RAM when running rails. Here are a few things I did to allow me to run multiple rails processes on one linode slice.

1: Install a 32-bit OS

Ruby uses twice as much RAM if you're in 64-bit than in 32-bit [and most other things do, too]. Use 32-bit! [linode has options to do this easily].

Savings: 50%

2: Don't use as many processes per rails app.
If you have low volume sites (and most of us do), then don't create too many processes per app.

PassengerMaxInstancesPerApp 1

3: If you are only using one process per rails app, then turn off the spawner process--it does you no help.

By default mod_rails spins up one "spawner process" and then "x" "actual working processes" The spawner thread just preloads rails so that it can be shared amongst instances of that app.

In my case, from top (~1 app process):
total mem, RSS, ... name

117m 49m 3156 S 0 13.8 0:00.93 ruby # spawner 49M
171m 89m 2240 S 0 24.7 0:00.61 ruby # instance 89m

fix:
use
RailsSpawnMethod conservative
in your apache2.conf

this results in [slower startup times and]
RAM, RSS...process name
143m 72m 3480 S 0 20.1 0:01.69 ruby
Savings: (assuming you only want one process per rails app): 49m (on 64-bit).

If you're worried about slow startup speed, you can set your rails processes to never expire[4].

4: If you use multiple processes per rails app, then set your spawner process to die quickly.

Though I haven't used it, theoretically the spawner process will be killed eventually--so set it to die quickly, to free up that expensive RAM[5].

You could do something like
ab -n10 -c10 http://yourhost
to ensure it fires up all the processes from the spawner, then the spawner is free to die quickly.

If you don't use REE, then don't use the spawner at all--little savings there RAM wise.

I have a suspicion that even with REE, you don't see much RAM savings, but having not used it, I can't say for certain.

4: use the MBARI patches to MRI.

Originally, one of my rails apps started at 89MB RSS then grew in total RAM usage by ~ 8MB per request, linearly. Odd? Yes. Fix: I recompiled ruby using the MBARI patches to 1.8.6/1.8.7. It now starts at 60MB RSS and stays there solid [1]. That's right--it stays solid at LESS than an unpatched ruby starts at. And with higher speeds [6].

Also avoid the ruby that comes bundled with ubuntu--though it uses the same amount of RAM as normal, it is compiled with pthreads enabled so it is slower.

Using 1.9 might also yield RAM savings like this. Haven't tried it though.

Savings: unknown since it appeared to growing forever (a lot though).

Overall Result:

With these suggestions in place, I can now run 3 or 4 rails apps on my "cheap grade" 360MB linode. Much better than the 1 I was able to originally.

Enjoy!

Other potential tricks:

use nginx instead of apache--faster, much better RAM usage. Potential savings:

Tweak mysql to use less memory [3]

Possibly tweak GC settings (37signals', evan weaver [2], etc.) though this appears to not increase speed too much over the MBARI patches [6].

refs:

http://groups.google.com/group/phusion-passenger/browse_thread/thread/df1fc1073dbef38
[1] http://www.ruby-forum.com/topic/170608#new
[2] http://blog.evanweaver.com/articles/2009/04/09/ruby-gc-tuning/
[3] http://articles.slicehost.com/2007/9/11/ubuntu-feisty-mysql-and-ror
[4] http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerPoolIdleTime
[5] http://www.modrails.com/documentation/Users%20guide%20Apache.html#_railsframeworkspawneridletime_lt_integer_gt
[6] http://www.nabble.com/-ruby-core:19846---Bug--744--memory-leak-in-callcc--to20447794.html#a21140287

Blog Archive

Followers