beyond clearplay

I've posted several comments on what I envision for the future of user content editable media.

http://programming-gone-awry.blogspot.com/2009/08/how-to-proxy-to-port-when-your-firewall.html

and

http://betterlogic.com/roger/?p=1043

Well the comments here are even more!

Pictures wrong sometimes?

Anybody else notice that pictures are sometimes...just wrong?

ping!

in reference to: Google News (view on Google Sidewiki)

Benefits of Firefox

The benefits of firefox:

the adblock plus plugin [plus myriad other plugins]
Reasonably speedy, well supported.

Drawbacks:

It takes up more RAM than Chrome (i.e. slower for older machines), renders, for me, slightly slower than Safari/Chrome [in windows, at least].

Written only to be able to say I was the first to sidewiki FF :)

in reference to: Firefox Browser | Free ways to customize your Internet (view on Google Sidewiki)

Beautiful

Google Earth is a beautiful thing--the stand alone app is smoother than the google maps web version.

[written just to be able to say I wrote the first sidewiki for google earth]

in reference to: Google Earth (view on Google Sidewiki)

how to compile ruby with increased garbage size

You may have noticed when you're running long-ish rails processes in windows that it sometimes pauses briefly, then continues on its merry way.

I have, at least.

The problem can be attributed to the garbage collector.

Ex: with one operation large_table.find(:all).each{||...}

I would receive something like 10 0.10 second pauses before it finished--increasing my overall time by at least 25%

Tweaking the malloc limit [the amount it will grow before doing a GC limit] changed that to something like 2 0.25 second pauses--which caused things to speedup considerably.

How:

install msysgit

c:> git://github.com/oneclick/rubyinstaller.git

cd rubyinstaller

set CCFLAGS=-DGC_MALLOC_LIMIT=60000000

rake # build ruby 1.8 in sandbox/ruby18_mingw

rake ruby19 # build ruby 1.9 in sandbox/ruby19_mingw

Installers hopefully coming soon.

Another option for speeding things up is building with GCC 4.4 instead of the 3.4.5 currently used.

Here's a download that does that for the adventurous:

http://all.faithpromotingstories.org/ruby_distros/ruby19_mingw_44.tgz

Anyway, watch this area as I'll post various ruby executables with different (faster) compiler options, and experimental like using tcmalloc, etc.

Thanks and good luck to all your windows+ruby users (like myself) out there!

Also much thanks to Luis and the one click group for making it possible to use these cool tools.

=r

how to proxy to a port when your firewall blocks that port [using mod_proxy instead]

For me, it was getting some external server to forward all requests on to that port.

I realize you can use ssh -Rport:host:port as well (through some other server), however ssh tunnelling was at times interrupted and didn't have a retry or restart, it seemed, at least that I knew of.

Anyway the final forward was:

1) create new subdomain that will forward it on.

2) edit apache's conf [apache.conf in my example] to forward things on

Order deny,allow

Allow from all

ServerName audio-mp3.ibiblio.org.8000.doachristianturndaily.info

ProxyPass / http://audio-mp3.ibiblio.org:8000/

DocumentRoot /home/rdp/www/faithpromotingstories.org

Order deny,allow

Allow from all

ServerName audio-mp3.ibiblio.org.8000.doachristianturndaily.info

ProxyPass / http://audio-mp3.ibiblio.org:8000/

3) install forwarders:

sudo a2enmod proxy

sudo a2enmod proxy_http

4) restart apache

5) enjoy the tunes [in my case].

how to setup a remote git repository using ssh

Finding the normal instructions slightly lacking on how to create a brand-new, empty git repository via SSH for a central location, here they are:


$ ssh myserver.com
Welcome to myserver.com!
$ mkdir /var/git/myapp.git && cd /var/git/myapp.git
$ git --bare init
Initialized empty Git repository in /var/git/myapp.git
$ exit
Bye!

Add the remote repository to your existing local git repo and push:

$ cd ~/Sites
$ mkdir myapp
$ git init
$ git remote add origin ssh://myserver.com/var/git/myapp.git
$ touch README
$ git add README
$ git commit -m "initial commit"
$ git push origin master

How to compile using visual studio for ruby

Extension building in ruby on windows is a little complicated, because the "mainstream" one click installer uses mingw for its build, which is based on a rather outdated msvcrt.dll [version 6 I believe--the one used by Visual Studio 6, no longer in production].

It is, however, according to Charlie Savage, possible to mix code built from newer compilers into it [1].

Mixing runtime libraries
========================

The most obvious criticism of this plan is that it will lead to mixing of microsoft runtime c libraries. From my experience this works as long as extension developers follow the rules described here:

http://msdn.microsoft.com/en-us/library/ms235460(VS.80).aspx

To be more concrete, this boils down to two simple rules:

* If you call ALLOC or ALLOC_N, use xfree and not free
* Don't call sprintf or printf in an extension, instead use rb_f_sprintf/rb_vsprintf/rb_io_printf

If an extension violate these two rules then its obvious, a segmentation fault happens. Thus these bugs are easy to find and easy to fix.

Since VC6 is thankfully no longer available, supporting msvc absolutely, positively requires mixing c runtime libraries and therefore extension writers must follow these two simple rules.

(Note that if you use rb_x_malloc and its siblings, they will use the right free call for you).

i.e. (summarizing) "If it uses ALLOC, xfree, doesn't use STDIN/STDOUT, it can link against any runtime it wants" [though nobu did note there may be a problem with errno's differing, probably not a problem.]

That being said it would appear that you need to also avoid the /MD compiler directive in MS compilers, as well [2].

Also, users may need to download a redistributable for the version of C compiler you used.

However, it should work.

Note that for distributing gems, you can use Luis Lavena's "rake compiler" to build your binaries for you [3].

Good luck!

[1] http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/22727

[2] http://groups.google.com/group/thin-ruby/browse_thread/thread/c15b2472eb97c2ba/ea7c5127035d193b

[3] http://github.com/luislavena/rake-compiler/tree/master ex: ext.cross_platform = ['i386-mswin32', 'i386-mingw32']

rdocs for all gems from rubyforge

I am pleased to announce the release of my site that has the rdocs for [almost] all existing rubygems.

Why?

Because it's convenient to have all rdocs installed in a single known place to be able to browse/search them. Because it's a central repository, too, it eliminates the need to install local rdocs for gems, which (when you turn it off) means you install local gems *much* more quickly. It makes me happy every time I do a "gem install" :)
It also eliminates the need for running a local gem server.

These gems' rdocs are all in the hanna theme, which provides for method search and an easy on the eyes layout. Though darkfish is also quite pretty, it isn't as easy to read because of font contrast.

Check it out!

http://allgems.faithpromotingstories.org/gems

core docs: http://coredocs.faithpromotingstories.org/

Feedback welcome.

Note also that they're using a temporarily subdomain url. If anybody is interested and could help me with a subdomain of a more ruby related url that would be cool. I hate to fork over that $10 a year for another domain you know me :)

Enjoy.
=r

How to save MUCH RAM when running rails (linode/slicehost) and mod_rails passenger

If you're using mod_rails on a VPS with little RAM [ex: linode, slicehost], then there are a few things you'll want to do to save RAM when running rails. Here are a few things I did to allow me to run multiple rails processes on one linode slice.

1: Install a 32-bit OS

Ruby uses twice as much RAM if you're in 64-bit than in 32-bit [and most other things do, too]. Use 32-bit! [linode has options to do this easily].

Savings: 50%

2: Don't use as many processes per rails app.

If you have low volume sites (and most of us do), then don't create too many processes per app.

PassengerMaxInstancesPerApp 1

3: If you are only using one process per rails app, then turn off the spawner process--it does you no help.

By default mod_rails spins up one "spawner process" and then "x" "actual working processes" The spawner thread just preloads rails so that it can be shared amongst instances of that app.

In my case, from top (~1 app process):

total mem, RSS, ... name

117m  49m 3156 S    0 13.8   0:00.93 ruby # spawner 49M
171m  89m 2240 S    0 24.7   0:00.61 ruby # instance 89m

fix:
use

RailsSpawnMethod conservative

in your apache2.conf

this results in [slower startup times and]

RAM, RSS...process name
143m  72m 3480 S    0 20.1   0:01.69 ruby

Savings: (assuming you only want one process per rails app): 49m (on 64-bit).

If you're worried about slow startup speed, you can set your rails processes to never expire[4].

4: If you use multiple processes per rails app, then set your spawner process to die quickly.

Though I haven't used it, theoretically the spawner process will be killed eventually--so set it to die quickly, to free up that expensive RAM[5].

You could do something like

ab -n10 -c10 http://yourhost

to ensure it fires up all the processes from the spawner, then the spawner is free to die quickly.

If you don't use REE, then don't use the spawner at all--little savings there RAM wise.

I have a suspicion that even with REE, you don't see much RAM savings, but having not used it, I can't say for certain.

4: use the MBARI patches to MRI.

Originally, one of my rails apps started at 89MB RSS then grew in total RAM usage by ~ 8MB per request, linearly. Odd? Yes. Fix: I recompiled ruby using the MBARI patches to 1.8.6/1.8.7. It now starts at 60MB RSS and stays there solid [1]. That's right--it stays solid at LESS than an unpatched ruby starts at. And with higher speeds [6].

Also avoid the ruby that comes bundled with ubuntu--though it uses the same amount of RAM as normal, it is compiled with pthreads enabled so it is slower.

Using 1.9 might also yield RAM savings like this. Haven't tried it though.

Savings: unknown since it appeared to growing forever (a lot though).

Overall Result:

With these suggestions in place, I can now run 3 or 4 rails apps on my "cheap grade" 360MB linode. Much better than the 1 I was able to originally.

Enjoy!

Other potential tricks:

use nginx instead of apache--faster, much better RAM usage. Potential savings:

Tweak mysql to use less memory [3]

Possibly tweak GC settings (37signals', evan weaver [2], etc.) though this appears to not increase speed too much over the MBARI patches [6].

refs:

http://groups.google.com/group/phusion-passenger/browse_thread/thread/df1fc1073dbef38
[1] http://www.ruby-forum.com/topic/170608#new
[2] http://blog.evanweaver.com/articles/2009/04/09/ruby-gc-tuning/
[3] http://articles.slicehost.com/2007/9/11/ubuntu-feisty-mysql-and-ror

[4] http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerPoolIdleTime

[5] http://www.modrails.com/documentation/Users%20guide%20Apache.html#_railsframeworkspawneridletime_lt_integer_gt

[6] http://www.nabble.com/-ruby-core:19846---Bug--744--memory-leak-in-callcc--to20447794.html#a21140287

state of the art in ruby compilation/JIT

There are several levels that one can take compilation of Ruby code to.
Ex: yarv compiles ruby code to yarv internal byte code. But there are other levels, which we hope to exploit in order to make a faster ruby. We'll discuss some different style of compilation (JIT and otherwise).


There are a few existent libraries that do translation.

ruby2c: translates code like

def fact(n)
 n*(fact(n-1))
end
fact(1000) # it needs this as a hint to know what type the fact method takes

to standard C (at least in all examples I've seen)

int fact(int n) {
  return(n*fact(n-1));
}

So their ansiC aspect is the most hard-core "I don't want this to lany of this in Ruby at all after it's done."

pros: fast as can be. cons: see below.

rubyinline, interestingly, wraps something like

"int fact(int n) {
  return(n*fact(n-1));
}"

with converters to call and return to and from Ruby, so you can call fact(1000) in your ruby code and it will work.

One interesting idea would be to pipe the output of ruby2c to rubyinline. Alas, ruby2c seems somewhat broken currently.

cons: you have to write in C. That is not what most of us want to ever have to do again.


I think where this perspectives fall apart is that they're not as flexible/dynamic as normal Ruby. It might have trouble with:

a = [/abc/, 'abc', 33]
b = 33
a.each{|thing| puts thing.inspect; b+= 1} # dynamic arrays probably not well handled, as well as ruby internal methods like 'inspect', blocks, etc.

or 

a = [/abc/, /def/]
a.each{|reg| if "abc" =~ reg then puts 'yes'; end}  # regex probably isn't supported, a lot of the rest of the stdlib

Then again, I haven't tried it since ruby2c doesn't even work for me.

pros: as fast as fast can be. also comes with a dependency walker so that it can "surmise" which classes of methods you'll be passing in, then it programs for just those.
cons: doesn't handle all ruby (AFAIK).
Maybe in its current state they're useful for doing mathematically intense operations? Dunno. Rubyinline is a little more useful, but requires writing in C.


Ruby2Cext takes a "ruby to ruby C" approach. Its original aim was to produce ruby compatible code equivalents in C.

i.e. the ruby:

def go
 3
end

translates to

VALUE go_c(VALUE self) {
 return INT2NUM(3); // uses all ruby types
}
void init_File {
 rb_method_register(someClass, "go", go_c, 0);
}

It also translates blocks and everything to its ruby c equivalents (using ruby c syntax) . For 1.8 that was said to yield a 2X speed increase.

They also added some plugins, one gives you the ability to "freeze" certain C method calls, i.e.
if it encounters String#strip it always calls straight to the C function for String#strip [thus avoids doing a ruby method call]

ex:

def go
 "".strip
end

Generates code that looks up String#strip and "saves off" the function's exact location [who would want to override String#strip, right?].

Thus the above ruby is converted roughtly to something like:

void *string_strip_function = Qundef;

VALUE go_c(VALUE self) {
  VALUE str = rb_str_new("");
  return strip(str);
}

VALUE strip(VALUE fromThis) {
   switch(GET_TYPE(fromThis)) {
     case STRING :
         return *(string_strip_function)(fromThis);// this avoids rb_funcall to lookup then call strip
     else:
 return rb_funcall3(rb_intern("strip"), fromThis);
   }
}    


void init_File {
 rb_method_register(someClass, "go", go_c, 0);
 strip_function = lookup_internal_ruby_c_method(rbString, "strip");
}

So you can see this avoids a few rb_funcalls to built-in methods, and is still almost entirely ruby compatible. This results in "up to 5x speedup" or so it says.

drawbacks to rb2cext: 1.9 compiles ruby to bytecode--perhaps rb2xext won't have as much a gain if used with a 1.9 VM, since it already does some of this work, though profiling would help evaluate this.

All the above examples were static compilers. Run once before runtime [or at eval time].

Another class would be JIT dynamic compilers. Jruby is really the only one that has anything like that currently, and yet somehow it doesn't yet seem to be quite as fast as 1.9 without a JIT [1][4].
There exists a "demo" JIT using ruby2c, as well [2]. It optimizes a single method, but at least shows you how you could do something more dynamic.

So where can/should the future of ruby interpreters lie? There's lots of things you could try.
A few are making it tighter in C, or making it more JIT'y, or making it moree "dependency walking" so it can pre-optimize the paths that are guaranteed to only have a certain class passed in to them.

re: making it tighter in C

One drawback currently to ruby2cext is that if you define

class A
 def this_method
    calls_this_other_method
 end
 def calls_this_other_method
 end
end

it will translate this (loosely) as

VALUE calls_this_other_method = rb_intern("calls_this_other_method"); // cache the symbol away
VALUE this_method_c(VALUE this) {
  check_right_parameters();
  return rb_funcall(calls_this_other_method);
end
VALUE calls_this_c(VALUE this) {
  check_right_parameters();
  return Qnil;
} 

Note that it did not optimize the call between this_method and calls_this_other_method, but required ruby do rb_funcall. This is the perfectly valid ruby way--you can override calls_this_other_method later and it will work with the new method, but in most cases after a warm up phase methods aren't overridden, so the rb_funcall could be avoided by calling the method directly. So we could hard code the calls to the other known ruby (now C) methods.

If we were to assume that class method definitions were "frozen" after a specific setup phase, then a few more optimizations would be available, the above being one of them. [3]
In other words, rb2cext could become more C-y, with direct calls from method to method.

Interestingly, ruby2c also has a RubytoRubyC component (I never saw any examples of it posted, and couldn't get it to run) which might attempt something similar.

At the same time, ruby2c could become more ruby-y, i.e. integrating with the stdlib [i.e. if you know a string is being passed in, and the command is

def go a
 a << '3'
end

then you could call rb_str_concat directly.

So there are two ways of action: one to make ruby to c translators more c-y, one to make them more ruby-y. Static compile time type analysis might make it so you can guarantee an object's class and optimize for it. That's an option.

Another option would be to create something more JIT'y. Profile to discover the "hot paths", then write C that optimizes for the common path (then you could make direct C calls outside your current class, too).

So what to do? Thoughts?
-=r

[1] http://blog.pluron.com/2009/05/ruby-19-performance.html
[2] http://github.com/seattlerb/zenhacks/tree/master
[3] For instance you could c-ify every ruby method in existence, with direct calls to any C methods within the same class (since we assume they all exist now). Ludicrous allows for this: http://betterlogic.com/roger/?p=1534 though I haven't experimented speed-wise.
[4] http://groups.google.com/group/ruby-benchmark-suite/browse_thread/thread/f56b4335cfd3ec57/c7babfb676d71450?lnk=gst&q=patch+gc#c7babfb676d71450 shows how 1.9 with a GC patch can be competitive to jruby.

how to install sqlite3 from source on ruby mingw (1.8 or 1.9)

here they are, inspired by a question from Charles Roper

1) install wget or curl in path

2) install unzip.exe (http://gnuwin32.sourceforge.net/packages/unzip.htm -- binaries link) in path, then put \bin in your path.

3) install rake compiler [gem install rake-compiler]

3.1) add the bin folder of GCC from the devkit to your  path [i.e. if you installed the straight devkit [1] then "set path=C:\ruby\devkit\gcc\3.4.5\bin;%PATH%"]
3.5) install msysgit, then git clone git://github.com/luislavena/sqlite3-ruby.git, then cd into it.

4) rake vendor:sqlite3

5) rake native gem

6) gem install pkg/xxx.gem

[1] http://programming-gone-awry.blogspot.com/2009/05/ruby-19-one-click-installer.html

mingw ruby 1.9/1.8 how to install mysql from source

inspired by a question from JD[1], here's the instructions:

Looks like the only way I know of is (assuming ruby 1.9 with devkit [2]) 

1) download mysql 5.0.x windows "without installer" 

2) unzip, put/rename it to c:\mysql 

3) download mysql 2.8.1 binary from tmtm http://www.tmtm.org/en/mysql/ruby/ 

4) edit extconf so that anywhere it says mswin32 it now says mswin32|mingw  

4a) make sure that mysql's bin are *not* in your path [might not be ncessary, but just in case] 

5) run ruby extconf.rb -- --with-mysql-include=c:/mysql/include --with-mysql-lib=c:/mysql/lib/opt 

6) run make, then run make install 

7) make sure libmysql.dll is in your path or in current directory 8) run ruby -e 'require "mysql"' -- should work.

[1] http://www.ruby-forum.com/topic/188199#new

[2] http://programming-gone-awry.blogspot.com/2009/05/ruby-19-one-click-installer.html

how to setup a complete mingw msys build environment for ruby (including dependencies)

Here's how: use Luis Lavena's pre built scripts to set it up for you (in this case to set it up for building a complete version of ruby), then tie in to the msys and mgit that it created for you.

steps:

download rubyinstaller's source, git clone git://github.com/oneclick/rubyinstaller.git

then build a version, "any version" of ruby with it, a la

cd rubyinstaller

rake # wait about 30 minutes :)

now set your path to "use" its newly downloaded, loaded with dependencies, versions of mingw and msys, a la build_env.bat contents (in my case):

set PATH=D:\dev\downloads\rubyinstaller\sandbox\mingw\bin;D:\dev\downloads\rubyinstaller\sandbox\msys\bin;D:\dev\downloads\rubyinstaller\sandbox\msys\usr\local\bin;%PATH%

now you can go anywhere and build anything (other versions of ruby, whatever),

example of building ruby trunk:

D:\dev\downloads\ruby_trunk>build_env

D:\dev\downloads\ruby_trunk>sh

Melissa@BLACKY /d/dev/downloads/ruby_trunk

$ autoconf && ./configure && make && make install

much thanks to luis for setting this up, and the mingw/msys team.

e text editor on virtualbox howto

howto compile e text editor on ubuntu, virtualbox:

install ubuntu on a virtualbox guest.

Now we need to create some extra swap file (more RAM, basically).

create a new disk (say 2G) -- you'll probably want it fixed size.

add it as secondary master to the image (shutdown guest-> settings -> hard drives).

add it as swap now

[in ubuntu]

sudo mkswap /dev/sdc

edit /etc/fstab

add in this line:

/dev/sdc none swap sw 0 0

check top to see the old swap size

reboot

check top -- your swap size should have increased considerly, now it should compile with default settings.

e text editor on andlinux howto

finally got the e text editor to work on andlinux! sweet!

how to:

1) install andLinux, choose coFS so you can see *all* your windows files.

2) resize the default partition to > 2GB default [1]

http://ext2resize.sourceforge.net/download.html

extract c:\program files\andlinux\toporesize.zip

shutdown andlinux

now run (for 10GB)

C:\Program Files\andLinux\ImageResizeTool\toporesize-0.7.1>tfile ..\..\Drives\base.vdi 10000

now restart andlinux

and within it:

sudo apt-get install build-essential -y # install gcc

now download ext2resize within linux http://ext2resize.sourceforge.net/download.html

install ext2resize, then run

    sudo ext2online /dev/cobd0 

3) "give yourself more swap ram" [at least 1GB for sure 2GB swap file works well] [2]

root@andLinux:~# swapoff /dev/cobd1

c:\Program Files\andLinux\Drives>type swap.vdi >>swap.vdi2

several times

or in ruby:

a = File.new 'swap.vdi2', 'w'

a.seek 2e9

a.write 'a'

4) checkout the git repository

sudo apt-get install git-core

git clone git://github.com/etexteditor/e.git

apply this patch:

http://e-texteditor.com/forum/viewtopic.php?p=13046#13046

install some unlisted dependencies

sudo apt-get install build-essential curl libgtk2.0-dev

then follow the directions within linux-notes.txt

Then follow the directions under the section "it all comes crashing down"

http://fixnum.org/blog/2009/e_on_fedora

And you *might* have a working andlinux e text editor.

[1] http://www.andlinux.org/forum/viewtopic.php?p=1581#1581

[2] http://www.andlinux.org/forum/viewtopic.php?p=956&sid=ebe8efafb867fe03519fcf166dbe104e

ruby 1.9 one click installer

Here's how to get a pretty easy working 1.9/1.8.6 mingw (from my post to ruby installers list)

for the files at

http://rubyinstaller.org/downloads/

quick install instructions (1.8.6, 1.9):

install a ruby one click installer to c:\Ruby

afterward, 
c:\Ruby now has
  bin\
  lib\
  share\

Now unzip the devkit.7z and move its contents into the same folder
(devkit is a bundled up mingw GCC+msys).

i.e.

c:\Ruby now has
  bin\
  lib\
  share\ 
 devkit\
INSTALL.txt

(note: the message "This folder already contains a folder named bin"
is expected--choose "yes" to confirm folder replace).

next add c:\Ruby\bin to the beginning of your path and enjoy the power
of a faster ruby with a compiler.

==More Detailed instructions==

Same as above--however, if you want to use a folder other than c:\Ruby
then please change devkit/msys/1.0.11/etc/fstab to properly point to
the right directories for mingw and /usr/local

How to add ruby\bin to your path:
either do this on the command line thus:
set PATH=c:\ruby-1.9.1-p0-i386-mingw32\bin;%PATH%

or change the PATH variable within my computer -> properties ->
advanced -> environment variables (add it to the front of path within
"system variables").

== How to determine success ==
If successful you should get an output like (for 1.8):

C:\>ruby -v
ruby 1.8.6 (2009-03-31 patchlevel 368) [i386-mingw32]

C:\>gcc -v
Reading specs from C:/Ruby/devkit/gcc/3.4.5/bin/../lib/gcc/mingw32/3.4.5/specs
...

and installing binary gems should work properly:
C:\>gem install json
Building native extensions. This could take a while...
Successfully installed json 
...

If a gem fails to compile (ex: because of lack of headers), you can at 
times get away with doing a
gem install gemname --platform=mswin32
this installs the msvc binaries for it. They are *typically* compatible. It's safer, however, to download the headers and compile it locally, and submit your process upstream so they can release a quality mingw binary :)

Enjoy!

how to create a new git repository (fresh one)

http://toolmantim.com/articles/setting_up_a_new_remote_git_repository finally something that works :)