Thursday, September 28, 2006

Why I often implement things from scratch

Once upon a time there was an Erlang programmer who needed an FTP server running on one of the hosts in a private network. In fact he didn't need an FTP server, he just needed to transfer files between a central server and his client machine, but he thought that he needed an FTP server to do this.

He searched for FTP servers, and indeed found several of them. They had to be free of course, because even though the organisation he worked for had loads of money the administrative procedures for buying such a product were considerable. The program also had to have the right form of license, so that the legal department would be happy.

He downloaded several such servers, some of them wouldn't compile, and even if they did compile they had to be correctly configured in order to work, and this was not easy.

Suddenly our programmer was stuck by the thought that he might be able to write an FTP server himself and that writing the FTP server might be quicker than finding and installing an FTP server that somebody else had written.

"What do I want to do?" he asked himself.

"List files on a remote directory, copy files, between a local and remote machine, etc"

Then he remembered that the project he was working on used distributed Erlang.

"This must be easy," he thought. He was right.

So ... on the server machine he give this command:

>$erl -name server -cookie ASDRTFERT
1> node().
'server@host1.somenet.com'
[See this link for a description of what these commands do]. Then he went to another machine and typed this:
>$erl -name client1 -cookie ASDRTFERT
1> node().
'client1@host23.somenet.com'
Now he had started two erlang nodes. Now the nice thing about distributed Erlang is that you can easily run code on any of the nodes, so to check that he could access the server node from the client node, the programmer typed this:

2> rpc:call(server@host1.somenet.com',
erlang, node, []).
'server@host1.somenet.com'
Which is what would have happened if the command had been issued on the first machine. Now our programmer knew that he could evaluate any function on the remote machine, just as if it had been on the local machine. The correspondence is as follows:

If the local command was:
> file:get_cwd()
Then to perform this on the server all he had to do was call:
> rpc:call(server@host1.somenet.com',file, get_cwd, []).
So to list files on the remote machine, he did this:
1> rpc:call(server@host1.somenet.com',
file, list_dir, ["."]).
{ok, ["Makefile",
"readme",
....
Then he decided that he wanted to copy the Makefile to his local machine so he wrote this:
2> {ok, Bin} = rpc:call(server@host1.smenet.com',
file, read_file, ["Makefile"]).
<<".SUFFIXES: .erl .beam .yrl" .....>>
3> file:write_file("Makefile", [Bin]).
ok
At this point all that typing in the shell became tedious, so he fired up emacs and wrote this:
-module(myftp).
-export([get_file/1]).

get_file(F) ->
{ok, B} = rpc:call(server@host1.smenet.com',
file, read_file, [F]),
file:write_file(F ++ ".copy", [B]).
Then he compiled and tested his program:
4> c(myftp).
{ok,myftp}
5> myftp:get_file("Makefile").
ok
He then added a few bells and whistles and was done.

Moral

If you have the right tools it's often quicker to implement something from scratch than going to all the trouble of downloading compiling and installing something that somebody else has written.

This is a true story, but I've just written this code. Writing this code and the blog entry took about the same time as it would to find and install an FTP server on my machine.

6 comments:

Jonathan Allen said...

> Once upon a time there was an Erlang programmer who needed an FTP server running on one of the hosts in a private network. In fact he didn't need an FTP server, he just needed to transfer files between a central server and his client machine, but he thought that he needed an FTP server to do this.

Can anyone name a single OS that doesn't have a built-in FTP or File Sharing system?

Anonymous said...

Joe,

Very true. Many people ignore the hidden costs of using third party tools, which are exactly as you describe, finding, testing, makeing sure its what you want, and worse, keeping up to date with new versions.

Like you Erlangers, as an APL programmer, I know I have the right tools ;)

PS I have been enjoying your writings on concurrency.(Dyalog APL has neatly integrated multithreading into APL)

Unknown said...

Justin, installing a ftp server is nice and fast, but programmatically scripting it will require more lines of code than the erlang code shown above.

Anonymous said...

I have written my own javascript libraries, my own database abstraction layers, and a host of other libraries. I do it for a different reason though. The first is the same as yours, it is often quicker to write your own than to find someone else's that is appropriate. Secondly, if something goes wrong I know where to look, and can fix it easily. And finally, if I want to add a feature, it only takes me a few minutes rather than a few hours traversing through someone else's unfamiliar code.

I have at times spent longer than I should have on these projects, especially in testing them, but I always come out on top and now I have a solid little library that I personally own. No problems with licenses either because it is all my work.

Food for thought :) I enjoyed your post!

Unknown said...

I can see where the author is coming from. It's not necessarily hard to install an FTP server; it's hard to pick one. How are you supposed to know which one has a huge bug that corrupts all downloads? How are you supposed to know which author has no concept of buffer overflow problems? Which program uses its own flat file for user auth instead of tying in with the operating system's PAM? It can be frustrating to wade through the mass of open source projects.

Alvaro Videla said...

Hi Joe, certainly this article is really cool. I'm reading your pragprog book to learn Erlang. When I tried the code in this post I found that the calls to rpc:call(server@host1.smenet.com' ... are missing the one quote before server.

When I first try the code it failed for me, but after I found the problem. Maybe another beginner like me finds this useful.