Ramblings Of a Madman

Yeah, like i would know a good description…

Posts for Tag: Uncategorized

Souls and determinism

Physical world Big bang Soul outside the system Everything would be deterministic Death never happens We are our fathers, mothers, and everyone else The concept of thing is a fallacy

Should go to bed now -)

Bought a new mac book pro. Like with most new toys, I cannot stop playing with it. It's 3.30 in the morning and I should go to bed. But somehow the evil apple still holds me tight in front of the laptop :) Posted via email from nocivus' ramblings

Some notes on scalability

I have been wanting to write this forever, also so I could remind myself of it whenever needed. And this is the kind of subject that traverses most programming languages/frameworks/systems as long as you are dealing with large quantities of data. There are probably more things one can do in order to allow an application to scale up easily, but these are the ones I have put in place in the past and it has worked out nicely so far: Read/write splitting (master/slave replication) If possible (e.g. if you are not using some kind of really restrictive database access framework (think django, as it is right now :)), put in place mechanisms to allow you to read data from multiple slave DB machines with some strategy to choose between them (round robin, last used, most load, etc.). This will allow you to scale up reading from your database. Streaming directly from the database to prevent a lot of in-memory data coming from DBI came in contact with this particular problem when having to send clients a bunch of processed data from a database. Whenever possible shift all the work on the data to the database (SQL) and then stream the data to the client through your application. What I mean by this is: don't keep data in memory in your application as most as possible. For instance MySQL, by default, will not stream the data to your application; it will actually keep the rows in memory and then send them to the client. If you have requests of 50 Mb worth of data you can see where your server is going after 30 concurrent requests (hint: morgue). CachingCache your requests whenever possible. Use an out of the box solution (like squid for example) or develop your own strategy to cache data, so you hit your databases as least as possible.Data discretization This one is related to caching. Figure out clever ways of discriminating your data so it's easier to cache. For instance, imagine you are serving a bunch of data, that people submit to you, to everyone else. Anytime someone submits something you will have data that can be sent to everyone else. Instead of making this data immediately available, make it discrete. Serve batches of data, say, once a day. This will allow you to cache daily requests easily with squid, for example. Hibernate second level cacheFor java/spring/hibernate developers out there: Do make use of hibernate's second level cache. Especially for content that does not change that often. It will pay off especially for web interfaces. For data distribution I would recommend not using hibernate at all. Think plain JDBC with some (earlier mentioned) different caching solution and streaming the results directly. Data redundancy where it makes senseDon't worry about sacrificing disk space on the database cluster in favor of faster query times. Especially good when searching content on your database. That one extra redundant column in your data table will make the search much faster than using joins all over the place. These are my thoughts on scalability. I would love to hear about your experiences and solutions on this field. Posted via email from nocivus' ramblings

236 hours until my first airplane jump

Getting excited, so I setup an online stopwatch hahahaha :Dhttp://www.7is7.com/otto/countdown.html?year=2008&month=12&date=20&hrs=10&ts=24&min=0&sec=0&tz=0&lang=en&show=h&mode=r&cdir=down&bgcolor=%23FFFFFF&fgcolor=%23000000&title=Countdown%20to%20my%20first%20airplane%20jump Posted via email from nocivus' ramblings

Way to wait for all threads to finish in Python

Not sure if it's the best way, but I found this example in an webarchive page and it kind of makes more sense than keeping a list of the threads and cycling all the time checking if they are active or not: while threading.activeCount()>1:    sleep(1)Not sure if this works, but have to try :)Posted by email from nocivus (posterous)

My computer

My computer is amazing. He (I will always refer to it as an entity, even though I am aware that *it* is still not aware of itself) is capable of executing a tremendous amount of operations per second and, yet, it's not capable of remembering and asking for (in a smart way) where do I want to store something I download. I am aware that there are probably a lot of (academic?) projects out there set to change this and make machines do what they are actually supposed to do: make my life easier. There is one movie that recently as reinforced this idea in my mind (no, not minority report) and that is the new Iron Man movie. That is exactly how I see computers: Working with us to actually help us be more creative, productivem, and efficient. Not just dumb machines that know how to store bits, but extensions to our own work. That being said, and my frustration put out in the open, I ask if there is anyone out there doing anything even remotely close to this: Creating some kind of smart interface on top of (any, preferably linux ;)) operating system that is gifted with some kind of, at least, memory and inquisitive engine that, when presented with something it does not understand, would ask and learn, instead of throwing an exception or a segmentation fault. Here's a use case (directly adapted from Iron Man): I'm browsing the web and I find a torrent for a new cool linux distro that I want to try out. I tell my computer "download this" and, by now, he already knows that a torrent is supposed to be download in a given way. What he does not know is where to store the actual contents of the file. I mean, there are just so many drives in my home network, all discoverable through avahi, bonjour, or other service discovery architectural flavour. So where should he put it? He asks and I tell him (tell being the keyword, think speech recognition, but not literally like "c:/somewhere/distros/linux", more like "create me a new place for linux distros"). The next time he should still ask, but this time if I want it in the linux distros also. And I should be able to say something like "from now on, store all distros in that place, unless I say otherwise when downloading", if I feel really lazy". That is the kind of stuff I would expect to be available today, when the amount of computational power in the world is imense, but personal computers are still as dumb as they were in the times of punctured cards. Alright, maybe an overstatement, but still... :)Posted by email from nocivus (posterous)

Optimizing external (USB) hard drives for performance on WinXP

I just found a really cool option on Windows XP for USB disks.  It seems that, if you go to the device manager and check the properties on an external (USB) hard drive, there is an option to optimize the drive for performance.  This means that windows will cache some data on the other drives to make the access faster. The downside is that you *always* should remove the drive using the proper "safely remove hardware" icon on the system tray.  But since I always did it anyways, it's a waste not make the drive faster :)Posted by email from nocivus (posterous)

Android development

Being the android emulator slow as it is, I decided to give it a try running it from the command line. This is just a mental guide, so bear with me. The first thing we should do is adding the android SDK tools folder to our path, so we can run the required commands from anywhere. On my Ubuntu I added the following to my ~/.bashrc file:

export PATH=$PATH:/home/{user}/development/android/sdk/android-sdk-linux_x86-1.0_r1/tools

Next, we can query the emulators running by issuing the following command:

adb devices

There should be none, so far. Now let's start the emulator. I pass some extra arguments to be able to make it boot a bit faster (remove the nojni parameter if you required jni stuff ;)):

emulator -nojni -netfast -no_boot_anim

If you query for devices you should get one on the list returned by adb.

Once the emulator is up and running, we can install our previously built application (if you used the default skeleton, created with activitycreator (google, google), then after using ant to build the app, go to the bin folder in the project). Issue the following command:

adb install {project name}.apk

Now the project is available on the emulator, under the menu.

Easy, uh? Posted by email from nocivus (posterous)

New pictures

After a lot of requests (Ok, only my mother asked for this) I decided to put online new photos. But first, a quick explanation:1. I finally finished my apartment's floor. Actually most of the things are now in place for a comfortable living on which looks like a normal apartment, at least by my Portuguese standards (and no, I'm not saying there are no old houses in Portugal, it's just that there are more new ones ;)); 2. My father and Aunt (his sister) came to visit the Netherlands for the first time (actually my aunt had already been to Amsterdam, though a long, long time ago). All in all it was a great time. We rented a car to be able to go to more places (like the windmills in Zanse Schans, and Scheveningen, in Den Haag) and it's actually cheaper than paying for train tickets. Mental note: Consider renting a car for going to the beach in the summer. Oh, wait, take that back, there is no summer in the Netherlands;

3. I got a cat! His name is Mangas. I wanted to name him Kili, after the mount Kilimanjaro, but my ex-girlfriend convinced me otherwise, so he ended up being called Mangas (which is kind of a dumb Portuguese joke/synonym about Kilimanjaro). The reason for the hommage is that he literally climbed 2 walls while I was trying to grab him, in the house I went to get him from. I'll repeat, he climbed 2 slick walls using nothing but his claws and he stayed close to the ceiling while I tried to get him to come down. So, those are the news, and without further delay, here are the pics for the finishing touches on the new apartment in Amsterdam, my father and aunt's visit to the Netherlands, and my new cat, Mangas.

Posted by email from nocivus (posterous)