[ Dear diary.... Yeah, I know. This is another way-too-long pointless blog. But I can't stop writing when I start. And I might find it funny to look back in a few years, at how my 'battles' have evolved... ]

Technology is wonderful.

Trying to get technology working exactly like I want it to, is a whole lot more frustrating.

For years, I've felt like I was constantly battling technology. Especially since 2006, when I started trying to use Linux with 'not very exotic but still not extremely common' hardware and peripherals. (Like my 'PVR' PC, a webcam, and a bluetooth headset.)

Lately I feel like I'm slowly gaining ground in those battles again, and can use technology the way I want it... mostly. But we'll see how long it lasts, because there can always be a new struggle right around the corner...

Linux

Using Linux is still a constant struggle for me. I hate to admit it, because I know the system pretty well so I'd like to keep using it... but it's true.
(Or at least it has been since 2006. In the 10 years before that, I guess I didn't care about peripherals that much. And in 1999 I started using Windows as my 'OS for daily use' when I started programming VB. So for a long time I only did 'network related stuff' in Linux, which it absolutely rocks at, compared to Windows. So I loved it.)

I have spent days, weeks, reading documentation on how stuff worked. Back in the days, I didn't care. I'm a tinkerer who wants to know how stuff works. But since 2006 it has taken the form of being constantly disappointed that I can't find how stuff works or that it doesn't work yet. And coming back again and again to retry or recompile stuff has been reallllly tiring.

Since last year (summer 2008) I am seriously considering switching to an iMac at the time my trustworthy daily-use Dell laptop needs to be replaced. I am just tired of the wasted time and frustrations, and just want stuff to work. And Linux, regrettably, is still at the "almost there but not yet" phase.

Since I don't have a lot of money to spare, I'm waiting - and we'll see what my verdict on Linux is when the time comes.

Postfix

My relationship with Postfix feels like a marriage, in part. I weighed all the options as much as I could beforehand, and decided that Postfix was the one for me. And I'm still fond of it, and never planned on switching to anything else. (Especially not after a Windows-loving friend switched his web and mail hosting to Apache+Postfix on Linux and is ecstatic about the lack of maintenance issues he's encountered.) But still... I've run into some 'personality quirks' of Postfix, which I have spent years trying to change. So often I thought I 'got it' and knew how to get it to work 'my way', and still failed. In the end just reconciled myself with the fact that the world isn't perfect, and adjusted my wishes to Postfix' personality & way of thinking.

It's just hard for me to believe that a piece of Linux server software is unable (or unwilling) to do something. But it is.

That particular thing is: I wanted to use one domain both for 'virtual mailboxes' (i.e. configure addresses in a database that have no corresponding UNIX account) and as the source & destination address of all system accounts on a server. This could apply to several 'one-server domains' I have, like wyz.biz. It would be good for sending automatically generated system mail (from e.g. 'www-data' and 'root' users) from a well known domain, handling replies to those mails in the same fashion as all other ('virtual') e-mail addresses/domains that are set up, and getting a 'catch-all' address for the domain(s) in $myhostname/$mydestination.

Well, it turns out you can get most of this done, but only if you approach (/ work around) it the exact right way. And there is no documentation pointing you in that exact right way.

Since the start of 2004, I have had three different servers and four entirely different postfix configs on which I have spent days and days of configuring, testing, documenting how all its options and how they exactly influenced each other, in my case... and then the next time discovered that there was some variable that I hadn't taken into account, and my whole world view collapsed again and I had to start all over.
Really, postfix isn't that straightforward. It's not apparent what 'address classes' are in all cases and there are no real world examples. It's not apparent that the relationship between $virtual_mailbox_domains and $virtual_mailbox_maps is not equivalent to the relationship between $virtual_alias_domains and $virtual_alias_maps. It's not apparent when $local_transport should be redefined and when $mailbox_transport should be redefined, and which other variables need changing, when you want to deliver 'local' mail into you IMAP server directories directly. It's not immediately apparent, which things (in mail headers and SMTP conversations) are governed by $myhostname, which by $mydomains, and which by $myorigin.

But I've made my peace with it now. With my fourth extensively-personally-commented config file, I think I have a point of reference where I can do everything I need, in the way postfix wants me to. And I can forget about how I totally went into the wrong direction the previous three times. What the problems exactly were, is a fading memory. I have grown to appreciate it again, for its stability and scalability and extensibility and configurability IF you treat it in exactly the right way. I again can say that don't need any other mailer.

I just hope it stays this way from now on.

(And ofcourse I'll forget about the constant unresolvable crashes from a few years ago for any $relay_domain, which caused me to just not relay anything anymore. And I've disregarded anti-spam setups, whose possibilities are getting nicer with each Postfix version but which take ages and ages of documentation reading and setup each time I review 'current best practice in anti spam land'...)

The LCD TV and my windows system

I hope and expect that nowadays, they suck less than when I bought one. That was at the start of 2006, when they were gaining popularity but weren't very widespread yet. After spending lots of time on consumer sites, I got a Toshiba 42WL58P, a 42'' that I could hook a computer up through VGA, which many TVs could not do (for €2850).

Back then there were no 'true HD' screens, only screens sold under the strange marketing term "HD Ready", which were 1366x768 pixels. (Or actually, there was a range of Philips TVs with a 1920x1024 display... but when you hooked up a computer to it, it could only display 1024x768. WTF!?)

So then the fun started. When I hooked up a spare desktop computer to it, I could only get a 800x600 screen to display correctly on the TV. And to my surprise, there was NO documentation with the TV that gave any clue... it said that anything over 1024x768 was unsupported over VGA!
I had already seen postings from people on TV forums, saying they got 1360x768 or 1368x768 to display. (Because 1366 was impossible for a computer.) I spent ages reading up on how someone did that with my video card... Finally, after tracking down and installing a special unlicensed video driver for Windows (and lots of warnings that monitors could be blown up with wrong settings), I got it done.

The 'new' PVR

Three months later, in may 2006, I finally bought a computer with decent power to display movies and a decent form factor, to sit under my TV. Because of course I didn't want some crippled media center box, I wanted a PC I could use exactly as I wanted it.

Even deciding on which one to get, took lots of time. According to the lack of models being offered and lack of feedback found, there were not as many people wanting/doing the same thing in my country, as I had hoped. And I didn't have that much time to waste at that point, so in the end I just went out and bought the only one I saw that looked like 'a real video recorder case', to fit with the rest of my stereo set.

Luckily, the Windows installation on that machine was set up already for 1368x768 display correctly, so I could immediately watch my downloaded TV series / movies in a nice way (Just using VLC from my desktop. There was Windows MCE on there, but I never really used the MCE part... The interface was nice, but I didn't see the point for just watching downloaded stuff. Just opening Explorer and doubleclicking the relevant icon was good enough for me.)

One thing was too bad, though: I could not get 'pixel-on-pixel' display through VGA. The TV's pixels were not properly aligned with the video card's output and were a little wider or smaller.. so there was some kind of strange 'wavey' artifact on the display, when displaying text (especially non-antialiased text).

Booting Linux on my PVR

Then Linux needed to be configured. Because of course I didn't want to mess with all kinds of separate codecs that had to be downloaded and installed separately from hopefully-not-dodgy sites, back in those days... (That's for Windows experts and I've never tried to be one.) I just wanted to use mplayer on Linux! (And I wanted this box to play my MP3s - on Linux!) And I wanted to see whether I could get a pixel-perfect image with X.

So I booted a Debian Stable (sarge) CD, and... the kernel hung immediately after booting.

So I booted the latest Debian Testing (etch) beta CD, and... again, the kernel hung immediately after booting.

This was an Intel motherboard, and not the latest one either. (This PVR had been taken out of production already and I had the last one from the store.) So I really did not expect this. I still don't know what was so special about this motherboard.

After a long time of trying different boot options, I could get it to boot. Unfortunately, then USB would not work - and this computer had no connections for non-USB keyboards, so I still could not run the installer! In the end, after many tries, I had burnt a modified 'etch' installer CD with a modified kernel that could boot and use the keyboard - and with some CD swapping I could install as system that at least booted, and fix everything from there.

Frustrating? Yes. Timewaster? Yes. But that struggle with Linux (or rather, Debian) was won in the end.

Configuring X on my PVR

Back then, configuring X had gotten to be less of a complete pain than in the years before. That much had changed for the better -- in general at least. Unfortunately, not for my 'exotic' screen needs of 1368x768 with an exceptionally low(?) vertical refresh rate. And I again spent ages reading up on 'ModeLines' and how to set them in my xorg.conf, looking for example calculations on the web, etc. In the end I discovered values that I felt more or less comfortable of trying, expecting my TV not to be blown to bits... But nothing worked. The screen ignored my ModeLines, without a proper hint as to why.

Finally, I found that some stupid setting in my xorg.conf needed to be set for my specific video card, in order to make ModeLines even work... Argh!!

So then the pain was over?

Well... I guess. Things weren't perfect because I could still not get 'pixel-on-pixel' display, also by tweaking ModeLines under X.

But I never really cared anymore, because... the movies and TV series I downloaded, seemed to display perfectly. Also a few years later, when the general video quality was 'upped' from DivXes that were 300MB per episode to 'x264's that were 1GB per episode... stuff still looked great. A little to my surprise, I have never really had to care that my display was not perfect 'pixel-on-pixel'. (I wasn't reading text files on my TV anyway.) So I just left it.

So I call that a draw between my and the TV. (And a whole lot of wasted time.)

By the way, my PVR-computer had a DVI output. My TV had HDMI inputs - and I had no fitting cable between output and input plugs. Since I almost everything I heard and read about my TV (including the official manual) said that the maximum HDMI input for my TV was 1024 pixels wide and messing with it could destroy the TV... I decided to not waste nearly 100 bucks on a cable just to try if I could get it working. It would probably have gotten me pixel-perfect display if it did... but I haven't cared enough about that yet. It was a draw. Let's leave it.

Linux and sound on my PVR.

I guess this was my biggest disappointment in Linux and overall time waster. The PVR had an Intel motherboard with integrated sound. So Linux should support it, right? Alsa had developed enough, right? By 2006, I could just buy any product from a well established manufacturer, and it would work, right?

Wrong. The D915POM motherboard (with 7.1 surround sound plus digital output) is apparently rare, or something. It uses the 'hda-intel' driver which is seeing lots of development since 2006, but the sound (ALC880 chip) on a D915POM motherboard didn't work in 2006 - and the corresponding bug there is still open, so I bet it'll never work. Just goes to show: you still always need to check exactly what you buy.

I've spent ages trying figure out what the exact problem was, looking through existing bug reports for clues, filing my own bug report... nothing worked. As happens more often with projects 'in progress', I could not figure out from the existing documentation exactly what was the problem. But I suspect I can't fix it myself without learning C and hacking hda-codec.c. Or something. I think. (And I don't have time for that.)

After a month or so, I admitted defeat. I bought a cheap little stereo sound card in a USB dongle, and have been using that ever since, to connect the PVR to my sound system. (Lucky I didn't have a surround set or decent speakers.) Hey, in the end, this way worked too. Just ignore the fact that I have a 7.1 surround sound card that I can't use.

Windows networking

I have to name this.

I hate it. I hate it. I hate it.

...maybe because I don't know the details of it... and I can't be bothered to get into the details of the ever-changing windows networking specs. I don't see why I should want to.

I just know that smbclient on my computer NEVER works to connect somewhere, whatever I do. And I just know that I could install Samba on my Linux PVR, and connect with a windows Desktop to it... but my Windows laptop with the exact same configuration, refused to connect. And I could find no clues in debug logs as to why. (Which I'm not used to. Usually, with Linux you can at least get to see weird unintelligible error messages that you can google on.)

Oh, the wasted evenings. I hate it. Maybe I just don't know enough about it, but still. I don't even want to. I want to cherish my hate.

So that's a battle lost. Luckily, I have stopped using Windows for anything real, since I stopped making money doing MS Access/VBA/Excel/SQL Server work.

Linux and webcams

Just like my PVR, I just rushed out and bought a cam of 'an established brand' when I needed one, expecting that to work on Linux.

And just like before, I was disappointed.

It turns out that Logitech cams and Logitech cams can be entirely different. They have different drivers. To make things worse, one of the more obscure drivers was forked. The fork started supporting different models of cameras that the original didn't, and vice versa. To make things worse, the forked version replaced another older driver (which was named differently), but this was not explained anywhere, causing all kinds of confusion that I spent some time to diffuse.

(Of course, none of these drivers were incorporated in the Linux kernel.)

Well... after spending time on figuring that out... I discovered that the webcam didn't really match my expectations anyway, and bought a better model.

...for which I needed yet another driver, that was incorporated in the kernel... at least, an older version that didn't work for my camera, was. I had to make sure I downloaded a newer version from somewhere else, every time I upgraded my kernel. (But that problem went away over time and my camera is supported by the stock kernel now.)

So, this battle was 'won' in the end... by support for my hardware being included in the kernel. Which is fine; I don't expect 100% of things to work immediately on Linux without some effort. It's just too bad that there was no real documentation on this, and I had to spend several evenings figuring things out and testing different drivers myself...

Linux sound in 2007: enter PulseAudio...

On the new laptop that I bought in 2007, everything worked. (Well, almost. There was a little problem with my screen (i915 videochip) not turning on after a suspend. But that was solved in 'only' one afternoon of googling, so I guess I can almost disregard that.)

But over the course of time, it seemed like things stopped working for me in the sound area. I don't know why... maybe it was my different usage, or newer versions of software that stopped working... but only later, I discovered that programs would hang. When I was playing an MP3 on my computer, Iceweasel(Firefox) would hang or crash at the moment it started to play sound. Same problems with Skype.

Apparently only a single program could access the sound device. (Or the programs were programmed incorrectly, so they would take control of the whole sound card instead of 'playing nice with others'. I'm unsure on the details of ALSA being able to handle this.) And the fact that both Skype and the only really working Flash player for Iceweasel (the Adobe one) were proprietary software, of course didn't help either.

Enter PulseAudio, a sound server for Linux which can act as an intermediary between programs and the sound card, and mix all audio streams together. It looked to be 'the future', to be adopted by major distributions. So I upgraded my laptop to the latest 'testing' Debian packages, got my music player working with it, and started configuring Flash & Skype with it.

Was that easy? Heck no. (Again, probably because of those things being proprietary software.) All kinds of ALSA/OSS tricks needed to be done to make Flash and Skype work. Sound kept stuttering for Skype, which could be fixed by changing some internal settings (whose meaning I don't get) in PulseAudio... but on the next PulseAudio upgrade, these didn't work anymore. Debian documentation referred to a 'pulseaudio flash support' library that stopped existing a year before.
Which was ofcourse again different after an upgrade from Flash 9 to Flash 10. So that became at least half a year of intermittendly reading all kinds of conflicting / possibly-outdated web pages with 'the best setup for...' and trying everything out. But there didn't seem to be any other way for me to both play MP3s and YouTube videos...

Bluetooth

And then I bought a stereo Bluetooth stereo headset for my phone so I could listen to music on my phone without hassle. And of course I wanted to use the headset + dongle on my computer too...

(One thing in between: this is one thing where Linux really was better than my Windows XP with its lack of a standard bluetooth stack. Not to say that Linux was good, but Windows was unworkable. My phone came with one bluetooth stack for Windows that I needed to install, and my headset with another one, and after I installed my headset, my phone's bluetooth wouldn't work anymore. What's worse, often my headset would just stop working and the computer would refuse to recognise it. When that happens on Linux, it's irritating - but at least if all else fails you can trace and kill all programs using bluetooth, unload the kernel driver, and load everything again. And then you can make a script to do that for you. Tedious, but it works. On Windows, you can't trace or unload anything. I regularly had to log out or reboot the computer if I wanted to get my headset working again. Which is stupid.)

Anyway, on Linux this was also back to square one, for 'making sound work the way I want it to'. Because of course, Bluetooth audio was only in its infancy on Linux.

So... lots of searching and trying and seeing no good documentation. Having to constantly upgrade PulseAudio + the bluetooth stack to try stuff out in desperation. Having to look through all the low level bluetooth programs on my computer to see what they actually did. And most of all, finding absolutely no way to specify the 'pin code' that was needed to establish the connection between my computer and the headset. Then finding a little c program to do that (which I had to compile myself)... only to discover that in a later version of the bluetooth Debian packages, even that didn't work anymore. *sigh*.

There was only one program at that moment, which I could use to reconnect my headset again. It was a Gnome panel applet (and I had no idea how it worked) so I was forced to run Gnome. (Just when I had settled on the Enlightenment DR17 default look as a nice fancy but clean looking environment that made other window managers look clunky )

Incidentally... this is the moment that I turned into 'just a user, who does not know how his programs exactly work'.)

Luckily, after some time, one of my desparate search efforts yielded a wonderful python-dbus script that could totally reset and reestablish the bluetooth connection. It was only a minor annoyance that I had to run this every time I turned the headset on; at least it worked!
(Again, I had no idea how it worked, because I don't know the internals of 'this dbus stuff'. Still 'just a user'.)

...until a few months later, a totally rewritten version of the bluetooth stack hit Debian testing. And this python script gave errors about dbus methods not existing anymore. AND the gnome applet didn't work either anymore. So I was stuck without any proper lead to get things working again!

After about two months, finally a graphical client (written for Gnome, but able to work without gnome-panel running1) hit Debian. So I can run a graphical program again, and click 'delete connection' and 'reconnect' buttons every time I turn my headset on. We're back to "slightly annoying but workable".

Current state of PulseAudio & Bluetooth

Since a few months (end of summer 2009) stuff works well. I can finally call my Linux sound setup 'mature', because I don't have to mess with 'custom emulated device setup in ALSA configs' anymore. Both Flash/IceWeasel and the latest Skype version have native PulseAudio support. And, PulseAudio now works well with the new bluetooth stack. It automatically detects my bluetooth headset as an extra PulseAudio output device, the moment I turn it on. (Still, often a bluetooth connection can't be established, so I need to do the 'forget device and reconnect' thing by hand in that graphical client... but I'll survive that.)

That is... until last week, when 'pavucontrol' stopped working when I have my headset installed. And again, I need this graphical client to change volume -- I'm such a "regular user" nowadays I know 'pacmd' and 'pactl' exist, and I tried all pactl volume-related commands, but the sound from the headset still is very bad (which probably happens when some volume-value somewhere is >100%). Anyway, I can't be bothered to dig around more for this, so I'll take my wired headset for now, and wait until the bug gets fixed...

WiFi

Another thing where I am 'just a regular user' (like bluetooth audio), is wireless networking. I need a graphical client to set up the connection, because I just don't know how the setup mechanism works and I don't know any command line scripts for it. I've only started using WiFi on Linux in the fall of 2008, when I moved to my 'new apartment' in Budapest; before that, I used either wires or Windows.

Now, being a 'regular user' without knowing all inner workings of the connection wouldn't bother me... if there actually were good support programs for it. But there was no graphical client for my needs. That is - there was only one (NetworkManager) but that not only seemed to work only half the time for me.... it also %&^%*^ required you to run Gnome, because it only worked as a Gnome applet. (And I don't hate Gnome... but I prefer my nice and clean Enlightenment environment, thanks.)

As a good Linux enthusiast, I proceeded to look at the internals of how things were set up, 'under the graphical hood'. And I actually found the config files and scripts to set up a connection on WEP networks. And I was lucky enough to have a crappy WiFi network with WEP encryption at my apartment. So at least I was able to connect and work from home, after some hours reading.

WPA was totally different. The man page of 'wpa-supplicant' on Debian was really horrible for me to get through. I just could not understand how this program ran and when it was called... because it wasn't just one script being called from the command line to set stuff up. Debian had made the key exchange mechanism all 'better'. Which meant that all documentation was out of date. That's one problem with Linux... in a system that is developing, good documentation only follows about a year afterward, and until that time it's a real gamble. Google didn't help me and all the documentation about WPA for Debian pointed to outdated documents. I was never sure if I was reading "the modified Debian way for calling wpa-supplicant", "old and outdated WPA documentation", or "wireless documentation that didn't have anything to do with WPA, but with the totally different way of setting up WEP connections".

This turned into about three full nights of pointless frustration. I've never been able to figure out how to configure WPA, and Gnome's NetworkManager was illogically behaving crap that totally stopped working for me. Also, even on the unsecured network in my local McCafe, I could not connect by command line scripts/config, as I could at home. And I had no good error messages to see why. Grr. At that point I (again) really wished I had a system that 'just worked', like a Mac.

Until... suddenly... I came across 'wicd', a wireless connection daemon plus graphical client with no strings attached, that worked in a logical way. I couldn't believe I hadn't found it months earlier! It wasn't in Debian then, but I found a .deb repository for it, and have been using it ever since. (It isn't perfect, sometimes it behaves in the wrong way i.e. when not putting network interfaces down before trying to switch to another network and then failing. But at least it's predictable and you can get used to its quirks. That's workable for me; after all, I am still a Linux user )

Then in the following months... my whole WiFi subsystem broke down a few times after a necessary kernel upgrade. Or for months on end it refused to connect after a hibernate/resume cycle, because the newer firmware would have bugs. The regular Linux frustrations, it seems.

But since 2009, I'm happy with it. This is not a 'battle with technology' that I won... Just one that I endured until my system finally matured.

Recurring system problems (pam_mount and hibernation)

I really really really hope that now the sound problems are over, the new bluetooth stack is properly working (and WiFi is working stably for nearly a year), I can stop tracking the Debian testing distribution and new kernels constantly.

Because the annoying thing about it is: stuff just stops working. Like was the case with WiFi last year, as mentioned.

I've had the pam-mount facility (which I use to mount my encrypted home directory) stop working twice in the past 2 years, because Debian upgraded its PAM mechanism. Suddenly my pam-mount password was not being asked anymore upon login. Which has had me 'log in as root, mount my home directory, log out and THEN log in as myself' for about a year in total, because twice I couldn't figure out where the configuration error was, after multiple sessions of reading and trying stuff. Just last week I got things working again. And documented things, again, hoping that I would recognise the problem if it occurred next time. AGAIN.

(I found by now that I should ditch pam-mount anyway... and encrypt my whole disk using dm-crypt+LUKS, or something. Instead of this one-device encryption with loop-aes, which 'was new and shiny 10 years ago'... I'll find the time to do a complete new setup. Sometime.)

And right now (that is, a few months ago), tuxonice (hibernation) stopped working. That has happened multiple times in the past too. It hibernates, but does not resume - and I can't see why. *sigh*.

It's not as bad when I am home (because I can leave my laptop on) but irritating when in the train or at a client. And I just don't want to delve into the system anymore, to trace the TuxOnIce errors... and to see why my kernel isn't outputting anything to the screen at the moment when I start up (because of funny not-working framebuffer stuff that I installed, probably)... I just want a system that just works. The way I want it. *slight sob*

So...

So yeah. I'm a bit wary of the battles.

And I may just switch over to Macs, with my next computer. On the other hand, I know what I have right now, and can't really be sure if I'll be driven crazy by Apple making me jump through some of their hoops, either.

Maybe now that the constant emergence of 'new battles' has subsided... if laptops don't have get any any real new hardware technologies anymore... I'll be able to live without surprises, after I solved the one remaining (hibernation) issue. Then maybe I'll just buy a nice trustworthy 'PC laptop' again, with proven-to-work hardware, and install Linux on it again. After all, it feels a bit like family. I've invested so much into it that it's hard to let it go.

(Although what I've heard about the small time it takes to suspend their laptops, is a very big plus... That's another story.)

• 1. or, at least one of the $#%#$^$#$ two half-documented forks that are currently in Debian testing, is... if they take that fork/gnome-panel-less functionality away again, I'll be stuck again or I'll have to run Gnome again...