They were looking for volunteers to marshal at the K2 rally around Kielder forest at the weekend, I decided after having enjoyed the enduro last year I helped with last year, I’d give it a go.

Even the trip up to Bellingham that morning turned out to have its interesting moments. The A68 is a very straight Roman road with lots of steep dips with blind summits. I had the bike on a bike rack on the back of the vehicle and naively set the cruise control. I soon found that it would plummet down the dips like a stone, then come to the other side, realise the incline needed throttle and that the speed was rapidly decreasing so it would open the engine right up. It would do this up to the summit of the hill with an effect like applying rocket boosters. The only way to describe the result is “big air”. Thankfully the air suspension does appear to be able to cope with this, much more gently and gracefully than I expected.

I’d not actually been able to get much information other than what time to turn up, even where was a little bit vague. The rally was two 80 mile loops of mostly forest fire road. The day was looking to get rather warm and the fire roads would be extremely dusty.

Firstly, I found out there were two refuelling points and the fuel trailer was leaving ASAP so I put 5L on each and hoped for the best since I had no idea where I’d be. At this point I didn’t even know what the organiser looked like. I figured out who he was and was assigned to the group manning the first section. We set off and followed the course around getting a feel for where we were supposed to be although the section with two way traffic where it loops back on itself confused us.

We shortcut some of the section and reached the first refuelling point. We decided to split up with some going to do traffic management on the two way section and some of us figuring out our shortcut to loop back to the start and the enduro loop we were responsible for.

I found the GPS invaluable at this point. We had a paper map of the route and comparing the trace on screen with that gave some idea of where we were, even if the scale seemed incorrect and the route had changed a bit from the one marked.

The final part of the course has some horrible bends covered in large rocks which I really hated, our enduro loop was good fun, starting off with brash, a drop and hill climb, then a nice green lane forest track.

I did have one interesting moment where both wheels lost traction and I was travelling along the track semi sideways doing a lowside in slow motion. I remember thinking that it felt like I could just tweak it back into line however there was a significant risk of turning it into a highside. I gently tried anyway and much to my surprise the front wheel started working properly again, even if the rear was fishtailing like crazy which was a much less serious issue. I think that stands as the single best recovery I’ve ever made on a motorcycle, I just wish I had it on camera.

So we made it back to the start and waited around a bit for the bulk of the riders to head through, then set off ourselves. We soon came to a fallen rider who was in a bad way. The fractured wrist was clear, the back pain worrying but they were conscious and alert which was a big plus. We had a few marshals around and some riders had summoned emergency services having been unable to get the satellite phone link to the organisers to work properly. I went back to the start and escorted paramedic to the scene who decided to call for air support. At this point there were too many people around the scene so I moved further down the course. The marshalls there were going to flush the first section so I skipped to the first refuel and refuelled, watching the air ambulance come in from a distance. We realised no marshal had been over the section to the first test so agreed I’d change plans and head over that way.

I rode for what seemed like hours without seeing another person, eventually reaching the special test. Great, except there was no safe route back to the start that I could see, so I did the only thing I could and continued onwards. I stopped and checked a couple of people dealing with punctures were ok. I also found someone who’d had an off and damaged their wrist, I advised just following the course since there was no shortcut out that section of forest I knew of. Eventually I came to the second refuel point and refuelled. I waited there for a bit and was able to direct the injured rider on a bit of a shortcut back.

The course was due to close at 2pm and I was supposed to be helping close it so I didn’t look at the enduro loops and headed back to the start, meeting up with another start marshal who’d had to find fuel point two since his fuel was there. We made it there slightly late but were the first there. Two others left to close the course, I was left manning the tape. I spent the next four hours there since I’d been told that I was not to move on account of anything, or let any rider onto the course. I was pleased when the closing marshals finally appeared.

Sunday started an hour earlier and for us, had the added fun of demarking. The riders would only do the first section on the first lap, then it was getting closed and demarked. The course would close at 1pm and then we’d sweep through and close/demark the remaining parts to the start of the special test.

The day started with ensuring the first section was clear and all the correct gates were open. It had rained overnight which meant there was more grip and less dust. There was a nasty bog in the first section which was interesting. I picked a line and took some reasonable speed into it. The bike threatened to stop half way but I really wasn’t keen on that, opened the throttle and put the weight on the back. Much to my surprise the front lifted to head height and the YZ did what it does best and powered onwards. I was in a deep rut at this point and using the front wheel for steering is overrated anyway . I was mildly concerned that there might be something in front but there wasn’t and I was able to land safely on the other side. Another of the marshals hadn’t faired quite as well but we all got through in the end.

Some of us went to figure out who the last man was so we could start the demarking, I went on with another to check the rest of the course to the first test. With riders setting off behind us, we were needing to get a move on but I was finally starting to remember how to ride fire roads (inside line on corners instead of road riding stick to the left for example). We were able to back track from the special test a little and short cut back to the start where we refueled.

When we got back, demarking had started but the gates needed locking. No problem, I can do that so I set off. This meant I was off by myself and reached that bog. This time around I tried a different line and managed to ground the bike out. I therefore spent quite a while extracting it and ended up rather muddy. Moments like that where the bike is on its side and your foot is trapped under the front wheel are always good fun. With the gates shut we went to the start for course closing. On the way there I found someone struggling with a stuck throttle after an off. I pointed out the cable was jammed by the barkbuster and helped them free it. A short while later I also loaned out an allen key to stop the guard catching their brake lever.

With the course closed, we set off, demarking as we went. I got the awkward signs to do since I had the sidestand. Obviously getting off the bike for each one is a pain so you attempt to ride the bike through some interesting obstacles. Occasionally you’d ride into what looked like a nice safe area, only to find for example it was full 1ft sized lumps of rock covered in moss. All good fun though. We demarked all the way to the special test, then passed on the batten to the team there. We’d passed a broken down rider so we went back with a tow rope for them and took some short cuts to pull them out the forest. I then manned the tape for half an hour before calling things done and headed back to camp and then home.

The bike is going to need some TLC now, the rear tyre was worn at the start, now its totally knackered. My number plate is also disintegrating and the sidestand spring gave out when I was unloading the bike. My hands now have blisters on the blisters and muscles I’ve never felt pains from before are aching. All things considered it was a good weekend .

The first weekend in June was the Northumbria TRF camping weekend (as it is each year). Sadly numbers were down this year, we’ve wondered if the switch to the electronic version of trail had something to do with this but there were still many old faces and some new ones.

I arrived at the site near Hexham on the Friday evening, most people had already arrived and a pleasant evening was had around the campfire. We were entertained by Neil’s outdoor jacuzzi which he’s long since threatened to build. I’m sure next year the minor design issues will be resolved and people will be queuing to use it!

Saturday morning started off looking a bit damp but it is Northumberland and it looked like it would burn off. The group was an interesting mix of bikes and people, some older riders and some riders who were relatively new to trail riding. The 690 on semi offroad tyres wouldn’t have been my choice for the day. I only wish I’m doing as well as Bob is if I manage to reach that age!

We headed off in a south easterly direction, quickly turning northwards, crossing the Tyne and off into the wilds of Northumberland. As we went North the lanes grew greener and more overgrown. There were a few nice river crossings but the water levels were well down compared to normal making them quite enjoyable. The aim of the day seemed to be that if you were going to fall off, don’t let anyone see and then it didn’t happen! Despite some valiant efforts some group members were spotted although nobody did spot the moment I had my legs flailing around with the bike veering in every direction but the one I wanted to go in. I never did actually come off although how, I have no idea.

The first part of the route was in a light annoying drizzle and it was hampering the readability of the GPS but I am starting to get a feel for where the lanes are. I managed to avoid the tour of a housing estate as featured the last time I lead this route though and thankfully the drizzle stopped once we started heading turning towards the west. The pace was whatever each rider felt comfortable with as we were waiting at the lane ends.

After a river crossing, some of the group failed to show up so after a while I went back to find out what was going on. It turned out Nic, my backmarker’s exhaust had fallen off. Those with a good memory would remember this featuring in the run I did last year too. We’d even been joking about this at the campsite that morning. I have started carrying a small reel of lockwire, I was wondering how much this was worth at this point. Equally, I wanted to get to lunch so we wired the exhaust back on and continued, stopping for fuel in Scots Gap a short while later.

Just before the lunch stop we bumped into one of the other groups going the other way. They’d already had lunch and were debating what to do about a breakdown but there wasn’t much we could do to help. We eventually made the lunch stop sometime after 3pm having found the first choice of cafe shut. It had been a tiring but enjoyably morning.

We set off from there through Wark forest, I believe Nic had to tweak the exhaust mounting again. We made the final fuel stop and then back to camp, being the last group back and having covered 138 miles.

On Sunday we were feeling the effects of the day before so we decided not to go as far and get back a bit earlier. The weather was lovely. One local rider decided they’d done enough the day before so wouldn’t be joining us. A trip over Plenmeller common and then looping around finding the A686 started the day. The 690 headed back to camp at this point as they wanted to get home for a particular time so then we were five. Since its good to have a challenge we went over Limestone Brae which is a steep decent, through a river and then a steep rocky climb though a set of switchbacks on the other side. We then continued up over Long Cross, another rocky climb which I quite enjoy in that direction (I dislike coming down it, not least with the bike on top of me as has happened at least once).

We stopped for fuel and some food in Alston which we took up to the stop on the top of Hartside where we talked with some other local trail riders and watched all the bikes that were out.

After lunch we headed back to Alston, looped around some trails there (note to self to reverse the loop next time as I prefer rocky climbs than descents) and then it was over to Nenthead and Tynehead. As we drew closer to Hexham, it was clear people were starting to get tired, making silly mistakes and dropping bikes, me included having cross rutted it trying to avoid something I should have just ridden through.

I think everyone enjoyed the weekend and everyone made it safely back, including Nic and his exhaust. I live in hope that he will fix the bracket for the next time so we could at least have some variety in the breakdowns! By the time I got home I was totally exhausted.

Yesterday I was asked to help solving a small sed problem. Considering that file (don’t look too closely on the engineering of the defined elements):

<root>
  <key>key0</key>
  <string>value0</string>
  <key>key1</key>
  <string>value1</string>
  <key>key2</key>
  <string>value2</string>
</root>

The problem was: How to change value1 to VALUE!. The problem here is that you can’t blindly execute a s command matching <string>.*</string>.

Sed maintains a buffer called the “pattern space” and processes commands on this buffer. From the GNU sed manual:

sed operates by performing the following cycle on each line of input: first, sed reads one line from the input stream, removes any trailing newline, and places it in the pattern space. Then commands are executed; each command can have an address associated to it: addresses are a kind of condition code, and a command is only executed if the condition is verified before the command is to be executed.

When the end of the script [(list of sed commands)] is reached, unless the -n option is in use, the contents of pattern space are printed out to the output stream, adding back the trailing newline if it was removed.3 Then the next cycle starts for the next input line.

So the idea is to first, use a /pattern/ address to select the the right <key> line, append the next line to the pattern space (with the N command) and finally run a s command on the buffer now containing both lines:

  <key>key1</key>
  <string>value1</string>

And so we end up with:

$ cat input 
<root>
  <key>key0</key>
  <string>value0</string>
  <key>key1</key>
  <string>value1</string>
  <key>key2</key>
  <string>value2</string>
</root>
$ sed -e '/<key>key1<\/key>/{N;s#<string>.*<\/string>#<string>VALUE!<\/string#;}' < input 
<root>
  <key>key0</key>
  <string>value0</string>
  <key>key1</key>
  <string>VALUE!</string
  <key>key2</key>
  <string>value2</string>
</root>

now, that was a title of a Decemberists song that I’d have never expected to use as a blog post title

I did announce it on foundation-list, given that it impacts my position on the Board of Directors of the GNOME Foundation, and I did a sneaky tweet as well, but I guess the blog (and Planet GNOME) is still the Old Fashioned Way™ to do these things — and seeing that Cosimo beat me to a punch, it’s worth saying that I have joined Endless Mobile as well.

my last blog post about my work life was a bit depressing, I guess; I received a ton of support and encouragement from many, many people — too many to thank effectively in the narrow margins of this blog. I did take the announced month off, and I was already on my way to recovery; then I met Matt, who told me about Endless, and what they were trying to do with GNOME, and I felt the absolute need to help them as much as I could. after all, aren’t we trying to make GNOME a viable proposition for OEMs and OSVs to take and put on their own devices? I’m sure we’ll be able to start telling the community at large more details about what we want to achieve, and how.

in the meantime, I expect to see people in San Francisco a bit more often (though I’m still going to be based on London for the foreseeable future), and I’ll obviously be at GUADEC in Brno.

For the last week I’ve been using a Geeksphone Keon as my only phone. There’s been no cheating here, I don’t have a backup Android phone and I’ve not taken to carrying around a tablet everywhere I go (though its use has increased at home slightly…) On the whole, the experience has been positive. Considering how entrenched I was in Android applications and Google services, it’s been surprisingly easy to make the switch. I would recommend anyone getting the Geeksphones to build their own OS images though, the shipped images are pretty poor.

Among the many things I missed (Spotify is number 1 in that list btw), I could have done with a countdown timer. Contrary to what the interfaces of most Android timer apps would have you believe, it’s not rocket-science to write a usable timer, so I figured this would be a decent entry-point into writing mobile web applications. For the most part, this would just be your average web-page, but I did want it to feel ‘native’, so I started looking at the new building blocks site that documents the FirefoxOS shared resources. I had elaborate plans for tabs and headers and such, but turns out, all I really needed was the button style. The site doesn’t make hugely clear that you’ll actually need to check out the shared resources yourself, which can be found on GitHub.

Writing the app was easy, except perhaps for getting things to align vertically (for which I used the nested div/”display: table-cell; vertical-alignment: middle;” trick), but it was a bit harder when I wanted to use some of the new APIs. In particular, I wanted the timer to continue to work when the app is closed, and I wanted it to alert you only when you aren’t looking at it. This required use of the Alarm API, the Notifications API and the Page Visibility API.

The page visibility API was pretty self-explanatory, and I had no issues using it. I use this to know when the app is put into the background (which, handily, always happens before closing it. I think). When the page gets hidden, I use the Alarm API to set an alarm for when the current timer is due to elapse to wake up the application. I found this particularly hard to use as the documentation is very poor (though it turns out the code you need is quite short). Finally, I use the Notifications API to spawn a notification if the app isn’t visible when the timer elapses. Notifications were reasonably easy to use, but I’ve yet to figure out how to map clicking on a notification to raising my application – I don’t really know what I’m doing wrong here, any help is appreciated! Update: Thanks to Thanos Lefteris in the comments below, this now works – activating the notification will bring you back to the app.

The last hurdle was deploying to an actual device, as opposed to the simulator. Apparently the simulator has a deploy-to-device feature, but this wasn’t appearing for me and it would mean having to fire up my Linux VM (I have my reasons) anyway, as there are currently no Windows drivers for the Geeksphone devices available. I obviously don’t want to submit this to the Firefox marketplace yet, as I’ve barely tested it. I have my own VPS, so ideally I could just upload the app to a directory, add a meta tag in the header and try it out on the device, but unfortunately it isn’t as easy as that.

Getting it to work well as a web-page is a good first step, and to do that you’ll want to add a meta viewport tag. Getting the app to install itself from that page was easy to do, but difficult to find out about. I think the process for this is harder than it needs to be and quite poorly documented, but basically, you want this in your app:

if (navigator.mozApps) {
  var request = navigator.mozApps.getSelf();
  request.onsuccess = function() {
    if (!this.result) {
      request = navigator.mozApps.install(location.protocol + "//" + location.host + location.pathname + "manifest.webapp");
      request.onerror = function() {
        console.log("Install failed: " + this.error.name);
      };
    }
  };
}

And you want all paths in your manifest and appcache manifest to be absolute (you can assume the host, but you can’t have paths relative to the directory the files are in). This last part makes deployment very awkward, assuming you don’t want to have all of your app assets in the root directory of your server and you don’t want to setup vhosts for every app. You also need to make sure your server has the webapp mimetype setup. Mozilla has a great online app validation tool that can help you debug problems in this process.

Visiting the page will offer to install the app for you on a device that supports app installation (i.e. a Firefox OS device). Not bad for a night’s work! Feel free to laugh at my n00b source and tell me how terrible it is in the comments

Day 1

it turns out that this week wasn’t the best possible to hold a hackfest in Boston and Cambridge; actually, it was the worst. what was supposed to be the first day passed with us hacking in various locations, mostly from home, or hotel lobbies. nevertheless, there were interesting discussions on experimental work, like a rework of the drawing and content scrolling model that Alex is working on.

Day 2

or Day 1 redux

with the city-wide lockdown revoked, we finally managed to meet up at the OLPC offices and start the discussion on Wayland, input, and compatibility; we took advantage of Kristian attending so we could ask questions about client-side decorations, client-side shadows, and Wayland compatibility. we also discussed clipboard, and drag and drop, and the improvements in the API that will be necessary when we switch to Wayland — right now, both clipboard and DnD are pretty tied to the X11 implementation and API.

after lunch, the topic moved to EggListBox and EggFlowBox: scalability, selection, row containers, CSS style propagation, and accessibility.

we also went over a whole set of issues, like positioning popups; high resolution displays; input methods; integrating the Gd widgets into GTK+, and various experimental proposals that I’m sure will be reported by their authors on Planet GNOME soon. it was mostly high level discussion, to frame the problems and bring people up to speed with each problem space and potential/proposed solutions.

we’d all want to thank OLPC, and especially Walter Bender, for being gracious hosts at their office in Cambridge, even on a weekend and the GNOME Foundation.

this is a PSA: if you’re thinking about submitting a talk for GUADEC 2013 in Brno, you have a week to do so.

One of the new features I was able to contribute to GNOME 3.8 was the sharing settings panel.

The goal of this panel is to provide the user with a way to control what is shared over the network. The sharing services are provided by various existing projects, including Vino, Rygel and gnome-user-share. If any of the services are not installed, the relevant settings are not displayed. The panel also allows the user to configure various options for the services.

More details about the design of the panel are on the wiki page.

Debugging problems which mean init goes crazy is tricky, especially so on modern Intel hardware that doesn’t have anything resembling a serial port you can connect to.

Luckily this isn’t a new problem, as Linux supports a network console which will send the console messages over UDP packets to a specific machine. This is mostly easy to use but there are some caveats that are not obvious.

The prerequisites are that netconsole support is enabled, and your ethernet driver is built in to the kernel and not a module. Luckily, the stock Yocto kernels have netconsole enabled and the atom-pc machine integrates the driver for my hardware.

Then, on the target machine, you pass netconsole=... to the kernel. The kernel documentation explains this quite well:

netconsole=[src-port]@[src-ip]/[],[tgt-port]@/[tgt-macaddr]

   where
        src-port      source for UDP packets (defaults to 6665)
        src-ip        source IP to use (interface address)
        dev           network interface (eth0)
        tgt-port      port for logging agent (6666)
        tgt-ip        IP address for logging agent
        tgt-macaddr   ethernet MAC address for logging agent (broadcast)

The biggest gotcha is that you (obviously) need a source IP address, and netconsole starts before the networking normally comes up. Apart from that you can generally get away with minimal settings:

netconsole=@192.168.1.21/,@192.168.1.17/

Note that apparently some routers may not forward the broadcast packets correctly, so you may need to specify the target MAC address.

On the target machine run something like netcat to capture the packets:

$ netcat -l -u -p 6666 | tee console.log

If you get the options wrong the kernel will tell you why, so if you don’t get any logging iterate a working argument on an image that works, using dmesg to see what the problem is.

Finally instead of typing in this argument every time you boot, you can add it to the boot loader in your local.conf:

APPEND += "netconsole=@192.168.1.21/,@192.168.1.17/"

(APPEND being the name of the variable that is passed to the kernel by the boot loader.)

Update: when the journal starts up systemd will stop logging to the console, so if you want to get all systemd messages also pass systemd.log_target=kmsg.

I’ve been asked to review the GNOME 3 Application Development Guide for Beginners; I went through the book in about half a day and wrote this somewhat short review afterwards, and published on G+; sadly, I used a limited distribution, and G+ does not allow changing that without resharing your own post. given that I wanted to push it on the blog, I took the chance to review some of the stuff I wrote, and expand it.

my initial impression of the GNOME 3 Application Development Guide for Beginners book is fairly positive: the topics covered are interesting, and the book never loses itself in them, so that beginners will not feel utterly stranded after the first three chapters, as it too often happens with “for beginners” books. I appreciated the “pop quiz” sections, as well as the small sections that recommended improvements to the example code.

obviously, writing a book enshrines a certain set of requirements and APIs, and that is problematic when there is high churn – like what happens in GNOME 3, especially in terms of development tools (libraries and applications) and overall experience. for instance, the section on Clutter (which is the one I can immediately give feedback on, given my position) still uses the deprecated “default stage”, and yet it uses the new ClutterActor easing state for animations; the default stage was deprecated at long last in Clutter 1.10, but its use was not recommended since the days of 1.0; the actor easing state API was introduced in the same release that deprecated the default stage. also, the example code published in the Clutter section does not use any of the layout managers provided by Clutter, preferring the fixed positioning of the actors, which is perfectly fine on its own; the book, though, then proceeds to mention the amount of code necessary to get something on the screen, compared to the equivalent code in GTK, that uses boxes and grids. in general, that’s an utterly fair thing to say: Clutter sits at a lower-level than GTK, and it doesn’t have complex constructs like GTK does; I’m pretty sure, though, there are better examples than a row of boxes that could have used a BoxLayout, or a FlowLayout, or a GridLayout, or a TableLayout; or better examples than using an explicit PangoFontDescription instance with a ClutterText to set a specific font name and size, instead of using the ClutterText:font-name property which wraps the whole thing for the developer’s convenience. in short: Clutter is more “raw” than GTK, but there are convenience APIs for developers.

it’s been a long time¹ since I started off as a beginner in developing with (and) GNOME, so all I can say about book for beginners is whether they are using what I write in the way I think it’s supposed to be used; as far as I’m concerned, apart from a couple of issues, this book is indeed providing a solid base for people that want to begin developing for GNOME and with GNOME.

the price is pretty accessible, compared to the cost of similar books: I’ve paid much, much more for an introductory book on Core Animation, or on Perl books; the ebook version, if you dislike the dead tree version, comes in various formats, including PDF and Kindle.

I’m not going to give votes or anything: it’d be a pointless number on an equally pointless scale; but if you’re a beginner, I think this book may be fairly interesting to you, if you want to start coding with GNOME technologies.

1. this year is actually my 10th bug-versary

It’s been raining all this weekend, so what I wanted to do was see if I could accompany myself playing a piece of music by making multiple videos and slowly building it up.

I used my Canon S100 to record a bunch of videos and then imported them into kdenlive (the only viable multi-track video editor on Linux Desktop), this allowed me to cut and paste and do all the editing to make sure each track correctly accompanied the previous ones.

Because I wanted to show all 4 video tracks at once I figured I could use the videomix GStreamer element to do this with a custom pipeline. So by toggling which video track was “muted” I exported each video in a raw format at 960×540 so that 960×2 + 540×2 = 1920×1080 (i.e. 1080p).

The next thing I wanted to do was clean up the audio as the little microphone in the camera picked up a fair bit of hissing (not from the audience of course), multiply that 4 times and it’s pretty bad. The UI for doing this in Audacity is a bit odd…

1) Go to Effect menu select “Remove noise” 2) In the dialog click “Get noise profile” 3) Use the selection tool to select a part of the audio which contains only noise. 4) Deselect (so nothing is selected) 5)  Go to Effect menu select “Remove noise” and click OK .
Or watch this tutorial. Either way It works pretty damn well. Export to WAV.

So now we have 4 video files and an audio track let’s combine them into a webm video:

gst-launch -e  \
 	filesrc location=piano.avi ! avidemux ! ffdec_huffyuv ! ffmpegcolorspace ! video/x-raw-rgb !  mix. \
 	filesrc location=sing.avi ! avidemux ! ffdec_huffyuv ! ffmpegcolorspace ! video/x-raw-rgb ! mix. \
 	filesrc location=guitar2.avi ! avidemux ! ffdec_huffyuv ! ffmpegcolorspace ! video/x-raw-rgb ! mix. \
 	filesrc location=guitar1.avi ! avidemux ! ffdec_huffyuv ! ffmpegcolorspace ! video/x-raw-rgb ! mix. \
 	videotestsrc pattern=2 ! video/x-raw-rgb,width=1920,height=1080 ! \
        videomixer name=mix sink_0::xpos=0 sink_0::ypos=0 sink_0::zorder=02  sink_1::xpos=960 sink_2::ypos=540 \
        sink_3::ypos=540 sink_3::xpos=960 sink_4::zorder=0 ! video/x-raw-rgb,height=1080,width=1920 ! \
        ffmpegcolorspace !  vp8enc threads=4 !  webmmux name=mux ! filesink location=full-render.webm  \
 	filesrc location=audiotrack.wav ! wavparse ! audioconvert ! vorbisenc ! mux.

So those who know a bit of GStreamer you’ll spot the “videotestsrc” here as the 5th video input, this is because I couldn’t see a way to set the surface size of the videomixer, it seems it was mostly designed for doing picture in picture, so the output is set to the largest video source, without the videotestsrc the video would be stuck at 960×540.

I did try a videobox element but couldn’t persuade the pipeline to roll if the left/right/top/bottom properties were > 100px. I also had problems with the order of the height/width caps causing “Internal Data flow” in the avidemux and problems using decodebin2 which I abandoned pretty early on.

The videotestsrc introduces a second problem in that the videotestsrc has no EOS, it will continue streaming forever. I knew how long my video should be so I just HUP’d the process when it got to the right progress, `gstreamer -e` makes sure it sends an EOS on the pipeline on HUP which closes the filesink properly. Obviously this is pretty lame so maybe there is a better solution?  you could use a python script to listen to the EOS signal on one of the filesrc elements and then propagate the EOS to the videotestsrc.

Anywhoo it kind of worked.

Linux is all about making it work for you?

Update: oops forgot to add the audio level right.. time to re-add the audio..

gst-launch -e filesrc location='full-render.webm' ! matroskademux ! \
 video/x-vp8 ! webmmux name=mux ! filesink location=final.webm \
 filesrc location='audiotrack.wav' ! wavparse ! audioconvert ! vorbisenc ! mux.

The question of mutually exclusive streams in PulseAudio came to mind earlier, and thanks to Arun and Jens in #gupnp I discovered the PulseAudio supports them already. The use-case here is a system where there are many ways of playing music, but instead of mixing them PA should pause the playing stream when a new one starts.

Configuring this with PulseAudio is trivial, using the module-role-cork module:

$ pactl load-module module-role-cork trigger_roles=music cork_roles=music

This means that when a new stream with the “music” role starts, “cork” (pause to everyone else) all streams that have the “music” role. Handily this module won’t pause the trigger stream, so this implements exclusive playback.

Testing is simple with gst-launch

$ PULSE_PROP='media.role=music' gst-launch-0.10 audiotestsrc ! pulsesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstPulseSinkClock

At this point, starting another gst-launch results in this stream being paused:

Setting state to PAUSED as requested by /GstPipeline:pipeline0/GstPulseSink:pulsesink0...

Note that it won’t automatically un-cork when the newer stream disappears, but for what I want this is the desired behaviour anyway.

If you sometimes find yourself confused about whether you should use the word ‘fewer’ or ‘less’, here is a handy table to help you:

It is extremely important to pick the right word. For example if you think there are too many pedantic people in your office you should say ‘my office has the manyest pedantic people’. If you were to incorrectly say ‘my office has the most pedantic people’, your listener would immediately assume you meant that the people in your office are more pedantic than any other people. This could cause untold confusion and misunderstanding. It is therefore essential that if you hear a friend incorrectly say ‘more’ when they mean ‘manyer’ you must immediately interrupt and correct them.

conferences

sadly, this year I missed FOSDEM — and I saw from the blog posts on PGO that the DX hackfest was also amazing — but I had a good excuse, as I had to give a talk about Unicorns and Rainbows at linux.conf.au, in Canberra. I had the chance to explain what are the plans for the toolkit(s) used in GNOME in the coming future, as well as (obviously) enjoying one of the best free and open source software conferences, so all in all, I think that it was a good trade off. plus, I didn’t have to deal with the FOSDEM flu, apparently, so: good stuff all around. the video of the talk is already available on the LCA website¹ so if you weren’t there, you can still watch it. I’d put the slides somewhere, but I’ll have to make sure that the notes are up to date first².

life

after LCA, I took a week off in Sydney with Marta; the city was really enjoyable, and we had a great time exploring it. when I say “a week off” I don’t mean a week off of work, given that I’m currently not working: January 22nd was my last day at Mozilla. I detailed a why I left, and what my current plans are, on G+, so I won’t repeat them here — the executive summary is that I burned out pretty badly before leaving Intel, and I made the mistake of not taking a clean break before starting to work for Mozilla. right now, I’m trying to get back in a productive groove, so I’m looking around for cool stuff to do, as well as fulfilling my roles in the GNOME community as best as I can.

happenings

while in Sydney, I got a notification from my VPS hosting, the kind of notification that you really don’t want to receive when you’re on holiday and on a hotel wifi connection: apparently, my WordPress installation got hacked, and it started serving malware. cue various profanities that I clearly cannot repeat on a family-friendly website. I learned some valuable lessons from the first security breach I’ve ever experienced in years, but the main one is definitely the old adage from the Hitchhikers Guide to the Galaxy: don’t panic³.

beers

apropós of events, there’s a new GNOME London Beers meet up scheduled for celebrating the 3.8 release — or, at least, that’s the main excuse for meeting up, drinking beer, and having pizza with the GNOMErs in (and around) London. if you are in the area on March 1st, sign up on the wiki!

1. and, seriously: how come the GUADEC videos are always, always late or not available even when we do record them? it makes us look really bad; GUADEC teams: fix this crap. it should be a hard requirement
2. I had my notes on the tablet, to avoid the focus back and forth when I had to advance them
3. the other, and equally important one is: never trust a pile of PHP brain damage

I’m starting to type this up as EdgeConf draws to a close. I spoke on the performance panel, with Shane O’Sullivan, Rowan Beentje and Pavel Feldman, moderated by Matt Delaney, and tried to bring a platform perspective to the affair. I found the panel very interesting, and it reminded me how little I know about the high-level of web development. Similarly, I think it also highlighted how little consideration there usually is for the platform when developing for the web. On the whole, I think that’s a good thing (platform details shouldn’t be important, and they have a habit of changing), but a little platform knowledge can help in structuring things in a way that will be fast today, and as long as it isn’t too much of a departure from your design, it doesn’t hurt to think about it. At one point in the panel, I listed a few things that are particularly slow from a platform perspective today. While none of these were intractable problems, they may not be fixed in the near future and feedback indicated that they aren’t all common knowledge. So what follows are a few things to avoid, and a few things to do that will help make your pages scroll smoothly on both desktop and mobile. I’m going to try not to write lies, but I hope if I get anything slightly or totally wrong, that people will correct me in the comments and I can update the post accordingly

Avoid reflow

When I mentioned this at the conference, I prefaced it with a quick explanation of how rendering a web page works. It’s probably worth reiterating this. After network and such have happened and the DOM tree has been created, this tree gets translated into what we call the frame tree. This tree is similar to the DOM tree, but it’s structured in a way that better represents how the page will be drawn. This tree is then iterated over and the size and position of these frames are calculated. The act of calculating these positions and sizes is referred to as reflow. Once reflow is done, we translate the frame tree into a display list (other engines may skip this step, but it’s unimportant), then we draw the display list into layers. Where possible, we keep layers around and only redraw parts that have changed/newly become visible.

Really, reflow is actually quite fast, or at least it can be, but it often forces things to be redrawn (and drawing is often slow). Reflow happens when the size or position of things changes in such a way that dependent positions and sizes of elements need to be recalculated. Reflow usually isn’t something that will happen to the entire page at once, but incautious structuring of the page can result in this. There are quite a few things you can do to help avoid large reflows; set widths and heights to absolute values where possible, don’t reposition or resize things, don’t unnecessarily change the style of things. Obviously these things can’t always be avoided, but it’s worth thinking if there are other ways to achieve the result you want that don’t force reflow. If positions of things must be changed, consider using a CSS translate transform, for example – transforms don’t usually cause reflow.

If you absolutely have to do something that will trigger reflow, it’s important to be careful how you access properties in JavaScript. Reflow will be delayed as long as possible, so that if multiple things happen in quick succession that would cause reflow, only a single reflow actually needs to happen. If you access a property that relies on the frame tree being up to date though, this will force reflow. Practically, it’s worth trying to batch DOM changes and style changes, and to make sure that any property reads happen outside of these blocks. Interleaving reads and writes can end up forcing multiple reflows per page-draw, and the cost of reflow can add up quickly.

Avoid drawing

This sounds silly, but you should really only make the browser do as little drawing as is absolutely necessary. Most of the time, drawing will happen on reflow, when new content appears on the screen and when style changes. Some practical advice to avoid this would be to avoid making DOM changes near the root of the tree, avoid changing the size of things and avoid changing text (text drawing is especially slow). While repositioning doesn’t always force redrawing, you can ensure this by repositioning using CSS translate transforms instead of top/left/bottom/right style properties. Especially avoid causing redraws to happen while the user is scrolling. Browsers try their hardest to keep up the refresh rate while scrolling, but there are limits on memory bandwidth (especially on mobile), so every little helps.

Thinking of things that are slow to draw, radial gradients are very slow. This is really just a bug that we should fix, but if you must use CSS radial gradients, try not to change them, or put them in the background of elements that frequently change.

Avoid unnecessary layers

One of the reasons scrolling can be fast at all on mobile is that we reduce the page to a series of layers, and we keep redrawing on these layers down to a minimum. When we need to redraw the page, we just paste these layers that have already been drawn. While the GPU is pretty great at this, there are limits. Specifically, there is a limit to the amount of pixels that can be drawn on the screen in a certain time (fill-rate) – when you draw to the same pixel multiple times, this is called overdraw, and counts towards the fill-rate. Having lots of overlapping layers often causes lots of overdraw, and can cause a frame’s maximum fill-rate to be exceeded.

This is all well and good, but how does one avoid layers at a high level? It’s worth being vaguely aware of what causes stacking contexts to be created. While layers usually don’t exactly correspond to stacking contexts, trying to reduce stacking contexts will often end up reducing the number of resulting layers, and is a reasonable exercise. Even simpler, anything with position: fixed, background-attachment: fixed or any kind of CSS transformed element will likely end up with its own layer, and anything with its own layer will likely force a layer for anything below it and anything above it. So if it’s not necessary, avoid those if possible.

What can we do at the platform level to mitigate this? Firefox already culls areas of a layer that are made inaccessible by occluding layers (at least to some extent), but this won’t work if any of the layers end up with transforms, or aren’t opaque. We could be smarter with culling for opaque, transformed layers, and we could likely do a better job of determining when a layer is opaque. I’m pretty sure we could be smarter about the culling we already do too.

Avoid blending

Another thing that slows down drawing is blending. This is when the visual result of an operation relies on what’s already there. This requires the GPU (or CPU) to read what’s already there and perform a calculation on the result, which is of course slower than just writing directly to the buffer. Blending also doesn’t interact well with deferred rendering GPUs, which are popular on mobile.

This alone isn’t so bad, but combining it with text rendering is not so great. If you have text that isn’t on a static, opaque background, that text will be rendered twice (on desktop at least). First we render it on white, then on black, and we use those two buffers to maintain sub-pixel anti-aliasing as the background varies. This is much slower than normal, and also uses up more memory. On mobile, we store opaque layers in 16-bit colour, but translucent layers are stored in 32-bit colour, doubling the memory requirement of a non-opaque layer.

We could be smarter about this. At the very least, we could use multi-texturing and store non-opaque layers as a 16-bit colour + 8-bit alpha, cutting the memory requirement by a quarter and likely making it faster to draw. Even then, this will still be more expensive than just drawing an opaque layer, so when possible, make sure any text is on top of a static, opaque background when possible.

Avoid overflow scrolling

The way we make scrolling fast on mobile, and I believe the way it’s fast in other browsers too, is that we render a much larger area than is visible on the screen and we do that asynchronously to the user scrolling. This works as the relationship between time and size of drawing is not linear (on the whole, the more you draw, the cheaper it is per pixel). We only do this for the content document, however (not strictly true, I think there are situations where whole-page scrollable elements that aren’t the body can take advantage of this, but it’s best not to rely on that). This means that any element that isn’t the body that is scrollable can’t take advantage of this, and will redraw synchronously with scrolling. For small, simple elements, this doesn’t tend to be a problem, but if your entire page is in an iframe that covers most or all of the viewport, scrolling performance will likely suffer.

On desktop, currently, drawing is synchronous and we don’t buffer area around the page like on mobile, so this advice doesn’t apply there. But on mobile, do your best to avoid using iframes or having elements that have overflow that aren’t the body. If you’re using overflow to achieve a two-panel layout, or something like this, consider using position:fixed and margins instead. If both panels must scroll, consider making the largest panel the body and using overflow scrolling in the smaller one.

I hope we’ll do something clever to fix this sometime, it’s been at the back of my mind for quite a while, but I don’t think scrolling on sub-elements of the page can ever really be as good as the body without considerable memory cost.

Take advantage of the platform

This post sounds all doom and gloom, but I’m purposefully highlighting what we aren’t yet good at. There are a lot of things we are good at (or reasonable, at least), and having a fast page need not necessarily be viewed as lots of things to avoid, so much as lots of things to do.

Although computing power continues to increase, the trend now is to bolt on more cores and more hardware threads, and the speed increase of individual cores tends to be more modest. This affects how we improve performance at the application level. Performance increases, more often than not, are about being smarter about when we do work, and to do things concurrently, more than just finding faster algorithms and micro-optimisation.

This relates to the asynchronous scrolling mentioned above, where we do the same amount of work, but at a more opportune time, and in a way that better takes advantage of the resources available. There are other optimisations that are similar with regards to video decoding/drawing, CSS animations/transitions and WebGL buffer swapping. A frequently occurring question at EdgeConf was whether it would be sensible to add ‘hints’, or expose more internals to web developers so that they can instrument pages to provide the best performance. On the whole, hints are a bad idea, as they expose platform details that are liable to change or be obsoleted, but I think a lot of control is already given by current standards.

On a practical level, take advantage of CSS animations and transitions instead of doing JavaScript property animation, take advantage of requestAnimationFrame instead of setTimeout, and if you find you need even more control, why not drop down to raw GL via WebGL, or use Canvas?

I hope some of this is useful to someone. I’ll try to write similar posts if I find out more, or there are significant platform changes in the future. I deliberately haven’t mentioned profiling tools, as there are people far more qualified to write about them than I am. That said, there’s a wiki page about the built-in Firefox profiler, some nice documentation on Opera’s debugging tools and Chrome’s tools look really great too.

This is part one of the series “The State of the CLI”. There’s a lot to write about, but let’s start small: how helpful are command line tools and how do we acquire help?

“Help!”, I mean, “Hilfe?”

You’d think that getting help on certain commands is the most straightforward thing ever, but I’ve found that this isn’t always the case. Most commands carry help screens and documentation, but the way you access these may vary.

If you’re a bit more familiar with the CLI, you’d probably start by running the command accompanied by arguments or options that may show you the help screen. These will be different depending on who provided it (eg. GNU, or BSD): help, --help, -help, -h, ?, and -? are all possibilities (I will discuss the different option styles in a different post). I’m trying to figure out how this new command works. This is a bit like someone nagging who’s asking for help because they didn’t say "please" correctly in the right language. This is mostly due to Linux distributions shipping tools from many different projects. Overall though, --help seems to be the safest bet.

A less proactive approach would be to pretend to be clueless and pass a bogus argument or option to a command. If you’re lucky, you’ll be shown some information on how to get to the help screen.

hbons@slowpoke:~$ ip --foo
Option "-foo" is unknown, try "ip -help".
hbons@slowpoke:~$ mkdir --foo
mkdir: unrecognized option '--foo'
Try `mkdir --help' for more information.
hbons@slowpoke:~$

We didn’t get what we wanted in either case here, but at least we’re being led into the right direction.

Lastly, and probably most obviously, you can launch the command without any arguments at all. Some commands will give you the help this way. There is a downside to this kind of investigation, however. Running a command blindly can have unintended consequences, and may actually be dangerous due to inconsistency between different commands (“will this trigger an action, or show me the help?”).

Handholding (or not)

The access to useful help differs greatly between commands. This is what the BSD version of the ls command does:

hbons@slowpoke:~$ ls --help
ls: illegal option -- -
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]
hbons@slowpoke:~$

It doesn’t look particularly useful at all. It lists all the option that you can possibly enter, but there’s no indication of what any of these do, nor are there any hints about how you can find out. Even if I was more experienced using this command, it’s hard to see how a line like this could be helpful.

The git command line tools are short but helpful when you do something wrong. You’re being pointed to the help:

hbons@slowpoke:~$ git unknown
git: 'unknown' is not a git command. See 'git --help'.
hbons@slowpoke:~$

Additionally, these tools have a nice mechanism built in that tries to guess your intentions when you’ve made a typo, or when you’ve entered a command that doesn’t exist:

hbons@slowpoke:~$ git pu
git: 'pu' is not a git command. See 'git --help'.

Did you mean one of these?
&nbsp;&nbsp;&nbsp;&nbsp;pull
&nbsp;&nbsp;&nbsp;&nbsp;push
hbons@slowpoke:~$

Documentation

In addition to help screens, most commands have a good (sometimes overwhelming) amount of well written documentation available. It’s just a matter of how you get to it. The vast majority of commands ship with what are called “man” (manual) pages. Documentation on a command can be retrieved by running the man command with a subject (command name) as an argument. There’s also the info command by the GNU project, but it doesn’t seem as commonly used. Some commands have pointers at the end of their help screens on how to get more detailed help and documentation. Good examples of this are git and GNU’s version of ls:

hbons@slowpoke:~$ ls --help
[...]
Report ls bugs to bug-coreutils@gnu.org
GNU coreutils home page: &lt;http://www.gnu.org/software/coreutils/&gt;
General help using GNU software: &lt;http://www.gnu.org/gethelp/&gt;
For complete documentation, run: info coreutils 'ls invocation'
hbons@slowpoke:~$ git --help
[...]
See 'git help &lt;command&gt;' for more information on a specific command.
hbons@slowpoke:~$

Do you have problems getting help with commands, or do you have opinions or tips on how to make things better? I look forward to your feedback, please let me know.

Lucas Rocha and I gave a talk at FOSDEM over the weekend on Firefox for Android. It went ok, I think we could have rehearsed it a bit better, but it was generally well-received and surprisingly well-attended! I’m sure Lucas will have the slides up soon too. If you were unfortunate enough not to have attended FOSDEM, and doubly unfortunate that you missed our talk (guffaw), we’ll be reiterating it with a bit more detail in the London Mozilla space on February 22nd. We’ll do our best to answer any questions you have about Firefox for Android, but also anything Mozilla-related. If you’re interested in FirefoxOS, there may be a couple of phones knocking about too. Do come along, we’re looking forward to seeing you

p.s. I’ll be talking on a performance panel at EdgeConf this Saturday. Though it’s fully booked, I think tickets occasionally become available again, so might be worth keeping an eye on. They’ll be much cleverer people than me knocking about, but I’ll be doing my best to answer your platform performance related questions.

Unlike most computer interfaces, the Command Line Interface (or CLI) hasn’t changed much over the last 30 years. Does this mean we’re in a good place?

Pros and cons

The CLI can be helpful in a lot of cases, but it has a bit of a learning curve. It allows you to do simple canonical things, but these things can be linked together in various ways that can’t often be done with Graphical User Interfaces (or GUIs). Thus sparing you from tedious repetitive labour.

Some applications also allow for command line interaction next to their graphical user interface, which makes them able to be integrated in automated processes.

In my opinion, the CLI sits about half way between raw programming and using a graphical application to get your tasks done. There are graphical applications that can give you a lot of control as well. A good example of this is OS X’s Automator.

It may be that your product provides (or requires) some interaction on the CLI level. Now how do you go about doing this right?

GNU and BSD

Linux distributions are a mixed bag of different programs, and so these programs don’t always behave in the same predictable ways. Most common commands originate either from the GNU or BSD projects.

Superficially, BSD sticks more to the older Unix days, whilst GNU tries to come up with new ideas that makes their programs a little more convenient to use.

Eric Raymond’s The Art of UNIX programming goes into CLI interface design a little bit. The GNU Coding Standards has something to say about the topic as well.

Overall, documentation and guidelines are sparse. As a developer writing software for Linux distributions, it’s not clear where one should go.

Best practises

The CLI is used most often by people that are more familiar with computers, software developers, system administrators, and those who like to tinker. This doesn’t mean that the CLI has to or is allowed be a suboptimal experience on that level.

What’s noticeable is that there doesn’t seem to be much documented reasoning or motivation behind any of the CLI design decisions. Things mostly seem the way they are because of “historic” reasons or tradition.

The next couple of months I’m going to to assess the CLI of various common software packages and write blog posts about them, to come up with a list of best practises that doesn’t conflict with the current state of utilities out there.

I’d like to explicitly mention that these will be best practises that are compatible with current conventions, not some new official “standard”. Having some rules that you can keep in mind, as well as knowing the reasons behind them. I don’t want to fall into the standards trap.

The usual suspects

If you administrate a server, you’ve probably used commands from the following packages:

• GNU coreutils, a collection of basic tools, such as cat, ls, and rm
• GnuPG, encryption tools
• openSSH, connectivity tools for remote access
• Apt, package management
• Git, source code management
• cURL, data transfer/download tool

These are some of the tools I’ll be looking at.

To wrap up: this is not about redefining the command line experience, or creating some revolutionary alternative. rather to identify issues, streamline the process, and come up with some design patterns that may help you write a usable CLI application for Linux.

If you have any CLI pet peeves, know of commands that are really good/bad, or have any other tips or links you think I should read, please let me know.

Like every year, touching a source file means you also need to update the year of the copyright notice you should have at the top of the file. I always end up forgetting about them, this is where a git pre-commit hook would be ultra-useful, so I wrote one:

#
# Check if copyright statements include the current year
#
files=`git diff --cached --name-only`
year=`date +"%Y"`

for f in $files; do
    head -10 $f | grep -i copyright 2>&1 1>/dev/null || continue
    
    if ! grep -i -e "copyright.*$year" $f 2>&1 1>/dev/null; then
        missing_copyright_files="$missing_copyright_files $f"
    fi
done

if [ -n "$missing_copyright_files" ]; then
    echo "$year is missing in the copyright notice of the following files:"
    for f in $missing_copyright_files; do
        echo "    $f"
    done 
    exit 1
fi

Hope this helps!

I’m delighted to announce the availability of SparkleShare 1.0!

What is SparkleShare?

SparkleShare is an Open Source (self hosted) file synchronisation and collaboration tool and is available for Linux distributions, Mac, and Windows.

SparkleShare creates a special folder on your computer in which projects are kept. All projects are automatically synced to their respective hosts (you can have multiple projects connected to different hosts) and to your team’s SparkleShare folders when someone adds, removes or edits a file.

The idea for SparkleShare sprouted about three years ago at the gnome Usability Hackfest in London (for more background on this read The one where the designers ask for a pony).

SparkleShare uses the version control system Git under the hood, so people collaborating on projects can make use of existing infrastructure, and setting up a host yourself will be easy enough. Using your own host gives you more privacy and control, as well as lots of cheap storage space and higher transfer speeds.

Like every piece of software it’s not bug free, even though it has hit 1.0. But it’s been tested for a long time now and all reproducable and known major issues have been fixed. It works reliably and the issue tracker is mostly filled with feature requests now.

The biggest sign that it was time for a 1.0 release was the fact that Lapo hasn’t reported brokenness for a while now. This can either mean that SparkleShare has been blessed by a unicorn or that the world will end soon. I think it’s the first.

Features

For those of you that are not (that) familiar with SparkleShare, I’ll sum up its most important features:

The SparkleShare Folder

This is where all of your projects are kept. Everything in this folder will be automatically synced to the remote host(s), as well as to your other computers and everyone else connected to the same projects. Are you done with a project? Simply delete it from your SparkleShare folder.

The status icon

The status icon gives you quick access to all of your projects and shows you what’s going on regarding the synchronisation process. From here you can connect to existing remote projects and open the recent changes window.

The setup dialog

Here you can link to a remote project. SparkleShare ships with a couple of presets. You can have mulitple projects syncing to different hosts at the same time. For example, I use this to sync some public projects with Github, some personal documents with my own private vps and work stuff with a host on the intranet.

Recent changes

The recent changes window shows you everything that has recently changed and by whom.

History

The history view let’s you see who has edited a particular file before and allows you to restore deleted files or revert back to a previous version.

Conflict handling

When a file has been changed by two people at the same time and causes a conflict, SparkleShare will create a copy of the conflicting file and adds a timestamp. This way changes won’t get accidentally lost and you can either choose to keep one of the files or cherry pick the wanted changes.

Notifications

If someone makes a change to a file a notification will pop up saying what changed and by whom.

Client side encryption

Optionally you can protect a project with a password. When you do, all files in it will be e encrypted locally using AES-256-CBC before being transferred to the host. The password is only stored locally, so if someone cracked their way into your server it will be very hard (if not impossible) to get the files’ contents. This on top of the file transfer mechanism, which is already encrypted and secure. You can set up an encrypted project easily with Dazzle.

Dazzle, the host setup script

I’ve created a script called Dazzle that helps you set up a Linux host to which you have SSH access. It installs Git, adds a user account and configures the right permissions. With it, you should be able to get up and running by executing just three simple commands.

Plans for the future

Something that comes up a lot is the fact that Git doesn’t handle large (binary) files well. Git also stores a database of all the files including history on every client, causing it to use a lot of space pretty quickly. Now this may or may not be a problem depending on your usecase. Nevertheless I want SparkleShare to be better at the “large backups of bulks of data” usecase.

I’ve stumbled upon a nice little project called git-bin in some obscure corner of Github. It seems like a perfect match for SparkleShare. Some work needs to be done to integrate it and to make sure it works over SSH. This will be the goal for SparkleShare 2.0, which can follow pretty soon (hopefully in months, rather than years).

I really hope contributors can help me out in this area. The Github network graph is feeling a bit lonely. Your help can make a big difference!

Some other fun things to work on may be:

• Saving the modification times of files
• Creating a binary Linux bundle
• SparkleShare folder location selection
• GNOME 3 integration
• ...other things that you may find useful.

If you want to get started on contributing, feel free to visit the IRC channel: #sparkleshare on irc.gnome.org so I can answer any questions you may have and give support.

Finally...

I’d like to thank everyone who has helped testing and submitted patches so far. SparkleShare wouldn’t be nearly as far as it is now without you. Cheers!

Python GI / pygi GUdev example

from gi.repository import GUdev
from gi.repository import GLib

mainloop = GLib.MainLoop ()

def on_uevent (client, action, device):
    print ("action " + action + " on device " + device.get_sysfs_path())

# empty array for all subsystems, char array for subsystem e.g. ["usb","video4linux"] etc
client = GUdev.Client (subsystems=[])
# or client = GUdev.Client.new ([])

client.connect ("uevent", on_uevent)

mainloop.run ()

1. You can’t change subsystems that you’re listening to after construction
2. Make sure you keep a reference to client, because even though you’ve connected to it’s signal it will still be unreferenced when it goes out of scope

Last month our friends at Codethink were guests on FLOSS Weekly, talking about Baserock. Baserock is a new embedded build system with some interesting features/quirks (depending on your point of view) that I won’t go into. What caught my attention was the discussion about build times for various embedded build systems.

Yocto, again, if you want to do a clean build it will take days to build your system, even if you do an incremental build, even if you just do a single change and test it, that will take hours.

(source: FLOSS Weekly #230, timestamp 13:21, slightly edited for clarity)

Now “days” for a clean build and “hours” for re-building an image with a single change is quite excessive for the Yocto Project, but also quite specific. I asked Rob Taylor where he was getting these durations from, and he corrected himself on Twitter:

I’m not sure if he meant “hours” for both a full build and an incremental build, or whether by “hours” for incremental he actually meant “minutes”, but I’ll leave this for now and talk about real build times.

Now, my build machine is new but nothing special. It’s built around an Intel Core i7-3770 CPU (quad-core, 3.4GHz) with 16GB of RAM (which is overkill, but more RAM means more disk cache which is always good), and two disks: a 250GB Western Digital Blue for /, and a 1TB Western Digital Green for /data (which is where the builds happen). This was built by PC Specialist for around £600 (the budget was $1000 without taxes) and happily sits in my home study running a nightly build without waking the kids up. People with more money stripe /data across multiple disks, use SSDs, or 10GB tmpfs filesystems, but I had a budget to stick to.

So, let’s wipe my build directory and do another build from scratch (with sources already downloaded). As a reference image I’ll use core-image-sato, which includes an X server, GTK+, the Matchbox window manager suite and some demo applications. For completeness, this is using the 1.3 release – I expect the current master branch to be slightly faster as there’s some optimisations to the housekeeping that have landed.

$ rm -rf /data/poky-master/tmp/
$ time bitbake core-image-sato
Pseudo is not present but is required, building this first before the main build
Parsing of 817 .bb files complete (0 cached, 817 parsed). 1117 targets, 18 skipped, 0 masked, 0 errors.
...
NOTE: Tasks Summary: Attempted 5393 tasks of which 4495 didn't need to be rerun and all succeeded.

real 9m47.289s

Okay, that was a bit too fast. What happened is that I wiped my local build directory, but it’s pulling build components from the “shared state cache”, so it spent six minutes reconstructing a working tree from shared state, and then three minutes building the image itself. The shared state cache is fantastic, especially as you can share it between multiple machines. Anyway, by renaming the sstate directory it won’t be found, and then we can do a proper build from scratch.

$ rm -rf /data/poky-master/tmp/
$ mv /data/poky-master/sstate /data/poky-master/sstate-old
$ time bitbake core-image-sato
Pseudo is not present but is required, building this first before the main build
...
NOTE: Tasks Summary: Attempted 5117 tasks of which 352 didn't need to be rerun and all succeeded.

real 70m37.298s
user 326m45.417s
sys 37m13.304s


That’s a full build from scratch (with downloaded sources, we’re not benchmarking my ADSL) in just over an hour on affordable commodity hardware. As I said this isn’t some “minimal” image that boots straight to busybox, this is building a complete cross-compiling toolchain, the kernel, X.org, GTK+, GStreamer, the Matchbox window manager/panel/desktop, and finally several applications. In total, 431 source packages were built and packaged, numerous QA tests executed and flashable images generated.

My configuration is to build for Intel Atom but a build for an ARM, MIPS, or PowerPC target would also take a similar amount of time, as even what could be considered “native” targets (targeting Atom, building on i7) doesn’t always turn out to be native: for example carrier-grade Xeon’s have instructions that my i7 doesn’t have, and if you were building carrier-grade embedded software you’d want to ensure they were used.

So, next time someone claims Yocto Project/OpenEmbedded takes “days” or even “hours” to do a build, you can denounce that as FUD and point them here!

Some more updates to the dawati-user-testing Insight recorder tool

New features:

• multiple webcam input support
• system installable
• required codec checker
• encoding finished notification
• various bug fixes

More info on latest release: http://belenpena.posterous.com/and-now-with-support-for-mobile-tests-hell-ye

project page: https://github.com/dawati/insight-recorder

p.s. we’re looking for some help making this  work on ubuntu/unity

Today is the first release of Rig. This is a project that Robert Bragg and I have been working for the past few months. I won't try to describe it here, but instead I will try to entice you with a screenshot and ask you to take a look at Robert's detailed blog post.

For anyone interested in graphics and in the design of user interfaces (UIs) I hope I can invite you to take a look at our release of Rig 1 today. Rig 1 is the start of a new design tool & UI engine that I've been working on recently with Neil Roberts and Damien Lespiau as a means to break from the status quo for developing interfaces, enabling more creative visuals and taking better advantage of our modern system hardware.

In particular we are looking at designing UIs for consumer devices with an initial focus on native interfaces (device shells and applications), but also with an eye towards WebGL support in the future too.

With Rig we want to bring interactivity into the UI design process to improve creativity and explore the possibilities of GPUs beyond the traditional PDF drawing model we have become so accustomed to. We want to see a design work flow that isn't constrained by the static nature of tools such as Photoshop or mislead by offline post-processing tools such as After Effects. We think designers and engineers should be able to collaborate with a common technology like Rig.

Lets start with a screenshot!


This gives you a snapshot of how the interface looks today. For Rig 1 our focus has been on bootstrapping an integrated UI design tool & rendering engine which can showcase a few graphics techniques not traditionally used in user interfaces and allows basic scene and animation authoring.

If you're wondering what mud, water, some trees and a sun have to do with creating fancy UIs; firstly I should own up to drawing them, so sorry about that, but it's a sneak peek at the assets for a simple "Fox, Goose and Corn" puzzle, in the style of a pop-up book, we are looking to create for Rig 2.

I'd like to highlight a few things about the interface:

It all starts with assets:

On the left of the UI you see a list of assets with a text entry to search based on tags and names. Assets might be regular images, they could be 3D meshes, they could be special ancillary images such as alpha masks or normal maps (Used for a technique called bump-mapping to give a surface a perturbed look under dynamic lighting).

Assets don't have to be things created by artists though, they might also be things like scripts or data sources in later versions of Rig.

The basic idea is that assets are the building blocks for any interface and so that's why we start here. Click an asset and it's added to the scene and assets can sometimes even be combined together to make more complex things, which I'll talk more about later.

Direct manipulation:

In the center is the main editing area where you see the bounds of the device currently being targeted and can directly manipulate scenes for the UI being designed. We think this ability to directly manipulate a design from within the UI engine itself is going to be a cornerstone for enabling greater creativity since there is no ambiguity about what's possible when
you're working directly with the technology that's going to run the UI when deployed to a device.

These are the current controls:

• Middle mouse button to rotate the whole scene
• Shift + middle mouse to pan the scene
• '+' and '-' to zoom in and out
• Left click and drag to move an object (object should not already be selected)
• Left click and drag a selected object to rotate
• Ctrl-Z and Ctrl-Y for Undo and Redo respectively
• Ctrl-S to Save the current document

Timelines:

Once you've built up a scene of Entities then authoring animations is pretty simple. Clicking the red record button of a property says that you want to change the property's value over time and a marker will pop up in the timeline view at the bottom. Left clicking and dragging on the timeline lets you scrub forwards and backwards in time. If you scrub forwards in time and then change a recorded property then a new marker pops up for that property at the current time. Left clicking markers lets you select and move marks. Property values are then automatically interpolated (tweened) for timeline offsets that lay in between specific markers.

These are the current timeline controls:

• Left click lets you scrub the current timeline position
• Left clicking markers lets you select marks; Shift + Left click lets you select multiple markers
• Left clicking and dragging lets you move selected markers left and right
• Delete lets you remove markers
• Ctrl-Z and Ctrl-Y also let you undo and redo timeline changes

Effects

We started with a fairly conservative set of effects for Rig 1, opting for effects that are well understood and widely used by game developers. This reduced some of the initial development risk for us but there is a chance that our choices will give the impression we're simply trying to make UIs that look like console games which isn't the intention.

Modern GPUs are extremely flexible pieces of hardware that enable an unimaginably broad range of visual effects, but they are also pretty complex. If you really want to get anything done with a GPU at a high level you quickly have to lay down some constraints, and make some trade-offs.

The effects we started with have a long history in game development and so we know they work well together. These effects emphasize building photorealistic scenes but there are lots of non-photorealistic effects and generative art techniques we are also interested in supporting within Rig. Since these are more specialized we didn't feel they should be our first focus while bootstrapping the tool itself.

Lighting

Shadow mapping

Bump mapping

Note: There are sometimes better ways calculate normal maps for specific use cases but this tool at least gives us one general purpose technique as a starting point. An artist might for example at least want to hand tune the bump map before generating a normal map.

This video shows the gist of what this effect enables, though for a more realistic use case I think it would deserve a more hand-tuned bump map, and more suitable texture mapping.

Alpha masks

This video shows a simple example of animating the threshold with two different mask assets:

Depth of field

Status

So back from layout into graphics again! For the last few weeks, I’ve been working with Benoit Girard on getting progressive tile rendering finished and turned on by default in Firefox for Android. The results so far are very promising! First, a bit of background (feel free to skip to the end if you just want the results).

You may be aware that we use a multi-threaded application model for Firefox for Android. The UI runs in one thread and Gecko, which does the downloading and rendering of the page, runs in another. This is a bit of a simplification, but for all intents and purposes, that’s how it works. We do this so that we can maintain interactive performance – something of paramount important with a touch-screen. We render a larger area than you see on the screen, so that when you scroll, we can respond immediately without having to wait for Gecko to render more. We try to tell Gecko to render the most relevant area next and we hope that it returns in time so that the appearance is seamless.

There are two problems with this as it stands, though. If the work takes too long, you’ll be staring at a blank area (well, this isn’t quite true either, we do a low-resolution render of the entire page and use that as a backing in this worst-case scenario – but that often doesn’t work quite right and is a performance issue in and of itself…) The second problem is that if a page is made up of many layers, or updates large parts of itself as you scroll, uploading that work to the graphics unit can take a significant amount of time. During this time, the page will appear to ‘hang’, as unfortunately, you can’t upload data to the GPU and continue to use it to draw things (this isn’t true in every single case, but again, for our purposes, it is).

Progressive rendering tries to spread this load by breaking up that work into several smaller tiles, and processing them one-by-one, where appropriate. This helps us mitigate those pauses that may happen for particularly complex/animated pages. Alongside this work, we also add the ability for a render to be cancelled. This is good for the situation that a page takes so long to render that by the time it’s finished, what it rendered is no longer useful. Currently, because a render is done all at once, if it takes too long, we can waste precious cycles on irrelevant data. As well as splitting up this work, and allowing it to be cancelled, we also try to do it in the most intelligent order – render areas that the user can see that were previously blank first, and if that area intersects with more than one tile, make sure to do it in the order that maintains visual coherence the best.

A cherry on the top (which is still very much work-in-progress, but I hope to complete it soon), is that splitting this work up into tiles makes it easy to apply nice transitions to make the pathological cases not look so bad. With that said, how’s about some video evidence? Here’s an almost-Nightly (an extra patch or two that haven’t quite hit central), with the screenshot layer disabled so you can see what can happen in a pathological case:

And here’s the same code, with progressive tile rendering turned on and a work-in-progress fading patch applied.

This page is a particularly slow page to render due to the large radial gradient in the background (another issue which will eventually be fixed), so it helps to highlight how this work can help. For a fast-to-render page that we have no problems with, this work doesn’t have such an obvious effect (though scrolling will still be smoother). I hope the results speak for themselves

]]>

The white horse enduro was on Sunday and I’d been invited to marshal. I’ve never held any illusions about my ability regarding an enduro, its something I’d just never do but this was a chance to see one up close. You know it will be interesting when you’re told which bike to bring and to put low gearing on it.

The view from the top of the hillside, one of the few spots with phone signal

The first part of Saturday was spent cutting out forest and marking out bits of course, hammering in road pins, putting up tape, stapling arrows and so on. It was hot work and I was using a heavy branch to smash through deadwood when the club like head on it it snapped and hit me hard in the nose which was the first incident of the day for me. I managed to embarrass myself stalling repeatedly in front of people in one section. Once you’re stopped on a slope, its a nightmare to get going again.

Later in the afternoon, after marking out was completed, they needed some course times. A group set out and I decided to tag on the back, at least I could check the marking out. In short order, I’d stalled the bike and they disappeared off into the distance, I never did see them again, several didn’t complete the course either. I continued on, coming across some interesting features like a nice decent with a steep, tree rooted climb on the exit to a stream. I gave this a go, mostly made it up but had to help the bike over a root. Later, people pointed out that this was the “expert” route and there was an easier alternative to the right but there was no track going that way visible at this point. This was taken at the stream crossing showing the choice of two routes on the Sunday after the signs had been tweaked:

So far the terrain was unknown and full of surprises, I then came to the bits we’d marked out and/or already ridden so this gave some let up. I was taking breaks periodically since otherwise, I was going to get tired and make more mistakes. I made a phone call on the top of the hillside with great views of the valley where there was phone signal. Shortly after than I managed to catch a rut badly, laid the bike down on its side and barrel rolled off it and down an incline. No damage to anything thankfully as I was going slowly. At some point I also ended up upsidedown on an incline with the bike on top of me which was fun. Periodically, there were interesting features like steep drop off sections. Again, apparently there were easier bits but I wasn’t understanding the marking of these correctly. In the special test section there was an interesting incline you had to go up which for some reason scared the living daylights out of me. They ended up taking out the crazy downhill then uphill section just after this as even though a few of us had made it around, it was going to cut up too easily with a few bikes around it.

Around this time, someone passed me and we exchanged a few words about me taking it easy and them suggesting “the best bits are to come”. I came to the piece of the course I’d been around so far and knew I could head back to the start from here. Since by this point I was suffering exhaustion, the sensible thing to do was to head back. I’d heard stories about having to jump ditches and a dry stone wall but I thought I’d done that part (that seemed easier than they were suggesting). There was definitely some steep hill climbs ahead though. Since I’d likely never do it again, I decided to continue, slowly and carefully. I soon came to the uphill section which was a relentless steep climb for a few hundred meters. Amazingly for me, I did make it to near enough the top without stopping. The bike was trying to stall, bogging down badly and I was determined we were not stopping and its probably one of the few times I’ve ever found the throttle stop. A decent followed, followed by what looked like more hard hill climbs but the hardest climb turned out to be behind me. A boggy trip through the trees brought me back to the start. I’d taken somewhere over two hours, the clubman time was about an hour. Experts would do four of the 12 mile laps, one was enough for me.

It was an early start on Sunday and I’d left the bike outside the van overnight so it was covered in frost. I had to scrape ice off the van windscreen too.

The start area

As a marshal, the idea was to roam around your section making sure everyone was ok, repair the route where possible and generally help out as needed. The section assigned to our group included the downhill section into the stream and another steep downhill section just after it. I did my best to roam around, yet keep out the way of the competitors and help out where needed. We’d also been asked to send people periodically to the starters point to help. I picked a good time to be there under the gazebo for the short downpour we had!

On the steep downhill, I managed to lose my balance on one trip down and ended up “choosing” a head-on collision with a tree into the headlight as the alternatives would have involved clipping the bars against a tree and leaving the bike. As it was, I stopped against the tree, restarted the bike and continued before the marshal behind slid into me (as there was no way he’d have been able to stop and I was conscious of this). Amazingly, there wasn’t too much damage as the light itself stayed intact:

This should be a straight bracket with the tax disc pointing outwards

The day’s activities were then complete and it was then time to demark the course and collect in all the arrows and tape. I helped out, ending up at the back of the group with a heavy paperbag filled with collected in material. I was about to set off once again, tried to kick the engine over to start it and found the kick start lever ended up on the ground. The kick start shaft itself had sheared leaving the casing in a mess too

Not how you expect a component to fail

Cracks = Not Good

I waited around but nobody came back far enough to find me although I did hear someone in the distance so I was on my own. No problem, I found a hill, bump started the bike and rode fire roads back to the start. Having a GPS on the bike helped!

It was a good weekend and I enjoyed it. I doubt I’ll ever enter an enduro, I think this weekend determined that for sure but I might marshal again if they’ll have me back though.

I can’t be the only “computer geek” who has someone tell them they need to get more fresh air or that they look pale and need some sunshine. In my case this got to the point my parents were making crypt and vampire jokes. it turned out there is a serious side to those comments which I’ve been meaning to write about for a while.

About this time last year, I was noticing that my joints were unhappy about something. In particular stairs were hard to climb up with my joints feeling at least twice my age with tearing sensations in my knees. This was along with various other aches and pains which were slowly getting worse. I’d also fractured my hand 9 years ago and ever since it had a dull ache at the fracture site, particularly after stirring gravy, painting or hammering in nails to a fence or similar activities.

On a different matter and nearly in passing, my GP suggested checking my vitamin D blood level and the test came back as borderline deficient suggesting Osteomalacia (in childhood much more commonly known as Rickets, eek!). Vitamin D is the “sunshine vitamin” being generated by exposure of skin to UV light. You get very little from food sources. A low level made sense in that I work indoors and when I do go out into the countryside and get exercise on the bike, I’m wearing layers of protective clothing and skin exposure is a bad idea. Sunlight exposure generating enough vitamin D in northern latitudes is questionable at the best of times and with a pale skin, any strong sunlight would trigger me to use suncream and hence block off the UV that would generate the vitamin.

Thankfully the “cure” is simple, either sunlight exposure (which here at this time of year isn’t going to work) or supplements. Having taken some supplements, I have to say it was amazing the difference they made with my joints no longer experiencing the tearing sensations and generally feeling a lot better. Also, totally unexpectedly, the fracture which ached when provoked stopped aching and has not done so since! There shouldn’t be any long term issue since the deficiency was mild and any issues reverse themselves with normal vitamin levels.

So if you spend lots of time in front of computers, make sure you get some sunshine too!

This may come as a shock to some, but I’ve just tagged Devil’s Pie 0.23 (tarball here).

tl;dr: don’t use this, but if you insist it now works with libwnck3

The abridged changelog:

• Port to libwnck3 (Christian Persch)
• Add unfullscreen action (Mathias Dalh)
• Remove exec action (deprecated by spawn)

I probably wouldn’t have ever released this as I’m generally not maintaining it and tend to push people towards Devil’s Pie 2 which funnily enough had a 0.23 release two days ago, but Christian asked nicely and I was waiting on a Yocto build to finish.

One of the perks of being a Virgin Media customer (beyond getting my name wrong and constant up-sell harassment) is that I got cheap, early-access Eurogamer Expo tickets! This was my first Eurogamer Expo, though I’m no stranager to ECTS or ATEI/EAG. The setup was quite good – perhaps a bit smaller than I expected, but nice to see a games show that’s actually aimed at gamers. I was always amused at the hoops you had to jump through to get tickets for ECTS and ATEI; more so when you actually visit the events and realise the majority of people there are gamers who have jumped through those same hoops. Good to see that the games industry, finally, after several years, got wise.

There was a fair amount on show. Lots of soon and not-so-soon to be released games, the WiiU, a surprising and pleasing amount of indie content and various bits and bobs. The WiiU was certainly the main attraction, but was managed terribly and was extremely disappointing. While most of the company reps were great and very helpful, a couple of Nintendo’s were oddly aggressive and patronising. I don’t think anyone at Eurogamer needs to be told how to play WiiU mini-games, or have buttons on their controllers pressed for them. The decision to dedicate three entire kiosks in the WiiU section to a video panorama viewer was baffling too. It’s almost as if no one at Nintendo has picked up a smart-phone in the last 5 years or so – this isn’t astounding stuff. Wonderful 101 seemed quite fun, but not as fun as I was expecting. The rest of the WiiU content was very disappointing. Pikmin 3 looking bland and boring was especially upsetting. It’s ironic that playing on the console has secured my decision not to buy it on release. I could easily write about how disappointing the WiiU was for a lot longer, but I just don’t care enough.

What was pleasantly surprising was how good Sony’s presence and content was. Reps were polite and helpful, not getting in the way where they weren’t needed and turning up when they were. Much like a good waiter. They had plenty of kiosks and space, and queues were minimal (not due to lack of interest, mind). Playstation All-stars Battle Royale, though clearly a Smash Bros. rip-off, is actually a very good one. We spent quite a while on it, and it was very enjoyable (possibly more so than Smash Bros. Brawl, but it doesn’t even approach the heights of Melee). The cross-play was especially impressive too, mirroring almost the exact same game frame-for-frame with only minor graphical omissions. Stand-out game of the show had to be When Vikings Attack, though. Incredibly simple concept, but perfect execution and impressive cross-play again. The only disappointment was that it doesn’t have a confirmed release date, but Clever Beans say it will be on PSN before the end of the year. This is definitely day-one purchase material.

Carmageddon definitely deserves a mention. It’s just as much fun as it was all those years ago, and the tablet/smartphone port has been handled perfectly. A shame that there was no demo or footage for the Carmageddon Reincarnation project, but hopefully it made a few more people aware. Also worth mentioning was God of War: Ascension, which although is more of the same, it’s a brilliant same that it’s more of. The multiplayer worked surprisingly well too, though a LAN setup is always going to be more fun than online. There were a few things that I’d have liked to have tried, but queues prevented me – nothing I would deem queue-worthy though. Hitman looked quite impressive, but the whole misogyny thing has put me off. Same goes for Tomb Raider. Dishonoured looked interesting, but not so interesting to queue for. Halo 4 looked like more of the same, though the considerable graphical upgrade certainly doesn’t hurt. Dead or Alive 5 was quite fun, and pleasing to see that they’ve returned to the mechanics of Dead or Alive 2 (clearly the series high). Disappointing amount of guys picking bikini-clad women to fight and leaving the camera aimed at crotch/chest areas; we evened the score a bit by playing as ridiculous-looking guys and aiming at the groin. Yes, I am 12. Disappointed to see that they’ve not included Zack’s weird sports-bra costume. The indie games arcade section is probably worth mentioning in that almost everything in it was terrible and just trading on a quirky look with zero gameplay to back it up. I conclude that there’s still plenty of room for ideas and innovation in the British indie games community.

All in all, a pretty fun event. Slightly disappointing that the industry still hasn’t moved on from the whole booth-babe thing, but it’s definitely far less prevalent than it used to be, so that leaves me with some hope. The graphical standard of console games is astounding, especially given there hasn’t been a hardware refresh in over 5 years. I’ll definitely be returning next year.

]]>

Been experimenting with time lapse

Had a fairly decent web cam attached to an old netbook and we have a good view out of our office so thought, hmm wonder if this would make a good timelapse.. wrote this to capture and encode:

#!/bin/bash

#Take a picture every INTERVAL for making time lapse
#see http://www.michaelwood.me.uk/wordpress/2012/09/26/time-lapse/

#start Configuration

INTERVAL=15
BASEDIR="$HOME/Pictures/timelapse/"
MAXSPACE=90
WIDTH=1920
HEIGHT=1080
DEVICE="/dev/video0"
VIDEOFPS=10

#end Configuration

function encode
{
 gst-launch-0.10 -e \
         multifilesrc \
                 location="%06d-timelapse.jpg" ! image/jpeg,framerate=$VIDEOFPS/1 \
         ! decodebin \
         ! videoscale \
         ! video/x-raw-yuv,width=$WIDTH,height=$HEIGHT \
         ! progressreport \
                 name=progress \
         ! vp8enc threads=4 \
         ! webmmux \
         ! filesink  location="timelapse.webm"

 notify-send "Timelapse encode done"
}

if [ "$1" == "--encode" ];
then
  encode;
  exit;
fi

num=0

while true
do
spaceused=`df / -H | grep -vE '^Filesystem' | cut -f 10 -d ' ' | cut -d '%' -f 1`
if [ $spaceused -gt $MAXSPACE ]; then
 echo "out of space";
 exit;
fi

workingdir="$BASEDIR/`date +%F`/"

if [ ! -d $workingdir ]
then
  #new day so reset to 0
  num=0;
  mkdir -p "$workingdir";
fi

filename="$BASEDIR/`date +%F`/`printf %04d $num`-timelapse.jpg"

# To easily relate $num to date you could echo "$num,`date +%H-%S` >> logfile
# or something like for i in  *.jpg; do new_j=`echo ${i: 0:4}`-timelapse.jpg; ln "$i" "$new_j"; done

gst-launch-0.10 v4l2src num-buffers=1 device=$DEVICE ! \
 video/x-raw-yuv,width=$WIDTH,height=$HEIGHT !\
 jpegenc quality=100 \
 ! filesink location=$filename >> /dev/null;
sleep $INTERVAL;

num=`expr $num + 1`

done
exit

(run as a bash script) should work on any recent Linux desktop (as of 6/9/12) with gstreamer tools, gstreamer plugins good and bad installed

And this is the result:

 

Update 17/3/13

Got a Microsoft Lifecam HD, it’s a slightly better camera than the one i’ve been using before, but it has a bunch of autofocus and light balance features which means the above method isn’t much good as I ended up with a bunch of pictures that were out of focus and at weird light levels. The solution was to keep the camera running all the time and get camerabin to take pictures.

#!/bin/python

import gst
import time

count = 0

jpeg = gst.element_factory_make ("jpegenc", "jpeg")
jpeg.set_property ("quality", 100)

vl4src = gst.element_factory_make ("v4l2src", "source")
vl4src.set_property ("device", "/dev/video1")

sink = gst.element_factory_make ("fakesink", "sink")

camerabin = gst.element_factory_make ("camerabin", "cam")
camerabin.set_property ("video-source", vl4src)
camerabin.set_property ("image-encoder", jpeg)
camerabin.set_property ("viewfinder-sink", sink)

camerabin.set_state (gst.STATE_PLAYING)

while 1:
    time.sleep (5)
    filename = "%06d-timelapse.jpg" % count
    count = count + 1
    camerabin.set_property ("filename", filename)
    camerabin.emit ("capture-start")

It seems, somehow, for the last few months, I’ve been working on layout. I’m not quite sure how it happened, as anyone who’s spoken to me or follows me on Twitter would know that I have a very healthy fear of the Gecko layout code. I still have that fear, but I’d like to think now that it’s coupled with a tiny amount of understanding; understanding that has, dare I say it, even let me have fun while working on this code!

Those of you that have used browsers on mobile phones (all of you?), especially not the very latest phones, may be familiar with an annoying problem. That is, elements that have position:fixed in their style tend to jump around the place like they’ve had too much coffee. You commonly see this on sites that have a persistent bar at the top or bottom of the page, or floating confirmation notifications, things like this. Brad Frost wrote about this far more eloquently than I could here. This has always annoyed me, especially after learning more about how browsers work. Certainly in Gecko, we have all of the context we need for this not to happen. It also ended up that this problem had been worked on long before I even joined Mozilla last year, so that made it doubly surprising that we suffered from this problem in all releases of Firefox Mobile.

When I first came across this last year, I discovered that the support was already there in the old Firefox Mobile, but disabled by default due to it causing test failures. I was working on other things then, and wasn’t at all acquainted with layout code, so I let it be. Revisiting it for the new, native Firefox Mobile though, these test failures didn’t exist anymore. Enabling this basic support that would let position:fixed elements move and zoom with user input correctly was not too big a deal – just flip an environment variable and write a small amount of support code. This landed in Firefox 15 and is tracked in Bug 607417. Just this is enough for a lot of mobile sites to start using position:fixed (I’m looking at you, Twitter and Facebook!).

This wasn’t enough for me though. Around this time, Android 3.x (Honeycomb) tablets had been around for quite a while and the Galaxy Nexus with Android 4.0 (Ice-cream Sandwich) had just come out, both with even better support for position:fixed. Not to mention the iPhone, which has excellent support. A problem with our implementation in Firefox 15, is that anything fixed to the bottom or right of the screen, or anything that doesn’t anchor to the top-left in any way, may become inaccessible after zooming in. In recent versions of the Android stock browser, not only do these remain accessible, but they zoom very smoothly too. Not wanting to be one-upped by what could be considered our main competition, I started to work on more comprehensive position:fixed support. This work was tracked in Bug 758620.

When zooming in our browser, we don’t change how the page is laid out, but fixed position elements are still rendered relative to the viewport. What we want (at least, for now) is for fixed position elements to lay out with respect to this viewport so that they always remain visible, and to transition smoothly between these states. To achieve this, I changed layout so that fixed position elements are laid out to what we call the scroll-port. When we zoom in, we change the scroll-port, otherwise you wouldn’t be able to scroll to the bottom-right of the page, but this only changes scrolling behaviour and nothing else. This change also made it so that fixed-position children of the document would be relayed out when this scrollport was changed. This fixed the inaccessibility problem, but left nasty-looking transitions when zooming in.

To fix the transitions was quite a bit more comprehensive and lead me down a long path of causing and fixing various layout bugs. When a page is rendered, the DOM tree is parsed into a frame tree, which better represents the layout of the page. This frame tree is then parsed into a display-list, which represents how to draw the page. This display-list is then optimised and parsed/drawn into a layer tree, which is the final representation before we composite it. There’s cleverness to make sure that layers aren’t recreated unnecessarily, but that’s another subject for another time. We wanted to be able to annotate the layer tree so that when compositing, we have enough information to determine how to place fixed-position layers when zooming. This involved creating a new display-list item with the information about how the element is fixed (to the top? left? right? bottom?), and also that would guarantee that the items representing this element would end up on their own layer. Once this was done, code in the compositor was added to leverage this information and draw layers in the right place.

This is an area that a lot of browsers have difficulty with, so it was a fun problem to work on. Try out a couple of my test-cases if you’re interested, they expose how different browsers handle this situation, and in the case of a few of them, some bugs We’re still not perfect, but we’re better than we were before – and to my feeling, we’re adequate now. This work landed in Firefox 16.

So is there work left to do? Well, unfortunately, yes. I’ve just finished off support for fixed backgrounds and backgrounds with multiple fixed/non-fixed layers, and this will arrive in Firefox 18. This is tracked in Bug 786502. I also think that the best behaviour would be for fixed position elements to layout to whichever is largest of the CSS viewport or the scroll-port, and for scrolling to be within the CSS viewport and push the edges when you reach them. I’m told this is what happens in IE10 on Windows 8, and is similar (but slightly better) to what gets done in Safari on iOS. I think it’s about time for a break from this particular feature for me, though.

]]>

As some people know, I’ve had a little bit of a health scare recently as the first knuckle on my index finger was swollen with a variety of other pains occurring in my hand joints. This combined with some long running unanswered questions and a family history had my GP say effectively “eeek, rheumatoid arthritis?” just from showing him my hand with no other comment. Thankfully blood tests for RA are negative, as are the other blood tests for more exotic diseases they decided to run just in case.

Six weeks on, this raises the question of what is wrong with it since its not really much better and one side of the joint is still swollen. The conclusion is its literally collateral damage, specifically a torn collateral ligament. Its likely this was pressuring tendons causing problems elsewhere and generally confusing things. Treatment is to keep it moving, gentle use. Time to heal, up to 18 months. It could be worse,at least I know what it is now.

Knowing what I do now, how do I suspect I did this? I have a suspicion trying to undo the front sprocket nut on the YZ might have been the trigger. That saw me jumping on a 2m pipe over the ratchet to loosen it in the end.

Last night I merged the work by our lovely interns Emilia and Mihai to add a media server image to Guacamayo. Basically this is a DLNA “Digital Media Server” implemented using Rygel’s media-export plugin.

It’s early days, mainly because it only exposes the demo content so far and there is no easy way to add or remove media (SFTP as root doesn’t count!), but it’s certainly a solid step in the right direction.

Thanks Emilia and Mihai!

Setting the browser to GNOME default.

Edit -> preferences -> Attachments -> http
Edit -> preferences -> Attachments -> https

Set to /usr/bin/gnome-open

Edit -> preferences -> Advanced -> Config editor -> search for “network.protocol-handler.app.http” set value gnome-open

search for “network.protocol-handler.app.https” set value gnome-open

Came across this problem due to migrating my settings and not having the “/usr/bin/firefox-bin” or what ever it was previously set to. This should ensure it works a bit more robustly on GNOME.

Thursday and Friday was Ixion@Cadwell 2012. Cadwell Park is a race track suited to motorcycles in the middle of rural Lincolnshire. The way it makes use of the natural landscape to form one of the UK’s most technical motorcycle circuits is to me, beautiful, I’ve loved it since I first saw it as a spectator. The event consists of a day, an evening and another day of track time in three groups (Tiggers, Piglets, Eeyores) and this was the eighth time I’ve been. As a piglet, I know my way around the circuit but am schizophrenic often showing eeyore tendencies to admire the scenery and only occasionally showing signs of the bouncy fun of the tiggers who tend to like the scenery so much they are known for engaging in close examination of it. This year they colour coordinated the wrist bands so mine was a lovely piglet pink.

The view from the clubhouse showing the track entry, the top of the mountain with the pit straight and the entry to the mountain in the background

I missed the event last year so it has been a while since I’ve been to Cadwell and I’ve not done as much riding of the Daytona as I’d like having given priority to the trail riding. Thursday was red hot sunshine. The first session is the infamous “no-brakes” drill where you ride around the circuit without touching the brakes. I love the fact you can still go over 100mph down the straight, scrub the speed off up the hill into Coppice and not have to touch the brakes, that has to be my favourite corner. I also discovered you can go around Mansfield quicker than I’ve typically done so in the past. It helped being as hot as there was no worry about any moisture affecting traction and the tyres were extremely sticky. Since I’m not commuting on the bike it does have sports compound tyres on it which helps (dual compound commuting tyres on the VFR were never as sticky).

It took some time to remind myself which gear to be in where on the track, I quickly remembered not to use first gear anywhere after I attempted it. Some corners I seemed better on (Hairpin, Mansfield), others I seem to have regressed (Charlies 1, Barn). In the first proper session I was surprised to find the front end going light over the mountain (front wheel in the air) as I hadn’t intended it and try and avoid it. The offroad riding does seem to have helped as I was much less unhappy about it than I would once have been.

The Hairpin

Hall Bends

I had a session with one of the new instructors following and then leading me which was interesting. It highlighted that I was disjointed between Coppice and Charlies which is something I tried to work on without an awful lot of success during the day. Other than that, the usual advice still applied, “just go faster”, my lines were basically ok.

At lunchtime I topped off the fuel tank with 10L from the cans which I was hoping would see me through the day which it did (the fuel light came on for the last two laps of the evening session). The evening session was split into two groups, Pigores and Tiglets of which I opted for the former. It highlighted that whilst I could circulate at a faster pace than some of the Eeyores, I have little experience of overtaking and was very reluctant to do so other than down the straights where I could use the 675’s acceleration and straight line speed to my advantage. Right at the end there wasn’t enough time for two sessions so they announced an open pit lane. I took a short break, then went back out for a final session. The organiser came past me at the end of park straight and I was quite interested in his line and the way he went into Park. The trouble was I was so busy paying attention to that I forgot to brake myself and steamed into the corner at a much increased speed. I did touch the brake slightly on the apex but managed to force myself to leave it alone and hope the bike would go around the corner which it did, being capable of much more than the rider. In total I covered 186 miles on the track on Thursday, which is quite some mileage given the concentration required for that kind of riding. I added a further 10 miles to refuel the bike at the end of the day.

Friday started with another no-brakes drill which I have to say are a useful learning exercise. The day was cloudy and the temperatures were much lower. After the no-brakes drill the front tyre didn’t look well, covered in peeled rubber that had stuck back on (“cold tearing”?). After the first session it was much worse and also I realised rather cold compared to the temperature it should be. I ended up dropping the front tyre pressure quite considerably to get it to match the wear and temperature of the much healthier looking and warmer rear.

Whilst I have no evidence of it, I think I did slowly make improvements to my speed around the circuit through out the day. Towards the end of the day, I was talking to someone about improving and whilst I could do some things they mentioned (e.g. find the throttle stop on the straight and avoid braking for coppice at least some of the time) there were other things I could definitely work on. They also mentioned an interesting tip about going deep into charlies 1 so that you could accelerate for park straight earlier. In the next session I went into that corner knocking off 15% speed to try the different line which did seem much nicer. It was so much nicer I arrived at the end of the straight with much more speed than I’d ever done before and had to brake considerably harder for park which made life interesting. They happened to be circulating in that session and I followed them into coppice as they overtook me at that point, roughly carrying the same speed at entry. What was clear was that I was losing speed at the corner apex at which point they steamed away from me at a point where I do feel I want to go faster but I’ve lost momentum. It showed exactly where I’d need to carry more speed and another rider who overtook there demonstrated the same effect. Something to work on next time. In the last session of the day, the fuel light came on again, beautifully timed.

I covered 285 miles on the track over the two days and 370 miles in the van getting there and back. There were a few crashes although thankfully it was mainly damage to people’s wallets. It was noticeable how many other 675s there were this year, they’re becoming common with about six of them there! They still look best in red to me though and despite being repeatedly told otherwise, there was only one red 675 there (which people were confusing with a red CBR600).

As anyone who runs oe-corePoky on a development distribution, you’ll get a warning when you start bitbake because the distribution is unsupported:

WARNING: Host distribution "Debian GNU/Linux unstable (sid)" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.

Fair enough, but I’m tough enough to deal with that. Luckily you can silence this warning. Take the distribution identifier out of the warning and then append it to SANITY_TESTED_DISTROS in your local.conf, for example:

SANITY_TESTED_DISTROS += "Debian GNU/Linux unstable (sid)"

Voilà, no more spurious warnings.

As I’m now working on the Yocto Project, I’ve a new i7 build machine which builds all of the distro with -j8 for speed (and builds up to 8 packages at once, just to make sure that all the cores are busy). I don’t actually know what -j level the autobuilders are using but they’ve 24 cores each… Anyway, lots of code is being built daily with high Make parallelism, so we’re good at finding subtle races in makefiles. Debugging these isn’t trivial or obvious at first, so I thought I’d blog about a few that I’ve encountered recently.

telepathy-glib

| Making all in telepathy-glib
| make[2]: Entering directory `/buildarea1/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/telepathy-glib-0.19.2-r0/telepathy-glib-0.19.2/telepathy-glib'
| /bin/mkdir -p _gen
| ( cd . && cat versions/0.7.0.abi [...] versions/0.19.2.abi  ) | \
| 		/bin/grep '^tp_cli_.*_run_.*' > _gen/reentrant-methods.list.tmp
| /bin/sh: line 1: _gen/reentrant-methods.list.tmp: No such file or directory
| make[2]: *** [_gen/reentrant-methods.list] Error 1

So it creates a directory, and then fails to create a file? The hint is that the error is “no such file or directory” which tells you that _gen/ isn’t present. What isn’t obvious from the output is that make is running the mkdir and the subshell containing reentrant-methods.list in parallel, which you can confirm by looking at the makefile. It’s rather large, but the gist of it is that the rule that does the mkdir isn’t a dependency of the code that generates reentrant-methods.list, so they must be dependencies of some higher target and are therefore being run in parallel.

Most of the time the mkdir happens first but occasionally the subshell wins the race and _gen/ doesn’t exist yet. Once this was understood it’s a simple matter to add some missing dependencies to the makefile.

gThumb

This was more fun. When building with any level of parallelism, make would busy-loop forever. Annoying on your desktop, not so funny on a build server.

When make is running tasks sequentially, it knows when the task has been completed it can check to see if files have appeared and so on. This logic changes with any level of parallelism because multiple things are happening at once. Strangely make solves this by busy-looping, watching for file changes (you can see this with --debug). Generally the expected files either appear or there is an error, but in this case make was spinning for ever.

Digging into the rules for the enumeration generator shows some dependenceis that are not required, and rather complex logic when putting the generated files in the right place. Complicated, and Doing It Wrong.

Writing to a temporary file and then atomically moving that to the right file is a good thing, and essential in parallel builds, as otherwise dependent rules could read a partially-written file. But this makefile is comparing the temporary file with the target and copying the file only if it’s different. This looks like an attempted optimisation to reduce rebuilds caused by the enum timestamp changing (won’t work: the enum re-generation is happening for a reason, so the rest of the source will rebuild too) and this is what is causing the problem: make is waiting for a file to change when it won’t ever change. Once this is understood the fix is simple and results in a cleaner makefile.

WebKitGTK

Oh, WebKit… The one package that you need to build with -j to get a build time less than two days, and it exposes a bug in Make 3.82 causing it to fail with -j. Thanks for that, Make. For reference this is the WebKitGTK+ bug and this is the two-year old Make bug.

Had a great time at GUADEC

Did a quick talk about the Dawati User testing tool Insight recorder

news of my death abandonment of the GNOME community have been greatly exagerated.

seriously: I’m still here at GUADEC (typing this from the common area); I’m still on the Board of the GNOME Foundation; and I’m still working on GNOME tech, like Clutter, GLib, and GTK+.

I am also working at Mozilla (and we’re hiring! ), but I also worked at OpenedHand and at Intel, and that never stopped me from actually doing stuff on the side; lots of people in this community do this — you don’t need to be full time employed with a company to contribute to GNOME, or to try and give the projects goals and direction.

on Sunday, I tweeted this:

[tweet https://twitter.com/ebassi/status/229551373044305920]

if it doesn’t show up, here’s what I wrote: we were always a bunch of friends working on stuff we loved in the face of unsurmountable odds. here’s to 15 more years.

it’s very true that we lack resources. we always did. it’s also true that we are competing in a space that does not leave us much room. we didn’t get 20% of the desktop market either, though. we’re doing what we do not because of the market share, or because of the mind share, or because we want to be paid. we write GNOME, we document GNOME, we design GNOME, we translate GNOME because we love GNOME. you would need to pay us not to work on GNOME.

everyone here at GUADEC is aware that hard times are upon us; we (presumably, though we don’t have any real metric to define that) have lost users. we definitely have lost sponsors. it’s not the first time, and I suspect it won’t be the last. what we haven’t lost are our passion for what we do; our mission, to provide a free environment for users to work with; and our willingness to drain all the swamps we have in the Free Software world.

if you want to work with us, join the GNOME Foundation — both as a member or on the advisory board if you are interested in sponsoring the project. help out in one of the many teams, not just with code, but with design, documentation, translation, marketing, web development, and mentoring.

we have so much work to do ahead of us to not only stay relevant, but to fullfill our mission, and blaze the trail to the future of Free and Open Source Software — we’ve got to get to it.

I’ve been pretty lax on blogging lately, but here’s something that’s troubling me. I haven’t really done any work directly related to Gnome since I started working at Mozilla. Ends up writing browsers is pretty hard, and any recreational programming time I get, I don’t particularly feel inclined to work on Gnome. I have, however, been attending Guadec this week. I haven’t missed one since 2006 and I don’t intend to. What’s troubling me, is that although Mozilla were kind enough to sponsor my presence here (we’re hiring!), Gnome doesn’t seem to be hugely relevant to us anymore. I’d love to be corrected of course, but judging by the amount of effort we’re putting into the Gtk+3 port, themeing and other Linux-related bugs, I’m pretty sure this is the case.

I have some ideas about this, but I’d like to be brief. For now. So, my simple question is, How can Mozilla and Gnome work better together?

[Edit]: Seems my blog’s commenting form is broken. Until it’s fixed, feel free to mail me your comments, I’d love to hear them! (address on the side of the page)

[Edit2]: Comments appear to be working again, but if they fail, do mail me!

]]>

Being a good Git citizen I often use git add -p and git rebase -i to produce a perfect patch series that doesn’t reflect the reality of creating it, but is far easier to review and understand later. Then, of course, a rebase can go terribly wrong…

Today I was doing one last rebase to squash a few typo fix commits, and accidentally squashed the wrong commits.  I was left with a single commit that claimed to do one thing, but actually has several unrelated (but more important) changes.  Urgh.  One solution would be to do another interactive rebase and use git add -p to split the commit, but I thought I’d try using the reflog.

The Git reflog is a log of all changes that have happened to the tree, including all the stages of apparently destructive and deep rebase operations.

$ git reflog
 ...
 9f46daa HEAD@{142}: rebase -i (finish): returning to refs/heads/gtkdoc
 ...
 8d7f266 HEAD@{166}: rebase -i (squash): glib-2.0: cleanup thanks to new gtk-doc.bbclass
 a8f06b1 HEAD@{167}: rebase -i (squash): updating HEAD
 0ac2f59 HEAD@{168}: rebase -i (fixup): # This is a combination of 3 commits.
 a8f06b1 HEAD@{169}: rebase -i (fixup): updating HEAD
 cccef4d HEAD@{170}: rebase -i (fixup): # This is a combination of 2 commits.
 a8f06b1 HEAD@{171}: rebase -i (fixup): updating HEAD
 c7ef1a6 HEAD@{172}: checkout: moving from gtkdoc to c7ef1a6
 14bdd14 HEAD@{173}: commit: glib squash
 ...

From the bottom up, the checkout of a hash is the start of the rebase. Then there are some fixups and squashes. whoops! The top squash shouldn’t be squashed, but picked. Never mind, we can checkout the tree as it was before the rebase with git checkout -b gtkdoc2 14bdd14, which I’ve highlighted. Double check that the tree is what you expect, and then the old branch can be deleted and the new one renamed. Braver people may want to simply git reset --hard 14bdd14, but the double safety net is useful.

Update: for a more concise method, see glandium’s reply in the comments.

Last weekend should have been the Northumbria TRF Forest Trail Ride. Unfortunately that got cancelled by the Forestry Commission due to the amount of water logged ground. We offered the attendees a trail riding weekend instead.

Friday night wasn’t promising with rain pouring down all night. The camping field drains really well but even that was covered in pools of standing water. Its times like this I appreciate having a decent tent. Saturday morning arrived and amazingly whilst everything was wet, it was not raining. I had a choice of leading a group of four other locals or taking some visitors out and opted for the locals. We set off and on the second trail, one of the members bikes stopped and wouldn’t start. We spent a while dismantling various bits of the bike, draining off the float bowl and ended up changing the plug. It started and seemed to be working again. At the next short trail, I reached the end and there was nobody following. I had seen them turn off for it. I headed back to find the bike had died again. We ended up arranging for him to get picked up and we continued with the route.

We soon ran into the another group who were basically doing a similar route. They’d given time to let us get ahead but the breakdown had set us back. We devised a plan to overtake them by filling up at a different fuel station and this went to plan and we overtook. We were also skipping some of the trails due to the amount of water and the likely bad ground conditions on them.

After a few more trails, I noticed on a road section that the bike was revving a lot more freely than the changes in speed would suggest. Either the wheel was slipping, or the clutch was. The next trail proved interesting with the bike not behaving as it should being hard to pull away, stalling and generally being snatchy. I also noticed the clutch cable had no slack in it, in fact being more like a banjo string. I stopped and checked the adjusters which were all maxed out, never a good sign.

I’ve had a complete clutch failure before and I vowed then that if I ever had signs of one again, I’d stop and head home whilst I still had drive. I therefore made my apologies to the group and headed back to the camping field on my own. I was back there by 1pm. Thankfully, because the group were locals, they could find their way to the lunch stop without me where they joined with group we were trying to stay ahead of. If I’d been leading the visitors there would have been a more significant problem.

The problem also meant that I was out of action for Sunday although with the light drizzle falling on Sunday morning, I wasn’t too upset about that. At least the problem should be a simple fix with a new set of clutch plates likely to resolve the problem. I did have a look at the clutch when I got back and the steels are in good condition with no notches on the clutch basket either.

Today, North Tyneside was hit by the kind of storm you usually only hear about or see on the news. I watched from my office window looking out over the bay as the sky got darker and darker to the point cars started using headlights, the sky was like a stroboscope and there was the gentle rumble of thunder in the background. The rain then arrived, hammering down and soon there was a river flowing down my street, on both the road and pavement. I couldn’t stop grinning, I can totally sympathise with storm chasers!

The police were soon on the scene to block off the road since it was flooding on the corner just down the road. I wasn’t too worried, there is a cliff and the water can only get so high. The back of the house is more of a risk but they’ve just put new special storm drains in there.

Its not often I get interrupted by a phonecall when in a meeting but I was, by my Mum who after discussing sausages mentioned in passing she was worried about their house and car and the rising water levels. With my Dad being away, I agreed to go up there. If its that bad out there, my only car option, a Mazda MX5 isn’t really appropriate. Easy, I’ll just use the YZ. I grabbed my road bike gear, put a change of clothes in plastic bags which in turn went into the backpack and set off. Traffic was chaotic as cars are allergic to more than 3″ of water and were driving all over to avoid it. I just rode through it all no problem, maybe 12″ on one roundabout.

So far so good until I tried to get into my parent’s estate to which the entrance was flooded. If I’m put off on a YZ, it must be bad. I did ride into it and made the mistake of going too slowly and carefully and the bike stalled. No chance of restarting it in water that deep (say 2.5-3ft) so I pushed it around to my parents house, rather annoyed with myself. The photo above is of the junction after the levels dropped a bit. Its hard to picture without knowing the area but the roads under there drop quite a bit level wise.

My Mum’s car, parked in the road was certainly at risk if the water got any higher so I moved it onto the drive. The house itself seemed to be ok, the risk would be flooding from the rear and the drains were holding up and draining.

The street around the corner was where the water was coming from and was also like a river. Basically, water off the fields behind the estate had nowhere to go other than through the estate.

I think a lot of people were lucky. Here it looks like the water made it up to the air bricks in the underfloor space but with any luck it didn’t make it into the house itself.

I went for a look around. This pond is where all the water above flowed to about half a mile away. There would be some very underwater houses here

Here, the road has been blown out by a sewer pipe bursting. When I rode over this on the way to my parents, there was water flowing through those cracks.

No surprise that this underpass was flooded. I didn’t try the bike through it, its deep in there even for the YZ.

The remains of the flood at the bottom of my street. I did ride through this one even it was cordoned off.

I went for a walk on the seafront and came across this on the upper promenade. Under Whitley Bay sea front, there is a 3m diameter storm sewer/storage pipe which runs for several miles. The idea is simple, when it rains, this gets used as a giant tank which fills and they can then slowly process the water later. Here, one of the access points to this huge sewer has been blown open. You can only imagine what that implies.

Here is an illustration of what it must have been like on a small scale. This is on the lower promenade where some of the storm drains were still lifting the covers and overflowing onto it. It looked like some kind of large scale water feature from this angle.

The water patterns in the sand here indicate the water must have been seriously flowing off the lower prom onto the sand.

I couldn’t resist ending with a shot of the sunset. This is basically the view from my office, the camera not doing the colours justice. The bay sweeping around ending with the lighthouse. Blyth’s piers and wind turbines silhouetted in the background. Beautiful.

I have an idea of where the app developers are.

they are in an ecosystem that put up with a language that calling niche 10 years ago was flattering.

or they are in an ecosystem that is so fragmented that you can forget complaining about three or four major distros, when you have to deal with five different major versions on dozens of devices and form factors, all with minimal or no upgrade paths.

more importantly, though, I think that the app developers are where stuff looks cool and where the community leaders don’t eat their children in constant whining on public venues.

public service announcement — I just revoked my trusty GPG key, which I also used to sign Clutter releases:

pub   1024D/A4320FF4 2000-09-18
      Key fingerprint = 4DD0 C90D 4070 F071 5738  08BD 8ECC DB8F A432 0FF4
uid                  Emmanuele Bassi 

the size of the key was ultimately too small to be trusted indefinitely. it’s been superceded by the following key:

pub   4096R/EA11BBB7 2012-06-22
      Key fingerprint = 53EF 3DC3 B63E 2899 271B  D263 22E8 091E EA11 BBB7
uid                  Emmanuele Bassi (GNOME) 
uid                  Emmanuele Bassi 

which will (hopefully) last at least as long as the old one, and possibly more.

today is my last day here, at Intel.

it’s been an honour and a privilege working with one of the best teams in one of the best companies in the world. as I leave behind people that humbled me and made me a better engineer, I cannot but feel a bit sad; still, all good things must come to an end — and I’m sure that the good things done will, in time, temper the bad things that happened during these nearly four years.

what will happen on Monday, I’ll leave for a latter blog post — but suffice it to say that I’m going to stick around the free and open source software community at large (and GNOME in particular) for a while longer.

I know what your question will be: what happens to Clutter’s maintainership? well, if the demise of Moblin and MeeGo, and the rise of Tizen, haven’t killed Clutter I suppose nothing short of an asteroid impact will. so if you were already planning to open the champagne bottle and celebrate me stepping down from my self-appointed position of Clutter (almost) benevolent dictator, I’m afraid you’ll have to postpone the party for a long while.

I also want to thank all the people that voted for me, as well as the other candidates in the GNOME Foundation elections for the Board of Directors.

finally, see you all at GUADEC 2012:

where I’ll give a talk about Clutter 2.0 and GTK+ 4.0.

Pleasantly surpised we made it into the great LWN

The esteemed editors at LWN have decided to write a small article about Guacamayo, giving an overview of the project and what our plans are.  At the moment it’s subscriber-only but on the 21st June it will be publicly available.  We’re quite happy with the result, Nathan contacted us earlier in the week to clarify some points that we hadn’t made clear so it’s probably the best source of information of Guacamayo now!

Part two of the coast to coast write up is now at http://www.rpsys.net/wp/rp/c2c/index2.

The day for the coast to coast arrived, we made it to the meeting point for 2am, set off in the Lake District from the west coast at 5:30am and headed for the east coast. The trip was not without incident although that wasn’t really a surprise.

People have been asking how it went and I’ve shared part one of a write up at http://www.rpsys.net/wp/rp/c2c. I’d warn its long and has lots of photos, mostly stills taken from the video footage. I took 20GB worth of video footage of the trip but there wasn’t time to stop and take photos, not that you could carry a decent camera trail riding anyway. Its taking some time to write up but I’ll try and get the rest of the write up completed soon.

“Coast to Coast” trips from the west to east coast of the UK are fashionable at the moment, be it walking, cycling or otherwise. For me, this weekend’s challenge is covering ~250 miles of green lanes by motorcycle, starting on the west coast near Ulverston and ending up near Alnwick on the east side, all in one day. Anyone with a map will notice this is a diagonal crossing, not a straight line and the route itself is even more convoluted to incoropate a good selection of lanes. There is a wide selection of terrain, including some deep fords which anyone reading this blog will know are associated with Alnwick.

All seemed to be going well with perparations, even my complete lack of transport for the bike on the day in question was to be resolved by dropping it off at a collection point in advance when I could borrow a van. I’ve decided to use the YZ over the CRM despite the CRM being the logical choice (better tank range, oil tank, comfy seat) mainly on fun factor although it also has the better tyres on it. I was about to put the YZ into the van to transport it and thought “I’d better just check it starts”. Turning on the fuel lead to a puddle forming under the bike with a continual stream of fuel flowing from the overflow pipes. Hmm, not good.

None of the normal tricks to resolve this helped so it was back into the garage where I removed the carb, along with half the bike or so it seemed after letting the person at the collection point know I’d be “a bit late”. I’ve never dismantled this part of the YZ before but that wasn’t going to stop me. There was gunge in the carb and the fuel valve seemed to operate ok with things cleaned up. The floats weren’t holed or anything nasty like that and once back together, it started and worked with no fuel leaking. I didn’t have any bits left over either which is always good.

I’m hoping this is my share of the mechanical gremlins and I’ve got them out the way now. The bike is waiting ready for us to meet at 2am on Saturday (tomorrow) at the collection point from where we set off to get dropped off in the Lake District. Then the fun begins!

For the last few weeks my spare cycles have been mostly spent on the Guacamayo Project; this is something that myself and Ross have been toying with for a while, and it’s probably time to say a bit about it.

In a gist Guacamayo is a specialised Linux distribution for networked multimedia devices; I say specialised, because the aim is not to produce yet another rehashed desktop distro with a bit of multimedia functionality on the side, but a system built from ground up for a pure multimedia experience.

The clearly defined focus allows us to do one thing in particular: we can ditch the traditional Linux desktop! The Guacamayo aim is to provide an intuitive gateway into a multi media world; the traditional desktop metaphor made up of workspaces, applications, documents and no end of toolbars and menus does nothing but stand in the way. Considering most of us have to put up with that sort of mess during working hours, I think we deserve better when it’s time to chill out.

Ditching the traditional Linux desktop has some other inherent benefits; we can forget about legacy technologies, not least the venerable X11 windowing system, and instead choose what makes best sense for creating that sort of user experience we are after.

So what are we doing:

• Distro based on Yocto; this was an obvious choice. Guacamayo targets multiple hardware architectures, so a proven cross-compilation environment is a must, plus Yocto makes crafting a carefully defined distro from scratch if not an easy, then a practical possibility,
• UPnP integration provided by rygel,
• PulseAudio for the sound,
• GStreamer as the core medial framework,
• Shell based on OpenGL, provided by MediaExplorer, aka MEX. Like Yocto, this has been an obvious choice, since MEX provides a really cool, modern, intuitive interface.

Supported HW? We are not focusing on any HW in particular. Our aim is to create a distro that could be used on a broad variety of suitable HW.  The current development is done using the Zotac zbox (Intel Atom) and the Beagleboard (Arm Cortex A8), and we fully intend to support Raspberry PI (eagerly awaiting HW).

We did our first release code named ‘See No Evil, Hear All You Want!’, aka 0.2, last week. As the name suggests, this is a limited functionality audio-only release, to get folk interested to come along and start testing and contributing. The next stable release is planed to include MEX running under X11, as a stepping stone toward a pure OpenGL system beyond that.

If you are interested, the source is here, you can drop by #guacamayo on Freenode, or follow @MetaGuacamayo on Twitter.

Last week we (well, mostly Tomas if I’m honest) made our first release of Guacamayo, a reference Linux distribution for media playback devices in a networked world. The core technologies are the usual suspects: Yocto for the distribution, GStreamer and PulseAudio, Rygel and Media Explorer.

The first release caters for the “connected speakers” use-case.  On boot it connects automatically to the network over ethernet (wi-fi coming soon) and using Rygel exposes a UPnP/DLNA MediaRenderer, with hot-plugged USB speakers automatically switched to if plugged in. I’ve been happily using it on a laptop with my Raumfeld system, and Tomas has tested it on a BeagleBoard with a UPnP control point app on his Android phone.

So, what’s next?  I’m working on a web-based configuration tool for the headless systems, and Tomas is integrating Media Explorer so we’ll have something you can plug into a TV.  Tomas is testing this on a Zotac ZBox at the moment, and any week now I’ll have a RaspberryPi to experiment with.

If you’re interested and want to know more, we have a small wiki at GitHub and you can often find us in #guacamayo on FreeNode.

I’ve been meaning to migrate away from pyblosxom to WordPress for a long time now, but have never found the time to work out how to migrate the many comments.  Then my blog host crashed and the comments were all lost, problem solved!  Some hacking and pain later, and here I am blogging with WordPress on hardware with support. Maybe I’ll blog more, maybe not.  Who knows what the future holds!

today is the deadline for submitting candidacies for the election of the GNOME Board of Directors.

I decided to run again this year: it took me a bit to get into the role, but I think I can work with the fellow Board members, as well with the rest of the people in GNOME, to ensure the proper functioning of the Foundation now.

remember: being on the Board of Directors doesn’t mean having a fancy title, or a shot at managing the GNOME project towards a technical goal; it mostly entails providing the means by which the +GNOME project can actually function. without the Board and the Foundation, we could not organize hackfests, or send people to GUADEC, or even have GUADEC.

being a Director is not a huge amount of work: a 1hr meeting every two weeks, and at most one hour every day for emails and IRC discussions; but it’s necessary work, and it makes all the difference between a functioning community that can provide infrastructure to a complex project and a project that can only use public and free services without any guarantee of continuity. imagine not relying on all the services on gnome.org; or imagine not having any funds to organize the successful hackfests we’ve had in the past few years.

if you are a GNOME foundation member, consider running – if not for a specific goal, just to help out making GNOME successful in bringing together people to provide a first class software platform.

One memory that springs to mind this evening is something tglx said at a plumbers conference about modern Linux hackers of the modern generation not being able to hold to traditional values, like properly enjoying beer!

I think this evening we proved that wrong. If the team here in Romania was trying to get me drunk, they’re going to have to try harder but it was an enjoyable evening and I think there are people who appreciate those traditional values, long may it continue!

For those that aren’t already aware, many Mozillians gathered last week in Toronto to Maximise Synergy. Seeing as there have been updates for the UX team and the Firefox/Mobile UI team, I think it’d be worth having a similar update for mobile platform. Some fantastic work is going on in this area, and people deserve kudos

• Kartikaya Gupta (aka kats) continues to do sterling work, fixing an inhuman amount of bugs. Most prominently, kats has been working on our display-port ‘strategies’, code that allows us to make the most of our rendering time and, thus, reduce checkerboarding. He’s also been working on our checkerboard measurement, augmenting and refining our tests and test-method to make sure we’re always pushing ourselves. This, alongside much general bug-fixing work.
• Benoit Girard (aka BenWa) is part of the gfx team, all of whom have been helping us out for the past few months, getting our performance back to a competetive level. Benoit’s tiled rendering patches finally landed, which make a huge difference to performance. Benoit has also been working on some useful remote profiling tools, which he blogs about here.
• James Willcox (aka snorp) continues to battle Flash to get us the best Flash experience in an Android browser (ironic, right?) snorp’s work to get more accurate positioning of Flash elements landed, along with his work to replace elements with a snapshot during page movement on Froyo/Gingerbread, completing the experience.
• Brad Lassey (aka blassey) managed to get enough time to do some coding alongside his usual cat-herding responsibilities, and got us a low-res page cache. This means that in the rare situation that our rendering can’t keep up, instead of showing checkerboard, we show a low-resolution rendering of the page. It’s possible that this content can go stale, but we find it provides a much better experience than showing either a blank colour or checkerboard. This works much better than any of us expected it to, and for me at least, was the turning point after which I find our browser much easier to recommend.
• And me? I managed to get my retained tiles patch landed. Currently, we render the page into system-memory tiles, which we then upload to the GPU. My patch keeps a hold of the GPU tiles and metrics for a while after they’re invalidated. Again, when our renderer can’t keep up, it renders these old tiles into the space instead of rendering nothing. This mostly helps quick changes of direction and zooming out after zooming in. I also spent some time fixing up the low-res cache work, then fixing up my fix-ups, as is my custom Hopefully, preliminary fixed-position layer support will also land soon.

Much other work also occured, and sorry for missing anyone out that I surely did. It’s worth mentioning that the gfx team have been a huge help and have been working extra-hard for months now, helping us get to where we are today (and hopefully beyond!) I had a great (and hard) time during my stay and am very much looking forward to our upcoming release

]]>

For reference for my future self, a few handy sed commands. Let’s consider this file:

$ cat test-sed
First line
Second line
--
Another line
Last line

We can extract the lines from the start of the file to the marker by deleting the rest:

$ sed '/--/,$d' test-sed 
First line
Second line

a,b is the range the command, here d(elete), applies to. a and b can be, among others, line numbers, regular expressions or $ for end of the file. We can also extract the lines from the marker to the end of the file with:

$ sed -n '/--/,$p' test-sed 
--
Another line
Last line

This one is slightly more complicated. By default sed spits all the lines it receives as input, '-n' is there to tell sed not to do that. The rest of the expression is to p(rint) the lines between -- and the end of the file.

That’s all folks!