I’m tired of fighting with WordPress post formatting, so here, have a post on Medium.

I’ve read and enjoyed parts of Michael O. Church’s take on Venkatesh Rao’s Gervais Principle, but he took his blog offline before I was able to read them all. I decided to snag copies of all of the posts from the Wayback Machine and turn them into an ebook for convenient reading. This appears to be a solved problem thanks to this handy little Python library, pypub, but just in case it winds up being useful to someone here is what I did.

Building the list of links

I googled for the first post, looked it up in the Wayback Machine, noted a working URL, and kept following links until I found all of the posts in the series.

For reference, here they are.
1: https://web.archive.org/web/20150325112004/https://michaelochurch.wordpress.com/2013/02/19/gervais-principle-questioned-macleods-hierarchy-the-technocrat-and-vc-startups/
2: https://web.archive.org/web/20150705103513/https://michaelochurch.wordpress.com/2013/02/20/1410/
3: https://web.archive.org/web/20151221082352/https://michaelochurch.wordpress.com/2013/02/22/gervais-rehash-part-iii-markov-and-management-plus-a-4th-culture/
4: https://web.archive.org/web/20151221143632/https://michaelochurch.wordpress.com/2013/02/26/gervais-macleod-4-a-world-without-losers/
5: https://web.archive.org/web/20151221143322/https://michaelochurch.wordpress.com/2013/02/28/gervais-macleod-5-interfaces-meritocracy-and-the-effort-thermocline/
6: https://web.archive.org/web/20151221143806/https://michaelochurch.wordpress.com/2013/03/01/gervais-macleod-6-morality-civility-and-chaos/
7: https://web.archive.org/web/20151018091056/https://michaelochurch.wordpress.com/2013/03/11/gervais-macleod-7-defining-organizational-health-the-mike-test-and-vc-istans-fate/
8: https://web.archive.org/web/20151221083041/https://michaelochurch.wordpress.com/2013/03/12/gervais-macleod-8-human-nature-theories-x-y-z-and-a/
9: https://web.archive.org/web/20151221144342/https://michaelochurch.wordpress.com/2013/03/14/gervais-macleod-9-convexity/
10: https://web.archive.org/web/20151221143541/https://michaelochurch.wordpress.com/2013/03/17/gervais-macleod-10-the-pull-of-lawful-evil/
11: https://web.archive.org/web/20151225141206/https://michaelochurch.wordpress.com/2013/03/18/gervais-macleod-11-alignment-and-careers/
12: https://web.archive.org/web/20151221143250/https://michaelochurch.wordpress.com/2013/03/19/gervais-macleod-12-growth-chaos-and-risk/
13: https://web.archive.org/web/20151221143502/https://michaelochurch.wordpress.com/2013/03/20/gervais-macleod-13-separability-work-and-play/
14: https://web.archive.org/web/20151221144410/https://michaelochurch.wordpress.com/2013/03/21/gervais-macleod-14-expanding-alignment-plus-well-adjustedness/
15: https://web.archive.org/web/20151221144210/https://michaelochurch.wordpress.com/2013/03/22/gervais-macleod-15-what-is-being-rich/
16: https://web.archive.org/web/20151221143221/https://michaelochurch.wordpress.com/2013/03/25/gervais-macleod-16-healthy-culture-vs-why-you/
17: https://web.archive.org/web/20151018092426/https://michaelochurch.wordpress.com/2013/03/26/gervais-macleod-17-building-the-future-and-financing-lifestyle-businesses/
18: https://web.archive.org/web/20151221144309/https://michaelochurch.wordpress.com/2013/03/28/gervais-macleod-18-more-on-trust-square-root-syndrome-brownian-and-progressive-time/
19: https://web.archive.org/web/20151221144051/https://michaelochurch.wordpress.com/2013/03/29/gervais-macleod-19-living-in-truth-fighting-the-lie/
20: https://web.archive.org/web/20150919091259/https://michaelochurch.wordpress.com/2013/04/01/gervais-macleod-20-bozo-bit-vs-simple-trust/
21: https://web.archive.org/web/20150910154001/https://michaelochurch.wordpress.com/2013/04/03/gervais-macleod-21-why-does-work-suck/
22: https://web.archive.org/web/20150919133351/https://michaelochurch.wordpress.com/2013/04/16/gervais-macleod-22-inferno/
23: https://web.archive.org/web/20151221143922/https://michaelochurch.wordpress.com/2013/04/30/gervais-macleod-23-managers-mentors-executives-cops-and-thugs/
24: https://web.archive.org/web/20151221082517/https://michaelochurch.wordpress.com/2013/06/04/gervais-macleod-24-fundamental-theorem-of-employment/
25: https://web.archive.org/web/20151221083106/https://michaelochurch.wordpress.com/2013/06/08/gervais-macleod-25-state-and-truth-seekers-plus-the-emergence-of-the-technocrat/
26: https://web.archive.org/web/20151221082619/https://michaelochurch.wordpress.com/2013/08/06/gervais-macleod-26-r-and-k-selection-in-organizations-and-capitalism/

Creating the ebook

pypub made this bit super easy!

import pypub, os
epub = pypub.Epub('Gervais Principle Questioned')
urls = open('links.txt').read()
real_urls = [x for x in urls.split() if x.startswith('https:')]
for u in real_urls:
  c = pypub.create_chapter_from_url(x)                                                                                                     
  epub.add_chapter(c)          
epub.create_epub(os.getcwd())  

Et voila!

In the immortal words of Samuel L. Jackson, hold on to your butts. I’m going to explain, in painful detail, just exactly what it took to get ProgressTwitch off the ground.

Watch live video from Comptona5 on www.twitch.tv

Genesis

This all started when my friend Camarice explained to me what Twitch was and how it worked. I was impressed by the technology, but it all seemed a bit frivolous to me at first. I thought, “Wouldn’t it be funny if there was a Twitch stream that just broadcast some static image, 24/7?” I was wracking my brain trying to think of things that would be appropriate to broadcast, and suddenly I remembered Progress Quest. For those of you who are not old, Progress Quest is a satire of the role-playing game genre. Like any RPG, you roll up a character and set out on your quest; but unlike any other RPG, Progress Quest plays itself for you, randomly generating a variety of quests, encounters, and loot. You just leave it running and check in every now and again to see how powerful your character has become.

This immediately seemed like a perfect fit: set up a system to run Progress Quest and stream it to Twitch 24/7.

What Didn’t Work

Twitch Streaming in Linux

The only computer that I leave on 24/7 is my homemade DVR, running Linux. My first thought was to set up Twitch streaming from Linux directly, to set up a virtual screen and run Progress Quest in WINE, and then to stream the whole shebang to Twitch. That plan disintegrated almost immediately; I was unable to get any Twitch broadcasting software running on my Linux box. It’s not particularly powerful, and it’s running a distribution that’s about three years old, so I wasn’t really surprised; nevertheless, this was definitely a dead end.

Virtualbox Windows VM in Linux

Next, I decided to try setting up a portable virtual machine with a Windows guest, configuring everything in it to start Twitch streaming at boot up, and then run that on my Linux server. Spoiler alert: this was also unsuccessful, although for a long time it seemed like it was going to work out OK.

I started by configuring a Virtualbox VM with a completely legitimate and legal copy of Windows 7. Once that was installed and running, I started installing the Twitch streaming software. It turns out that Open Broadcaster Software requires some DirectX features that are not supported by the DirectX emulation available in Virtualbox, even with the guest extensions and custom drivers installed. I then tried XSplit Gamecaster, with which I was able to get most of the way to a working setup. It took a lot of tweaking to get everything running the way it was supposed to, looking right, and starting up on boot, but eventually I had a working portable VM that would stream Progress Quest to Twitch automatically. I copied that VM to the Linux server and launched it, and lo and behold… a strange graphics corruption bug that put a big black rectangle near the middle of my screen, and also in the stream. I tried to update the drivers on my Linux box, but short of a full distribution upgrade to something more recent, it wasn’t going to happen; and I don’t want to tweak my DVR too hard for fear of blowing it up and having to rebuild it. On to the next option!

What Did Work

Amazon Web Services Windows VM

When that failed, I decided to abandon my plan of running anything on my existing Linux server, and instead set up a Windows VM in Amazon EC2. That actually went relatively smoothly, with one big exception: EC2 virtual machines do not have sound hardware, and OBS requires a sound card be present in order to function. VB-CABLE Virtual Audio Device to the rescue! After installing that virtual device driver and setting it as the default, I was able to leave OBS running even when I wasn’t connected to the Remote Desktop session.

One setup note here: the VPC Amazon made for me did not have a default route out to the internet, as pointed to here. I was unable to access my VM until I went in to the VPC configuration and added one by hand.

Final Setup

For future reference, this is what I’ve got running now:

  • Windows Server 2012 (ami-830ee0c7) on a t2.micro EC2 instance
  • Open Broadcaster Software 0.651 beta
  • Progress Quest 6.2
  • VB-CABLE Virtual Audio Device v.42b

I’ve been dealing with a very strange problem with my MythTV HTPC for a while now. Occasionally, for no good reason that I could see, both the frontend and backend processes would spike to 100% CPU utilization and hang out there until they were restarted. This caused the fans to spin up, which was annoying.

I had some time today to investigate this problem, and it turns out (at least so far) to be related to the UPnP feature. Read on for more about how I fixed it.

Read the rest of this entry »

Tags: ,

Quick one: today I learned that, when installing Windows 7 on a machine with multiple hard drives, it will refuse to install to a drive which is not set as the default boot device in the BIOS.

It doesn’t tell you this is why it won’t install, of course. It just says “I couldn’t find a valid drive to install to, go read the setup log to find out more.” If you’re inclined to figure out this problem, you then have to Google around a bit to actually find the location of the setup log files, read them through “type setupact.log”, and then happen to run into an error that could be interpreted as being related to BIOS boot order.

Hopefully this note saves somebody the several hours that I wasted today on this. =P

Tags: ,

For my wife’s birthday, I got her an Ouya game console so she could play an old emulated game (for which we own the cartridge, but our console is broken). I also recently ordered a Chromecast since hey, why not, it’s only $35. This required two more HDMI ports than our TV has, so I ordered a three-way HDMI switcher to plug everything into.

And it all works.

Read the rest of this entry »

Tags:

I wrote this one-liner today, and that it exists makes me sad.

rrdtool update $tmpfile $(curl "http://127.0.0.1:4242/api/query?start=$(($(date +'%s')+$start))&m=sum:$metric\{cluster=$cluster,hostname=$hostname\}" | python -c "import sys, json; j=json.loads(sys.stdin.read())[0]['dps']; sys.stdout.write(' '.join(['%d:%s' % (int(x), j[x]) for x in sorted(j.keys())]))")

That it actually works just downright terrifies me.

It seemed like a simple enough desire. I wanted a small, functional wiki with (roughly) the following features:

  • Use a git repository for page storage and version control
  • Map files on the filesystem directly to web pages (making it equally easy to edit from a CLI or from the web UI)
  • Render various wikitexts (particularly Markdown) into some appropriate output format
  • Support intra-page links and image includes, etc.
  • Be written in a language I am comfortable with

I considered several projects, including:

  • Hatta: Backed by a Mercurial repo, non-trivial to change
  • Giit: Written in Haskell. I had neither the time nor the inclination to learn that for this project.
  • Gollum: Mostly suitable, but written in Ruby (which isn’t my strong suit). Plus I wasn’t super-fond of the way it handled subdirectories and other complex filetypes.

But none of them quite scratched my itch. Thus, I was left with no choice but to write my own.

Introducing valet

valet is a script that turns any directory into a simple wiki, complete with wikitext rendering, editing, and automatically committing changes to version control.

Here’s a short list of valet‘s features:

  • Single file, depending only on bottle
  • Handles navigating around the filesystem rooted where you start it
  • Automagically detects the type of various files, and (if you have the supporting Python libraries installed) renders them as they ought to appear. As a bonus, if you have pygments installed, source code will be syntax-highlighted where possible.
  • Changes from the web UI are automatically saved into the git repo (if desired)
  • Runs as a standalone service, serving from your local machine
  • Optionally supports CGI and WSGI; rename the script to something.cgi or something.fcgi and it will Do The Right Thing TM. This makes it suitable(-ish) for serving through Apache or some such with SSL, authentication, access control, etc. etc.
  • Installable with pip!
pip install valet

Hopefully this proves as useful to other people as it has been for me.

Tags: ,

Yahoo runs a little-known service called Yahoo Pipes, which lets you take inputs from a wide variety of sources (including, but specifically not limited to, RSS feeds), spindle, fold, and mutilate them, and then get an RSS feed out the other end. This is particularly useful if you have an RSS feed from a source that does some of what you want it to do, but isn’t quite perfect.

Case in point: the head designer for Magic: The Gathering started posting podcast episodes on his Tumblr blog. However, they’re mixed in with other non-podcast posts, and Tumblr’s RSS feed doesn’t link the podcast files up properly so they work in a podcast reader. Obviously, this situation cannot abide. I created a pipe to magically transmute the plain RSS feed into a fancy podcast feed, and this post will show you how!

TL;DR: This is the podcast feed I wound up with.

Quick Introduction to Yahoo Pipes

Per Yahoo’s description:

Pipes is a powerful composition tool to aggregate, manipulate, and mashup content from around the web.

Like Unix pipes, simple commands can be combined together to create output that meets your needs:

  • combine many feeds into one, then sort, filter and translate it.
  • geocode your favorite feeds and browse the items on an interactive map.
  • power widgets/badges on your web site.
  • grab the output of any Pipes as RSS, JSON, KML, and other formats.

Getting Started

You’ll need a Yahoo account to create a pipe; that’s beyond the scope of this tutorial. Click on Create a pipe to get started.

The basic mechanism of this tool is adding new components from the left-hand pane (by either dragging-and-dropping them into the canvas, or clicking on the little “+” button next to each component). You can also link components together by dragging from the circle at the bottom of one component to the circle at the top of another component; the canvas will draw a nice line to indicate the flow of data within the pipe.

The other important thing to know is that clicking on a pipe component will change its color (to orange) and display whatever the pipe’s current output would be at that point in the “Pipe Output” pane at the bottom of the window.

When you’re done making a pipe, click Save, and then Back to My Pipes to see how it worked out.

Fetching Feed Items

For this feed, I used the XPath Fetch Page “Source” component, with an XPath expression that gave me just the body DIVs I wanted:

Building a RSS Feed

The next step is to turn that list of XPath results into a proper RSS feed. Add the Create RSS “Operator” component, and link it to the XPath Fetch Path component. Here are the parameters I used to create the RSS feed:

  • Title: item.div.0.h3.a.content
  • Description: item.div.1.p
  • Link: item.div.0.h3.a.href
  • media:content url: item.div.0.h3.a.href
  • media:content type: item.”audio/mp4a-latm”

To break this down a little bit, each result from the XPath Fetch Page component becomes an item in the Create RSS component. You can poke around the original page content to figure out exactly how to specify the particular bits you want to use to fill the various fields. Create RSS will then iterate through each XPath result and make a new post in the RSS feed with that XPath item as its source.

Simple example

Let’s say the XPath Fetch Page component returned three items, which looked like so:

<div>
    <h1>First Post Title</h1>
    <p>Wonderfully clever post goes here.</p>
</div>
<div>
    <h1>Second Post Title</h1>
    <p>Second post; slightly less clever</p>
</div>
<div>
    <h1>Third Post Title</h1>
    <p>At this point we're just phoning it in.</p>
</div>

For this example, the Create RSS component will make three objects. If you specify the Title as item.div.h1, then the resulting RSS feed will have three <item>s, each of which will have a title child node with the appropriate title text.

Making the RSS feed into a Podcast

It’s not enough to have a fancy RSS feed of your posts; if you want to get them recognized by most podcast readers, you’ll need to add some metadata that indicates where the podcast file can be found, what kind of media it is, etc. etc.

For this step, you’ll need a Loop “Operator” component. Within the Loop component, drag and drop a Item Builder “Sources” component.

We want to add two attributes to each item in the RSS feed: url and type.

  • url: item.div.0.h3.a.href
  • type: audio/mp4a-latm
  • assign first results to item.enclosure

Note that this url definition is the same as the “Link” and “media:content:url” definitions of the Create RSS component. Probably at least one of these is overkill.

Link this component to the Pipe Output component, and enjoy your new podcast feed!

Other Applications

Yahoo Pipes is wildly flexible; I’ve used it to create RSS feeds for lots of different things:

  • Pages which update regularly in a structured format but which do not provide an RSS feed
  • Excluding authors I don’t care to hear from in an existing RSS feed
  • Replacing the short text slug in an abridged RSS feed with the full article text from the page (Loop + Fetch Page are your friends here)

Feel free to post in the comments any pipes you make with these instructions, or any suggestions you have to improve them (or the podcast feed I made).

When I was working recently on changing poodledo‘s parser from using a lexer to using regular expressions, I found myself implementing some crazy logic to try and ignore the results from matched groups whose matches I didn’t actually want (since I was using parentheses for their “pick one of these two alternatives” function instead of their “pull out this specific chunk as a match” function).

For example, take the string “task title @context name #due date”. Starting with the at-sign, I want to match all words until some other symbol is reached; or, to avoid confusing the parser, match everything following the at-sign and between two square brackets. Here’s the regex I came up with:

[sourcecode language=”python” light=”true”] (^|\W)@([\b ]?(([\w\"\’.@]+ )*[\w\"\’.@]+)[\b]??|[\b(.+?)\b]) [/sourcecode]

Using this regex, though, gave me a lot of extraneous matching groups:

[sourcecode language=”python” light=”true”] >>> q = r'(^|\W)@([\b ]?(([\w\"\’.@]+ )*[\w\"\’.@]+)[\b]??|[\b(.+?)\b])’ >>> task = ‘task title @context name #due date’ >>> re.findall(q, task) [(‘ ‘, ‘context name’, ‘context name’, ‘context ‘, ”)] [/sourcecode]

How do I determine programmatically which of these groups is the one I want?

Digging around in the Python regex documentation, I found a section about non-capturing groups, which seemed like it would save me a significant amount of sanity. Apparently, Perl-5-compatible regular expression parsers often implement a regex extension syntax by following an open parenthesis with a question mark, and then some other symbols that indicate which extension is in use. In this specific case, “(?:” indicates a group whose matches shouldn’t be captured in the output list. Eureka!

[sourcecode language=”python” light=”true”] >>> q = r'(?:^|\W)@(?:[\b ]?((?:[\w\"\’.@]+ )*[\w\"\’.@]+)[\b ]??|[\b(.+?)\b])’ >>> re.findall(q, task) [(‘context name’, ”)] [/sourcecode]

Now the only logic I need is “Ignore matching groups which are empty”, my sanity is preserved, and all is well with the world.

Tags: ,

« Older entries