Wire 1.5

There have been several recent updates to Wire focused on making the app more responsive and easier to use.

The original goal of Wire was to build an RSS reader that renders content in the format of the website, instead of a stylized view of the text. The whole idea is that a website is more than just what’s in the <content:encoded> tags in an RSS feed, but the CSS and JavaScript that browsers render as well.

There are downsides of loading the URL of an article in a web view though — namely, the overhead of downloading the article and then rendering it. On a fast connection, it’s noticeable. On a slow connection, it can be annoying.

To that end, the last couple of releases have been focused on improving that aspect of the experience. As of version 1.4, Wire downloads every article, which improves performance and makes offline viewing possible. As of version 1.5, articles are pre-rendered to make the transition from the article list to the web view as fast as if we were just rendering displaying the text from the RSS feed.

I’ve been using this for a couple weeks and the more responsive feels like magic.

Spying TVs are getting cheaper

But the most interesting and telling reason for why TVs are now so cheap is because TV manufacturers have found a new revenue stream: advertising. If you buy a new TV today, you’re most likely buying a “smart” TV with software from either the manufacturer itself or a third-party company like Roku.

Noah Kulwin in The Outline

It is so creepy when Roku TVs show a message to “continue watching from the beginning” when you’re watching something on an Apple TV. I assume the TV is constantly sending frames of whatever is on screen to Roku servers for analyzing. It seems unlikely that the TV is capable of doing this recognition on its own.

The first time this happened, I finally broke down and bought a Raspberry Pi so I could set up Pi-hole.

I can’t believe this spying is not a huge story.

One Month of AirPods Pro

The initial reviews of the AirPods Pro were incredible, if not a little hard to believe.

It’s true that the noise cancellation is very good though. I work in cafes regularly and still find it sort of incredible how good they are at cancelling out background noise. If someone is having a loud conversation right next to you, you can kind of hear it if the volume is low enough. The background buzz of people talking is completely gone though.

They also, predictably, stay in my ears much more reliably than the previous AirPods. If you get them, definitely try all the tips. I used the medium ones for two weeks and they were fine, but the smaller ones fit even better.

Transparency mode is the killer feature I feel like nobody is really talking about — it’s audio AR. Paired with a future pair of AR glasses and maybe a watch, you can start to see the path to making smartphones obsolete.

The only (small) problem I have so far is that transparency mode is unusable with any kind of hat that covers your ears, which means I won’t be using it very much for the next few months.

Overall, I find the AirPods Pro very exciting.

josh.blog 2.0

I must have a problem staying away from code for too long. After being on vacation last week, I decided to redo the CSS on this site over the weekend.

The main focus this time was keeping the design as simple as possible and ensuring the text is easy to read. The craziest design element is three little dots that divide articles.

For the colors, I used the Duck Duck Go Color Picker — literally search for “color picker”.

Search & Replace PHP Serialized Strings

Or how we improved the WordPress database migration speed by 40x with Go.

At VIP we move WordPress databases around many times per day and as you can probably imagine, our customers tend to have of a lot of data. When we move data between environments, one of the bottlenecks has historically been search and replace of the domain names. The two main use cases are importing data containing development domain names and moving production data to a hosted development environment.

The best option in most situations is WP-CLI’s search-replace command. WP-CLI is fairly ubiquitous at this point and it’s easy to use. The problem that we tend to have is that it’s not quite fast enough on really huge datasets.

You may be wondering how a simple search and replace could be the bottleneck. WordPress stores lots of PHP serialized strings in the database. Since object lengths are encoded in serialized strings, you can’t simply search and replace domain names in the database unless they happen to be the same length.

php > echo serialize( "google.com" );
s:10:"google.com";

WP-CLI deals with this by pulling the objects out of the database, unserialzing them, running recursive search and replace on the resulting object, reserializing them, and then updating the database. While this only takes a few seconds on most WordPress sites, it can take many minutes or even hours on sites with millions of post objects.

Since our migration process is based on .sql files, we thought it might be faster to run the search and replace outside WordPress as long as we could reliably fix the encoded lengths. So, I wrote Automattic/go-search-replace to do that.

There are two main things that happen:

  1. The search and replace. We simply replace one domain with another.
  2. Fix encoded string lengths as necessary.

It turns out that dealing with the string lengths is not as hard as we originally thought. Modifying nested objects is not a concern because we’re not changing the number of nested items. We only need to look at the string length immediately preceding the string we’re replacing.

Another problem that could have been hard to deal with is maintaining the integrity of the .sql file. It would be easy to replace the characters that are used by the MySQL importer to delineate records, like ),\n(. Our solution is to limit the search domain roughly to the characters that make up valid domain names.

Using this new command line tool, we were able to improve the search and replace process by about 25x. I wondered if we could make it even faster using concurrency.

One challenge is that we need to ensure every line in the resulting file is in the same place. For that, we use a buffered channel of channels. For each line, we write a new string channel to the channel and asynchronously write to that channel. Effectively, we put a placeholder on the channel synchronously, and write to it asynchronously. Here’s that code on GitHub.

We’ve been using this in production for over a year now. In some tests we’ve seen the search and replace performance improved by up to 40x. Since we use STDIN and STDOUT, we can use go-search-replace as part of an import pipeline. In some cases, the search and replace runs at line rate and has no effect on the total import time.