SwiftUI Webview with a Progress Bar

If you’re not familiar, SwiftUI is the new UI framework for building user interfaces in Swift apps. If you squint, the declarative syntax is vaguely reminiscent of contemporary JavaScript frameworks like React.

With a declarative Swift syntax that’s easy to read and natural to write, SwiftUI works seamlessly with new Xcode design tools to keep your code and design perfectly in sync. Automatic support for Dynamic Type, Dark Mode, localization, and accessibility means your first line of SwiftUI code is already the most powerful UI code you’ve ever written.

developer.apple.com/xcode/swiftui/

The API is still fairly immature. When you get beyond the sample apps, it’s hard to build anything real with the components that come out of the box. Components that you would expect to see if you’re familiar with UIKit, like WKWebView and UIActivityIndicatorView don’t exist in SwiftUI yet.

Luckily, it’s not that hard to create them yourself.

To get started with a basic view, you need an object that implements UIViewRepresentable. A simple Webview could look like this:

struct Webview: UIViewRepresentable { let url: URL func makeUIView(context: UIViewRepresentableContext<Webview>) -> WKWebView { let webview = WKWebView() let request = URLRequest(url: self.url, cachePolicy: .returnCacheDataElseLoad) webview.load(request) return webview } func updateUIView(_ webview: WKWebView, context: UIViewRepresentableContext<Webview>) { let request = URLRequest(url: self.url, cachePolicy: .returnCacheDataElseLoad) webview.load(request) } }

Progress Bar Example

It’s also possible to model a UIViewController by implementing UIViewControllerRepresentable.

For example, a view controller that renders a web view with a progress bar:

class WebviewController: UIViewController { lazy var webview: WKWebView = WKWebView() lazy var progressbar: UIProgressView = UIProgressView() override func viewDidLoad() { super.viewDidLoad() self.webview.frame = self.view.frame self.view.addSubview(self.webview) self.view.addSubview(self.progressbar) self.progressbar.translatesAutoresizingMaskIntoConstraints = false self.view.addConstraints([ self.progressbar.topAnchor.constraint(equalTo: self.view.topAnchor), self.progressbar.leadingAnchor.constraint(equalTo: self.view.leadingAnchor), self.progressbar.trailingAnchor.constraint(equalTo: self.view.trailingAnchor), ]) self.progressbar.progress = 0.1 webview.addObserver(self, forKeyPath: "estimatedProgress", options: .new, context: nil) } // MARK: - Web view progress override func observeValue(forKeyPath keyPath: String?, of object: Any?, change: [NSKeyValueChangeKey : Any]?, context: UnsafeMutableRawPointer?) { switch keyPath { case "estimatedProgress": if self.webview.estimatedProgress >= 1.0 { UIView.animate(withDuration: 0.3, animations: { () in self.progressbar.alpha = 0.0 }, completion: { finished in self.progressbar.setProgress(0.0, animated: false) }) } else { self.progressbar.isHidden = false self.progressbar.alpha = 1.0 progressbar.setProgress(Float(self.webview.estimatedProgress), animated: true) } default: super.observeValue(forKeyPath: keyPath, of: object, change: change, context: context) } } }

Then you can implement that as a UIViewControllerRepresentable like so:

struct Webview: UIViewControllerRepresentable { let url: URL func makeUIViewController(context: Context) -> WebviewController { let webviewController = WebviewController() let request = URLRequest(url: self.url, cachePolicy: .returnCacheDataElseLoad) webviewController.webview.load(request) return webviewController } func updateUIViewController(_ webviewController: WebviewController, context: Context) { let request = URLRequest(url: self.url, cachePolicy: .returnCacheDataElseLoad) webviewController.webview.load(request) } }

You can see a working example of this view controller over on GitHub.

Highly Available Node

At VIP, we run a highly available Node service that powers much of our platform. One of the biggest challenges we see teams face is the question of how to scale a highly available API.

That’s a broad problem to solve, but let’s assume we already have adequate test coverage and everything in front of the API taken care of for us. We only care about things we can change about the Node app itself.

Our typical answer looks something like this:

  1. Use Node’s cluster module to fully take advantage of multiple CPUs
  2. Gracefully reload worker processes for deploys and uncaught exceptions

Node Cluster

Node’s cluster module uses child_process.fork() to create a new process where communication between the main process and the worker happens over a unix socket.

The TCP module’s server.listen() function hands off most of the work to the main process, allowing child processes to act like they’re all listening on the same port.

HTTP Server Example

Let’s take a simple http server as an example. Here we have a server that listens on port 3000 by default and returns Hello World!. It also throws an uncaught exception 0.001% of the time to simulate a bug we haven’t accounted for.

/** * External dependencies */ const { createServer } = require( 'http' ) module.exports = createServer( ( req, res ) => { if ( Math.random() > 0.99999 ) { // Randomly throws an uncaught error 0.001% of the time throw Error( '0.001% error' ) } res.end( 'Hello World!\n' ) } ).listen( process.env.port || 3000 )

Obviously a real server would be much more complex, but this toy example will be adequate for this example. We could run this server with node server.js and we’d have an http server running on our server.

The first thing we’ll do is use Node’s cluster module to start one copy of the server per CPU, which will automatically load balance between them.

#!/usr/bin/env node /** * External dependencies */ const cluster = require( 'cluster' ) const WORKERS = process.env.WORKERS || require( 'os' ).cpus().length if ( cluster.isMaster ) { for ( let i = 0; i < WORKERS; i++ ) { cluster.fork() } cluster.on( 'listening', ( worker, address ) => { console.log( 'Worker %d (pid %d) listening on http://%s:%d', worker.id, worker.process.pid, address.address || '127.0.0.1', address.port ) } ); } else { const server = require( './server' ) }

This will start one copy of the server for each CPU in our system. The operating system will take care of scheduling these processes across the CPUs.

Graceful Reload

Now that we have multiple processes, we can gracefully reload these in case of errors and for deploys.

Errors

In case of errors, we terminate the worker process and spawn a new one. This is important because an uncaught exception means the process is now in an inconsistent state. In other words, an exception occurred that was not accounted for and we’re not sure what side effects that will have.

First, we’ll ensure that worker processes are restarted if any exit unexpectedly. In the isMaster branch:

cluster.on( 'exit', ( worker, code, signal ) => { if ( ! worker.exitedAfterDisconnect ) { console.log( 'Worker %d (pid %d) died with code %d and signal %s, restarting', worker.id, worker.process.pid, code, signal ) cluster.fork() } } )

Here worker.existAfterDisconnect would be true if we call worker.disconnect() or worker.kill(), but false if the worker itself calls process.exit(). That becomes important in this next step, where we automatically terminate the worker process in the case of an uncaught exception.

const SHUTDOWN_TIMEOUT = process.env.SHUTDOWN_TIMEOUT || 5000 process.on( 'uncaughtException', error => { console.log( error.stack ) // Stop accepting connections and exit server.close( () => process.exit( 1 ) ) // Force shutdown after timeout setTimeout( () => { process.exit( 1 ) }, SHUTDOWN_TIMEOUT ) } )

We stop connecting new connections with server.close() and terminate the process with process.exit( 1 ) when all existing connections are closed. Since we want to ensure this worker is stopped within a reasonable timeframe, we force it to close after 5 seconds.

Deploys

For deploys, we gracefully reload all the worker processes one at a time to avoid any downtime in the process.

In the worker, we look for the main process to send a message that simply says “shutdown”. This again calls server.close() to stop accepting new connections and terminates the process when all active connections have closed.

const server = require( './server' ) process.on( 'message', message => { switch( message ) { case 'shutdown': server.close( () => process.exit( 0 ) ) return } } )

Upon SIGHUP we create one new worker for each active worker and gracefully shutdown the old worker when the new one is ready to accept connections.

process.on( 'SIGHUP', () => { console.log( 'Caught SIGHUP, reloading workers' ) for ( const id in cluster.workers ) { cluster.fork().on( 'listening', () => { gracefulShutdown( cluster.workers[ id ] ) } ) } } )

Gracefully shutting down a worker involves a few steps.

First, we send the shutdown signal that the worker is listening for and disconnect. As mentioned before, when all the connections are closed, the worker process will terminate itself. Again, since we want to ensure this worker is stopped within a reasonable timeframe, we force it to close with worker.process.kill() after 5 seconds.

const SHUTDOWN_TIMEOUT = process.env.SHUTDOWN_TIMEOUT || 5000 const gracefulShutdown = worker => { worker.send( 'shutdown' ) worker.disconnect() const shutdown = setTimeout( () => { worker.process.kill() }, SHUTDOWN_TIMEOUT ) worker.on( 'exit', () => clearTimeout( shutdown ) ) }

Upon SIGINT or ^C, we’ll perform a similar graceful shutdown routine. The only difference is that we don’t need to restart each worker this time.

process.on( 'SIGINT', () => { console.log( 'Caught SIGINT, initiating graceful shutdown' ) for ( const id in cluster.workers ) { gracefulShutdown( cluster.workers[ id ] ) } } )

To prevent the initial SIGINT from propagating to worker processes and immediately terminating them, we’ll handle the signal separately there. The first one is ignored, but if you press ^C or otherwise send SIGINT twice, all threads are closed immediately, bypassing the graceful shutdown.

process.on( 'SIGINT', () => { // Ignore first SIGINT from parent process.on( 'SIGINT', () => { process.exit( 1 ) } ) } )

I hope this was helpful. You can see the full example on GitHub.

Wire 1.5

There have been several recent updates to Wire focused on making the app more responsive and easier to use.

The original goal of Wire was to build an RSS reader that renders content in the format of the website, instead of a stylized view of the text. The whole idea is that a website is more than just what’s in the <content:encoded> tags in an RSS feed, but the CSS and JavaScript that browsers render as well.

There are downsides of loading the URL of an article in a web view though — namely, the overhead of downloading the article and then rendering it. On a fast connection, it’s noticeable. On a slow connection, it can be annoying.

To that end, the last couple of releases have been focused on improving that aspect of the experience. As of version 1.4, Wire downloads every article, which improves performance and makes offline viewing possible. As of version 1.5, articles are pre-rendered to make the transition from the article list to the web view as fast as if we were just rendering displaying the text from the RSS feed.

I’ve been using this for a couple weeks and the more responsive feels like magic.

Search & Replace PHP Serialized Strings

Or how we improved the WordPress database migration speed by 40x with Go.

At VIP we move WordPress databases around many times per day and as you can probably imagine, our customers tend to have of a lot of data. When we move data between environments, one of the bottlenecks has historically been search and replace of the domain names. The two main use cases are importing data containing development domain names and moving production data to a hosted development environment.

The best option in most situations is WP-CLI’s search-replace command. WP-CLI is fairly ubiquitous at this point and it’s easy to use. The problem that we tend to have is that it’s not quite fast enough on really huge datasets.

You may be wondering how a simple search and replace could be the bottleneck. WordPress stores lots of PHP serialized strings in the database. Since object lengths are encoded in serialized strings, you can’t simply search and replace domain names in the database unless they happen to be the same length.

php > echo serialize( "google.com" ); s:10:"google.com";

WP-CLI deals with this by pulling the objects out of the database, unserialzing them, running recursive search and replace on the resulting object, reserializing them, and then updating the database. While this only takes a few seconds on most WordPress sites, it can take many minutes or even hours on sites with millions of post objects.

Since our migration process is based on .sql files, we thought it might be faster to run the search and replace outside WordPress as long as we could reliably fix the encoded lengths. So, I wrote Automattic/go-search-replace to do that.

There are two main things that happen:

  1. The search and replace. We simply replace one domain with another.
  2. Fix encoded string lengths as necessary.

It turns out that dealing with the string lengths is not as hard as we originally thought. Modifying nested objects is not a concern because we’re not changing the number of nested items. We only need to look at the string length immediately preceding the string we’re replacing.

Another problem that could have been hard to deal with is maintaining the integrity of the .sql file. It would be easy to replace the characters that are used by the MySQL importer to delineate records, like ),\n(. Our solution is to limit the search domain roughly to the characters that make up valid domain names.

Using this new command line tool, we were able to improve the search and replace process by about 25x. I wondered if we could make it even faster using concurrency.

One challenge is that we need to ensure every line in the resulting file is in the same place. For that, we use a buffered channel of channels. For each line, we write a new string channel to the channel and asynchronously write to that channel. Effectively, we put a placeholder on the channel synchronously, and write to it asynchronously. Here’s that code on GitHub.

We’ve been using this in production for over a year now. In some tests we’ve seen the search and replace performance improved by up to 40x. Since we use STDIN and STDOUT, we can use go-search-replace as part of an import pipeline. In some cases, the search and replace runs at line rate and has no effect on the total import time.

Wire

Wire is an RSS reader for iOS that displays articles with their native formatting — or an optimized mobile version for sites that support AMP — and doesn’t require yet another account to sign up for.

As mentioned previously, it had been quite some time since I wrote any iOS apps, so I decided to use some of my free time this summer to build an app that I wanted to use.

There are other RSS readers. It’s not exactly new territory. But like most apps, the good ones all seem to require you to sign up for another account. I don’t know about you, but I find the number of accounts I already have to worry about somewhat overwhelming. I don’t need anymore. Relying on an iCloud account won’t solve this problem for everyone, but in this case it solves it for me.

The other main feature I wanted in an RSS reader was the ability to disable the monotonous E-reader-like view that has become so common. So many of the websites I read look great on mobile.

As you can see, it’s a pretty standard design. There’s a simple list of articles that you can group however you want. The article view loads a selected article in a web view. I really prefer this real view of the site to the reader-ized version other apps use.

I have found Wire to be quite good for what I want. It doesn’t do much more than aggregate articles from the websites you want to keep up with, but I think it does that well.