The Connectors/Loaders Trap

Feb 23, 2023

There's a pattern in enterprise SaaS to accumulate as many "connectors" or "loaders" to your application as possible. Usually, these connectors are small snippets of code that make API calls to fetch data, usually via client credentials and oauth2. It's the underpinning of most ETL, reverse ETL, and workflow tools but finds itself in most application-level SaaS tools. Product owners do it for the following reasons:

  • It's easy. These are short snippets of code that don't do much besides move small amounts of data around. You can implement a few different connectors or loaders in an afternoon.
  • Distribution. Each connector and loader theoretically brings in a new audience that was previously blocked from your using your tool. You might not know a priori what services are complementary to your application.
  • Shipping product. Adding new connectors gives a sense of momentum to a product. Each new connector can be an announcement. The page of connectors can be a selling point to customers.

But the strategy is often a trap. It not only doesn't bring in the distribution you want, but it is a net negative for your product.

At the core of the problem is that connectors are easy to write but hard to maintain. APIs change over time (yours and theirs), libraries need to be updated, and code changes need to be made. You probably don't have an extensive test suite of connector or loader tests, and even if you did, how could you test across the wide variety of API functionality across multiple services (both documented and undocumented)? The least used connectors are perpetually broken, leaving the users who try them with a bad experience.

  • If it's core to your business (e.g., ETL platforms), you put humans to work. This is the strategy behind some of the largest ETL companies like Fivetran or Zapier. These support engineers might have relationships with the downstream providers or simply build out the tests and ops layer for their set of connections. You might choose to do this for the most popular connections.
  • Outsource it to one of the companies that do it as their main business (Fivetran, Mulesoft, Zapier, etc.).
  • Let the end-users write and run the code. Much easier if your product is open-source, but possible in proprietary SaaS via WebAssembly extension platforms nowadays. Of course, things will still break, but it's on the customers to maintain them.
  • First-class connections that become part of your application and test suite. Only doable when the number of services is small and the type of services are similar.
  • Open-source your framework and let the applications maintain their own connectors. You need to have a lot of platform power for this (you're driving distribution to your partners) for them to maintain the code. There might be some shared responsibility across connector consumers, but it's hard to drive ownership, and the connectors usually are left unmaintained (even if they're popular).

My advice? Discover complementary services in a code-free way – GitHub issues, support requests, forums, etc. Make the bar high to support connections. When there's a real unlock between services, try for a partnership first. Otherwise, keep the bar high and support the best services "in-tree." Remember, you're signing up for maintenance.