« Back to home

Reclaiming the URL from ember-data

Posted on

It's great that ember-data 1.0 is coming. It's long overdue. But lest you get lulled into thinking that ember-data 1.0 is a finalized product, it's time to take a step back and look where it's deficient and at everything it's missing.

The adapter pattern is amazing. Produce a highly standardized thing (A), provide an interface for connecting it to any other thing (C), and add an easy ability to write an adapter (B) that connects the two. End result? Some serious magic.

The new Macbook doesn't have X port

A cheap adapter connects you to anything, and in a few years as technology progresses you forget why you ever even needed that adapter in the first place, if you ever took it out of the box at all that is.

Ember Data has embraced adapters, a decision that should be applauded; however, ember data's adapter pattern is broken.

Imagine this. Your Macbook only comes with a single port. No worry, you think, Apple sells an adapter that connects that single port to any other conceivable port! Now imagine that you buy your Macbook with that expectation, only to discover that while the adapter can connect to USB 3.0 devices, it can't talk to USB 2.0 devices. Or that while it can talk to Thunderbolt, it can't talk to Firewire. Or while it can talk to HDMI, it can't talk to Mini-HDMI.

Because you need your adapter to work properly, you contact Apple. "Don't Worry", Apple says, we built our single port around solid standards. It can easily connect to any other port that implements those standards. "But I need to connect to a port that doesn't", you say. "Oh." Says Apple. "In that case, you just need to connect to this other device, which can talk to the port you need. Then you need to stream data from that device to yours wirelessly."

I'm not alone in saying that if Apple were to fail this miserably, we would consider their adapter pattern broken. Why then do we give Ember Data a free pass?

How do I mean?

From a recent Slack conversation:

fivetanley [11:47 AM] want to use your own url? $.getJSON().then(payload => store.pushPayload(payload))

Stanley Stuart's argument is essentially that if Ember Data's adapter pattern can't suite your needs, you should just make plain old ajax requests and push the data directly into the store.

Let's ignore for a moment that this pattern currently results in you often needing to normalize data outside of the adapter, or for bizarre hack-arounds to do adapter lookups. That's something that ember-data can fix easily.

This is a core contributor's answer on how to deal with situations when your API doesn't conform to ember-data's expectations in a way manageable by your adapter. The intention is for you to just know that if you are dealing with an edge case, you fallback to ajax and manually push records.

The problem is, these edge cases are far more common than I think the core ember-data team realizes. Personally, I suspect even the most well maintained and standard conforming APIs will have roughly 10% of their endpoints contain oddities. API oddities are a fact of life for developers.

Assuming, for a moment, that the number of oddities is small, and that manually pushing records is easy. What might be the drawback of doing so?

For one, you must now maintain two different approaches for retrieving data for the same model. One for most cases (in your adapter), and one for edge cases (outside of your adapter). This leads to assumption and maintainability problems. Was this the model I needed to use $.getJSON => pushPayload to retrieve? Or can I call this.store.fetch?

The more edge cases you have, the more this confusion grows, and the harder it becomes to refactor code to keep up with the latest ember-data, and the harder it becomes to progressively refactor your own code as your APIs improve or change.

pushPayload is not a solution, it's a patch to a pernicious problem plaguing ember-data: the URL.

There's no reason ember-data's adapter pattern couldn't work with more flexible URLs. Separating out buildURL by request type was a great first step. Separating it out by find request type will be another.

Even with the current changes, ember-data is missing out on the power of the URL. Ironic, given that ember was the front-end framework to first truly grasp the power, meaning, and usefulness of URLs.

Ember Data needs to give control over the URL back to the developer.

ember-data should give developers a single, readable, maintainable API for data retrieval.

It could do this and keep the adapter pattern, but it would have to significantly alter it's surface APIs.

Instead of just fetch('foo', 1), also fetch('foo', 1, 'http://example.com/foo/1').

What does this do? It provides an easy override to buildURL for all those hard to generalize endpoints, while still utilizing the appropriate adapter method for fetch fetchAll fetchQuery or fetchOne (or whatever the final find and fetch API ends up being).

The problem with ember-data adapters isn't the adapter pattern, it's that the adapter pattern wrongly assumes easily generalized url patterns.

By focusing only on the payload, Ember Data misses out on the power and benefits of URLs.

By abstracting the URL away from the developer, ember-data has actually created a significant set of challenges where none existed before, or where there were existing solutions that could have been built over.

A few of the challenges created by ember-data's disregard for the URL:

  • Record Invalidation
  • Ordering
  • How to deal with non-standard or related URLs
  • How to deal with One-off endpoints
  • How to deal with Search endpoints
  • Endpoints which return partial (incomplete) Records
  • Detecting (server side) Deletions

You might disagree with me on a few of these right now, but you won't in a minute.

Record Invalidation

Let's start with record invalidation. Currently Ember-data has no solution to this. There needs to be a lot done in this area, but just from the URL perspective, what about cache-expiry or etags? This is data that could be used to determine how long data in the store should be considered valid.

Server Side Sorting

What about Ordering? Can't you just sort client side? If your server already returns sorted records, or contains a specialized sort order, ember-data destroys that order unless you use findQuery. You will find more situations in which you don't want to restrict yourself to findQuery than situations in which you do. And what if you are making the same findQuery request twice? Couldn't you simply return the result of the first? Except you can't, because even if ember-data did pay attention to data validity, it doesn't know about the correct server-side sort order. If ember-data were more URL aware, it would be able to preserve server ordering.

How to deal with non-standard URLs

The next challenges are all closely related. What happens when a URL in an API doesn't conform to the rest of it? Let's say, for instance, that your API can access the same data in multiple ways.

For instance, in an API all three of the requests below could be valid for the same comment list.

/api/comments?post=1
/api/posts/1/comments
/api/users/1/posts/1/comments

In this case, you pick one standard and move on. I don't disagree with ember-data's current implementation here. But what about the following situation.

500 /conversations  
200 /conversations?dealer=X&department=Y  

Here, the standard "list" API needs two query params. This is the equivalent of findAll but looks a lot like a findQuery, depending on how you implement it, you may have a lot less flexibility with what you can do with the return.

Both the above and below are examples of endpoints I use but don't control.

/customers/5550129876/phonenumber?dealer=X

Above is the search for a customer by the given phone number at X dealership. This is a findQuery in which the params are serialized via two different methods.

In fact, any search-like endpoint does not play well with ember-data. Go ahead, try. You'll consistently find the answer is to not use the adapter, which makes ember-data a hell of a lot less useful. You'll spend so much time outside of ember-data, you'll inevitably begin questioning why you bother pushing records into the store at all.

Endpoints which return partial (incomplete) Records

Sometimes endpoints lie. Take for instance our conversations endpoint. If it were to return every message, or even every message ID associated with each conversation, the payload could quickly become enormous.

Conversations are paginated, their messages are paginated. The /conversations index returns a page of results, each with just the lastMessage contained. To ember-data, this looks a lot like these conversations all have just one record, but the developer knows this is false.

My solution to this was rather, erm, erm, erm. Just take a look.

import Conversation from "../models/conversation";  
import DS from "ember-data";  
import Ember from "ember";

const {  
  observer,
  computed,
  run
  } = Ember;

function autoUnloadConversationPreview() {  
  Ember.run.schedule('backgroundPush', this, function () {
    var json = this.toJSON({ includeId : true});
    json.conversation = json.id;
    json.active = false;
    this.store.unloadRecord(this);
    this.store.push('inactive-conversation-preview', json);
  });
}

export default Conversation.extend({

  __destroy: observer('theLastMessage', 'theLastMessage.timestamp', function() {
    var now = (new Date()).getTime();
    var timestamp = this.get('theLastMessage.timestamp');
    var conversationTimeout = (1000 * 60 * 15 - (now - timestamp));
    run.debounce(this, autoUnloadConversationPreview, conversationTimeout);
  }),

  conversation: DS.belongsTo('conversation', {async: true}),

  unreadCount: computed.alias('conversation.unreadCount'),

  theLastMessage: computed.alias('conversation.theLastMessage')

});

The partial becomes it's own record. Once the full record is loaded, we map to it. This has all sorts of crazy inefficiencies, but solves a lot of data integrity / syncing issues that other approaches had. Ideally, only one record would be involved, an isPartial flag would be set, and a full copy requested when needed if isPartial=true. This is, in fact, what a newer version of this code will do, and how a similar problem was solved (better) elsewhere in the app.

Notice also, the automatic record invalidation that's been added here. This is one shout out to ember-data's flexibility, which is really a tribute to the power of Ember's object model.

Detecting (server side) Deletions

Ember Data doesn't know. That's all on you bud. But wait! you exclaim. If my index no longer contains a record, it should be removed. Well, honestly, in any data situation that sort of decision implementing that behavior is going to be on you. While not uncommon, that's just not standard.

Does that mean Ember Data can't make use of this information? No. It could. If it were more URL aware, if it knew what records it received from a specific URL and in what order it received them, it could infer that data had been removed. If it allowed you to specify the merge strategy, it could easily drop those missing records.

Ember Data needs to be more URL aware.

When I say this, I mean it twofold. It (1) needs to give more control over the URL to the developer. And (2), it needs to keep track of and pay more attention to the data it receives from a URL.

By doing the former, ember-data would give developers a common API for data retrieval. Code written around ember-data would be more maintainable, readable, and understandable.

By doing the latter, ember-data would gain easy access to a wealth of new features: better local cacheing, server sort orders, deletes, invalidations.