Beeminder Forum

API: goal resource escapes unicode in "title"

#1
GET https://www.beeminder.com/api/v1/users/me/goals/water.json?auth_token=nononogofetchyourown HTTP/1.0

gives:

{
  "slug": "water",
  "title": "\u0026 Co.",

That’s a & Co..

whereas the "delta_text" of:

GET https://www.beeminder.com/api/v1/users/me/goals/workout.json?auth_token=nononogofetchyourown HTTP/1.0

right now is:

  "delta_text": "✔ +0 +0.43",

Here’s what the RFC says about this: https://tools.ietf.org/html/rfc7159#section-7

I found this working on the dashboard. The json library I use choked on the title and reports the above as

{
  "slug": "water",
  "title": null
}

and ends the json document there. To me that looks like a bug on the library’s side. The RFC does explicitly say characters MAY be escaped.
Still, I’m reporting this here because I think the escaping in the title is unintended given that a) it needs no escaping and b) other attributes are not escaped either.
I entered the title via the website in top part of the goal page:
image

And by “title” I mean… description?

1 Like

#2

Taking a look at what’s stored in the database, we’re storing the unicode un-escaped, and the api is not explicitly escaping things. I just ran a test with one of my own things, and other unicode characters are fine, but when I copy in the description from your water goal, it’s giving the escaped unicode with curl, and showing up unescaped in other libraries… so something with the json parsing / library on the other end perhaps?

Here’s what I see via curl
"ϡ and ⎈ ⏰ ◒ \u0026 and hurray!"

and via my ruby library of choice
"ϡ and ⎈ ⏰ ◒ & and hurray!"

2 Likes

#3

Here’s my working theory:

  1. The API sends out ϡ and ⎈ ⏰ ◒ \u0026 and hurray!
  2. That’s what curl receives and that’s what the library I am using receives
  3. Your ruby library of choice then goes ahead and unescapes the \u0026 again before giving it to you
  4. The lib I’m using recently added a flag to control this behaviour. It is disabled by default because it puts additional strain on the already tiny processor in an Arduino, I assume. Using this flag the \u0026 does get translated back to & just as it should and the library does not choke on it.

I wouldn’t know why the API sends out the ampersand escaped but I have a suspicion which I hope is wrong: It might be that they think they do you a “favour” by escaping the oh-so-dangerous ampersand so it doesn’t to funny things in your HTML by accident. I hope I’m wrong :wink:

1 Like

#4

Ah, I suppose that’s probably it. Anyhow, you’re correct; I was looking in slightly the wrong place previously, and yes, Beeminder’s JSON library is “helpfully” escaping just-that-one unicode character.

I guess that means “It is the opinion of this Authoritative Source [i.e. a widely used JSON gem that’s probably the default thing that most Rails use], that escaping ampersands is the Correct Behavior”, in which case, I’m loath to change that default behavior without good reason.

2 Likes

#5

Totally agree. Either way, there is no “wrong” behaviour here as far as I see. It’s technically correct to escape all characters inside a JSON string as far as the RFC is concerned. If that’s desired or not is of course a different and subjective question :wink: One that ultimately depends on the… wait for it… character. :see_no_evil: :stuck_out_tongue:

1 Like