Proposal+spec for a "Plain Data" integration

[gbear605]

I’m proposing what I’m initially calling a “Plain Data” integration - that is, a Beeminder data source that can make requests to an arbitrary URL that returns just a number, and uses that number as a datapoint for a goal. It’s a bit like URLminder, and after discussion with @dreev on Discord it might make sense as just an extension on URLminder, but for now I’m proposing it as an entirely separate integration.

I’ve written up a three page spec here.

For my personal usage, it’d allow me to keep track of my flashcards with Mnemosyne. I have two goals, one for “scheduled” flashcards and one for “unmemorized” flashcards. My scheduled flashcards goal is currently powered by Make, but it was frustrating to set up, has been hard to maintain, and only pushes data twice a day. My unmemorized goal is currently just manual data, since I didn’t want to go to the effort of setting up another Make integration. Both of them would be trivial to set up with this type of integration.

[skorytnicki]

I like it. It’s very similar to, or the same as I proposed here:

Easier data pull proposal

Hey, I think it would be super useful to have something like this: [image] Currently, in order to create a custom data source that is called by beeminder, you have to set up the application, generate some tokens. If you do it, it’s not obvious how to share your integration with the others. You have to prepare UI for your app and store your users’ data in some safe way. How about we could simply insert the URL that beeminder would call couple of times a day? Nowadays there are plenty of fre…

There were some legit concerns in this thread:

Prioritizing autodata integrations

I think before we make it dramatically easier for users to create integrations, we need to improve our reporting. If we don’t expose what we’re calling, when we’re calling it, how long it took, what the response was, and how that affected the datapoints, I worry we’d be setting ourselves up for a massive amount of support–but if we had those things exposed, it’d be much easier for folks to “self-service”. I worry that without more of these details available, it could be more support per integr…

[dreev]

Ah, thank you so much for writing this up! (I bet against it in your market about it in order to add to your incentive, you’re welcome!)

One thing that has occurred to me is how elegant a generalization of the existing URLminder this could be. Like, suppose that URLminder asks for not just a list of URLs but what metric to glean from those URLs.

Brainstorming:

1. raw wordcount (status quo)
2. first raw number that appears
3. sum of all numbers that appear

We have lots of use cases for 1 and @gbear605 has a great one for 2. Of course coolest of all would be to somehow specify an arbitrary function, but let’s not get ahead of ourselves!

[gbear605]

A different “function” that might be helpful would be a simple JSON path, or maybe even something like JsonLogic to parse it. A lot of APIs might return something like {“my value”: 123}, and it would be nice to have some way to get at that. I guess first raw number would technically work for that, but a slightly more complicated API (eg. from a pedometer for “all time” data, {“steps”: 60000, “floors”: 703}) would need a way to get at a specific value.

That all gets more complicated, but if you’re already supporting multiple “functions” then it might not be so bad.

[dreev]

What if the “function” were a regex that transformed arbitrary text/html into a string that can be eval’d as arithmetic (not necessarily to be exposed that way in the UI)?

Then case 3 is just s/[^\d]+/\+/g. Cute!

But as @skorytnicki pointed out, we have some concerns from @adamwolf on this (even the easiest version) that means we’re not quite at green light for shipping something, sadly.

[theospears]

While I agree what Adam outlines would be the “gold star” version of this, I wonder if there are ways to reduce the risk to a tolerable level without doing all that work.

Some ideas, though you can probably come up with better ones:

• You have to ask support to turn this on for your account
• It’s only available in an expensive paid plan
• You have to pinky promise to accept all derailments and not contact support for issues or call non legit
• You can only set it up via the API

[skorytnicki]

You can also delegate the work of adding a datapoint to a client. So beeminder would just ping the URL without waiting for response or scraping the datapoint.

It’s worth mentioning that it’s perfectly doable right now, but you have to create your own beeminder app. It’s simply a bit steep. How about we create a template of a Vercel application (or AWS Lambda, or…) to facilitate it? I imagine scenario, where you just have to do:

1. git clone
2. set up your env variables (beeminder_user_token)
3. write datapoint logic
4. connect to beeminder

Just make it easier to use existing stuff?

[gbear605]

Part of my hope with this is that it could just be plug+play, without needing to run or host anything Beeminder specific, but something like a Vercel template that you can clone would be a decent middle ground at least.

[adamwolf]

Sliiightly offtopic (I haven’t used mnemosyne since maybe 2008 IIRC, so forgive me if this is obvious to everyone else): Is there an endpoint we can pull data from for users, or is that just an example for the spec?

[gbear605]

Good question! There is an endpoint now. I added it a couple years ago, to use for Beeminder with a Make integration. I love that that’s a possibility with open source software, though admittedly it does leave a question of how applicable this integration type would be for other data sources that don’t have an endpoint and can’t just have one added.

The Make integration I set up kind of works but I’ve had a lot of frustration with getting Make to do it correctly at the right timezone, especially when I’ve changed timezones and had to update:

• The Beeminder goal due time
• The Mnemosyne start of day time
• The timing of Make’s run to push data to Beeminder