Tracking Focusmate Completed Sessions on Beeminder

[felixm]

@adamwolf

Beeminder counts sessions where you’re ghosted by the other person. Focusmate probably does not. (I mean, they do track them–that’s where we get the data from.)

Hmm, yes. This:

#!/bin/bash
#
# Define the API_KEY variable
# API_KEY=<key>

# Define the years you want to query
years=("2021" "2022" "2023")

# Initialize a variable to store the total count of sessions
total_sessions=0

for year in "${years[@]}"; do
    start="${year}-01-01T00%3A00%3A00Z"
    end="$((year + 1))-01-01T00%3A00%3A00Z"

    # Get the session count for the current year and store it in a variable
    session_count=$(curl --location "https://api.focusmate.com/v1/sessions?start=${start}&end=${end}" \
                         --header "X-API-KEY: $API_KEY" | jq '.sessions | length')

    # Add the session count to the total_sessions variable
    total_sessions=$((total_sessions + session_count))
done

# Print the total count of sessions
echo "Total sessions: ${total_sessions}"

Gives me:

Total sessions: 842

FocusMate itself shows 800 and Beeminder 831. Ha!

Any comments, @nathansudds ?

It should have just been a simple endpoint with total completed sessions.

Edit:

Also, thanks to our AI friend, because clearly I don’t know Bash foo like this, haha:

for year in "${years[@]}"; do
    start="${year}-01-01T00%3A00%3A00Z"
    end="$((year + 1))-01-01T00%3A00%3A00Z"

Edit 2:

Also, 800 sessions? Yay!

[felixm]

I got this response from the Focusmate team:

So without doing any extensive verification, my guess it this is that this is expected. I took a look at the thread you linked and what Adam Wolf said is correct. In addition, Beeminder does not take into account the “completed” flag for each session. I forget the exact reasoning behind this, but I believe it’s because Beeminder prefers not to penalize someone for being the victim of a no-show (and the “completed” flag requires a partner).

As far as I understand, this means if I schedule a session at 8:00, the partner does not show up and I do not get a new partner, and then schedule another session at 8:15, Beeminder will count this as two sessions.

I think this defeats the purpose of the integration. @adamwolf could the integration (optionally) take the “completed” flag into consideration?

[adamwolf]

From a technical perspective–totally, it could. I think this is a
Danny/Bee question, though.

In general, we want to match what the upstream data provider shows
users as much as possible. This helps everyone debug things and
figure things out much more easily, which seems to push this to “use
the completed flag”.

However, if we switch to using the completed flag, is it going to be a
pain if you are at the last possible session time, trying to get your
minutes in, and some person ghosts you and now you’re outta luck? I
don’t use Focusmate so I don’t know the details here–what does “get a
new partner” mean? Is it usually relatively easy to get a new partner
if your first one doesn’t show up?

[felixm]

And it’s also in the spirit of the goal and Beeminder to only count completed flags.

Basically, a couple of things can happen:

1. You schedule a 8:00 session at 7:59 and don’t get a partner. This currently counts as a session, I think. I didn’t verify that but otherwise, I can’t explain the 40 sessions difference in my data.
2. You scheduled a session, get a partner, but your partner doesn’t show up. In this case, you can either a) finish the session by yourself (aka wait for your partner), or b) ask to be assigned to a new partner after two minutes. If no new partner is available, the session ends.

Case 1 is like going to the gym at 9:55 PM and then complain when they don’t let you in because the close at 10 PM and you complain because now you are failing your Beeminder goal.

Case 2 is more of an issue because if you schedule a session for 10 PM in the morning, get a partner, and they don’t show up, it’s not your fault. However, I think this can also be countered with response 2a.

To summarize, using completed sessions is the Beemindery thing to do, and there aren’t any problems that cannot be handled by a motivated user.

The only issue is that some people (like myself) will derail when you make the change, but we can either call non-legit or accept the improvement.

[felixm]

Okay, since apparently I am the only one who cares about data integrity [edit: this sounds passive aggressive, but I am not really upset], I gave in to my “fine, I will do it myself moment”. Que music for epic montage of GPT and yours truly coding.

First, we came up with this beauty of a script. I created a new goal /focusmate_ for the data. To use the script, update the global variables and pip install keyring and pydantic.

Focusmate data to Beeminder completed sessions only

import requests
import os
import keyring
import sys
from datetime import datetime, timedelta
from pydantic import BaseModel
from typing import List, Optional


BEEMINDER_AUTH_TOKEN = keyring.get_password("beeminder", "felixm")
FOCUSMATE_API_KEY = keyring.get_password("focusmate-api-key", "felixm")
FOCUSMATE_BASE_URL = "https://api.focusmate.com/v1"
USER = "felixm"
GOAL = "focusmate_"
BEEMINDER_URL = f"https://www.beeminder.com/api/v1/users/{USER}/goals/{GOAL}/datapoints.json"


class User(BaseModel):
    userId: str
    requestedAt: Optional[str]
    joinedAt: Optional[str]
    completed: Optional[bool]
    sessionTitle: Optional[str]


class Session(BaseModel):
    sessionId: str
    duration: int
    startTime: str
    users: List[User]

    def get_timestamp(self) -> int:
        return int(datetime.fromisoformat(self.startTime).timestamp())


class BeeminderDatapoint(BaseModel):
    id: str
    timestamp: int
    daystamp: str
    value: int
    comment: str
    updated_at: int
    requestid: Optional[str]

    def get_timestamp(self) -> int:
        return self.timestamp


def get_user_name(user_id: str) -> Optional[str]:
    response = requests.get(
        f"{FOCUSMATE_BASE_URL}/users/{user_id}",
        headers={"X-API-KEY": FOCUSMATE_API_KEY}
    )

    if response.status_code != 200:
        print(f"Unable to fetch user data. Status code: {response.status_code}")
        return None

    data = response.json()
    return data['user']['name']


def get_sessions(start_date: str, end_date: str) -> List[Session]:
    response = requests.get(
        f"{FOCUSMATE_BASE_URL}/sessions?start={start_date}&end={end_date}",
        headers={"X-API-KEY": FOCUSMATE_API_KEY}
    )

    if response.status_code != 200:
        print(f"Unable to fetch session data. Status code: {response.status_code}")
        return []

    data = response.json()
    sessions = [Session.parse_obj(session) for session in data['sessions']]
    return sessions


def post_to_beeminder(session: Session) -> Optional[dict]:
    if not session.users[0].completed:
        print(f"Did not add uncompleted session {session}.")
        return None

    if len(session.users) > 1:
        partner_user = session.users[1]
        partner_name = get_user_name(partner_user.userId)
    else:
        partner_name = "No partner."

    session_title = session.users[0].sessionTitle
    session_start_time = datetime.fromisoformat(session.startTime)

    start_day = session_start_time.strftime("%a") # format start time as day of the week
    duration_mins = session.duration // 60000  # milliseconds to minutes
    comment = f"{start_day}, {duration_mins} mins, {partner_name}"
    if session_title:
        comment += f", {session_title}"

    timestamp = session.get_timestamp()

    # Convert session start time to "daystamp" (date in "yyyymmdd" format)
    daystamp = session_start_time.strftime("%Y%m%d")

    data = {
        "auth_token": BEEMINDER_AUTH_TOKEN,
        "value": 1,
        "comment": comment,
        "timestamp": timestamp,
        "daystamp": daystamp
    }

    response = requests.post(BEEMINDER_URL, data=data)
    if response.status_code != 200:
        print(f"Unable to post to Beeminder. Status code: {response.status_code}")
        return None
    else:
        print(f"Posted '{comment}' to Beeminder.")

    return response.json()


def get_latest_datapoint() -> Optional[BeeminderDatapoint]:
    params = {
        "auth_token": BEEMINDER_AUTH_TOKEN,
        "count": 1,
        "sort": "timestamp"
    }

    response = requests.get(BEEMINDER_URL, params=params)

    if response.status_code != 200:
        print(f"Unable to fetch datapoint from Beeminder. Status code: {response.status_code}")
        return None

    data = response.json()
    try:
        # We asked for 1 datapoint, so we take the first one
        return BeeminderDatapoint.parse_obj(data[0])
    except IndexError:
        return None


def update_sequentially():
    # Step 1: Get latest datapoint
    latest_datapoint = get_latest_datapoint()
    if latest_datapoint is None or latest_datapoint.value == 0:
        latest_timestamp = datetime(2015, 1, 1)
    else:
        # Convert timestamp to datetime object
        latest_timestamp = datetime.fromtimestamp(latest_datapoint.timestamp)

    current_timestamp = datetime.now()
    while latest_timestamp < current_timestamp:
        # Step 2: Get sessions
        start_date = latest_timestamp.strftime("%Y-%m-%dT%H:%M:%SZ")
        end_date = (latest_timestamp + timedelta(days=365)).strftime("%Y-%m-%dT%H:%M:%SZ")
        sessions = get_sessions(start_date, end_date)

        # Step 3: Add sessions to Beeminder
        print(f"Got {len(sessions)} sessions between {start_date} and {end_date}.")
        sessions.sort(key=lambda s: datetime.fromisoformat(s.startTime))
        for session in sessions:
            post_to_beeminder(session)

        # Step 4: Set latest_timestamp to latest_timestamp + 1 year
        latest_timestamp += timedelta(days=365)


if __name__ == "__main__":
    update_sequentially()

That worked great, except:

FML. So, we wrote 100 additional lines of debugging code

Focusmate data to Beeminder but with debug information

import requests
import os
import keyring
import sys
from datetime import datetime, timedelta
from pydantic import BaseModel
from typing import List, Optional


BEEMINDER_AUTH_TOKEN = keyring.get_password("beeminder", "felixm")
FOCUSMATE_API_KEY = keyring.get_password("focusmate-api-key", "felixm")
FOCUSMATE_BASE_URL = "https://api.focusmate.com/v1"
USER = "felixm"
GOAL = "focusmate_"
BEEMINDER_URL = f"https://www.beeminder.com/api/v1/users/{USER}/goals/{GOAL}/datapoints.json"


class User(BaseModel):
    userId: str
    requestedAt: Optional[str]
    joinedAt: Optional[str]
    completed: Optional[bool]
    sessionTitle: Optional[str]


class Session(BaseModel):
    sessionId: str
    duration: int
    startTime: str
    users: List[User]

    def get_timestamp(self) -> int:
        return int(datetime.fromisoformat(self.startTime).timestamp())


class BeeminderDatapoint(BaseModel):
    id: str
    timestamp: int
    daystamp: str
    value: int
    comment: str
    updated_at: int
    requestid: Optional[str]

    def get_timestamp(self) -> int:
        return self.timestamp


def get_user_name(user_id: str) -> Optional[str]:
    response = requests.get(
        f"{FOCUSMATE_BASE_URL}/users/{user_id}",
        headers={"X-API-KEY": FOCUSMATE_API_KEY}
    )

    if response.status_code != 200:
        print(f"Unable to fetch user data. Status code: {response.status_code}")
        return None

    data = response.json()
    return data['user']['name']


def get_sessions(start_date: str, end_date: str) -> List[Session]:
    response = requests.get(
        f"{FOCUSMATE_BASE_URL}/sessions?start={start_date}&end={end_date}",
        headers={"X-API-KEY": FOCUSMATE_API_KEY}
    )

    if response.status_code != 200:
        print(f"Unable to fetch session data. Status code: {response.status_code}")
        return []

    data = response.json()
    sessions = [Session.parse_obj(session) for session in data['sessions']]
    return sessions

def post_to_beeminder(session: Session) -> Optional[dict]:
    if not session.users[0].completed:
        print(f"Did not add uncompleted session {session}.")
        return None

    if len(session.users) > 1:
        partner_user = session.users[1]
        partner_name = get_user_name(partner_user.userId)
    else:
        partner_name = "No partner."

    session_title = session.users[0].sessionTitle
    session_start_time = datetime.fromisoformat(session.startTime)

    start_day = session_start_time.strftime("%a") # format start time as day of the week
    duration_mins = session.duration // 60000  # milliseconds to minutes
    comment = f"{start_day}, {duration_mins} mins, {partner_name}"
    if session_title:
        comment += f", {session_title}"

    timestamp = session.get_timestamp()

    # Convert session start time to "daystamp" (date in "yyyymmdd" format)
    daystamp = session_start_time.strftime("%Y%m%d")

    data = {
        "auth_token": BEEMINDER_AUTH_TOKEN,
        "value": 1,
        "comment": comment,
        "timestamp": timestamp,
        "daystamp": daystamp
    }

    response = requests.post(BEEMINDER_URL, data=data)
    if response.status_code != 200:
        print(f"Unable to post to Beeminder. Status code: {response.status_code}")
        return None
    else:
        print(f"Posted '{comment}' to Beeminder.")

    return response.json()


def get_latest_datapoint() -> Optional[BeeminderDatapoint]:
    params = {
        "auth_token": BEEMINDER_AUTH_TOKEN,
        "count": 1,
        "sort": "timestamp"
    }

    response = requests.get(BEEMINDER_URL, params=params)

    if response.status_code != 200:
        print(f"Unable to fetch datapoint from Beeminder. Status code: {response.status_code}")
        return None

    data = response.json()
    try:
        # We asked for 1 datapoint, so we take the first one
        return BeeminderDatapoint.parse_obj(data[0])
    except IndexError:
        return None

def get_all_datapoints() -> List[BeeminderDatapoint]:
    params = {
        "auth_token": BEEMINDER_AUTH_TOKEN,
        "sort": "timestamp"
    }

    response = requests.get(BEEMINDER_URL, params=params)
    if response.status_code != 200:
        print(f"Unable to fetch datapoint from Beeminder. Status code: {response.status_code}")
        return None

    data = response.json()
    try:
        return [BeeminderDatapoint.parse_obj(o) for o in data]
    except IndexError:
        return None


def update_sequentially():
    # Step 1: Get latest datapoint
    latest_datapoint = get_latest_datapoint()
    if latest_datapoint is None or latest_datapoint.value == 0:
        latest_timestamp = datetime(2015, 1, 1)
    else:
        # Convert timestamp to datetime object
        latest_timestamp = datetime.fromtimestamp(latest_datapoint.timestamp)

    current_timestamp = datetime.now()
    while latest_timestamp < current_timestamp:
        # Step 2: Get sessions
        start_date = latest_timestamp.strftime("%Y-%m-%dT%H:%M:%SZ")
        end_date = (latest_timestamp + timedelta(days=365)).strftime("%Y-%m-%dT%H:%M:%SZ")
        sessions = get_sessions(start_date, end_date)

        # Step 3: Add sessions to Beeminder
        print(f"Got {len(sessions)} sessions between {start_date} and {end_date}.")
        sessions.sort(key=lambda s: datetime.fromisoformat(s.startTime))
        for session in sessions:
            post_to_beeminder(session)

        # Step 4: Set latest_timestamp to latest_timestamp + 1 year
        latest_timestamp += timedelta(days=365)


def get_all_sessions() -> List[Session]:
    sessions = []
    latest_timestamp = datetime(2015, 1, 1)
    current_timestamp = datetime.now()
    while latest_timestamp < current_timestamp:
        start_date = latest_timestamp.strftime("%Y-%m-%dT%H:%M:%SZ")
        end_date = (latest_timestamp + timedelta(days=365)).strftime("%Y-%m-%dT%H:%M:%SZ")
        sessions += get_sessions(start_date, end_date)
        latest_timestamp += timedelta(days=365)
    sessions.sort(key=lambda s: datetime.fromisoformat(s.startTime))
    return sessions


def update():
    def check_duplicates(datapoints: List):
        timestamps = {}
        for datapoint in datapoints:
            try:
                timestamps[datapoint.get_timestamp()].append(datapoint)
            except KeyError:
                timestamps[datapoint.get_timestamp()] = [datapoint]

        all_okay = True
        for timestamp, datapoints in timestamps.items():
            if len(datapoints) > 1:
                all_okay = False
                print("[error] duplicates")
                for d in datapoints:
                    print(d)
        if all_okay:
            print("[ok] no duplicates")

    def post_missing_sessions(datapoints: List[BeeminderDatapoint], sessions: List[Session]):
        """ Get FocusMate sessions that have not been posted to Beeminder yet and post them. """
        beeminder_timestamps = [datapoint.timestamp for datapoint in datapoints]
        missing_sessions = [session
                            for session in sessions
                            if session.get_timestamp() not in beeminder_timestamps]
        for session in missing_sessions:
            if session.users[0].completed == True:
                post_to_beeminder(session)
            else:
                pass
        print("[ok] All Focusmate sessions are in Beeminder")

    def find_overposted_datapoints(datapoints: List[BeeminderDatapoint], sessions: List[Session]):
        """ Get Beeminder sessions that are not in FocusMate. """
        focusmate_timestamps = [session.get_timestamp() for session in sessions]
        assert(len(focusmate_timestamps) == len(set(focusmate_timestamps)))
        additional_datapoints = [datapoint
                                 for datapoint in datapoints
                                 if datapoint.timestamp not in focusmate_timestamps]
        if additional_datapoints:
            print("[error] Additional datapoints")
            for a in additional_datapoints:
                print(a)
        else:
            print("[ok] No additional Beeminder datapoints")

    def check_beeminder_values(datapoints: List[BeeminderDatapoint]):
        all_okay = True
        for d in datapoints:
            if d.value != 1:
                print(f"[error] Beeminder datapoints with value != 1 {d=}")
                all_okay = False
        if all_okay:
            print("[ok] All Beeminder datapoints have value == 1")

    total_sessions = get_focusmate_total_sessions()
    datapoints = get_all_datapoints()
    sessions = get_all_sessions()
    print(f"Beeminder sessions: {len(datapoints)}")
    print(f"Focusmate official count: {total_sessions}")
    print(f"Focusmate all session: {len(sessions)}")

    check_duplicates(datapoints)
    check_duplicates(sessions)

    check_beeminder_values(datapoints)
    post_missing_sessions(datapoints, sessions)
    find_overposted_datapoints(datapoints, sessions)


def get_focusmate_total_sessions() -> int:
    response = requests.get(
        f"{FOCUSMATE_BASE_URL}/me",
        headers={"X-API-KEY": FOCUSMATE_API_KEY}
    )
    if response.status_code != 200:
        print(f"Unable to fetch user data. Status code: {response.status_code}")
        return 0
    data = response.json()
    return data['user']['totalSessionCount']


if __name__ == "__main__":
    update()

And got this:

[ok] no duplicates
[error] duplicates
// two sessions with identical timestamp, both completed=False
[error] duplicates
// two sessions with identical timestamp, one completed=True and one completed=False
[ok] All Beeminder datapoints have value == 1
[ok] All Focusmate sessions are in Beeminder
Traceback (most recent call last):
  File "/home/felixm/tmp/focusmate/sync.py", line 289, in <module>
    update()
  File "/home/felixm/tmp/focusmate/sync.py", line 264, in update
    find_overposted_datapoints(datapoints, sessions)
  File "/home/felixm/tmp/focusmate/sync.py", line 232, in find_overposted_datapoints
    assert(len(focusmate_timestamps) == len(set(focusmate_timestamps)))
AssertionError

Which completely threw me off because there is no reason why there couldn’t be two sessions at the same timestamp (as long as they are not both completed, I guess).

I removed the useless:

assert(len(focusmate_timestamps) == len(set(focusmate_timestamps)))

And, sure enough:

[ok] no duplicates
[ok] All Beeminder datapoints have value == 1
[ok] All Focusmate sessions are in Beeminder
[error] Additional datapoints
id='64651b00f0168a430fe3d3b7' timestamp=1675141199 daystamp='20230130' value=1 comment='Mon, 50 mins, Jian L.' updated_at=1684347973 requestid=None

I don’t know how Jian got added twice, but it has something to do with me locking my device and the script kinda getting stuck, and then after restarting it, the latest_timestamp approach didn’t work as expected?

Anyways, I have removed the data point, and now I finally get a satisfactory:

Beeminder sessions: 856
Focusmate official count: 856
Focusmate all session: 899
[ok] no duplicates
[ok] All Beeminder datapoints have value == 1
[ok] All Focusmate sessions are in Beeminder
[ok] No additional Beeminder datapoints

I will use post_missing_sessions(datapoints, sessions) to sync this goal in the future. It sucks that I have to query for all data points and sessions, but since I only do it once a day, it’s okay. It should be easy to fix update_sequentially. If I do that I will post the update.

Now, I have a beautiful graph with retro-fitted red line:

[howtodowtle]

Do we have a time-sensitive Focusmate integration yet? I disabled my Focusmate goal when we did not have it but really cannot read it from this thread.

[felixm]

No, we don’t. All sessions count as ‘1’. You could adapt the Python script, though.

edit: I was wrong.

[adamwolf]

You can email support and get a duration-based Focusmate goal.

[felixm]

My bad.

[felixm]

In my pursuit of becoming a more authentic version of myself, I feel the need to share that I’m somewhat irritated. It seems like nobody else is bothered by the way this integration currently works, and I feel like I’m not being taken seriously.

As it stands now, I could start a Focusmate session, leave it after just ten minutes, begin a new one, and repeat this pattern. Each of these sessions would still be recognized by the integration. Clearly, this isn’t how it’s supposed to work.

If anyone could jump in and point out why I might be wrong, I would appreciate it. I’m not bitter or angry (anymore), and my script works great now (without needing to load the complete Beeminder or Focusmate history), but still. I hope that by sharing this my ego will let go of this somewhat irrelevant (in the grand scheme of things) discussion.

Edit:

Since I have already crossed the threshold to being obnoxious, here is another example from today which brought me to make this post:

What happened is that I had a session at 1:15 PM. My partner had trouble joining and gave up eventually meaning I was in the session by myself at 1:25 PM.

Now, if I desperately need the datapoint to not derail on my goal, I could just finish the session by myself. Since I didn’t need the datapoint and working by myself defeats the goal of Focusmate, I cancelled the on-going session, and created a new one for 1:30 PM.

With the current integration, the 1:15 PM and the 1:30 PM session would both be counted in Beeminder. This could be avoid by considering the completed flag (like my script does).

[adamwolf]

Hi Felix!

I hope nothing I said made it sound like I’m not taking you seriously. It seems like maybe you are curious why the integration would be like it is today, and how could anyone do that on purpose I may be able to help.

First, a few notes:

• As a person who works on Beeminder, I don’t really drive priorities (and I don’t typically do support), and as a user, I don’t use Focusmate (and I’m not super active on the forum and don’t read every post).
• There are a bunch of ways that a person can make the data in Beeminder not match some objective Truth. I’d guess this is true for every integration. Most folks aren’t bothered by most of the ways. @dreev has written some about this here: Ice Cream Truck Loopholes | Beeminder Blog
• The fact that today’s integration bothers you and Beeminder lets you rig up your own integration that fits you better seems like a complete win! Nice work!

In general, Beeminder integrations try to give credit when things are started rather than when they’re ended.

The way the completed flag works today in the Focusmate API is not the same way it worked when we wrote the integration. IIRC, there was a different system where all sessions, even completed ones, looked the same, but sessions were deleted “some time after” the time they would have completed if they weren’t completed. (i.e a session could show up in the output, look completed, the user could hit their goal deadline, and then the session gets removed since it wasn’t completed. Do we give credit for started sessions, and then remove it some time later when it’s removed? Do we wait some extra time after the session would have ended before giving credit, so we remove fewer things we gave credit for? How long should we wait? An extra 15 minutes after the session would be done? No guarantee that’d be long enough–and that means that a user could complete a session, hit refresh on their goal, and see no data updates for 15 minutes. That also means that a “midnight” deadline in the Beeminder UI is actually an 11:45 deadline in real life. )

Combine the complexity there with “typically Beeminder give credit at the start of things, not the end” and that’s how we got the initial configuration.

Having a completed flag makes it so we could count sessions only after they’re completed in a reasonable way! Great!

The whole point of Focusmate is to work with other folks, right? Suggesting honest folks (who want to work with other people) spend their ghosted session working alone in order to not derail seems at least a little bogus to me. It might be, but it isn’t obvious to me that reducing the number of ways that a person could abuse the system by one is guaranteed to be worth that tradeoff. (Add in “the number on Focusmate and on Beeminder will match”, though, and maybe I think that Beeminder should use the completed flag…)

Alright, I know the above might sound negative, and I don’t want this to be, so another note: Beeminder typically launches its integrations as barebones as possible, and adds to them based on user feedback! Keep it coming! (even things you know Beeminder has heard before!)

(@dreev, writing this made me realize I don’t know if we have a “how Beeminder makes and thinks about integrations” that we can point folks at)

[felixm]

Hi Adam,

Thanks so much for taking the time to write this and providing the background information! I have a much better understanding now and feel like my voice has been heard. I feel much better now. Thanks for your work and the great product.