Any way to control exponential moving average history?


#1

Hey folks, quick question:

I have a tracking goal I use for my weight. I use beeminder for this because it’s easier for me to limit the number of services I have to add data to, and I like the implementation of EMA that the purple line represents.

My problem: over the holidays, I was away from a scale and gained a bit of weight. My moving average reacted predictably to the data (see picture), but the way it’s implemented it’s been playing catch-up for the last week and a half.

I’d like to be able to have it “forget” about history before a certain date so it’ll go back to tracking my actual weight, not my presumptive, exponentially-creeping weight from four weeks ago, and I can use it as a good metric for weight gain or weight loss again by checking whether the data points are mostly over or mostly under the line.

Does this make sense? Is there any way to do this with currently available settings?


#2

Moving averages are always a lagging indicator, yes? Weights above the line mean it’s going up, weights below the line mean it’s going down. In our case:

This is an exponentially weighted moving average with smoothing constant 0.1. It shows the overall trend of your data, but lagged. If most of your daily numbers are below this trend line then you are inexorably dragging it down.

No.

I can imagine a world in which derailments mean that the average starts over, though that’d also introduce a discontinuity in the thin purple line.


#3

Yes, I realize it’s a lagging indicator. My problem was that because my data was too sparse for several weeks, it lagged much farther behind than I’d have liked.

This is pretty much solved now, though, since the EMA has caught up to my data after a week and a half.


#4

ooh, sparse data is interesting. @dreev would know what impact that might have on the moving average.

Some folks get a lot of mileage out of having a complementary weigh-in goal that forces them to stand on the scale with some frequency. (You can automate that with IFTTT.)


#5

I’ve since started doing that, hence the recent uptick in weigh-in frequency. And why the road is so wide on this goal; it’s more of a tracking goal than anything else.


#6

I almost answered that you could just set x-min but apparently the purple moving average line doesn’t care about that. I guess the simplest answer, as you suggest, is to just wait. It doesn’t take many days for the influence of old datapoints to fade to negligibility…


#7

If your smoothing constant is 0.1, I think that’s waaay too smooth. If you want N previous values to be included in a moving average, you set your smoothing constant to 2/(N+1). A smoothing constant of 0.1 means

0.1 = 2/(N+1)

0.1N + 0.1 = 2

0.1N = 1.9

N = 19

so nearly three weeks are being averaged into each datapoint.

If you want to include 7 days, set ⍺ = 0.25, or if you want it to include 14 days, set ⍺ = 0.13.

Why ⍺ = 0.25 is better for these purposes

With an EWMA, if you have a constant rate of change, the change in the moving average should eventually converge on the raw rate of change. So if you have a goal to lose 0.1 lbs per day and you lose exactly 0.1lbs per day, eventually, the value of any day’s EWMA should be 0.1 less than the previous day’s value.

If you are 200lbs on Day 0 and you lose 0.1lb every day after that, we’ll have these values with ⍺ = 0.1:

| Day | Weight | Wt. Δ | EMA Value | EMA Δ
| 75  | 192.5  | -0.1  | 193.39967 |
| 76  | 192.4  | -0.1  | 193.29970 | -.09997
| 77  | 192.3  | -0.1  | 193.19973 | -.09997

But with ⍺ = 0.25, we’ll have:

| Day | Weight | Wt. Δ | EMA Value | EMA Δ
| 30  | 197.0  | -0.1  | 197.29994 |
| 31  | 196.9  | -0.1  | 197.19995 | -.09999
| 32  | 196.8  | -0.1  | 197.09996 | -.09999
...
| 39  | 196.1  | -0.1  | 196.40000 |
| 40  | 196.0  | -0.1  | 196.30000 | -.10000

What this means is that with a smoothing constant of 0.1, we don’t even hit convergence to five decimal points after 77 days; with 0.25, we hit it after 40.

This also has further implications for what the graph will look like. With a smoothing rate of 0.25 and a constant weight loss of 0.1lbs per day, the EWMA would always be 0.3lbs above any day’s datapoint past the 40 day mark. In contrast, a smoothing constant of 0.1 and weight loss of 0.1lbs per day will have an EWMA graph that’s 0.9lbs above the datapoint each day.

So, in conclusion, 19 days is a lot of datapoints to factor into a moving average, and a 75+ day lag for the EWMA rate of change to converge on the raw rate of change is a reeeeeally long time.


#8

This is super persuasive, @nickanderegg! Sorry it took me so long to just try it (it’s a 2-character change after all!) but I’ve finally done so. How’s it look now?


#9

On my moving average graphs, it looks really great to me. It now follows the curve much more closely and accurately reflects my changing trends while also smoothing out noise in the data!


#10

After this change, don’t forget to update the text in the graph legend!