Temporal Patterns in Self-Tracking Data
In this post, I briefly discuss three temporal patterns that are likely to be present in every self-tracking data: trend, social rhythm and intra-day variability. Overlooking or ignoring these patterns when analyzing data may result in erroneous conclusions.
Continuous tracking can help you see how certain aspects of your life change over time. It can also help you discover relationship between various variables.
The most common mistake that people make when analyzing their self-tracking data is treating it as stationary (i.e., independent of time). Some variables, however, may be slowly changing over time - and as a result, they share time as a common factor. If this confounding factor is not removed, you may end up finding spurious correlations between othrwise unrelated variables.
The best way to remove time as a confounding factor is to detrend the data. The easiest way to detrend data is using differencing. You take a value of X at point in time t: X(t) and subtract it from the value of X at the previous poin in time X(t-1).
group_by(daypart) %>% mutate(healthdt=health-lag(health)) %>% mutate(energydt= energy-lag(energy)) %>% mutate(stressdt=stress-lag(stress)) %>% mutate(happinessdt=happiness-lag(happiness)) %>% mutate(flowdt=flow-lag(flow)) %>% mutate(emovardt=emovar-lag(emovar)) %>% ungroup %>% mutate_if(is.numeric, ~replace(., is.na(.), 0))
This is, for example, how my daily happiness looks like before and after detrending:
And this is how correlations look before and after detrending:
In 1990s, several scientists from Western Psychiatric Institute developed a questionnaire for measuring regularity of major daily behaviors, called Social Rhythm Metric. The idea behind the SRM is that certain everyday routines, like waking up, having breakfast/lunch/dinner, going to bed, etc. are performed at a relatively consistent schedule. The deviations from the normal rhythm may be a result or a cause of major changes in lifestyle, physical, cognitive or psychological health, performance, and so on.
For most of us with regular full-time jobs social rhythm on weekdays is different from social rhythm on weekends. Our work days are more structured, consistent and “streamlined” when it comes to sleep, meals, and social schedule, whereas weekends and holidays are more flexible and unpredictable. This often translates into different patterns for the same variables across workdays and weekend data.
Chart below shows differences in my situational context across days of the week.
More details on situational metrics in another blog post. In the meantime we can see that some days of the week can be more/less active/social/positive/typical than others when it comes to situational context.
Intra-Day is a special case of social rhythm. Some variables may have different values when measured at different times of the same day, due to circadian rhythms, lifestyle or other factors. Such variability is typical not only for latent variables like mood or fatigue. Many objectively measured metrics (body temperature, mental alertnesss) also exhibit intra-day variability.
Intra-day variability is not a bad thing. By taking multiple measurements thoroughout the day I can discover interesting patterns across mornings, afternoons, and evenings.
Here is an example of how my energy changes throughout the day: