Does screwing up correlate with sleeping badly?

This is part 2 in series of blog posts about a React-application that I built.

  1. Introduction
  2. Data validation – client side
  3. Data validation – server side
  4. Adding screw-up data
  5. Parsing Withings health data
  6. Unimplemented plans
  7. Displaying data
  8. Displaying data – charts
  9. Things learned
  10. Epilogue
  11. References

How can React and NodeJS give us answers

At my job (which is being a truck driver), our tasks vary daily, so there isn’t exactly a routine that is repeated every day. Therefore to get paid, we need to write down what we have done each day and how many hours we have clocked as we are paid by the hour. About a year ago, I bought a Udemy React-course by Andrew Mead and after getting to know the basics, I started making a web app which allows me to save daily work tasks to a database and then easily browse each months workdays.

Also, as a joke, I have been writing down (or gathering “big data
“) of whenever I screw up at work, such as forget something. I only did this for two of the three summer months that I worked, so there isn’t exactly much data to work with. I wanted to make it possible for me to easily add this data into the app, so I could view screw ups along with daily tasks.

My Nokia smartwatch on the other hand logs everything into their app, from where one can view them from their apps or API, or by downloading the data manually (thanks to GDPR). Sadly, their API isn’t the best in the world, as mentioned in this gist. Therefore, I couldn’t even register my app in their site, and had to scratch my original plans of integrating the API straight into the app, so that it would continuously log my health data. Luckily, however, I could download the GDPR data package and parse it manually to the database.

The task names are just jargon that only I and 20 other employees can understand

The interface ended up being relatively simple for two things: I didn’t have time and I’m not so good visual designer anyway.

Basically, the user can add their workdays into the punch-card, after which they are not added to the database yet until the user presses the button for adding them to the database. They are also added to local storage, which means that if the user refreshes the page or leaves and comes back, the data is still there. Until he clicks on ’empty punch-card’.

Data validation is not as simple as the interface

Even though the interface is.. what it is, the back end isn’t that simple. The data has to be validated in many different ways. For example, when user enters an end date for their workday, it has to be after the begin date. It also can’t be before another workday (unless it also ends before..), and it can’t be between a previous workday. Also, the app has to check all of the above also with workdays that are already added into the database.

To implement this validation, I first made the html input fields as date. On modern browsers, users are prevented from sending the form, unless It’s a valid date. This however isn’t enough validation, cause one can bypass this with various ways such as having an older browser that doesn’t support date fields. So I also had to write some code.

validateDate = () => {
      if (utils.date1IsAfterDate2(this.state.workday_date_begin, this.state.workday_date_end)) { 
        this.setState({'error': 'Tarkista aloitus- ja lopetusajat.'}) // User tries to enter a workday end date that is before begin date, or vice versa
        return false 
      }
      else {
        return true
      }

Whenever user leaves the input field, this calls a function (in a separate .js file) that checks whether the end date is before begin date, or vice versa. Then, when the user clicks ‘add to punchcard’ -button, the app does a check whether there already is a workday in the database with overlapping dates. This is done by calling parent component’s ‘workdayIsValid’ -function, which then does a POST request that fetches all workdays from the database. Then, the new workday is compared to each and one of them. If it overlaps, the user gets an error. The function that does most of the magic is this:

const workdayOverlaps = (arr, workdayObj) => { // returns an error object with error = true IF workday DOES overlap
    if (arr.length === 0) {
        return {error: false}
      }
      for (let item of arr) {
        const workdayBeginsBeforeComparedBegins = (moment(workdayObj.workday_date_begin).diff(moment(item.workday_date_begin), true) < 0)
        const workdayBeginsBeforeComparedEnds = (moment(workdayObj.workday_date_begin).diff(moment(item.workday_date_end), true) < 0)
        const workdayEndsBeforeComparedBegins = (moment(workdayObj.workday_date_end).diff(moment(item.workday_date_begin), true) < 0)
        const workdayEndsBeforeComparedEnds = (moment(workdayObj.workday_date_end).diff(moment(item.workday_date_end), true) < 0)
  
        // Don't laugh, I was bashing my head in because of these
        const condition1 = (workdayBeginsBeforeComparedBegins && workdayBeginsBeforeComparedEnds && workdayEndsBeforeComparedBegins && workdayEndsBeforeComparedEnds) 
        const condition2 = (!workdayBeginsBeforeComparedBegins && !workdayBeginsBeforeComparedEnds && !workdayEndsBeforeComparedBegins && !workdayEndsBeforeComparedEnds)
        if (!condition1 && !condition2) {
            console.log('error')
          return {error: true, conflictingWorkday: item}
        }
      }
    return {error: false}
}

And this is what the user gets if it fails.

But hold on! There’s more..

Client-side validation isn’t simply enough. This app sends the user-submitted data to the server via a POST request. As you can see in my previous blog post, ‘Bullying phishers with Python‘, anyone can send whatever they want via POST, bypassing all fancy client-sided validation and possibly fill the database with all sorts of things that don’t belong there. Especially since I’m using SQLite, which is It’s own story, which I will address below.

Basically, the POST endpoints also need to validate whatever is sent to them. So here is the endpoint that handles the adding of workdays:

app.post('/addworkdays', async (req, res) => {
    let result = null
    if (utils.isValidRequest(req, 'workdays')) {
        for (const workdayObject of req.body.params.workdays) { // TIL You can't use foreach with async\await
            if (!utils.isValidWorkdayObj(workdayObject)) { // Validate each object
                res.sendStatus(400)
                return
            }
           ...

The library that I used most in this project is probably MomentJS, which is a handy date and time handling library that can do many crazy things. Basically this just for-loops each of the workday objects and does checks if each one of them are valid. If validation fails, the user gets a FORBIDDEN 400 -http error. I could have done a better job with the validation and it isn’t exactly bulletproof, but at least I tried.

Adding the screw ups

Adding the screwing-up is basically the same thing, except that I decided to take a shortcut and implement it only halfway. As in, all screw-ups are added straight into the database, after being validated, of course. Also, you can’t delete them.. which is next on the feature list, if I ever decide to continue this.

The user can optionally add a time of day for their screw up, add severity of screw up in a scale of 1-3, add how much time they have lost due to this accident (optional) and also a description of the event.

Adding health data into the app

As I said before, I couldn’t register my app to Withings (Nokia) and get access to the API, so I had to result into manual data parsing, as opposed to using OAUTH to really integrate Withings api into this. Withings provides everyone a zip full of .csv files with both raw and cooked data, so I just had to parse that to the database, which was quite simple to do.

const parseSleep = () => {
    fs.readFile('./withingsData/sleep1.csv', 'utf8', ((err, contents) => {
        if (err) {
            console.log('error ', err)
            return
        }
        try {
            const rows = contents.split('\n').slice(1)
            rows.forEach((row) => {
                row = row.split(',')
                if (row.length === 14) {
                    let fromDate = row[0]
                    let toDate = row[1]
                    ...
                 db.addSleepData([fromDate, toDate, lightSleepAmount, deepSleepAmount, rem, awake, wakeUp, DurationToSleep, DurationToWakeUp, snoring, snoringEpisodes, averageHeartRate, heartRateMin, heartRateMax])
                   }
                     ...

Nothing fancy, just splitting each line on a comma and adding the rows one-by-one to the database.

Displaying submitted data

Here is the ‘all data’ page where the user can browse their submitted data. They can enter their desired time frame and get a list of each days tasks and hours, as well as the grand total. Again, this could be a little bit clearer and contain a bit nicer style. But it works. How it works is that it takes the user-submitted dates and then performs a query with these two dates, returning the requested workdays.

Because everybody loves charts

At least I do. Therefore, I implemented a page that shows nice little, colorful charts of various types. This I did with a nice JavaScript library called c3.js, which is easy to use and contains great documentation, such as this interactive demo.

Here we have a chart showing workday length (in blue), screw-ups (in orange) and sleep amount (in green). What judgments can we make from this chart? That you probably shouldn’t take this very seriously.

Then there is also this chart which shows you average amount of tasks (in orange), workday length (in blue) and amount of sleep (the green line). From this chart, we can notice that I am the least productive on Fridays and Mondays. What you can’t notice from the chart that this is only over a roughly-2-month period.. so there is around 6 days for each day to count an average from.

This chart is my favorite. It contains colors.. data that isn’t very useful.. everything. What it shows is my nutrition for each day, along with steps (divided by 1000),compared with workday length (blue bar) and screw ups (the gray line). What we can notice from this is that.. I don’t know. Oh and just to add, we have a leaderboard in the Nokia Health App where you can compete with amount of steps against other people. I wanted to win it so I walked a lot every day on my spare time.

What I was planning to add..

Exporting workdays to different formats

Originally, my plan was to add some sort of API for the workdays, because everyone loves API’s. Also, the validation of POST requests is a bit half-assed and doesn’t take into account some things. I also wanted to make the ‘show all workdays’ table to be a bit more clearer or printable, such as to contain some sort of ‘export to X’ -type or a print-formatting feature which would look like something running SQL from a command line. A bit something like the one below:

+------------------+----------------+
|       PVM        | Workday length |
+------------------+----------------+
|    16.6.2018     |      8,5h      |
|    17.6.2018     |      9h        |
|    18.6.2018     |      9,5h      |
+------------------+----------------+

Users

Obviously, this app works only for me, so if this was a real thing, I’d need to allow different users to use it in their own session by logging in, etc. There could also be Admin users who can see total statistics, who has done what, etc.

More data

Modern trucks provide a lot of big data, so this could be integrated in the app as well in some form, along with data that is written into each driver’s ‘driver card’, which is used to monitor when the driver takes their mandatory breaks, etc.

Use a real database

My original plan was to use SQLite only for the developing part, and switch to PostgreSQL at one point, but that never happened. SQLite works fine for this particular application, but there are some nuances which is why I probably will switch straight to PostgreSQL next time I develop something. More on this below.

What I learned.. the hard way

Even though this project was mostly a joke, it has been been a learning experience. React is something unlike anything I’ve ever done before. Along with that, there were also many pretty basic programming-related things that I hadn’t thought of before.

One simply does not copy an object that easily

Considering the below code:

let car = {make: "Mazda",
					model: "RX-8",
          color: "Red"}
let secondCar = car

secondCar.color = "Blue"

console.log(car.color)
console.log(secondCar.color)

Outputs ‘blue’ on both cases. This is because I’m not actually copying the object to a new variable, I’m just referencing to it. It’s pretty basic programming 101 on certain languages, but I hadn’t ran into this before so it caused me to scratch my head and then proceed to bash it against a wall at one point. You can read more on the subject here. However, now if I get a mysterious problem that doesn’t make any sense, such as when some data is not what it’s supposed to be, I can quickly identify that It’s probably due to this. Such as what happens here:

 async handleAddWorkday(e) {
    let eCopy = Object.assign({}, e) // element disappear somewhere and I cba to search why, possibly references to state object which gets emptied
    e.preventDefault()
    const resultObj = await this.props.workdayIsValid(this.state)
    if (resultObj.error && resultObj.message) {
...

The ‘e‘ argument is passed on with the onClick event, and it initially is there. But when the code makes the call to this.props.wordayIsValid, e‘s values disappear somewhere. I didn’t get to the root of the problem, but I suspected that the state changes and e along with it.

SQLite is really lite

Al thought I knew what SQLite was, I didn’t think It’d be that different from other databases. The fact that there are only text, integer, real, blog and numeric data types I can totally get, as you can still add dates as text and then call a date(text_column) to get a date value, but what made me lose more hair was a problem caused by this create table query:

						"CREATE TABLE workdays" + 
							"(" +
								"workday_begin text," +
								"workday_end text," +
                                                                 ...
								"other_info text" +
								"urakka integer DEFAULT 0" + // Boolean True\False
							")",

I had made a mistake on the row where other_info column is created, as I forgot to add a comma after the line. In other databases, this should cause an error, but apparently not in SQLite. What happened here is that it a column ‘other info’ with datatype ‘texturakka integer’

What the hell kind of a datatype is that?

Apparently, you can give the columns whatever datatypes you want. You can guess that this caused some head-scratching when inserting data.

Some things are asynchronous

Another moment that caused some Что?? -moments is realizing that setState for example is asynchronous. This means that the code works in parallel with another, such as when you fetch something from a database in this code. Unless you await for the query to complete, you’re going to have a bad time. This is because the code will keep on executing while it is fetching the data, and when it has fetched it, rest of the code has already been read and nobody cares about that fetched data anymore. So controlling the code flow was a major learning point.

There’s a thing called MAX_SAFE_INTEGER

In JavaScript, there’s a limit to how long integer you can have. This is explained in detail here. The following code for example:

let integeeri = 123456789101112131415
console.log(integeeri)

Will output 123456789101112130000. I guess this is also a pretty basic programming thing and I luckily had a hunch of it when I encountered it.

Epilogue

The reason why I couldn’t completely polish and finish this project is the fact that I have to start working on my road to Reaktor -project. Before I dived into that, I wanted to have at one somewhat-finished project to make a blog post about, so I could start applying for summer jobs and possibly show “hey I did this” to said people. Even though this is what it is, it has been a great learning experience.

I also highly recommend buying the Udemy course, as React is something that is completely different at least from anything I’ve ever seen in my 0 day working career as a programmer. Because of this, having someone hold your hand and show you things from 0 to 100 was essential to me, even though I stopped the course after learning the basics (which is about 30% of the course)

Special thanks to my man Henry for helping me out. Probably the reason why I have learned so much over the past few years is the fact that I’ve always had someone to ask ‘why is this like this’ from.

Reaktor’s pre-apply assignment has been published since December, but here we are once again, two weeks away from deadline and I haven’t done pretty much anything. Which means that I’m going to implement the One Punch Man coding technique: 100 craft beers, 100 commits and
10 kilometers of code per day.

References

My project on github
https://github.com/Thomaxius/Tuntikortti
Live demo

Andrew Mead, Udemy, React 2nd edition:
https://www.udemy.com/react-2nd-edition/

User ‘katemonkeys’ on github, “Everything Wrong With The Withings API”
https://gist.github.com/katemonkeys/e17580777b57915f5068

Stackoverflow topic, “Is JavaScript a pass-by-reference or pass-by-value language?”
https://stackoverflow.com/questions/518000/is-javascript-a-pass-by-reference-or-pass-by-value-language/3638034#3638034

Mozilla docs, “Number.MAX_SAFE_INTEGER”
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER

Javascript libraries I used the most:
Charts: https://c3js.org/
Handling of time and dates: https://momentjs.com/docs/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close