-End-

After a whole semester of non-stop struggling, CS3216 had at last come to an end. Now that things had calmed down somewhat, I think it’s good to reflect on why I had struggled.

Assignment 1

For the most part of assignment one, I think my main struggle was to figure out nodeJS and Squelize. During the second lecture, when Prof Ben advised rather straight forwardly that we should just use a framework we’re familiar with, I gave a nervous smile. Me group and I are already halfway into a framework i have never ever touched before. The prof was right of course; if even then had we switched to a more familiar framework we might actually had gotten done slightly a bit more then what we eventually had.

I guess the idea was this: choosing the right tool for the job is not the simple task of looking at the problem picking out the best tool for it. One should also consider the skills of the team one has and how much they can achieve within that time period. Sometimes, a tool might not be the absolute best given the problem, but one the timeline is take into account, it might be the best choice.

Assignment 2

Assignment 2’s stuggle was…. jk A2 was easy. Complaining about other apps is easy as cake.

Assignment 3

Looking back, the hardest part about assignment 3 was that we tried to push the boundary of tech a bit too far. When our app is up, it basically turns on every other sensor in your phone. Best part was that it depends on everyone one of them to work. When one fails, everything fails. One of my friend’s had a phone with a wonky GPS, the app didn’t work. Another friend’s gyroscope was weird, the app didn’t work either. Then, we had Apple who won’t let us get the camera, the app didn’t work either.

The idea was born out of a brainstorming session. The method was to list down everything we have in a PWA and see what we can use. In the end, we just combined everything and the project was born. I would say now that when an idea depends totally on a whole bunch of tech to work flawlessly, one should really tread carefully before declaring it a viable idea, no matter how possible it seems when it was dreamt up.

Final Project

Given that I had been ranting quite a bit on what went wrong for my final project, I think i just have one last thing to say. When doing a project in a impossibly short period of time, think about what it is really that you are trying to sell to your users. Everything you decide to sink some time into must contribute to that goal one way or another. Beyond that, what the user wouldn’t be seeing, he’ll never see anyway.

My final project was a patchwork of many parts scrambled together, but thanks to the abundant guidance we’ve received, they could make up a coherent story in the end on steps day.

Conclusion

The 80-20 rule seem to apply to so many things. Let me add one more to the list: 80% of the selling point in a app comes from 20% of the features. Spend time to make the other 80% up to scratch, but make those 20% kick ass.

More rants, and on documentation

This is going to be a short one, since our project blew up once again and we’re re-coding like 80% of it, so I really need to get back to it. Basically the user flow was pretty much invalid, because we grossly misunderstood how the tuition is supposed to work. Oh well, it seems we’re really bad at that kind of stuff.

We’ll probably have to scale back on the marketing somewhat, given how stuff died so horribly.

That’s the rant. The second part is on documentation.

When you third party code, this is the ideal work flow:

1. Find library/framework
2. Read some short yet comprehensive tutorial to get started
3. Look at the awesome API documentation that explains every single thing whenever anything else is needed
4. Code is fully self contained and never blows up anything else

Of course no such thing exists. The closest I’ve seen is Rails itself, where the .rb files is like 80% documentation and 20% code.

When documentation is lacking, people do tend to get pissed off. “How the *** am I supposed to get this value?” In some cases the library gets dropped totally after that since one thinks it’s probably faster to roll our your own library.

There is actually another path: go read the code. If you’re shifting about uncomfortably in your seats right now, that’s normal; I too think reading code to be a pretty depressing prospect. What I’m saying though is that if the library is even any good, the code should be clear enough for you to make it out alive. And spending that 15mins diving the code can allow you to make some super complex library work; once which you’ll spend a few hours to do on your own. (The caveat of course is if the code is too, unreadable. Then yes, that’s a bad library, move along now…)

In case if you’re wondering if there was a library that made me write this, it’s IceCube. Documentation is pretty much non-existent. But the code is quite easy to understand. Saved me from trying to wrestle recurring times manually.

SQL vs NoSQL (Document based)

I actually went through this decision process quite a while back, when we were still doing HuntBot for PearComms, but I’ll thought it wouldn’t hurt to post it here now (even though it is not as relevant to our current project).

Back then, we were attempting to build a directory scraper. Needless to say, there were huge amounts of data, but none very relational (scrape data, put in db, repeat: one table no associations). As such, we thought it might be better we use  NoSQL database like mongodb to speed things up.

Of course we thought of drawbacks. What if later we somehow came up with something relational? Are we going to do the foreign key constraints ourselves? (Last time that happened, the result was a ton of corrupted data and a very painful migration I’m still stuck with). Furthermore, we’re all hooked onto Rails (with it’s ActiveRecord quite tightly integrated with SQL), and Rails do not take kindly to you messing with it – try doing React withing Rails itself, I’ll wait.

We eventually found a adapter for mongodb in Rails that takes care of both Rails integration and associations, and went ahead with it. But throughout the process I had some thoughts on when to use NoSQL (document-based DB in this case).

Wind back even a little longer and we’re at CVWO. Long story short, it was a system complicated enough such that even the home page fires off enough queries to make you give up on scrolling through the logs. Why? Because the client wanted a whole load of aggregated information all at once.

A document based DB is better at this kind of stuff. Information is kept together, all denormalized, in a single record, a document. For example, if a person has multiple roles, all those roles are kept and retrieved together as a single record instead of 2 tables. This part forms a huge component of the “performace”.

That of course doesn’t mean we should convert everything to documents. Documents are great if  and only if the stuff inside them belongs solely to the document alone. Our directory listing is pretty compatible: each entries’ information, both contact details and meta data, is relevant to the entry itself. A negative example will be a university course system. Student has and belongs to many modules. If you park the module under each student, you need to hunt down each of them when you update the module. Same if you park student under modules. You can have them as separate documents and use keys, and then implement foreign key constraints, and then ensure it works concurrently, and then realize you’re just doing it like SQL anyway. tldr; if you’re sub-elements are in principle shared, don’t do documents.

The key idea here is that NoSQL dbs like mongodb is not merely a “drop-in” replacement for relational databases with some tradeoffs. You actually need to adopt a whole new paradigm if you want to use them properly.

The Subtle Art of Ideation

Legend has it that some CS3216 group in the past pivoted 4 times for their project. We just had our 3th pivot last Friday, and it’s the third week. I guess we’re on schedule for the record!

In all seriousness though, we’re honestly not very good at this business of coming up with cool stuff; I think we’re somewhat better at making cool stuff cooler.  We targeted external projects as a result.

Of course, after all that pivoting, we all learnt quite a bit on what will not work. So allow me to iterate though the list, and examine what did not work.

1) Treatsure

Idea

The first project was a app that allowed food vendors to sell off their leftovers at a discounted price.

What’s good

The idea is a pretty sound one. There is indeed a problem of leftovers from the vendors, and there are definitely people who would want discounted stuff. Viable, and for a good cause.

Problem

The party who pitched it is still in its early stages of finding vendors. Furthermore, the owner prefers to keep the details confidential until he’s sure it’ll work. There will be communication problems, as well as the issue that the business process will stop us from launching before the deadline.

Lesson

We all need to work with constraints. There are things that can be sped up, and there are things that can’t be. When on a tight deadline, good ideas that eventually has potential can become bad ideas, for crucial steps will be rushed and screwed up.

2) Huntbot

Idea

To make a bot to scrape directories to automatically find large amounts of sales leads.

What’s good

It’s a project from Pearcomms, and based on their research, there is essentially nothing out there that does this task properly. Previously they hired an intern to sort though hundreds of rows of directory data, and that’s just not something anyone should be doing in this day and age. Thus, it’s a doable project that is really useful.

Problem

We make many scrapers, organise and data, and er… idk, then what? This is one of those projects where though useful, all the action occurs behind the scenes. We can do some wonderful data organisation at the backend if necessary,  but they, alas, will remain there; there’s not much to show. Going beyond simply directories will meaning venturing into unstructured data; apparently even the intern couldn’t deliver that one consistently.

Lesson

Some ideas may be useful, but there’s also little space for extension. There are many small problems out there, and they need to be solved, but don’t expect everyone of them to be able to turn into something epic.

3) Sales Marketplace

Idea

A marketplace for sales services. Producers sell sales services or even sales services (like Huntbot can actually be sold on this platform), consumers grab them, in one dedicated portal.

What’s good

There’s no go-to place for such things. Right now hunting for someone to do your marketing means hanging out on everything from Gumtree to Hardwarezone and hoping you bump into someone. It’s a market need.

Problem

Prof Ben. Ok, not really, but Prof Ben happened to hear of it, and advised against it. We think it’s pretty solid advice. For starters, where the heck are we gonna get the producers? Telemarketers/Email-marketers usually register with some agent to find jobs. We are competing directly with them, they’ll never happily hand the list over. It’ll be a marketplace full of consumers with no producers, standard two-sided problem. Not gonna be solved in 4 weeks.

Lesson

When something doesn’t exists at all, it best to consider long and hard why it does not exists. Most likely it’s because it requires more resources than 4 college kids could ever hope to amass to pull it off.

5)

We’re going a online tuition software right now. Probably gonna annex some WebRTC interactive tool and build a complete organisation system around it. Looking forward (hopefully never) to the next pivot.

Third-party code

Sidenote: Before we began, contrary to the last post, we didn’t pivot; we did a compromise for iOS instead. Hopefully they release WebRTC soon though, it’ll be very useful.

Not too long ago, there was this article: http://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/. TLDR: There was this npm package that leftpads strings with characters, and everyone used it. One fine day the author was angry and pulled it out. The internet broke. (Not to worry, there’s a API version now)

In the aftermath though, there was a discussion about the extend of third-party code usage. Sure, using packages reduces the amount of work; you don’t have to reinvent the wheel. But is it a bit too much when half the internet depends on a few lines of code that almost any dev could cook up in a few minutes?

Indeed, third party code may seem magical and god-sent at times, but there are caveats: 1) you end up not really knowing what’s going on, and 2) you give up control over some parts of your code. Number 2 was what caused the crisis above.

In our assignment 3, we used Rails as our back-end, and keeping in line with the spirit, we found a gem to do token based authentication. We hooked it up, thinking we’re done.

Well I wouldn’t be writing this blog post if that was the whole story. See, the gem is called devise-token-auth. So we depend on that gem, and that gem depends on another gem called devise, and everything depends on Rails.

Now Rails 5 just came out not too long ago. Somewhere along the line devise changed  a little to adapt (not quite enough though). devise-token-auth had bugs to begin with, and didn’t really catch up either.

So what happened was that numerous things broke everywhere along the chain, and it took me a day and a fork of devise-token-auth to get things in check.

I actually asked Emmanuel (from another team) how he got things up, since he was using devise-token-auth too. He said he gave up and wrote his own login flow; it was about 20 lines in the back-end.

Well… there. Sometimes, no matter how magic is promised to you, you’re probably better making your own magic.

 

 

Falling off the edge

We thought we had a pretty decent idea: an augmented reality navigation tool, right from the browser. It lets you track your friend, as well as any location directly with your camera; map reading skills not required. We did some digging, and found that we can get device orientation, location, and camera all from the browser. Great, that’s all we need. Sure it pretty much fires up every sensor on your phone, but it’ll work.

Except that it didn’t, and we found out pretty late. The issue here is that in this app, there is no single fatal point of failure, there are multiple single fatal points of failure. Fail to get any of the three working perfectly and pretty much nothing works.

And of course, if depending totally on one thing not failing is bad, depending totally on 3 things not failing is even worse. The story is that we actually managed to get all three working pretty well: https://meebleforp.com/projects/pinpoint/threevr-test/. And when we started to fell good about it and proceeded to go full steam with development, iOS happened.

See, the issue with iOS is that, even though it seems that you have a choice of browsers, when it comes to features under the hood, there isn’t really much difference between them. Due to a policy by Apple, all browsers must use the iOS Webkit kernel. Thus, Chrome on iOS is basically Safari with a different UI.

So here’s our problem. Apple does not (yet) support WebRTC on its Webkit kernel. What that means is that every single App Store approved browser ever is not able to stream images from the camera on the iPhone in anyway. Apple is working on it, but it missed the latest iOS launch.

So here we have it, one point of failure just imploded. We could simply not have AR, and do purely map and street views for iOS, but that kinds of undermines the whole purpose of the app. While trying to be on the bleeding edge, we might have just overshot and fell off the cliff.

Oh wells, time to pivot.

Post-mortem

So assignment 1 didn’t really go too well for my team. We were unable to implement quite a few of our features in time. As a result, a few of our user flows are rather intuitive, since they have more or less been stripped of some of their original planned functionality.

I admit I am at fault here. For me personally, I have underestimated the amount of focus required for the assignment. Instead I focused on doing CVWO and sort of neglected the assignment for the first two weeks. As a result, there was a crunch in the end, and the dev process essentially degraded into a waterfall model all over again. Whatever it was it wasn’t fun.

Besides that, I think there was two more mistakes we made.

Firstly, I feel that we did not really select the proper framework. The ORM that we used, sequelize, was actually very different to what I was used to in Rails. On top of that, it’s documentation wasn’t the best and it took me a very long time to figure out how to execute the queries. In that sense, things might have been a bit different if we chose to work with Rails instead, since I would then have saved a lot of time with the queries. The point here is that when selecting a framework, one might want to take the deadline into consideration, and use a more familiar framework if its tight.

Secondly, we should have checked our assumptions before making them. One of the assumption we made was that we can push notification to Facebook. As such, we planned to have our app fire Facebook notifications when needed as part of the user flow. However, it turned out that only canvas apps can do that. We only found out near the end, and as a result we had no notifications. Case in point: actually try things out before making assumptions.