-End-

After a whole semester of non-stop struggling, CS3216 had at last come to an end. Now that things had calmed down somewhat, I think it’s good to reflect on why I had struggled.

Assignment 1

For the most part of assignment one, I think my main struggle was to figure out nodeJS and Squelize. During the second lecture, when Prof Ben advised rather straight forwardly that we should just use a framework we’re familiar with, I gave a nervous smile. Me group and I are already halfway into a framework i have never ever touched before. The prof was right of course; if even then had we switched to a more familiar framework we might actually had gotten done slightly a bit more then what we eventually had.

I guess the idea was this: choosing the right tool for the job is not the simple task of looking at the problem picking out the best tool for it. One should also consider the skills of the team one has and how much they can achieve within that time period. Sometimes, a tool might not be the absolute best given the problem, but one the timeline is take into account, it might be the best choice.

Assignment 2

Assignment 2’s stuggle was…. jk A2 was easy. Complaining about other apps is easy as cake.

Assignment 3

Looking back, the hardest part about assignment 3 was that we tried to push the boundary of tech a bit too far. When our app is up, it basically turns on every other sensor in your phone. Best part was that it depends on everyone one of them to work. When one fails, everything fails. One of my friend’s had a phone with a wonky GPS, the app didn’t work. Another friend’s gyroscope was weird, the app didn’t work either. Then, we had Apple who won’t let us get the camera, the app didn’t work either.

The idea was born out of a brainstorming session. The method was to list down everything we have in a PWA and see what we can use. In the end, we just combined everything and the project was born. I would say now that when an idea depends totally on a whole bunch of tech to work flawlessly, one should really tread carefully before declaring it a viable idea, no matter how possible it seems when it was dreamt up.

Final Project

Given that I had been ranting quite a bit on what went wrong for my final project, I think i just have one last thing to say. When doing a project in a impossibly short period of time, think about what it is really that you are trying to sell to your users. Everything you decide to sink some time into must contribute to that goal one way or another. Beyond that, what the user wouldn’t be seeing, he’ll never see anyway.

My final project was a patchwork of many parts scrambled together, but thanks to the abundant guidance we’ve received, they could make up a coherent story in the end on steps day.

Conclusion

The 80-20 rule seem to apply to so many things. Let me add one more to the list: 80% of the selling point in a app comes from 20% of the features. Spend time to make the other 80% up to scratch, but make those 20% kick ass.

More rants, and on documentation

This is going to be a short one, since our project blew up once again and we’re re-coding like 80% of it, so I really need to get back to it. Basically the user flow was pretty much invalid, because we grossly misunderstood how the tuition is supposed to work. Oh well, it seems we’re really bad at that kind of stuff.

We’ll probably have to scale back on the marketing somewhat, given how stuff died so horribly.

That’s the rant. The second part is on documentation.

When you third party code, this is the ideal work flow:

1. Find library/framework
2. Read some short yet comprehensive tutorial to get started
3. Look at the awesome API documentation that explains every single thing whenever anything else is needed
4. Code is fully self contained and never blows up anything else

Of course no such thing exists. The closest I’ve seen is Rails itself, where the .rb files is like 80% documentation and 20% code.

When documentation is lacking, people do tend to get pissed off. “How the *** am I supposed to get this value?” In some cases the library gets dropped totally after that since one thinks it’s probably faster to roll our your own library.

There is actually another path: go read the code. If you’re shifting about uncomfortably in your seats right now, that’s normal; I too think reading code to be a pretty depressing prospect. What I’m saying though is that if the library is even any good, the code should be clear enough for you to make it out alive. And spending that 15mins diving the code can allow you to make some super complex library work; once which you’ll spend a few hours to do on your own. (The caveat of course is if the code is too, unreadable. Then yes, that’s a bad library, move along now…)

In case if you’re wondering if there was a library that made me write this, it’s IceCube. Documentation is pretty much non-existent. But the code is quite easy to understand. Saved me from trying to wrestle recurring times manually.

SQL vs NoSQL (Document based)

I actually went through this decision process quite a while back, when we were still doing HuntBot for PearComms, but I’ll thought it wouldn’t hurt to post it here now (even though it is not as relevant to our current project).

Back then, we were attempting to build a directory scraper. Needless to say, there were huge amounts of data, but none very relational (scrape data, put in db, repeat: one table no associations). As such, we thought it might be better we use  NoSQL database like mongodb to speed things up.

Of course we thought of drawbacks. What if later we somehow came up with something relational? Are we going to do the foreign key constraints ourselves? (Last time that happened, the result was a ton of corrupted data and a very painful migration I’m still stuck with). Furthermore, we’re all hooked onto Rails (with it’s ActiveRecord quite tightly integrated with SQL), and Rails do not take kindly to you messing with it – try doing React withing Rails itself, I’ll wait.

We eventually found a adapter for mongodb in Rails that takes care of both Rails integration and associations, and went ahead with it. But throughout the process I had some thoughts on when to use NoSQL (document-based DB in this case).

Wind back even a little longer and we’re at CVWO. Long story short, it was a system complicated enough such that even the home page fires off enough queries to make you give up on scrolling through the logs. Why? Because the client wanted a whole load of aggregated information all at once.

A document based DB is better at this kind of stuff. Information is kept together, all denormalized, in a single record, a document. For example, if a person has multiple roles, all those roles are kept and retrieved together as a single record instead of 2 tables. This part forms a huge component of the “performace”.

That of course doesn’t mean we should convert everything to documents. Documents are great if  and only if the stuff inside them belongs solely to the document alone. Our directory listing is pretty compatible: each entries’ information, both contact details and meta data, is relevant to the entry itself. A negative example will be a university course system. Student has and belongs to many modules. If you park the module under each student, you need to hunt down each of them when you update the module. Same if you park student under modules. You can have them as separate documents and use keys, and then implement foreign key constraints, and then ensure it works concurrently, and then realize you’re just doing it like SQL anyway. tldr; if you’re sub-elements are in principle shared, don’t do documents.

The key idea here is that NoSQL dbs like mongodb is not merely a “drop-in” replacement for relational databases with some tradeoffs. You actually need to adopt a whole new paradigm if you want to use them properly.

The Subtle Art of Ideation

Legend has it that some CS3216 group in the past pivoted 4 times for their project. We just had our 3th pivot last Friday, and it’s the third week. I guess we’re on schedule for the record!

In all seriousness though, we’re honestly not very good at this business of coming up with cool stuff; I think we’re somewhat better at making cool stuff cooler.  We targeted external projects as a result.

Of course, after all that pivoting, we all learnt quite a bit on what will not work. So allow me to iterate though the list, and examine what did not work.

1) Treatsure

Idea

The first project was a app that allowed food vendors to sell off their leftovers at a discounted price.

What’s good

The idea is a pretty sound one. There is indeed a problem of leftovers from the vendors, and there are definitely people who would want discounted stuff. Viable, and for a good cause.

Problem

The party who pitched it is still in its early stages of finding vendors. Furthermore, the owner prefers to keep the details confidential until he’s sure it’ll work. There will be communication problems, as well as the issue that the business process will stop us from launching before the deadline.

Lesson

We all need to work with constraints. There are things that can be sped up, and there are things that can’t be. When on a tight deadline, good ideas that eventually has potential can become bad ideas, for crucial steps will be rushed and screwed up.

2) Huntbot

Idea

To make a bot to scrape directories to automatically find large amounts of sales leads.

What’s good

It’s a project from Pearcomms, and based on their research, there is essentially nothing out there that does this task properly. Previously they hired an intern to sort though hundreds of rows of directory data, and that’s just not something anyone should be doing in this day and age. Thus, it’s a doable project that is really useful.

Problem

We make many scrapers, organise and data, and er… idk, then what? This is one of those projects where though useful, all the action occurs behind the scenes. We can do some wonderful data organisation at the backend if necessary,  but they, alas, will remain there; there’s not much to show. Going beyond simply directories will meaning venturing into unstructured data; apparently even the intern couldn’t deliver that one consistently.

Lesson

Some ideas may be useful, but there’s also little space for extension. There are many small problems out there, and they need to be solved, but don’t expect everyone of them to be able to turn into something epic.

3) Sales Marketplace

Idea

A marketplace for sales services. Producers sell sales services or even sales services (like Huntbot can actually be sold on this platform), consumers grab them, in one dedicated portal.

What’s good

There’s no go-to place for such things. Right now hunting for someone to do your marketing means hanging out on everything from Gumtree to Hardwarezone and hoping you bump into someone. It’s a market need.

Problem

Prof Ben. Ok, not really, but Prof Ben happened to hear of it, and advised against it. We think it’s pretty solid advice. For starters, where the heck are we gonna get the producers? Telemarketers/Email-marketers usually register with some agent to find jobs. We are competing directly with them, they’ll never happily hand the list over. It’ll be a marketplace full of consumers with no producers, standard two-sided problem. Not gonna be solved in 4 weeks.

Lesson

When something doesn’t exists at all, it best to consider long and hard why it does not exists. Most likely it’s because it requires more resources than 4 college kids could ever hope to amass to pull it off.

5)

We’re going a online tuition software right now. Probably gonna annex some WebRTC interactive tool and build a complete organisation system around it. Looking forward (hopefully never) to the next pivot.