Over the last month I’ve been really busy with our product launch at work. The name of the project is Permission Slip, and at its heart is an app that acts as an agent for people excising their online privacy rights.
The main app is on ios, with a brand-new version now on android, and a backend made in python/django and postgres. The main development was contracted out to an external software house in Canada. There’s been some churn over there, and we’re on the third round of managers and engineers. I’ve been doing tech leadership with the team, which coming to the end meant lots of code reviews, acquiring credentials for all the different systems, coordinating with the product and marketing teams, and with apple and google, and doing develops, CI/CD, setting up pipelines from github to our deploy servers. Lots of extra drama about goggle ad tags, goggle auth keys, and back’n’forth with legal over the privacy policy. And oh yeah, building the web site.
Building the web site was actually kinda cool and fun, if not for the deadline pressure. Got to learn about QR codes, and do some nice responsive mobile layout in CSS. By the end of the last week we were in QA, fixing bugs to the very last minute. We did a pre-launch deploy of the web site and backend, and submitted the app to the apple and google stores. Everything came together and went fine and there were no bugs or glitches. Monday we got approved and for sale on google (the ios app already was released) and updated the web site with the goggle links. We were live, and could take a deep breath of relief.
Tuesday morning our app went live with the “true launch”. The marketing push included an article in the Washington Post, and on NPR. Around 11:30 in the morning the app is getting slow due to heavy and we start investigating.
The app had previously gone thru a beta phase, then a soft launch last winter, and we had about 12,000 users. In about six hours we had over 20,000 new users. Two days later we were above 50,000. That was our goal for the whole year. Over the weekend we passed 100,000.
Being deployed on the cloud, we scaled up our app dynos and added workers, and migrated the DB to a container with 4x the ram. Investigating, we discovered that the database was the critical bottleneck. We looked at what are the heaviest queries in terms of both invoked the most often and most expensive to run, and began optimizing the code there and pushing new changes on the backend into production. Amazingly, all this actually worked, and within a few hours the mischief was managed and things were trending back to normal. It took until after midnight to get all the loose ends tidied up. A long day that started with panic, but ended with a big victory. Being more popular than expected by an order of magnitude is a good problem to have.
Over the next few days we reviewed all the patches we made, and deeper, more robust fixes where necessary. We were able to deploy and roll back the commits one at a time to really understand the performance impact. I’m certainly glad now I spent time upfront to develop a deploy pipeline integrated with our code repo; it really paid off. A few months ago the devs were just deploying from their local dev environments, that would’ve been a huge disaster. I’m also happy about having in metrics and analytics in place that gave use info we could use and respond to with code changes in real time. Most of all, I’m very impresses with how everyone on the team came together in problem solving mode and got it done quickly and effectively.
You should know that my job is running a software R&D group within the company. We have a peer group, that’s more directly tasked with commercialization and productization of R&D projects, and indeed they worked closely with us on the marketing and other things. But they lost a few key people in the tech and leadership areas in the last few months, so we had to do what was necessary on our own. And, like I said, we made twice the target number of new users for the year in just three days. Happily, now our corporate enterprise department wants to migrate our app into their infrastructure, so down the road my team won’t have to worry about devOps and can get back to doing R&D.
That was just one adventure last week. The second was that it was time to make the class schedule for the upcoming Pacific Coast Origami Conference, happening in San Francisco at the end of October. This has actually gotten fairly routine. The tool that Robert Land and I build it working and stable, with the latest round of improvements making it easier to match teachers that want A/V equipment to classrooms that have it. Also this convention is alot smaller than the OUSA New York Convention in June. Still the work is over a weekend and tends to be late and night, and there’s always some last minute changes, shuffling, and special considerations to be accommodated. Anyway, we got it completed without any problems.
Also over the weekend we took a trip up to Buffalo to visit my parents and my kids. It was a pretty quick trip, we drove up Friday night and home Sunday night. Saturday we visited Michelle on campus, saw her new apartment, which is quite nice, walked around the campus and later went out to dinner at Pizza Plant at Canal Side. Pizza Plant used to be one of our favorite places when Jeannie and I were dating. It’s nice that they’re still around and their food is yummy. Sunday we watched the Bills game with my parents, which for some strange reason was being played in England, where they have an entirely different game called football, and was on at nine in the morning. After that Lizzy came over for dinner and we all enjoyed and nice afternoon. And wouldn’t you know, it was rainy on the drive up and home again.
In other news, we’re closing in on the release date for The Global Jukebox 3.0, and I’ve turned the corner from tacking to mixing on my song A Plague of Frogs. Today I layered up a nice fat, 80’s style synth sound for the part called “Synth 1”, using an analog lead sound, synth brass and strings.