You might have noticed a few glitches and or a delay when opening pages over the course of the last week when accessing Pop Price Guide. And while we encourage everyone to slow down during the holiday season, this is not what we had in mind. We’re always looking for ways to improve site speed and quality without compromising functionality.
But sometimes the bugs have other plans.
If you manage your collection or wish list on PPG or are one of our sellers we thank you for your patience as we work to improve the site’s technical infrastructure. In the coming weeks, we’ll be focusing all of our development capacity on infrastructure work.
The remaining time will go towards resolving check-out bugs with the Request Shipping process and finishing our Variant Group project, which aims to reduce and clean up Subvariants, Autograph and Customs. Read more about the project here.
Among other issues diligently being worked on by our team is making sure your collection value updates every day. That’s in addition to making sure items appear sooner when added to your Collection as well as the Pop Price Guide Marketplace.
What’s the Plan?
If you’re interested in some details on what’s happening under the hood (totally okay if you’re not, just skip this section), here’s what we know and what we’re planning to do.
hobbyDB and Pop Price Guide have “write” servers that write to the database whenever a change is made, like adding Price Points or editing a Database Item, and “read” servers that deliver the content whenever visitors load a page on the site. The read servers are what handle most of the traffic.
On Tuesday morning (21 November at about 6 a.m. MST) our read server started to hit 100 percent load capacity (called CPU usage %), and has stayed maxed out almost continually. For context, the normal load on these servers usually hovers around 30-40 percent.
That’s the reason for the slowness and we are still trying to work out exactly why the servers are maxing out. A few possibilities are DNS issues, some change at Microsoft Azure, bots, or someone trying to scrape data from PPG, but we don’t know the root cause yet and work through all of these issues in turn.
As we investigate and troubleshoot we are also making headway with infrastructure optimizations so we don’t get hit so hard like this if problems come up in the future. One of those optimizations was removing 15 gigabytes of unneeded information from database tables yesterday (that process, funnily, is called “vacuuming” tables).
Here’s what we plan to do in the next few weeks –
- Vacuum all other tables
- Remove some tables and other old code that are not needed anymore
- Upgrade Rails (our web application framework) to a higher version
- Deploy Pgbouncer (a tool that allows better usage of our servers) which needs that higher Rails version
- Refactor encoding of our database
- Improvements to a number of other frequently used queries
- Follow other suggestions that we are receiving from the technical support team at MS Azure team (we host with Microsoft)
Thank you again for your patience as we work to restore and improve PPG. As always, we want to know your thoughts. Message us at firstname.lastname@example.org and we’ll do our best to address any questions or concerns.