WeighMyRack

WeighMyRack is an American price comparison website that helps climbing amateurs and professionals find gear at the cheapest price.

What was the goal of the business?

The performance of the website was struggling. We need to import data from over 10 external data feeds and it wasn't scaling well. At one point, this total import time crossed 24 hours which meant that we couldn't update our stock every day. In addition there was a build up of technical debt that needed to be addressed. We had been working with an external agency to support the site but we weren't happy with their service so we needed to find a replacement as soon as possible.

How did you communicate the requirements?

We did some research on Upwork at the end of 2018. We were looking for someone with a lot of Drupal expertise because this was a complex Drupal 7 project. We put together some high-level requirements which made it clear what results we would expect from the work.

What approach was taken to delivery?

Having conducted a technical interview and a couple of chats, ADCI Solutions Drupal developers were chosen as the main website developers. Since we trusted their expertise we agreed to work on a Time and Materials basis. We set the scope based on our high level requirments but we were flexible on the time it would take to deliver as we knew that some unexpected issues may arise.

How did you structure the team?

We went out into the market to look for a partner, not just a supplier. We wanted someone to work with us so we created a hybrid team. ADCI Solutions provided the Drupal developers and our project developer was the one who have ultimate responsibility and take part in actively suggesting the solutions.

What infrastructure did you use?

We use 2 servers for the site. The first one serves website users, the second server, the sync-server, deals with retailers’ feeds. This allowed us to ensure that end user performance wouldn't be impacted by the data imports. But this introduce new complexities - keeping the data in-sync and reducing the import times.

For the import times, we separated the feeds from each other and started running them in parallel and simultaneously. Of course, we immediately faced some issues because, in Drupal, several queries tried to access to the same information which was causing database locks. When we tested updates for 1000 products, only 500-600 were updated. We implemented a lock protection mechanism to keep control over the data being imported and doesn’t let other processes intervene until this data isn’t needed for a current process anymore.

The other challenge was around ensuring that the data was in-sync. The WeighMyRack content manager would be maintaining product information on the on the main server, while the imports are taking place on the sync server. We were afraid that one of the servers would fail and we wouldn't have relevant and updated information, or worse still we could lose data.

We created a custom backup storage on a remote server in addition to the standard backups that we were performing. That custom backup storage provided an extra layer of redundancy to ensure that the imported data will be processed to production.

What other challenges did you encounter?

Drupal Module Audit

Drupal is an extensible CMS so the site used many thrid-party modules, in addition to some custom code. Some of these modules were doing tasks not aligned to their orginal purpose so the performance suffered. In addition, security updates were not applied for 2-3 years.

We didn't want to re-write everything so we cleaned the modules enough so that we could update the site to the latest version.

DDOS-attacks prevention

The website experienced some DDoS attacks. The MySQL database would become overloaded and the website would go down. We set up Monit, a system for monitoring and reanimating  processes if they get blocked.

Now, if MySQL, Apache or httpd don’t respond within 3 minutes, Monit restarts the respective process/service and the website goes back to normal. It is especially helpful when a DDoS attack starts in the middle of the night.

What Outcomes were achieved?

The biggest problem was the import time. We managed to reduce the import time from 26-28 hours down to 12-15 hours. This allowed us to connect more stores which helps us to increase revenue and conversions.

We also improved the user experience. Google Insights used to show 1-2 out of 100 but now we've improved this number several times through server optimization, CSS and JavaScript aggregation and implementing lazy load on heavy pages.