Integrating Shopify and Raiser's Edge CRM for Museum Members

The Art Gallery of South Australia (AGSA) has a long-running membership program that includes discounts within their physical shop. As part of AGSA’s digital transformation, the gallery wanted to provide a means for their members to also receive discounts within AGSA’s new online shop. The membership system is managed within the Raiser’s Edge CRM and the new online shop is run on the Shopify platform. If we can automate an integration between Raiser’s Edge and Shopify, it will help to lighten the load on AGSA staff as well as providing a better service for their members.

This article may seem a little grim because this integration was difficult. It took a lot of time and effort from both our internal team and the team at the gallery to get it to work. We think it's important to be authentic about what was tough – rather than the more common, super-positive, hyped-up tone of many (even technical) articles.

The Raiser’s Edge CRM is focused on providing support for nonprofits, such as the gallery. Recent versions of the CRM have shifted to a cloud-based storage system which allowed us to access its data via an API.

Accessing the data in Raiser’s Edge turned out to be a bit more difficult than expected mostly due a mismatch between what we were trying to do and the use cases the authorisation system is designed for. The system used by the API appears to be designed for reusable applications – to be used by multiple clients – rather than a one-off integration for a single client. These design decisions have resulted in a system with a lot of fairly onerous steps to perform:

• Application registration by the developer.
• Application approval by the customer.
• Public and private key combinations.
• OAuth token generation that requires a manual, in-browser step to generate a token.

Once all the steps were completed and we had an authorisation token in hand, we hit another roadblock as the API is designed to automatically invalidate tokens after a short period of time. To work around this issue, we’ve used a background task that is continually hitting the Raiser’s Edge API to revalidate the token and preserve our access.

Once we were through the authorisation system and were able to retrieve data, we were then able to build a periodic task that would automatically synchronise the membership data into AGSA’s main CMS. Unfortunately, we found that the periodic task bumped into some issues with the API – it is frequently offline and has sporadic issues handling requests without throwing errors. To prevent our automated error detection from alerting us every second day, we’ve added multiple layers of error handling and backoffs on every connection attempt.

To identify the members within Raiser’s Edge, gallery staff rely on some complicated queries that were built within the CRM’s GUI. Unfortunately, some of the features in those queries don’t have an exact 1:1 equivalent in the API, so we ingested the entire data set and then reimplemented the query.

Once our automated synchronisation had been running for a few weeks, we noticed that there were some inconsistencies in the data. Specifically, the API provides a way to filter records by their modified time, which would have allowed us to perform a much faster synchronization as we’d only need to handle new or updated records. However, in practice the system seemed to omit records – seemingly at random – possibly due to rounding or precision issues. As a workaround, we now perform a much slower, but more exhaustive sync that consumes all of the available data and then compares the records locally to determine which have changed since the last synchronisation.

An important part of AGSA’s digital transformation was extending their physical shop into an online space. They chose the Shopify platform to host their service and wanted to include the same discount system that their members enjoyed in the physical shop. Our task was to take the membership data from Raiser’s Edge and provide a way for the members to receive their discounts.

The conception phase of the discount mechanism involved a number of alternatives that were considered. Initially, we discussed having a unified login on the main website that would extend to the shop. However, this approach requires members to login and there was some anticipation that this would add more complexity to the system as well as more overheads for the members. Ultimately, we pushed for a simpler system that relied only on the unique identifier used for each member. This would allow a member to simply punch in their details from their membership card.

With a plan set and the data from Raiser’s Edge now in our hands, we could use Shopify’s API to create discount codes for each member. Given that memberships are created continually, we also had to build a periodic task that would automatically synchronise membership changes to Shopify. With each synchronisation, we create discount codes for new or rejoining members and we remove the codes for expired memberships.

While our integration with Shopify has gone reasonably well, we have run into an issue with the API’s usage limits. Shopify only allows a small number of calls to fetch all the discount codes created so far. Once our periodic task had run for a few days, we found that we passed that limit and the system began to crash during the initial call to the Shopify API. To work around the issue, we began to persist the state of Shopify’s discount codes locally. This added some minor complexity to the synchronisation code, but it allows us to perform the necessary checks quite cheaply. As a side-effect, the workaround has also massively boosted the performance of the synchronisation to Shopify.

Our work to automate the integration of AGSA’s membership system with their online shop, has helped to reduce overheads on AGSA’s staff, so that they can focus on more important tasks. For AGSA’s members, they’re now able to easily access their discounts through both AGSA physical shop and their online shop.

Integrating two systems can superficially seem simple, but the issues illustrated are fairly typical. Assumptions in one system, inconsistencies in features across APIs, incompatibilities in data models - these are all the typical problems that arise. Unfortunately, it’s rarely as simple as connecting two things together. While the challenges can be steep, it is quite satisfying to bring multiple systems together and see something new come to life.