Race Conditions in Workflows (Unexpected Behavior)

While rare, a race condition is when two servers update the status of a contact within seconds of each other. One server updates the contact, and the other server overwrites the first one. This can result in duplicate messages or an action showing that it has been executed but does not reflect the contact. This can cause loop lockouts and other unexpected behavior to occur with workflows.

Important note: Elimination of Racing Conditions for Triggers

To address issues where a contact could be enrolled in multiple workflows simultaneously due to racing conditions, we have updated our system to ensure only one entry for a contact is processed at a time. This enhancement prevents duplicate workflow executions and ensures more reliable and consistent operations. The main benefit of this update is the elimination of redundant entries, which improves the accuracy and efficiency of your workflow processes.

Points to Remember:

• The updated functionality does not apply to appointment, invoice, and IVR triggers, where re-entry checks are not implemented.

Please take a look at our Loop Lockout article, as it’s a related topic.

Common Race Condition causes:

• Multiple triggers for the same action
• Actions being executed at the same time for a contact
• Executions happening within automation for the same contact with similar timing

Example 1: Using both the Appointment Status and Customer Booked Appointment triggers to perform the same function will cause conflicts. Keep in mind that the Customer Booked Appointment trigger is deprecated, and you should use the Appointment Status trigger.

Example 2: Using opportunity changes and appointment triggers in the same workflow that fires off simultaneously.

Example 3: A contact tag is added at the same time in multiple workflows.

How do you know it's a race condition?

• Investigating workflow execution History/Status + Time and finding issues based on similar timing
• Repeating the same process and the tag is being added without it reflecting in the cont
• Duplicate messages and workflow history reflect a different trigger than intended due to timing
• Unexpected behavior due to timing

Example 1: A Contact tag trigger firing off instead of an appointment trigger because they are in the same workflow and happened at a similar time.

Example 2: You are adding the contact to a workflow with a tag, but you also have other workflows that will fire at the same time for the same contact. Using a Contact tag trigger and then expecting the appointment data to display correctly in an email using custom values. (Membership and Appointment custom values are dependent on their related triggers)

How can I prevent them from happening?

• Adding a 2 minute Wait step above the race-affected actions will allow the system to process the action correctly
• Do not use the same trigger or similar triggers on one/multiple workflows for the same actions
• Be specific with your filters and keep in mind the action the lead would take and how the workflow would fire when that happens.
• Do not use unspecific and similar triggers in one/multiple workflows
• Pay attention to the timing and watch out for triggers firing at the same time
• Use the PASS INPUT TRIGGER PARAMETERS with the Add To Workflow action when adding to another workflow when attempting to pass in values and data

Unexpected Behavior

• An execution showing that it happened in the workflow, but it does not reflect in the contact card
• Loop Lockouts
• Duplicated messages sending
• Multiple workflows firing off unexpectedly
• Membership or Appointment information not populating or displaying undefined in a message

Best Practices and Troubleshooting

• Check Enrollment History and Execution Logs inside the workflow
• Check the Added to Workflow execution in the workflow and the action details, especially regarding timing

• Add a wait step of 2 minutes before actions that may cause conflicts to offset these issues

• Pay attention to the timing and watch for triggers firing at the same time
• Get more specific with your filters as an example filter for a specific calendar when using appointment triggers

• Investigate multiple workflows that fire off the same or similar triggers that may fire at the same time when an action by the customer is taken
• Use unique trigger-to-action implementations instead of multiple workflows for the same actions
• If you need to pass data between workflows, remember passing data on to other workflows won't happen with an Add tag action and its related trigger in a separate workflow. Use the Pass input parameters toggle with the Add To Workflow action. This will take all the data in the workflow and pass it into the other workflow

• If you are using Zoom and see "undefined," Check that the Zoom user is integrated with their profile, and check if Zoom is added to the user's group calendar. After that, if race conditions apply, this may be due to the incorrect trigger adding a contact to the workflow.
• For appointments, use the Appointment Status trigger + correct filters, including STATUS
• Check added to workflow and what trigger was involved to know if data was passed correctly

FAQs: Race Conditions in Workflows

1. What is a race condition in the context of workflows?

• A race condition occurs when two servers or processes attempt to update the status of a contact simultaneously. This can lead to one update overwriting the other, causing issues like duplicate messages, incorrect action statuses, or conflicts in workflow executions.

2. What are the common causes of race conditions?

• Multiple triggers for the same action: Using different triggers that perform similar functions simultaneously can cause conflicts.
• Actions executed at the same time for a contact: When multiple actions occur simultaneously, it can lead to unexpected results.
• Executions happening within automations with similar timing: Automations that overlap in timing can create race conditions.

3. Can you provide examples of race conditions?

• Example 1: Using both the Appointment Status and the deprecated Customer Booked Appointment triggers for the same action can cause conflicts.
• Example 2: Simultaneous use of opportunity changes and appointment triggers within the same workflow can result in issues.
• Example 3: Adding a contact tag in multiple workflows at the same time can lead to race conditions.

4. How can I identify if a race condition is occurring?

• Investigate Execution History: Review workflow execution history and timestamps to detect issues related to timing.
• Check for Duplicates: Look for duplicate messages or actions in workflow history.
• Unexpected Behavior: Notice if the contact card does not reflect expected updates or if actions occur out of sequence.

5. How can I prevent race conditions in workflows?

• Add Wait Steps: Include a 2-minute wait step before actions that may cause conflicts to allow proper processing.
• Avoid Duplicate Triggers: Ensure that the same or similar triggers are not used across multiple workflows for the same actions.
• Use Specific Filters: Be precise with filters and triggers to avoid overlaps.
• Monitor Timing: Pay attention to the timing of triggers and actions to prevent simultaneous executions.
• Pass Input Trigger Parameters: When adding a contact to another workflow, use the Pass Input Trigger Parameters with the Add To Workflow action to ensure proper data handling.

6. What should I do if I encounter unexpected behavior due to a race condition?

• Check Enrollment History: Review the enrollment history and execution logs within the workflow to understand the issue.
• Verify Trigger and Action Details: Look at the details of the 'Added to Workflow' execution, especially focusing on timing and data passed.
• Adjust Workflow Timing: Implement a wait step to offset conflicts and adjust workflow configurations as needed.

7. How can I address specific issues related to race conditions?

• Zoom Integration: If using Zoom and seeing "undefined" issues, verify that the Zoom user is properly integrated with their profile and group calendar. Incorrect triggers might cause these issues.
• Appointment Triggers: Use the Appointment Status trigger with correct filters and STATUS to ensure accurate data handling.
• Pass Data Correctly: For passing data between workflows, use the Pass Input Parameters toggle with the Add To Workflow action to maintain data integrity.

8. What are best practices to avoid race conditions?

• Unique Trigger-to-Action Implementations: Use distinct triggers and actions to minimize conflicts.
• Single Workflow Approach: Where possible, consolidate actions into a single workflow rather than using multiple workflows for the same actions.
• Thorough Testing: Regularly test workflows to identify and resolve potential race condition issues.