Weeldi vs. Traditional RPA

This is an update to a post originally made in February 2020.
​
11 obstacles you will encounter using traditional RPA (Robotic Process Automation) bot platforms for process automation on websites

RPA on websites

Traditional RPA bot platforms are powerful tools, however, if you are attempting to use them for automation on websites you will need to ensure you have solutions for the following 11 obstacles.

1. Logins

The processes you want to automate may be on the Deep Web (i.e. behind login credentials). To automate these processes you’ll need to ensure your solution can securely store and/or access your credentials. This means both vetting the security posture of your bot automation platform and supporting integrations to 3rd party credentials vaults like Basis Theory, 1 Password, Amazon Secrets, and others. In addition, credentials change and may not always be updated in your vault or bot automation platform, which means your bots must be smart enough to recognize legitimate invalid credentials and not only surface visibility to your operations personnel for action, but also ensure that your bots don’t inadvertently lock logins by reattempting invalid credentials.

2. MFA - Multi-Factor Authentication

Websites may require MFA when logging in from a new or unrecognized web browser. In addition, this step can be triggered at the discretion of a website at any time. You’ll need a solution to securely minimize, keep up with and resolve various types of MFA including OTP (One Time Passcode) to email, text, or authenticator apps.

3. CAPTCHA

Websites may require you to solve CAPTCHA before proceeding with your automation. You’ll need a solution to solve, avoid and keep up with various types of CAPTCHA, which are constantly evolving to be ever more intricate and even invisible to the human eye.

4. Bot Deterrents CDNs - Content Delivery Networks

More and more Websites deploy CDN’s (e.g. Cloudflare, Akamai, etc.) which may block bots intentionally to minimize bot traffic or unintentionally through their efforts to prevent DDOS (Distributed Denial of Service) attacks and improve general website performance. These deterrents are triggered by accessing a website in a way that distinguishes a bot from a human such as too fast, too frequently, concurrently, from non-trusted IP addresses, while using headless browsers, and various other reasons. You’ll need a solution to avoid and adjust to constantly evolving bot deterrents.

5. Website Changes

When you automate on websites you’ll need to be prepared to manage and adapt to change. Page layouts are rarely static due to alerts, ads, and popups. Website DOMs (Document Object Models) are constantly evolving with element tags regularly changed or inconsistently named. Items on webpages may change their position rendering XY coordinates as a sole method for locating targets insufficient. To automate stably on websites, without overwhelming your resources, you’ll need a solution that can automatically solve for these changes and only surface the changes to operations that require human judgment.

6. Infrastructure

As your automation efforts gain traction, eventually you’ll need to scale, which means you’ll need to move your bots to the cloud. This will require DevOps time and expertise to deploy, manage and scale your bots in the cloud. You will also need to adjust your bots to work with the various nuanced configurations, security, and monitoring differences you’ll experience running automations in the cloud.

7. Scheduling, Intermittent Downtime, Time-Outs, and Latency

You’ll need a flexible solution to schedule the frequency of bot attempts and reattempts (for failed) unattended automations. Reattempts are particularly important with automation on websites, as websites may experience temporary downtime or slow page loading, which can cause bots to fail. Reattempt scheduling needs to be built into your bots and configurable to minimize manual intervention and maximize the efficiency of your automations. You’ll need a solution that can handle this detail and flexibility for scheduling.

8. New Front End Frameworks and Web Widgets

As the web evolves so do the frontend frameworks and web components. Your bots should be able to run in any framework such as AngularJS and ReactJS and compatible with a broad array of web components such as modals, hidden fields, shadow DOM, and sophisticated navigation and search components. In addition, your bots should evolve to account for modern web development practices. With traditional RPA, this may mean building new bots or rebuilding your existing bots.

​9. A/B Testing

Websites can have multiple versions depending on region, customer type, and A/B testing. To adequately support this common scenario you will have to build and support multiple versions of your bot and ensure your automation tool is intelligent enough to timely identify which version of the Bot should be used for each Job.

10. Rogue Scripts, Insecure Connections, and Mixed Content

Some websites include rogue scripts or libraries, which will trigger timeouts, and others insecure connections or content, which result in modern web browsers rejecting all or part of the page. You’ll need a solution to manage these obstacles.

11. Compliance

The processes you want to automate may require or access sensitive information like PII (personally identifiable information) or financial data (e.g. credit card, banking information, etc.). This means you must validate that your solution is both secure and within compliance for your automation use case. (e.g. PCI DSS, GDPR, SOC2 Type II, etc.).

Why Weeldi

While traditional RPA platforms can be powerful tools, they often require costly, intricate configuration and even custom development to deliver an enterprise-ready solution for stable, scalable process automation on websites. Alternatively, considering purpose-built solutions for this use case could save you time and money. As an example, companies like Weeldi provide solutions to these 11 obstacles and many others, out-of-the-box, with no costly configuration and with no coding required.