EXECUTIVE SUMMARY:

Motus is the definitive leader in solutions for businesses with mobile-enabled workforces helping over 2,300 companies and 300,000 mobile workers simplify reimbursement and management of mileage, mobile devices and remote work.

After a recent acquisition, Motus faced the challenge of automating its acquisition’s data collection processes from over 20 vendor websites, spread across 13 countries and in 10 languages. Motus had a simple objective: quickly create automated processes to identify, capture, and consolidate the critical business data it needed from anywhere on the web it was living.

So Motus explored every opportunity to automate data collection on the web.

After a pilot with a well known RPA (robotic process automation) platform, and a failed experiment with an open source tool, Motus chose Weeldi for a pilot project.

When Weeldi delivered five stable automations within its ten-day risk free pilot, Motus chose Weeldi as a partner. The company currently enjoys more than 90 automations in 10 different languages, collecting endless amounts of critical usage and spend data for their customers.

The problem: complex automation of manual processes on the web to better handle scale

The nature of the mobile workforce is often chaotic. Different devices used by different staff; working in different places and timezones; different needs for tech, transportation, and just about everything.
​
As the company that simplifies management for all that (and more) for over 2,300 customers, Motus has to think and act at scale. In this case, it meant being able to consistently and completely collect critical data for its clients from multiple vendor websites regardless of the hurdles at each step (CAPTCHAs, Two-Factor Authentication, instability, different website frameworks, etc.).
​
Prior to acquisition, the acquired company’s processes could be handled with the right people, manually. But after being acquired and integrated into Motus, the need for scale increased exponentially, and manually combing through thousands of webpages a month was no longer an option: automated processes were needed to prepare for the increase in volume, while the manual labor was directed to higher value tasks.

The search for a solution, part 1: start, sputter, stop

After working with a well-known RPA (robotic process automation) company to create a proof-of-concept for a single automation, Motus turned to Selenium, an open-source browser automation tool (primarily used for app testing) to build it.

After a development process that took the better part of two months, one automation was built. However, it proved to be a house of cards, unstable and unable to accomplish the single task it was developed to execute on the web.

So Motus looked elsewhere and decided on a 10-day pilot with Weeldi, a San Francisco-Bay Area company that enables companies of any size or technical ability to stably automate processes on the web, at scale, with no coding required.

The solution, part 2: automation at a record-setting pace, and we’re just getting started

In the 10-day pilot, Weeldi delivered five full-featured automations to help Motus automate complex manual backend processes on vendor websites, and another 40 processes just days after the official start of the engagement.

Weeldi was able to create stable automations on the web for Motus (without requiring any coding work from the client), solving challenges such as login, OTP resolution, task scheduling/completion, and more.

Motus now enjoys more than 90 automations, in 13 countries, and 10 languages. Other “Weeldi wins” for Motus include:

• Using the Weeldi API to create, update and run Automations
• Automation results pushed to a Motus sFTP
• A complete transition of all manual Telco data processes to Weeldi within just four months.
• Increases in scalability, timeliness of data, and data precision
• The ability to add more customers with less manual labor

With Weeldi, Motus enjoys the freedom to quickly create dependable automations for endless tasks across the enterprise and can better and more completely utilize the skill sets of its staff. Perhaps most importantly, because of Weeldi’s flexibility, Motus essentially has every automation on the web it will ever need at its fingertips.

About Weeldi

Headquartered in the San Francisco Bay Area Weeldi enables companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required.

8 areas where you can leverage AI today for automation on the web.

Artificial Intelligence (AI) is a hot and highly marketed topic, but the application of AI for automation on the web is still in its infancy. This means those laying the right foundation today will be rewarded with the compounding value of less friction and more success with future automations.

Here are 8 areas where you can begin leveraging AI for automation on the web.

1) Login

Many automations on the web start by getting through a login page.

AI can be leveraged to simplify the login step by looking up standard login components such as username, password, and “remember me” functionality, minimizing obstacles like MFA (Multi-Factor Authentication) and CAPTCHAs, and increasing the probability of automation success when there are login page changes.

2) Flow selection

A growing number of websites have multiple versions based on region, customer type, M&A (Mergers & Acquisitions), and/or A/B testing. 

Using AI you can simplify the support of multiple website versions. This can be achieved by recording multiple automation flows, by website version, then using AI to select and navigate to the correct flow. This approach adds scale and stability to the automation by eliminating the need for a human to micro-manage variability of website versions. 

3) Managing website issues

Traditionally, website instability is a key culprit for breaking automations on the web.

AI can help solve the most common instability issues like slow-loading webpages and webpage timeouts. It can do this by intelligently adjusting bot wait times by website, proceeding when critical components of a webpage are available (but noncritical components are not) and automatically reloading webpages with timeouts. 

4) Formatting

Bot automation flows are guided by data such as dates, account numbers, and SKUs. However, the way this data is presented on websites may vary from how you store it in your database, and even if you harmonize the two, eventual changes to the presentation of data on the web will break your automations.

AI can help solve this problem by recognizing data patterns when websites make changes to the way they present data. Common examples are changes in date formats (i.e. 01/17/2021 vs. 17/01/2021), account numbers (i.e. 001234567 vs. 1234567), and SKUs (i.e. TESLA-S vs. Tesla Model S).

5) Nomenclature 
 
Change on the web is constant as websites evolve to meet their users' expectations. For example, you may download your utility bill each month by clicking on a link that states “Invoice”, but next month that same link may state “Bill”. These types of seemingly obvious changes to the human eye will break traditional bot automations. 

By using AI you can better recognize relevant nomenclature changes such as synonyms and increase your probability of success automating on the web.

6) Navigation

Website changes go beyond nomenclature changes. Sometimes nomenclature stays the same, but the elements in the User Interface change. A common example is a website modernizing clickable links by transitioning them into buttons. This kind of change will break traditional bot automations. 

By storing the goal/intent of your automation rather than just the HTML element location and raw browser interactions you can leverage AI to successfully navigate website changes.

7) Pop-ups

Websites use pop-ups to gain the attention of their users. Unfortunately, this can break brittle, bot automations. 

By using AI you can intelligently recognize pop-ups, the content within them and derive whether your automation should fail or close (or acknowledge) the pop-up and proceed with the automation.

8) Scheduling 

Some Bot automations run on a preset schedule based on an understanding the automation can be executed successfully at that time. But, what if your scheduled date is wrong. A common example is you command your bot automation to download a report on a specific date, but continually the report isn’t available until 2-3 days later. 

AI can help you solve this problem by learning the optimal schedule based on historical automation success and suggest the appropriate schedule adjustment. 
​
Why Weeldi?

As companies attempt to automate predictably on the web they must consider the web's dynamic nature and look to pattern recognition technologies like AI to help manage this variability. Companies like Weeldi have started laying the groundwork by designing automations that focus on user intent, rather than exclusively on static navigation steps, and then using AI to successfully navigate to the users expected outcome. This approach proves to be a sturdy alternative for automation on the web where change is constant, broad, and unpredictable. 

6 key security items to consider when selecting a cloud-based automation vendor

1) Encryption

Sensitive data like passwords and PII (Personally Identifiable Information) required to run automations should be encrypted in transit and at rest with a cloud-based automation vendor. Also, sensitive data like passwords should never be displayed in clear through the application UI. 

2) Transparency

Access logs should be made easily available to you showing every time-sensitive data is accessed, including username, time of access, and the data accessed. Neither your users nor a cloud-based automation vendor should not have the ability to edit or remove these access logs.

3) 2FA (2-Factor Authentication)

2FA adds another layer of time-sensitive, security to the login process, reducing the chances of your account being hacked. This layer can be an OTP sent to your mobile device or an auto-generated code via a 2FA app. However, it should be noted, as hackers get more sophisticated in their ability to hack into SMS via flaws in SS7 protocols 2FA apps are preferred for optimal security. Cloud-based automation vendors should provide the flexibility to require 2FA at the organization level or individual user level. Commonly used 2FA apps are Twilio Authy or Google Authenticator.


4) KMS (Key Management Service) Integration

KMS Integration allows you to integrate with your current KMS system to create and control the encryption keys used to encrypt your sensitive data. It also enables transparency into who is accessing your sensitive data and provides you with full and immediate control over who can continue to access your sensitive data. Commonly used KMS tools include Amazon KMS and HyTrust.

5) Secret Manager Integration

Secret Manager integration gives you the ability to use your Secret Manager tool of choice to store sensitive data like passwords, control access to sensitive data, maintain transparency into when sensitive data is accessed and centrally manage sensitive data. Commonly used Secret Manager Tools include Amazon Secrets Manager and Google Secret Manager.

6) SSO (Single Sign-On)

SSO integration allows your users to log into a cloud-based automation provider with a single ID and password following the SAML 2.0 protocol, usually centrally managed through an Identity Management provider. Commonly used Identity Management Providers include OKTA and OneLogin.

​Why Weeldi?
​
As companies automate more tasks requiring sensitive data like passwords or PII (Personally Identifiable Information) they are faced with the challenge of ensuring their automations run securely. Cloud-based automation solutions like Weeldi provide transparency into sensitive data access, best in class security using tools like encryption, 2FA (2-Factor Authentication) and SSO (Single Sign-On), as well as integration with 3rd party KMS (Key Management Services) and Secret Manager tools to ensure your sensitive data is secure and in your control.

Mobile Solutions Partners with Weeldi to Enhance Managed Mobility Software Automations

San Francisco, CA and Centennial, CO – September 25 2020: Weeldi, a San Francisco Bay Area-based company innovating the way processes are automated on the web, and, a leader in Managed Mobility Services, delivering expense optimization, unified endpoint security, device management, and best in class support, have partnered to add additional automations to MAX, Mobile Solutions proprietary mobility management portal.

​Automations can be difficult due to the dynamic nature of the web. Partnering with Weeldi has enabled Mobile Solutions to easily and quickly add new automations, while also providing the flexibility to manage and run automations through Weeldi’s web service API and UI. With Weeldi, Mobile Solutions is automating about 50,000 tasks critical to managing their customers’ expenses, inventory and usage monthly.

MAX, Mobile Solutions innovative mobility management solution, was designed from the ground up built with the user experience in mind offering in-cycle analytics and delivering a unified platform for all mobility management needs. Weeldi’s API strengthens Mobile Solutions scale and frequency of automations, running a high volume of daily automations in a condensed timeframe of only a few hours. Weeldi also integrates with Mobile Solutions Amazon Key Management Services for encryption of their customer credentials.
​
“Mobile Solutions is committed to using innovation as a vehicle to scale and deliver the best possible experience for their customers. This pushes us and makes the opportunities in our partnership almost limitless. We’re excited about our partnership.” said Moe Arnaiz, Co-Founder and CEO of Weeldi.
​
Jim Johnson, Co-Founder and CEO of Mobile Solutions, said, “We couldn’t be happier with our partnership and experience with Weeldi. From the fast, low risk deployment to running over 1,500 daily automations, Weeldi elevates our ability to automate tasks, providing up to date and timely data for our customers.”

About Mobile Solutions

Mobile Solutions is a SaaS company that provides a complete managed mobility practice. We make it easy for our clients to optimize IT spend, reduce data consumption, secure mobile endpoints, and simplify the management of their entire fleet of devices, all with best in class support. For more information, visit https://mobilesolutions.net/. 

About Weeldi

Headquartered in the San Francisco Bay Area Weeldi enables companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required. To learn more, visit https://www.weeldi.com/

5 approaches to automation that are too rigid for the web.

Miles Davis could pick up his trumpet, step into a quartet, listen, adapt and produce beautiful Jazz that worked.

Similar to a great Jazz musician, a good automation approach for the web has the flexibility to process dynamic inputs, adapt in near real-time and deliver just the right output.

However, most traditional automation approaches are rigidly designed with little ability to adapt to change, making them brittle when deployed on the web.

Below are 5 approaches to automation that are too rigid for the web.

1) Using X-Y coordinates to locate objects

Similar to longitude and latitude on a map, this approach relies on the objects on the screen to be predictably in the same location time and again, but the web is innately dynamic and objects move or could be presented differently based on the browser, device type and resolution. Advertisements, message notifications, report filter options and UI design modifications are examples of changes that will break any automation that relies on X-Y coordinates.

2) Extracting the DOM structure and looking for elements by name or structure

The DOM is a tree structure of all the elements that make up a web page.

Traditional automations extract the DOM structure and look for elements precisely by their id, class or position. This approach is prone to instability as many websites eschew naming DOM elements entirely, arbitrarily change ids and class names or duplicate ids in violation of DOM specifications, causing automations to break.

Also, web development trends are moving toward websites that load dynamically as you scroll through the webpage, defer loading portions of a web page, or dynamically build portions of the page after it is loaded, to provide a more seamless experience to customers with lower bandwidth. While this improves the user experience, it confuses and breaks automations that depend on a static, predictable DOM structure.

Finally, most DOM violations aren’t a stumbling block for the user experience because web browsers like Chrome and Firefox do a fantastic job of masking poor adherence to web standards. But, these DOM violations will break automations and require arduous collaboration from operations and software development to investigate and resolve.

3) Javascript containers (e.g. React, Angular, etc.)

Javascript frameworks help software developers build consistent user experiences, however, they also hijack user interactions like mouse clicks and scrolling, which can render automations powerless, causing them to fail.

4) Using shortcuts to bypass the User Experience

To increase the speed of automations, some software developers decide to bypass the website user experience by taking shortcuts to complete automations faster. Taking these shortcuts will inevitably lead to issues like slowing down (or crashing) your target website, IP Blocking, getting auto-logged out and/or accessing incomplete data. Short-cuts appear to be great until they don’t work and they then become very difficult to investigate and repair as they were not based on user experience to begin with.

5) Using web testing automation tools (e.g. Selenium, etc.)

To accelerate automation, rather than start from scratch, software developers may choose to utilize web testing automation tools like Selenium. However, these tools have a propensity to get jammed-up by rogue javascript libraries, such as the ones often embedded in website analytics tools.

So when you’re automating on the web you must ditch the traditional rigid approach for an adaptive modern approach that produces beautiful automations that work.

Why Weeldi?

As companies attempt to automate stably on the web they must consider its dynamic nature. Weeldi’s modern approach to automation is flexible and adaptive, enabling companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required.​​

10 obstacles you need to consider when using Robotic Process Automation (RPA) bots for unattended automations on the public internet. 

RPA on the public web

​RPA bots are powerful tools, however if you are attempting to use RPA bots for unattended automations on the public web you’ll need to ensure you have solutions for the following 10 obstacles.

1. Log-Ins

The processes you want to automate may be on the Deep Web (behind login credentials). To automate processes behind logins you’ll need to make sure your RPA bots can securely access the credentials necessary to complete your unattended automations.

2. Two-Factor Authentication

Websites may require two-factor authentication when logging in from a new or unrecognized web browser. In addition, this step can be triggered at the discretion of a website at any time. You’ll need a solution to avoid and resolve two-factor authentication.

3. Captchas

Websites may require you to solve captchas before granting access. You’ll need a solution to solve or avoid captchas.

4. IP Blacklisting

Websites may blacklist critical IP addresses if you are accessing them too fast, too frequently or concurrently. You’ll need a solution to ensure you do not trigger IP blacklisting.

5. Scheduling
​
You’ll need a flexible solution to schedule frequency of RPA bot attempts and reattempts (for failed) unattended automations.

6. Schedule-and-Return

Some scenarios will require you to first initiate your automation and then return within a defined time frame to complete it. You’ll need a flexible solution to manage schedule-and- return automations.

7. A/B Testing

Websites can have multiple versions of their site depending on region, customer type and/ or A/B testing. To adequately support this common scenario you will have to build and support multiple versions of your bot.

8. Intermittent Downtime, Time-Outs and Latency

Websites on the public internet can be unstable so your RPA bots will need to be equipped to manage instability to ensure you can complete your unattended automations successfully. This means not aborting unattended automations too early and automatically reattempting others at a later time. In some instances, it may take several attempts over multiple hours before an automation completes successfully on a public website. You’ll need a solution to manage this website instability carefully.

9. New Front End Frameworks and Elements

As the web evolves so do the frontend frameworks and elements. Your RPA bots should be able run in frameworks such as AngularJS and ReactJS and elements such as modals and hidden fields. In addition, your RPA bots should evolve to account for new frameworks and web elements as they are introduced. In the world of RPA bots this may mean building a new bot.

10. Rogue Scripts, Insecure Connections and Mixed Content

Some websites include rogue scripts, which will trigger timeouts and others insecure connections or content, which result in modern browsers rejecting all or part of the page. You’ll need a solution to manage through these obstacles.
​
Why Weeldi
​
While RPA bots can be powerful tools, they often require costly, intricate configuration and even custom development to deliver an enterprise-ready solution for automating process on the public internet. Alternatively, considering purpose-built solutions for automating process on the public internet could save you time and money. As an example, companies like Weeldi provide solutions to these 10 obstacles and many others, out-of-the-box, with no costly configuration or coding required. In addition, you pay by the results, not per bot hour or the individual bot.

How businesses are moving from risky home grown “scraping” solutions into secure, comprehensive Web Data Integration (WDI) 

The current state of web data

Web data is the boundless unstructured data living on websites, web portals and SaaS applications–simply put, it’s the data in your web browser. For years, web data has been something you can see, but not really “touch” or incorporate into your business systems without accessing each website via an API or building homegrown scraping solutions. But that’s about to change.

Finding the fuel to move forward

Often, the data a business needs to fuel success can be accessed by APIs, but not always, as there are countless non-API sources of information on websites, web portals, and even some SaaS applications.

Take for example many Mobile Telecom Companies who offer APIs to customers to help manage expenses, but leave out critical pieces of data, like current international usage, which is invaluable for avoiding costly roaming charges. This usage data may not be in the API, but is instead sitting as unstructured web data, trapped in a customer web portal...ripe for consumption if it could only be easily accessed at scale.

Moving past relying (solely) on APIs

While APIs are a standard means of data transfer, that doesn’t necessarily make them the most expedient or ideal. Some API access processes can take weeks, months, or longer to approve and set-up, and these delays raise risks of failed deployments, missed revenue opportunities, and poor customer service.

But assume API setup goes smoothly: what if your API vendor sunsets support, or decides to make significant changes that make it more difficult to work with their API?

The ability to independently access web data moves from being a “nice to have” to a must, as does the need for a modern Web Data Integration (WDI) tool that can extract web data across multiple websites with the same scalability, reliability and structure of an API.

According to 2019 research from Opimas, spending on web data extraction was $2.5B in 2017 and predicted to reach nearly $7B by 2020, with the majority of spend on internal homegrown solutions. But while the overall web data extraction spend is trending up, Opimas expects a significant transition from internal spending (with expected growth at 30%) to external spending (with expected growth at a whopping 70%). (1)

So why the move toward more external solutions?

In their current state, internal solutions for web data extraction are often unstable and send valuable software development resources down a proverbial rabbit hole of custom coding and neverending maintenance in a attempt to manage: data extraction templates for every data source, IP blacklisting, CAPTCHA, 2 factor authentication, pacing, data access across multiple pages / websites or data behind logins, file downloads and scheduling — to name just a few.

Signs are pointing toward use of modern Web Data Integration (WDI) solutions that quickly transform any unstructured web data into structured APIs — with no coding required. Specifically, faster, more reliable, more scalable, and more affordable alternatives to legacy homegrown and outsourced web data extraction solutions.

How modern Web Data Integration (WDI) differs from “scraping”

Modern Web Data Integration (WDI) tools are built to deliver scalable, reliable web data extraction out of the box without requiring software development for either configuration or maintenance.

Whereas scraping is a disjointed process of collecting web data that requires software developers to build and reactively maintain custom scraping routines, typically built with a hodgepodge of open source tools.

A modern Web Data Integration (WDI) tool gives you “out of the box” ability to:

• configure web data extraction routines via a user interface with no coding
• required
• pay for the solution based on results / successful web data extraction
• extract web data across multiple websites and page flows
• merge data from multiple sources
• download multiple file formats (PDF, excel, csv, images, video, audio)
• deliver structured web data via an API or download
• extract your web data behind login screens, two-factor authentication or CAPTCHA
• automatically adjust pace and frequency to avoid IP blacklisting
• schedule web data extraction (including schedule and return to retrieve)
• ensure data is transmitted and stored securely
• circumvent web scraping defenses
• keep up with quickly evolving web browsers
• validate web data extraction accuracy
• provide audit trails of extracted web data
• leverage experience extracting billions of web data records

Scraping requires hourly, salary and/or contract specialist in product management, software development and QA to build and maintain the same functionality that is available out of the box with Web Data Integration (WDI) tools as listed above.

Plus you to take on the risk of:

• delays or never reaching a production-ready solution 
• being out of compliance with data privacy and regulation policies such as GDPR and California Consumer Privacy Act
• instability as you encounter frequent changes and challenges
• cost overruns with building and maintenance
• obsoleteness as severity of changes and challenges require a full rebuild tying up scarce and valuable software development resources
• having critical IP addresses blacklisted
• insecurely transmitting or storing data
• overwhelming target web data source resources, resulting in blocking outsourcing to “gray hat” locales

​
To DIY, or Not DIY?

​SMB and enterprise adoption of modern Web Data Integration (WDI) tools is on the rise. While there are many legacy, home-grown, outsourced and on-premise solutions still deployed to solve the “web data extraction needs” of competitive businesses, most do not deliver the modern ease of use, reliability and scalability companies need to make web data an integral part of their business.

Consider this: just as Stripe has done for online payments and Twilio for business communication, modern Web Data Integration (WDI) tools are now doing to empower organizations in all markets and of all sizes to capture and act on the endless ocean of web data.

REFERENCES
1: A. Griem, & O. Marenzi, Web Data Integration — Leveraging the Ultimate Dataset [White paper], retrieved March 2019, Opimas Research