How an up and coming machine learning technology overcame every barrier to automatically collect subscription and service data from vendor web portals and SaaS applications
MISO3’s line of business requires getting data in hard-to-reach places An intelligent subscription and service tracking company serving the needs of enterprise-level companies, MISO3 gives its clients insight into its vendor subscriptions and usage. Think about it: vendors like Zoom, Microsoft 365, Salesforce, Amazon AWS, CenturyLink, Bloomberg, and the list goes on (and on and on) — there are few businesses left that are not reliant on licenses, SaaS, PaaS, and/or IaaS subscriptions to power their business. The data required to manage these vendors spans inventories, portals, and invoices, making visibility as well as the ability to make data-driven decisions almost impossible. For most clients, MISO3 was able to deliver on its promise rapidly and with few hurdles. And then… one day… MISO3 came up against a challenge, and a big one. A Fortune 100 client and one of the best-known financial and credit card companies in the world came to MISO3 with the challenge of understanding and optimizing subscription costs for their financial data and intelligence provider. MISO3 knew exactly what it needed from the provider to better understand the client’s spend and utilization of their subscriptions and services. However, with no inventory-targeted APIs available and data scattered throughout a clunky provider portal, they needed a predictable and scalable way to get control of this multi-million-dollar expenditure. They turned to Weeldi. Headquartered in the San Francisco Bay Area, Weeldi is the easiest way to automate any process on the web with no coding required. A common use case for Weeldi is filling gaps (where vendor portals and/or SaaS applications lack API functionality) by enabling customers to automate tasks directly through the vendor portal or SaaS application user interface, common tasks include extracting data, executing transactions, and paying bills. For MISO3, Weeldi’s web automation platform would be the cornerstone of meeting its client’s need to get its vendor data out of multiple locations quickly, accurately, and with as little friction as possible. MISO3 prides itself on being the first “easy-button” in vendor inventory discovery, extraction, and alignment. What once took businesses months of labor-intensive research and data cleaning, can now be done in only hours; minimizing user-prone errors and discovering hidden costs that often uncover waste and security vulnerabilities. (The alternatives were hours of manual labor every month or attempting to engineer a homegrown web automation solution with all the costs, delays, and opportunities for error that creates — rarely an efficient path to client satisfaction.) Data: it’s everywhere, but still so hard to find A little background on the problem faced by MISO3’s client (and, really, all of its clients, and almost every business everywhere). It’s all about “the other guys”: the SaaS companies, the IaaS companies, and the PaaS companies —and the mixed blessing they provide. In 2020, the overall spend on SaaS products was up by 50% compared to only two years before. (Data has yet to be collated, but we’d bet it’s going up again in 2021.) The average company uses 137 different SaaS apps, with each employee using at least 8 different ones. (Thanks to Blissfully for this data.) With subscriptions varying from person to person and from department to department across some or all of those 137 vendors, most companies are tilting at windmills when it comes to effectively managing subscriptions. They’re oversubscribed, leaving ex-employees licensed (sometimes, to do ill), and generally overbuying across the board. They do have access to data to see what’s what, of course. But — because they need to manually chase and capture it — the data is rarely in real-time… or complete… or accurate. It makes a difference in the same way Monday-morning quarterbacking makes a difference. This is where MISO3’s M3 platform powered by data extracted from Weeldi helps. Overcoming logins and CAPTCHAs and other roadblocks—automatically Weeldi was brought in by MISO3 to solve these problems and more. What Weeldi does that others can’t is to create stable automations on vendor portals and SaaS applications that neither Weeldi nor its clients control. Weeldi ’s platform allows customers to create automations on the web when APIs simply aren’t enough, or when APIs simply aren’t available (e.g., for billing data and subscription info — the exact kind of data MISO3 collects for its clients). Weeldi has a menu of available vendor portal automations and provides the flexibility of “built to order” automations that customers can create in minutes with no coding required. Automations can uncover and capture the data or execute transactions buried in vendor portals (multiple pages, behind a login, multi-factor authentication, CAPTCHAs, requiring scheduling or filtering, on the user interface, in emails, on unstable websites, etc.) or within SaaS applications. Weeldi’s flexible and reliable automations for the web were the ideal fit for a client like MISO3, a company with a business model reliant on retrieving information from vendor portals and SaaS applications, and were central in MISO3 sustainably supporting the business of one of the best-known credit card and financial companies in the world. A solution delivered in days; results that last a lifetime Within days of receiving specifications from MISO3, Weeldi delivered complex automations that extract — from behind logins (and tricky multi-factor authentication) — critical data, some requiring filters by date and account number, some from the user interface, and even some embedded in emails. Files are then automatically renamed, and folders are created in MISO3’s sFTP to feed their ETL (extract, transform, load) process back into the M3 platform. This was not a “one and done” for MISO3, either. The automations created by Weeldi give it the ability to quickly create connectors to any vendor portal or SaaS application. Additionally, Weeldi provides precision, actionable visibility to MISO3 (and all Weeldi clients) on how their data is flowing. Weeldi can say exactly where and why the data isn’t flowing and provide direction (along with screenshots) and tools to fix it. In the end, MISO3 had a solution that served its immediate need with its client, and a solution it could use repeatedly as it continues to build its business. “Before you can make data-driven decisions for your organization, you must first know what you have,” says David Baule, CEO of MISO3. “Gathering this data manually is both cumbersome and time-consuming. Weeldi gives us the power to quickly scale to any web-based vendor portal or SaaS application at a fraction of the time of doing this manually. The result, quick and frictionless ingestion of customers’ data so M3 Users can make informed decisions on all of their vendor commitments.” Weeldi transformed disparate, dynamic, and sometimes unstable Vendor Portals into stable, scalable APIs, providing MISO3 with the detailed data required to analyze and optimize their customers' subscription costs. All delivered with no coding required by MISO3. A solution delivered in less than two weeks that will last a lifetime. About Weeldi Headquartered in the San Francisco Bay Area Weeldi enables companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required. Use "Input Values" to identify the key variables that drive your Weeldi automation. This enables one automation to power many scenarios. 🎥 Learn how to record one Weeldi Template that will extract as many files as you need — using the "Add-Lookup" feature. 🎥 This video takes you through setting up email notifications in Weeldi. Use the Weeldi Chrome extension to build and maintain web automations in near real-time. EXECUTIVE SUMMARY:
Motus is the definitive leader in solutions for businesses with mobile-enabled workforces helping over 2,300 companies and 300,000 mobile workers simplify reimbursement and management of mileage, mobile devices and remote work. After a recent acquisition, Motus faced the challenge of automating its acquisition’s data collection processes from over 20 vendor websites, spread across 13 countries and in 10 languages. Motus had a simple objective: quickly create automated processes to identify, capture, and consolidate the critical business data it needed from anywhere on the web it was living. So Motus explored every opportunity to automate data collection on the web. After a pilot with a well known RPA (robotic process automation) platform, and a failed experiment with an open source tool, Motus chose Weeldi for a pilot project. When Weeldi delivered five stable automations within its ten-day risk free pilot, Motus chose Weeldi as a partner. The company currently enjoys more than 90 automations in 10 different languages, collecting endless amounts of critical usage and spend data for their customers. The problem: complex automation of manual processes on the web to better handle scale The nature of the mobile workforce is often chaotic. Different devices used by different staff; working in different places and timezones; different needs for tech, transportation, and just about everything. As the company that simplifies management for all that (and more) for over 2,300 customers, Motus has to think and act at scale. In this case, it meant being able to consistently and completely collect critical data for its clients from multiple vendor websites regardless of the hurdles at each step (CAPTCHAs, Two-Factor Authentication, instability, different website frameworks, etc.). Prior to acquisition, the acquired company’s processes could be handled with the right people, manually. But after being acquired and integrated into Motus, the need for scale increased exponentially, and manually combing through thousands of webpages a month was no longer an option: automated processes were needed to prepare for the increase in volume, while the manual labor was directed to higher value tasks. The search for a solution, part 1: start, sputter, stop After working with a well-known RPA (robotic process automation) company to create a proof-of-concept for a single automation, Motus turned to Selenium, an open-source browser automation tool (primarily used for app testing) to build it. After a development process that took the better part of two months, one automation was built. However, it proved to be a house of cards, unstable and unable to accomplish the single task it was developed to execute on the web. So Motus looked elsewhere and decided on a 10-day pilot with Weeldi, a San Francisco-Bay Area company that enables companies of any size or technical ability to stably automate processes on the web, at scale, with no coding required. The solution, part 2: automation at a record-setting pace, and we’re just getting started In the 10-day pilot, Weeldi delivered five full-featured automations to help Motus automate complex manual backend processes on vendor websites, and another 40 processes just days after the official start of the engagement. Weeldi was able to create stable automations on the web for Motus (without requiring any coding work from the client), solving challenges such as login, OTP resolution, task scheduling/completion, and more. Motus now enjoys more than 90 automations, in 13 countries, and 10 languages. Other “Weeldi wins” for Motus include:
With Weeldi, Motus enjoys the freedom to quickly create dependable automations for endless tasks across the enterprise and can better and more completely utilize the skill sets of its staff. Perhaps most importantly, because of Weeldi’s flexibility, Motus essentially has every automation on the web it will ever need at its fingertips. About Weeldi Headquartered in the San Francisco Bay Area Weeldi enables companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required. 8 areas where you can leverage AI today for automation on the web.
Artificial Intelligence (AI) is a hot and highly marketed topic, but the application of AI for automation on the web is still in its infancy. This means those laying the right foundation today will be rewarded with the compounding value of less friction and more success with future automations. Here are 8 areas where you can begin leveraging AI for automation on the web. 1) Login Many automations on the web start by getting through a login page. AI can be leveraged to simplify the login step by looking up standard login components such as username, password, and “remember me” functionality, minimizing obstacles like MFA (Multi-Factor Authentication) and CAPTCHAs, and increasing the probability of automation success when there are login page changes. 2) Flow selection A growing number of websites have multiple versions based on region, customer type, M&A (Mergers & Acquisitions), and/or A/B testing. Using AI you can simplify the support of multiple website versions. This can be achieved by recording multiple automation flows, by website version, then using AI to select and navigate to the correct flow. This approach adds scale and stability to the automation by eliminating the need for a human to micro-manage variability of website versions. 3) Managing website issues Traditionally, website instability is a key culprit for breaking automations on the web. AI can help solve the most common instability issues like slow-loading webpages and webpage timeouts. It can do this by intelligently adjusting bot wait times by website, proceeding when critical components of a webpage are available (but noncritical components are not) and automatically reloading webpages with timeouts. 4) Formatting Bot automation flows are guided by data such as dates, account numbers, and SKUs. However, the way this data is presented on websites may vary from how you store it in your database, and even if you harmonize the two, eventual changes to the presentation of data on the web will break your automations. AI can help solve this problem by recognizing data patterns when websites make changes to the way they present data. Common examples are changes in date formats (i.e. 01/17/2021 vs. 17/01/2021), account numbers (i.e. 001234567 vs. 1234567), and SKUs (i.e. TESLA-S vs. Tesla Model S). 5) Nomenclature Change on the web is constant as websites evolve to meet their users' expectations. For example, you may download your utility bill each month by clicking on a link that states “Invoice”, but next month that same link may state “Bill”. These types of seemingly obvious changes to the human eye will break traditional bot automations. By using AI you can better recognize relevant nomenclature changes such as synonyms and increase your probability of success automating on the web. 6) Navigation Website changes go beyond nomenclature changes. Sometimes nomenclature stays the same, but the elements in the User Interface change. A common example is a website modernizing clickable links by transitioning them into buttons. This kind of change will break traditional bot automations. By storing the goal/intent of your automation rather than just the HTML element location and raw browser interactions you can leverage AI to successfully navigate website changes. 7) Pop-ups Websites use pop-ups to gain the attention of their users. Unfortunately, this can break brittle, bot automations. By using AI you can intelligently recognize pop-ups, the content within them and derive whether your automation should fail or close (or acknowledge) the pop-up and proceed with the automation. 8) Scheduling Some Bot automations run on a preset schedule based on an understanding the automation can be executed successfully at that time. But, what if your scheduled date is wrong. A common example is you command your bot automation to download a report on a specific date, but continually the report isn’t available until 2-3 days later. AI can help you solve this problem by learning the optimal schedule based on historical automation success and suggest the appropriate schedule adjustment. Why Weeldi? As companies attempt to automate predictably on the web they must consider the web's dynamic nature and look to pattern recognition technologies like AI to help manage this variability. Companies like Weeldi have started laying the groundwork by designing automations that focus on user intent, rather than exclusively on static navigation steps, and then using AI to successfully navigate to the users expected outcome. This approach proves to be a sturdy alternative for automation on the web where change is constant, broad, and unpredictable. Move past RPA and (other legacy tools) to automate successfully on telco web portals. 1) Encryption Sensitive data like passwords and PII (Personally Identifiable Information) required to run automations should be encrypted in transit and at rest with a cloud-based automation vendor. Also, sensitive data like passwords should never be displayed in clear through the application UI. 2) Transparency Access logs should be made easily available to you showing every time-sensitive data is accessed, including username, time of access, and the data accessed. Neither your users nor a cloud-based automation vendor should not have the ability to edit or remove these access logs. 3) 2FA (2-Factor Authentication) 2FA adds another layer of time-sensitive, security to the login process, reducing the chances of your account being hacked. This layer can be an OTP sent to your mobile device or an auto-generated code via a 2FA app. However, it should be noted, as hackers get more sophisticated in their ability to hack into SMS via flaws in SS7 protocols 2FA apps are preferred for optimal security. Cloud-based automation vendors should provide the flexibility to require 2FA at the organization level or individual user level. Commonly used 2FA apps are Twilio Authy or Google Authenticator. 4) KMS (Key Management Service) Integration KMS Integration allows you to integrate with your current KMS system to create and control the encryption keys used to encrypt your sensitive data. It also enables transparency into who is accessing your sensitive data and provides you with full and immediate control over who can continue to access your sensitive data. Commonly used KMS tools include Amazon KMS and HyTrust. 5) Secret Manager Integration Secret Manager integration gives you the ability to use your Secret Manager tool of choice to store sensitive data like passwords, control access to sensitive data, maintain transparency into when sensitive data is accessed and centrally manage sensitive data. Commonly used Secret Manager Tools include Amazon Secrets Manager and Google Secret Manager. 6) SSO (Single Sign-On) SSO integration allows your users to log into a cloud-based automation provider with a single ID and password following the SAML 2.0 protocol, usually centrally managed through an Identity Management provider. Commonly used Identity Management Providers include OKTA and OneLogin. Why Weeldi? As companies automate more tasks requiring sensitive data like passwords or PII (Personally Identifiable Information) they are faced with the challenge of ensuring their automations run securely. Cloud-based automation solutions like Weeldi provide transparency into sensitive data access, best in class security using tools like encryption, 2FA (2-Factor Authentication) and SSO (Single Sign-On), as well as integration with 3rd party KMS (Key Management Services) and Secret Manager tools to ensure your sensitive data is secure and in your control. Weeldi and Mobile Solutions (now Brightfin) partner to further automate mobility management9/25/2020
Mobile Solutions (now Brightfin) Partners with Weeldi to Enhance Managed Mobility Software Automations
San Francisco, CA and Centennial, CO – September 25 2020: Weeldi, a San Francisco Bay Area-based company innovating the way processes are automated on the web, and, a leader in Managed Mobility Services, delivering expense optimization, unified endpoint security, device management, and best in class support, have partnered to add additional automations to MAX, Mobile Solutions (now Brightfin) proprietary mobility management portal. Automations can be difficult due to the dynamic nature of the web. Partnering with Weeldi has enabled Mobile Solutions (now Brightfin) to easily and quickly add new automations, while also providing the flexibility to manage and run automations through Weeldi’s web service API and UI. With Weeldi, Mobile Solutions (now Brightfin) is automating about 50,000 tasks critical to managing their customers’ expenses, inventory and usage monthly. MAX, Mobile Solutions (now Brightfin) innovative mobility management solution, was designed from the ground up built with the user experience in mind offering in-cycle analytics and delivering a unified platform for all mobility management needs. Weeldi’s API strengthens Mobile Solutions (now Brightfin) scale and frequency of automations, running a high volume of daily automations in a condensed timeframe of only a few hours. Weeldi also integrates with Mobile Solutions (now Brightfin) Amazon Key Management Services for encryption of their customer credentials. “Mobile Solutions (now Brightfin) is committed to using innovation as a vehicle to scale and deliver the best possible experience for their customers. This pushes us and makes the opportunities in our partnership almost limitless. We’re excited about our partnership.” said Moe Arnaiz, Co-Founder and CEO of Weeldi. Jim Johnson, Board Member of Mobile Solutions (now Brightfin), said, “We couldn’t be happier with our partnership and experience with Weeldi. From the fast, low risk deployment to running over 1,500 daily automations, Weeldi elevates our ability to automate tasks, providing up to date and timely data for our customers.” About Mobile Solutions (now Brightfin) Mobile Solutions (now Brightfin) is a SaaS company that provides a complete managed mobility practice. We make it easy for our clients to optimize IT spend, reduce data consumption, secure mobile endpoints, and simplify the management of their entire fleet of devices, all with best in class support. For more information, visit https: www.brightfin.com/ About Weeldi Headquartered in the San Francisco Bay Area Weeldi enables companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required. To learn more, visit https://www.weeldi.com/ 5 approaches to automation that are too rigid for the web.
Miles Davis could pick up his trumpet, step into a quartet, listen, adapt and produce beautiful Jazz that worked. Similar to a great Jazz musician, a good automation approach for the web has the flexibility to process dynamic inputs, adapt in near real-time and deliver just the right output. However, most traditional automation approaches are rigidly designed with little ability to adapt to change, making them brittle when deployed on the web. Below are 5 approaches to automation that are too rigid for the web. 1) Using X-Y coordinates to locate objects Similar to longitude and latitude on a map, this approach relies on the objects on the screen to be predictably in the same location time and again, but the web is innately dynamic and objects move or could be presented differently based on the browser, device type and resolution. Advertisements, message notifications, report filter options and UI design modifications are examples of changes that will break any automation that relies on X-Y coordinates. 2) Extracting the DOM structure and looking for elements by name or structure The DOM is a tree structure of all the elements that make up a web page. Traditional automations extract the DOM structure and look for elements precisely by their id, class or position. This approach is prone to instability as many websites eschew naming DOM elements entirely, arbitrarily change ids and class names or duplicate ids in violation of DOM specifications, causing automations to break. Also, web development trends are moving toward websites that load dynamically as you scroll through the webpage, defer loading portions of a web page, or dynamically build portions of the page after it is loaded, to provide a more seamless experience to customers with lower bandwidth. While this improves the user experience, it confuses and breaks automations that depend on a static, predictable DOM structure. Finally, most DOM violations aren’t a stumbling block for the user experience because web browsers like Chrome and Firefox do a fantastic job of masking poor adherence to web standards. But, these DOM violations will break automations and require arduous collaboration from operations and software development to investigate and resolve. 3) Javascript containers (e.g. React, Angular, etc.) Javascript frameworks help software developers build consistent user experiences, however, they also hijack user interactions like mouse clicks and scrolling, which can render automations powerless, causing them to fail. 4) Using shortcuts to bypass the User Experience To increase the speed of automations, some software developers decide to bypass the website user experience by taking shortcuts to complete automations faster. Taking these shortcuts will inevitably lead to issues like slowing down (or crashing) your target website, IP Blocking, getting auto-logged out and/or accessing incomplete data. Short-cuts appear to be great until they don’t work and they then become very difficult to investigate and repair as they were not based on user experience to begin with. 5) Using web testing automation tools (e.g. Selenium, etc.) To accelerate automation, rather than start from scratch, software developers may choose to utilize web testing automation tools like Selenium. However, these tools have a propensity to get jammed-up by rogue javascript libraries, such as the ones often embedded in website analytics tools. So when you’re automating on the web you must ditch the traditional rigid approach for an adaptive modern approach that produces beautiful automations that work. Why Weeldi? As companies attempt to automate stably on the web they must consider its dynamic nature. Weeldi’s modern approach to automation is flexible and adaptive, enabling companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required. 6 + (1) tips for deploying your automations in the cloud at scale.
Automation improves how your business scales. But, how do you ensure your automations can scale? 1) Scale-Up Your automations may vary by volume or include tight time constraints. To ensure on-time, successful delivery of automations, you must be able to predict server capacity for peak usage and have the flexibility to deploy additional server capacity in real-time to guarantee timely execution. 2) Scheduling You must integrate the scheduling and prioritization components of your automation engine with your servers to provision adequate capacity to complete your automations on time. The scheduling and prioritization components to consider include: urgency, probability of success, concurrent login acceptance (if automating behind logins), the number of reattempts required for automations to complete successfully, amongst several others. 3) Scale-Down / Reduce Costs To run your automations economically, you must have the ability to reallocate server resources in real-time for automations that require them, or turn down servers during periods of lower automation activity. 4) Configuration Flexibility Not all websites are optimized for all browsing environments. One website may be friendlier on Chrome and another on Firefox. Further complicating matters, the process you’re trying to automate (i.e. downloading a file vs. submitting a web form) may have different probabilities of success depending on the configuration. By having multiple browsing configurations deployed in the cloud you can systematically A+B test your automations to identify configurations with the highest probability of success and use those configurations moving forward. Common configuration variables include: using Proxy IPs, using older browser versions, turning off 3rd party tracking amongst several others. 5) Security To protect your data, it's important to disconnect data storage from the actual processing of the automation. This enables you to secure the automation independently of the storage of the automation result, ensuring that anything bad your automation might pick up on the web does not contaminate your stored data. 6) Monitoring On the web, many items can degrade or even crash your automations' performance. To run stable and timely automations, it a requirement to have real-time visibility and monitoring of server health and usage 6*) Target Website Acceptance To ensure your automation is accepted by the target website, you must have IP flexibility including provisioning dedicated IPs, untainted IPs, a range of customer IPs or in some instances leveraging Proxy IPs. Why Weeldi? As companies increase the volume of tasks they automate, they are faced with the infrastructure challenges of ensuring their automations run successfully, on time and at scale. Cloud-based automation solutions like Weeldi provide out-of-the-box functionality that enables you to automatically scale to meet your workload. Weeldi empowers market leader, vCom Solutions “to automate processes, where RPA falls short”3/18/2020
Web data extraction solution eliminates the need for relying on APIs or using costly RPA tools to cleanly extract data from the web.
SAN FRANCISCO, California — March 18, 2020: Weeldi, a San Francisco Bay Area-based company innovating the way processes are automated on the web, has partnered with vCom, the premier IT spend management firm for the mid-market, providing vCom with a fast path to automate processes for data extraction on the web enabling them to immediately capture critical business data from thousands of web pages at scale via the Weeldi API. Cofounded in 2019 by Moe Arnaiz and Mathieu Guilmineau whose last SaaS company was acquired by a $6.7B private equity firm — Weeldi provides customers a fast and reliable means of automating processes on the web, filling gaps where API’s are unavailable or incomplete and tools like Robotic Process Automation (RPA) are difficult to deploy and costly to support. “RPA is top-of-mind for businesses as they look to automate repetitive tasks and while it can work well in a stable and static private server environment, automating processes using RPA tools quickly gets unwieldy in the wilds of the web where you run into common challenges like OTP, Captchas, IP Blocking, multiple website versions, multiple web frameworks, scheduling, website timeouts, and the list goes on. Trying to solve these problems with RPA leads to an endless, expensive cycle of customization and maintenance, as companies struggle to reliably automate processes on the web” said Arnaiz. “With Weeldi, we’re offering an out-of-the-box solution, deliberately leveraging technologies like AI and modern web drivers to solve the challenges of automating processes on the web all in one platform, to make automation of processes on the web more stable and predictable from project and cost standpoint,” said Guilmineau. “I couldn’t be happier with Weeldi,” says vCom COO Sameer Hilal. “All the data we need from invoices, reports, and portals are being collected automatically. We’re getting exactly what we need, faster and cleaner, and freeing up our development team for other projects. And the best part is we’re only paying for successful outcomes: we’re not paying for development hours or RPA bot hours: we’re only paying for the extracted data.” A SaaS tool, Weeldi has simplified data extraction for vCom across more than 30 telco and technology vendor websites, extracting text and capturing data from web pages and transforming it into an easily consumable API, enabling integration at scale into vCom’s award-winning vManager software. Embedded in Weeldi’s platform are solutions to solve a range of challenges synonymous with the web. (Read more about Weeldi vs RPA) “vCom is using the Weeldi API to run more than a thousand automations on various schedules from monthly to on-demand, and able to build new automation instructions in minutes,” said Hilal. “We deploy RPA to automate other internal processes, but for this use case, on the web, there’s no comparison.” About vCom Solutions vCom is a cloud-based software and managed services company focused on helping enterprises manage IT spend from procure-to-pay. vCom improves visibility and control within a single software while decreasing expenses for networks, mobile, cloud, collaboration, and SaaS technologies. To learn more, visit http://vcomsolutions.com About Weeldi Headquartered in the San Francisco Bay Area Weeldi enables companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required. To learn more, visit https://www.weeldi.com/ 10 obstacles you need to consider when using Robotic Process Automation (RPA) bots for unattended automations on the public internet.
RPA on the public web RPA bots are powerful tools, however if you are attempting to use RPA bots for unattended automations on the public web you’ll need to ensure you have solutions for the following 10 obstacles. 1. Log-Ins The processes you want to automate may be on the Deep Web (behind login credentials). To automate processes behind logins you’ll need to make sure your RPA bots can securely access the credentials necessary to complete your unattended automations. 2. Two-Factor Authentication Websites may require two-factor authentication when logging in from a new or unrecognized web browser. In addition, this step can be triggered at the discretion of a website at any time. You’ll need a solution to avoid and resolve two-factor authentication. 3. Captchas Websites may require you to solve captchas before granting access. You’ll need a solution to solve or avoid captchas. 4. IP Blacklisting Websites may blacklist critical IP addresses if you are accessing them too fast, too frequently or concurrently. You’ll need a solution to ensure you do not trigger IP blacklisting. 5. Scheduling You’ll need a flexible solution to schedule frequency of RPA bot attempts and reattempts (for failed) unattended automations. 6. Schedule-and-Return Some scenarios will require you to first initiate your automation and then return within a defined time frame to complete it. You’ll need a flexible solution to manage schedule-and- return automations. 7. A/B Testing Websites can have multiple versions of their site depending on region, customer type and/ or A/B testing. To adequately support this common scenario you will have to build and support multiple versions of your bot. 8. Intermittent Downtime, Time-Outs and Latency Websites on the public internet can be unstable so your RPA bots will need to be equipped to manage instability to ensure you can complete your unattended automations successfully. This means not aborting unattended automations too early and automatically reattempting others at a later time. In some instances, it may take several attempts over multiple hours before an automation completes successfully on a public website. You’ll need a solution to manage this website instability carefully. 9. New Front End Frameworks and Elements As the web evolves so do the frontend frameworks and elements. Your RPA bots should be able run in frameworks such as AngularJS and ReactJS and elements such as modals and hidden fields. In addition, your RPA bots should evolve to account for new frameworks and web elements as they are introduced. In the world of RPA bots this may mean building a new bot. 10. Rogue Scripts, Insecure Connections and Mixed Content Some websites include rogue scripts, which will trigger timeouts and others insecure connections or content, which result in modern browsers rejecting all or part of the page. You’ll need a solution to manage through these obstacles. Why Weeldi While RPA bots can be powerful tools, they often require costly, intricate configuration and even custom development to deliver an enterprise-ready solution for automating process on the public internet. Alternatively, considering purpose-built solutions for automating process on the public internet could save you time and money. As an example, companies like Weeldi provide solutions to these 10 obstacles and many others, out-of-the-box, with no costly configuration or coding required. In addition, you pay by the results, not per bot hour or the individual bot. How businesses are moving from risky home grown “scraping” solutions into secure, comprehensive Web Data Integration (WDI)
The current state of web data Web data is the boundless unstructured data living on websites, web portals and SaaS applications–simply put, it’s the data in your web browser. For years, web data has been something you can see, but not really “touch” or incorporate into your business systems without accessing each website via an API or building homegrown scraping solutions. But that’s about to change. Finding the fuel to move forward Often, the data a business needs to fuel success can be accessed by APIs, but not always, as there are countless non-API sources of information on websites, web portals, and even some SaaS applications. Take for example many Mobile Telecom Companies who offer APIs to customers to help manage expenses, but leave out critical pieces of data, like current international usage, which is invaluable for avoiding costly roaming charges. This usage data may not be in the API, but is instead sitting as unstructured web data, trapped in a customer web portal...ripe for consumption if it could only be easily accessed at scale. Moving past relying (solely) on APIs While APIs are a standard means of data transfer, that doesn’t necessarily make them the most expedient or ideal. Some API access processes can take weeks, months, or longer to approve and set-up, and these delays raise risks of failed deployments, missed revenue opportunities, and poor customer service. But assume API setup goes smoothly: what if your API vendor sunsets support, or decides to make significant changes that make it more difficult to work with their API? The ability to independently access web data moves from being a “nice to have” to a must, as does the need for a modern Web Data Integration (WDI) tool that can extract web data across multiple websites with the same scalability, reliability and structure of an API. According to 2019 research from Opimas, spending on web data extraction was $2.5B in 2017 and predicted to reach nearly $7B by 2020, with the majority of spend on internal homegrown solutions. But while the overall web data extraction spend is trending up, Opimas expects a significant transition from internal spending (with expected growth at 30%) to external spending (with expected growth at a whopping 70%). (1) So why the move toward more external solutions? In their current state, internal solutions for web data extraction are often unstable and send valuable software development resources down a proverbial rabbit hole of custom coding and neverending maintenance in a attempt to manage: data extraction templates for every data source, IP blacklisting, CAPTCHA, 2 factor authentication, pacing, data access across multiple pages / websites or data behind logins, file downloads and scheduling — to name just a few. Signs are pointing toward use of modern Web Data Integration (WDI) solutions that quickly transform any unstructured web data into structured APIs — with no coding required. Specifically, faster, more reliable, more scalable, and more affordable alternatives to legacy homegrown and outsourced web data extraction solutions. How modern Web Data Integration (WDI) differs from “scraping” Modern Web Data Integration (WDI) tools are built to deliver scalable, reliable web data extraction out of the box without requiring software development for either configuration or maintenance. Whereas scraping is a disjointed process of collecting web data that requires software developers to build and reactively maintain custom scraping routines, typically built with a hodgepodge of open source tools. A modern Web Data Integration (WDI) tool gives you “out of the box” ability to:
Scraping requires hourly, salary and/or contract specialist in product management, software development and QA to build and maintain the same functionality that is available out of the box with Web Data Integration (WDI) tools as listed above. Plus you to take on the risk of:
To DIY, or Not DIY? SMB and enterprise adoption of modern Web Data Integration (WDI) tools is on the rise. While there are many legacy, home-grown, outsourced and on-premise solutions still deployed to solve the “web data extraction needs” of competitive businesses, most do not deliver the modern ease of use, reliability and scalability companies need to make web data an integral part of their business. Consider this: just as Stripe has done for online payments and Twilio for business communication, modern Web Data Integration (WDI) tools are now doing to empower organizations in all markets and of all sizes to capture and act on the endless ocean of web data. REFERENCES 1: A. Griem, & O. Marenzi, Web Data Integration — Leveraging the Ultimate Dataset [White paper], retrieved March 2019, Opimas Research |
WeeldiTransform a website into an API in seconds w/ no coding required. Archives
December 2024
Categories |