weeldi
  • Why Weeldi?
  • Pricing
  • Get in Touch
  • Sign In

Mobile workforce tech company automates processes in days instead of months, with Weeldi’s “no-code” automation solution.

2/10/2021

 
Picture
EXECUTIVE SUMMARY:

Motus is the definitive leader in solutions for businesses with mobile-enabled workforces helping over 2,300 companies and 300,000 mobile workers simplify reimbursement and management of mileage, mobile devices and remote work.

After a recent acquisition, Motus faced the challenge of automating its acquisition’s data collection processes from over 20 vendor websites, spread across 13 countries and in 10 languages. Motus had a simple objective: quickly create automated processes to identify, capture, and consolidate the critical business data it needed from anywhere on the web it was living.

So Motus explored every opportunity to automate data collection on the web.

After a pilot with a well known RPA (robotic process automation) platform, and a failed experiment with an open source tool, Motus chose Weeldi for a pilot project.

When Weeldi delivered five stable automations within its ten-day risk free pilot, Motus chose Weeldi as a partner. The company currently enjoys more than 90 automations in 10 different languages, collecting endless amounts of critical usage and spend data for their customers.

The problem: complex automation of manual processes on the web to better handle scale

The nature of the mobile workforce is often chaotic. Different devices used by different staff; working in different places and timezones; different needs for tech, transportation, and just about everything.
​
As the company that simplifies management for all that (and more) for over 2,300 customers, Motus has to think and act at scale. In this case, it meant being able to consistently and completely collect critical data for its clients from multiple vendor websites regardless of the hurdles at each step (CAPTCHAs, Two-Factor Authentication, instability, different website frameworks, etc.).
​
Prior to acquisition, the acquired company’s processes could be handled with the right people, manually. But after being acquired and integrated into Motus, the need for scale increased exponentially, and manually combing through thousands of webpages a month was no longer an option: automated processes were needed to prepare for the increase in volume, while the manual labor was directed to higher value tasks.

The search for a solution, part 1: start, sputter, stop

After working with a well-known RPA (robotic process automation) company to create a proof-of-concept for a single automation, Motus turned to Selenium, an open-source browser automation tool (primarily used for app testing) to build it.

After a development process that took the better part of two months, one automation was built. However, it proved to be a house of cards, unstable and unable to accomplish the single task it was developed to execute on the web.

So Motus looked elsewhere and decided on a 10-day pilot with Weeldi, a San Francisco-Bay Area company that enables companies of any size or technical ability to stably automate processes on the web, at scale, with no coding required.

The solution, part 2: automation at a record-setting pace, and we’re just getting started

In the 10-day pilot, Weeldi delivered five full-featured automations to help Motus automate complex manual backend processes on vendor websites, and another 40 processes just days after the official start of the engagement.

Weeldi was able to create stable automations on the web for Motus (without requiring any coding work from the client), solving challenges such as login, OTP resolution, task scheduling/completion, and more.

Motus now enjoys more than 90 automations, in 13 countries, and 10 languages. Other “Weeldi wins” for Motus include:

  • Using the Weeldi API to create, update and run Automations
  • Automation results pushed to a Motus sFTP
  • A complete transition of all manual Telco data processes to Weeldi within just four months.
  • Increases in scalability, timeliness of data, and data precision
  • The ability to add more customers with less manual labor

With Weeldi, Motus enjoys the freedom to quickly create dependable automations for endless tasks across the enterprise and can better and more completely utilize the skill sets of its staff. Perhaps most importantly, because of Weeldi’s flexibility, Motus essentially has every automation on the web it will ever need at its fingertips.

About Weeldi

Headquartered in the San Francisco Bay Area Weeldi enables companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required.

AI. Believe the hype and get started.

1/26/2021

 
Picture
8 areas where you can leverage AI today for automation on the web.

Artificial Intelligence (AI) is a hot and highly marketed topic, but the application of AI for automation on the web is still in its infancy. This means those laying the right foundation today will be rewarded with the compounding value of less friction and more success with future automations.

Here are 8 areas where you can begin leveraging AI for automation on the web.

1) Login

Many automations on the web start by getting through a login page.

AI can be leveraged to simplify the login step by looking up standard login components such as username, password, and “remember me” functionality, minimizing obstacles like MFA (Multi-Factor Authentication) and CAPTCHAs, and increasing the probability of automation success when there are login page changes.

2) Flow selection

A growing number of websites have multiple versions based on region, customer type, M&A (Mergers & Acquisitions), and/or A/B testing. 

Using AI you can simplify the support of multiple website versions. This can be achieved by recording multiple automation flows, by website version, then using AI to select and navigate to the correct flow. This approach adds scale and stability to the automation by eliminating the need for a human to micro-manage variability of website versions. 

3) Managing website issues

Traditionally, website instability is a key culprit for breaking automations on the web.

AI can help solve the most common instability issues like slow-loading webpages and webpage timeouts. It can do this by intelligently adjusting bot wait times by website, proceeding when critical components of a webpage are available (but noncritical components are not) and automatically reloading webpages with timeouts. 

4) Formatting

Bot automation flows are guided by data such as dates, account numbers, and SKUs. However, the way this data is presented on websites may vary from how you store it in your database, and even if you harmonize the two, eventual changes to the presentation of data on the web will break your automations.

AI can help solve this problem by recognizing data patterns when websites make changes to the way they present data. Common examples are changes in date formats (i.e. 01/17/2021 vs. 17/01/2021), account numbers (i.e. 001234567 vs. 1234567), and SKUs (i.e. TESLA-S vs. Tesla Model S).

5) Nomenclature 
 

Change on the web is constant as websites evolve to meet their users' expectations. For example, you may download your utility bill each month by clicking on a link that states “Invoice”, but next month that same link may state “Bill”. These types of seemingly obvious changes to the human eye will break traditional bot automations. 

By using AI you can better recognize relevant nomenclature changes such as synonyms and increase your probability of success automating on the web.

6) Navigation

Website changes go beyond nomenclature changes. Sometimes nomenclature stays the same, but the elements in the User Interface change. A common example is a website modernizing clickable links by transitioning them into buttons. This kind of change will break traditional bot automations. 

By storing the goal/intent of your automation rather than just the HTML element location and raw browser interactions you can leverage AI to successfully navigate website changes.

7) Pop-ups

Websites use pop-ups to gain the attention of their users. Unfortunately, this can break brittle, bot automations. 

By using AI you can intelligently recognize pop-ups, the content within them and derive whether your automation should fail or close (or acknowledge) the pop-up and proceed with the automation.

8) Scheduling 

Some Bot automations run on a preset schedule based on an understanding the automation can be executed successfully at that time. But, what if your scheduled date is wrong. A common example is you command your bot automation to download a report on a specific date, but continually the report isn’t available until 2-3 days later. 

AI can help you solve this problem by learning the optimal schedule based on historical automation success and suggest the appropriate schedule adjustment. 
​
Why Weeldi?

As companies attempt to automate predictably on the web they must consider the web's dynamic nature and look to pattern recognition technologies like AI to help manage this variability. Companies like Weeldi have started laying the groundwork by designing automations that focus on user intent, rather than exclusively on static navigation steps, and then using AI to successfully navigate to the users expected outcome. This approach proves to be a sturdy alternative for automation on the web where change is constant, broad, and unpredictable. 

Automate Or Die ETMA Virtual Conference Weeldi Co Founder CEO, Moe Arnaiz

11/22/2020

 
Move past RPA and (other legacy tools) to automate successfully on telco web portals.

Successful automation on the web starts with secure automation.

11/11/2020

 
Picture


1) Encryption

Sensitive data like passwords and PII (Personally Identifiable Information) required to run automations should be encrypted in transit and at rest with a cloud-based automation vendor. Also, sensitive data like passwords should never be displayed in clear through the application UI. 

2) Transparency

Access logs should be made easily available to you showing every time-sensitive data is accessed, including username, time of access, and the data accessed. Neither your users nor a cloud-based automation vendor should not have the ability to edit or remove these access logs.

3) 2FA (2-Factor Authentication)

2FA adds another layer of time-sensitive, security to the login process, reducing the chances of your account being hacked. This layer can be an OTP sent to your mobile device or an auto-generated code via a 2FA app. However, it should be noted, as hackers get more sophisticated in their ability to hack into SMS via flaws in SS7 protocols 2FA apps are preferred for optimal security. Cloud-based automation vendors should provide the flexibility to require 2FA at the organization level or individual user level. Commonly used 2FA apps are Twilio Authy or Google Authenticator.


4) KMS (Key Management Service) Integration

KMS Integration allows you to integrate with your current KMS system to create and control the encryption keys used to encrypt your sensitive data. It also enables transparency into who is accessing your sensitive data and provides you with full and immediate control over who can continue to access your sensitive data. Commonly used KMS tools include Amazon KMS and HyTrust.

5) Secret Manager Integration

Secret Manager integration gives you the ability to use your Secret Manager tool of choice to store sensitive data like passwords, control access to sensitive data, maintain transparency into when sensitive data is accessed and centrally manage sensitive data. Commonly used Secret Manager Tools include Amazon Secrets Manager and Google Secret Manager.

6) SSO (Single Sign-On)

SSO integration allows your users to log into a cloud-based automation provider with a single ID and password following the SAML 2.0 protocol, usually centrally managed through an Identity Management provider. Commonly used Identity Management Providers include OKTA and OneLogin.

​Why Weeldi?
​

As companies automate more tasks requiring sensitive data like passwords or PII (Personally Identifiable Information) they are faced with the challenge of ensuring their automations run securely. Cloud-based automation solutions like Weeldi provide transparency into sensitive data access, best in class security using tools like encryption, 2FA (2-Factor Authentication) and SSO (Single Sign-On), as well as integration with 3rd party KMS (Key Management Services) and Secret Manager tools to ensure your sensitive data is secure and in your control.

Weeldi and Mobile Solutions (now Brightfin) partner to further automate mobility management

9/25/2020

 
Picture
Mobile Solutions (now Brightfin) Partners with Weeldi to Enhance Managed Mobility Software Automations

San Francisco, CA and Centennial, CO – September 25 2020: Weeldi, a San Francisco Bay Area-based company innovating the way processes are automated on the web, and, a leader in Managed Mobility Services, delivering expense optimization, unified endpoint security, device management, and best in class support, have partnered to add additional automations to MAX, Mobile Solutions (now Brightfin) proprietary mobility management portal.

​Automations can be difficult due to the dynamic nature of the web. Partnering with Weeldi has enabled Mobile Solutions (now Brightfin) to easily and quickly add new automations, while also providing the flexibility to manage and run automations through Weeldi’s web service API and UI. With Weeldi, Mobile Solutions (now Brightfin) is automating about 50,000 tasks critical to managing their customers’ expenses, inventory and usage monthly.

MAX, Mobile Solutions (now Brightfin) innovative mobility management solution, was designed from the ground up built with the user experience in mind offering in-cycle analytics and delivering a unified platform for all mobility management needs. Weeldi’s API strengthens Mobile Solutions (now Brightfin) scale and frequency of automations, running a high volume of daily automations in a condensed timeframe of only a few hours. Weeldi also integrates with Mobile Solutions (now Brightfin) Amazon Key Management Services for encryption of their customer credentials.
​
“Mobile Solutions (now Brightfin) is committed to using innovation as a vehicle to scale and deliver the best possible experience for their customers. This pushes us and makes the opportunities in our partnership almost limitless. We’re excited about our partnership.” said Moe Arnaiz, Co-Founder and CEO of Weeldi.
​
Jim Johnson, Board Member of Mobile Solutions (now Brightfin), said, “We couldn’t be happier with our partnership and experience with Weeldi. From the fast, low risk deployment to running over 1,500 daily automations, Weeldi elevates our ability to automate tasks, providing up to date and timely data for our customers.”

About Mobile Solutions (now Brightfin)

Mobile Solutions (now Brightfin) is a SaaS company that provides a complete managed mobility practice. We make it easy for our clients to optimize IT spend, reduce data consumption, secure mobile endpoints, and simplify the management of their entire fleet of devices, all with best in class support. For more information, visit https: www.brightfin.com/

About Weeldi

Headquartered in the San Francisco Bay Area Weeldi enables companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required. To learn more, visit https://www.weeldi.com/

Automating on the web is like playing Jazz

7/21/2020

 
Picture
5 approaches to automation that are too rigid for the web.

Miles Davis could pick up his trumpet, step into a quartet, listen, adapt and produce beautiful Jazz that worked.

Similar to a great Jazz musician, a good automation approach for the web has the flexibility to process dynamic inputs, adapt in near real-time and deliver just the right output.

However, most traditional automation approaches are rigidly designed with little ability to adapt to change, making them brittle when deployed on the web.

Below are 5 approaches to automation that are too rigid for the web.

1) Using X-Y coordinates to locate objects

Similar to longitude and latitude on a map, this approach relies on the objects on the screen to be predictably in the same location time and again, but the web is innately dynamic and objects move or could be presented differently based on the browser, device type and resolution. Advertisements, message notifications, report filter options and UI design modifications are examples of changes that will break any automation that relies on X-Y coordinates.

2) Extracting the DOM structure and looking for elements by name or structure

The DOM is a tree structure of all the elements that make up a web page.

Traditional automations extract the DOM structure and look for elements precisely by their id, class or position. This approach is prone to instability as many websites eschew naming DOM elements entirely, arbitrarily change ids and class names or duplicate ids in violation of DOM specifications, causing automations to break.

Also, web development trends are moving toward websites that load dynamically as you scroll through the webpage, defer loading portions of a web page, or dynamically build portions of the page after it is loaded, to provide a more seamless experience to customers with lower bandwidth. While this improves the user experience, it confuses and breaks automations that depend on a static, predictable DOM structure.

Finally, most DOM violations aren’t a stumbling block for the user experience because web browsers like Chrome and Firefox do a fantastic job of masking poor adherence to web standards. But, these DOM violations will break automations and require arduous collaboration from operations and software development to investigate and resolve.

3) Javascript containers (e.g. React, Angular, etc.)

Javascript frameworks help software developers build consistent user experiences, however, they also hijack user interactions like mouse clicks and scrolling, which can render automations powerless, causing them to fail.

4) Using shortcuts to bypass the User Experience

To increase the speed of automations, some software developers decide to bypass the website user experience by taking shortcuts to complete automations faster. Taking these shortcuts will inevitably lead to issues like slowing down (or crashing) your target website, IP Blocking, getting auto-logged out and/or accessing incomplete data. Short-cuts appear to be great until they don’t work and they then become very difficult to investigate and repair as they were not based on user experience to begin with.

5) Using web testing automation tools (e.g. Selenium, etc.)

To accelerate automation, rather than start from scratch, software developers may choose to utilize web testing automation tools like Selenium. However, these tools have a propensity to get jammed-up by rogue javascript libraries, such as the ones often embedded in website analytics tools.

So when you’re automating on the web you must ditch the traditional rigid approach for an adaptive modern approach that produces beautiful automations that work.

Why Weeldi?

As companies attempt to automate stably on the web they must consider its dynamic nature. Weeldi’s modern approach to automation is flexible and adaptive, enabling companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required.​​

Scaling Automations In the Cloud (with AWS)

5/31/2020

 
Picture
6 + (1) tips for deploying your automations in the cloud at scale.

Automation improves how your business scales. But, how do you ensure your automations can scale? 

1) Scale-Up

Your automations may vary by volume or include tight time constraints. To ensure on-time, successful delivery of automations, you must be able to predict server capacity for peak usage and have the flexibility to deploy additional server capacity in real-time to guarantee timely execution.

2) Scheduling

You must integrate the scheduling and prioritization components of your automation engine with your servers to provision adequate capacity to complete your automations on time. The scheduling and prioritization components to consider include: urgency, probability of success, concurrent login acceptance (if automating behind logins), the number of reattempts required for automations to complete successfully, amongst several others.

3) Scale-Down / Reduce Costs

To run your automations economically, you must have the ability to reallocate server resources in real-time for automations that require them, or turn down servers during periods of lower automation activity.


4) Configuration Flexibility

Not all websites are optimized for all browsing environments. One website may be friendlier on Chrome and another on Firefox. Further complicating matters, the process you’re trying to automate (i.e. downloading a file vs. submitting a web form) may have different probabilities of success depending on the configuration. By having multiple browsing configurations deployed in the cloud you can systematically A+B test your automations to identify configurations with the highest probability of success and use those configurations moving forward. Common configuration variables include: using Proxy IPs, using older browser versions, turning off 3rd party tracking amongst several others.


5) Security

To protect your data, it's important to disconnect data storage from the actual processing of the automation. This enables you to secure the automation independently of the storage of the automation result, ensuring that anything bad your automation might pick up on the web does not contaminate your stored data.
​

6) Monitoring

​On the web, many items can degrade or even crash your automations' performance. To run stable and timely automations, it a requirement to have real-time visibility and monitoring of server health and usage 


6*) Target Website Acceptance

To ensure your automation is accepted by the target website, you must have IP flexibility including provisioning dedicated IPs, untainted IPs, a range of customer IPs or in some instances leveraging Proxy IPs.

Why Weeldi?

As companies increase the volume of tasks they automate, they are faced with the infrastructure challenges of ensuring their automations run successfully, on time and at scale. Cloud-based automation solutions like Weeldi provide out-of-the-box functionality that enables you to automatically scale to meet your workload.

Weeldi empowers market leader, vCom Solutions “to automate processes, where RPA falls short”

3/18/2020

 
Picture
Web data extraction solution eliminates the need for relying on APIs or using costly RPA tools to cleanly extract data from the web.

SAN FRANCISCO, California — March 18, 2020: Weeldi, a San Francisco Bay Area-based company innovating the way processes are automated on the web, has partnered with vCom, the premier IT spend management firm for the mid-market, providing vCom with a fast path to automate processes for data extraction on the web enabling them to immediately capture critical business data from thousands of web pages at scale via the Weeldi API.

Cofounded in 2019 by Moe Arnaiz and Mathieu Guilmineau whose last SaaS company was acquired by a $6.7B private equity firm — Weeldi provides customers a fast and reliable means of automating processes on the web, filling gaps where API’s are unavailable or incomplete and tools like Robotic Process Automation (RPA) are difficult to deploy and costly to support.
 
“RPA is top-of-mind for businesses as they look to automate repetitive tasks and while it can work well in a stable and static private server environment, automating processes using RPA tools quickly gets unwieldy in the wilds of the web where you run into common challenges like OTP, Captchas, IP Blocking, multiple website versions, multiple web frameworks, scheduling, website timeouts, and the list goes on. Trying to solve these problems with RPA leads to an endless, expensive cycle of customization and maintenance, as companies struggle to reliably automate processes on the web” said Arnaiz.
 
“With Weeldi, we’re offering an out-of-the-box solution, deliberately leveraging technologies like AI and modern web drivers to solve the challenges of automating processes on the web all in one platform, to make automation of processes on the web more stable and predictable from project and cost standpoint,” said Guilmineau.
 
“I couldn’t be happier with Weeldi,” says vCom COO Sameer Hilal. “All the data we need from invoices, reports, and portals are being collected automatically. We’re getting exactly what we need, faster and cleaner, and freeing up our development team for other projects. And the best part is we’re only paying for successful outcomes: we’re not paying for development hours or RPA bot hours: we’re only paying for the extracted data.”
 
A SaaS tool, Weeldi has simplified data extraction for vCom across more than 30 telco and technology vendor websites, extracting text and capturing data from web pages and transforming it into an easily consumable API, enabling integration at scale into vCom’s award-winning vManager software. Embedded in Weeldi’s platform are solutions to solve a range of challenges synonymous with the web. (Read more about Weeldi vs RPA)
 
“vCom is using the Weeldi API to run more than a thousand automations on various schedules from monthly to on-demand, and able to build new automation instructions in minutes,” said Hilal. “We deploy RPA to automate other internal processes, but for this use case, on the web, there’s no comparison.”
 
About vCom Solutions
 
vCom is a cloud-based software and managed services company focused on helping enterprises manage IT spend from procure-to-pay. vCom improves visibility and control within a single software while decreasing expenses for networks, mobile, cloud, collaboration, and SaaS technologies. To learn more, visit http://vcomsolutions.com
 
About Weeldi
 
​
Headquartered in the San Francisco Bay Area Weeldi enables companies of any size or technical ability to stably automate processes on the web, at scale, through its web service API or user interface — with no coding required. To learn more, visit https://www.weeldi.com/

Weeldi vs. RPA

2/3/2020

 
Picture
10 obstacles you need to consider when using Robotic Process Automation (RPA) bots for unattended automations on the public internet. 

RPA on the public web

​
RPA bots are powerful tools, however if you are attempting to use RPA bots for unattended automations on the public web you’ll need to ensure you have solutions for the following 10 obstacles.

1. Log-Ins

The processes you want to automate may be on the Deep Web (behind login credentials). To automate processes behind logins you’ll need to make sure your RPA bots can securely access the credentials necessary to complete your unattended automations.

2. Two-Factor Authentication

Websites may require two-factor authentication when logging in from a new or unrecognized web browser. In addition, this step can be triggered at the discretion of a website at any time. You’ll need a solution to avoid and resolve two-factor authentication.

3. Captchas

Websites may require you to solve captchas before granting access. You’ll need a solution to solve or avoid captchas.

4. IP Blacklisting

Websites may blacklist critical IP addresses if you are accessing them too fast, too frequently or concurrently. You’ll need a solution to ensure you do not trigger IP blacklisting.

5. Scheduling
​

You’ll need a flexible solution to schedule frequency of RPA bot attempts and reattempts (for failed) unattended automations.

6. Schedule-and-Return

Some scenarios will require you to first initiate your automation and then return within a defined time frame to complete it. You’ll need a flexible solution to manage schedule-and- return automations.

7. A/B Testing

Websites can have multiple versions of their site depending on region, customer type and/ or A/B testing. To adequately support this common scenario you will have to build and support multiple versions of your bot.

8. Intermittent Downtime, Time-Outs and Latency

Websites on the public internet can be unstable so your RPA bots will need to be equipped to manage instability to ensure you can complete your unattended automations successfully. This means not aborting unattended automations too early and automatically reattempting others at a later time. In some instances, it may take several attempts over multiple hours before an automation completes successfully on a public website. You’ll need a solution to manage this website instability carefully.

9. New Front End Frameworks and Elements

As the web evolves so do the frontend frameworks and elements. Your RPA bots should be able run in frameworks such as AngularJS and ReactJS and elements such as modals and hidden fields. In addition, your RPA bots should evolve to account for new frameworks and web elements as they are introduced. In the world of RPA bots this may mean building a new bot.

10. Rogue Scripts, Insecure Connections and Mixed Content

Some websites include rogue scripts, which will trigger timeouts and others insecure connections or content, which result in modern browsers rejecting all or part of the page. You’ll need a solution to manage through these obstacles.
​
Why Weeldi
​

While RPA bots can be powerful tools, they often require costly, intricate configuration and even custom development to deliver an enterprise-ready solution for automating process on the public internet. Alternatively, considering purpose-built solutions for automating process on the public internet could save you time and money. As an example, companies like Weeldi provide solutions to these 10 obstacles and many others, out-of-the-box, with no costly configuration or coding required. In addition, you pay by the results, not per bot hour or the individual bot.

Taking Complete Command of Web Data

1/2/2020

 
Picture
How businesses are moving from risky home grown “scraping” solutions into secure, comprehensive Web Data Integration (WDI) 

The current state of web data

Web data is the boundless unstructured data living on websites, web portals and SaaS applications–simply put, it’s the data in your web browser. For years, web data has been something you can see, but not really “touch” or incorporate into your business systems without accessing each website via an API or building homegrown scraping solutions. But that’s about to change.

Finding the fuel to move forward

Often, the data a business needs to fuel success can be accessed by APIs, but not always, as there are countless non-API sources of information on websites, web portals, and even some SaaS applications.

Take for example many Mobile Telecom Companies who offer APIs to customers to help manage expenses, but leave out critical pieces of data, like current international usage, which is invaluable for avoiding costly roaming charges. This usage data may not be in the API, but is instead sitting as unstructured web data, trapped in a customer web portal...ripe for consumption if it could only be easily accessed at scale.

Moving past relying (solely) on APIs

While APIs are a standard means of data transfer, that doesn’t necessarily make them the most expedient or ideal. Some API access processes can take weeks, months, or longer to approve and set-up, and these delays raise risks of failed deployments, missed revenue opportunities, and poor customer service.

But assume API setup goes smoothly: what if your API vendor sunsets support, or decides to make significant changes that make it more difficult to work with their API?

The ability to independently access web data moves from being a “nice to have” to a must, as does the need for a modern Web Data Integration (WDI) tool that can extract web data across multiple websites with the same scalability, reliability and structure of an API.

According to 2019 research from Opimas, spending on web data extraction was $2.5B in 2017 and predicted to reach nearly $7B by 2020, with the majority of spend on internal homegrown solutions. But while the overall web data extraction spend is trending up, Opimas expects a significant transition from internal spending (with expected growth at 30%) to external spending (with expected growth at a whopping 70%). (1)

So why the move toward more external solutions?

In their current state, internal solutions for web data extraction are often unstable and send valuable software development resources down a proverbial rabbit hole of custom coding and neverending maintenance in a attempt to manage: data extraction templates for every data source, IP blacklisting, CAPTCHA, 2 factor authentication, pacing, data access across multiple pages / websites or data behind logins, file downloads and scheduling — to name just a few.

Signs are pointing toward use of modern Web Data Integration (WDI) solutions that quickly transform any unstructured web data into structured APIs — with no coding required. Specifically, faster, more reliable, more scalable, and more affordable alternatives to legacy homegrown and outsourced web data extraction solutions.

How modern Web Data Integration (WDI) differs from “scraping”

Modern Web Data Integration (WDI) tools are built to deliver scalable, reliable web data extraction out of the box without requiring software development for either configuration or maintenance.

Whereas scraping is a disjointed process of collecting web data that requires software developers to build and reactively maintain custom scraping routines, typically built with a hodgepodge of open source tools.

A modern Web Data Integration (WDI) tool gives you “out of the box” ability to:
  • configure web data extraction routines via a user interface with no coding
  • required
  • pay for the solution based on results / successful web data extraction
  • extract web data across multiple websites and page flows
  • merge data from multiple sources
  • download multiple file formats (PDF, excel, csv, images, video, audio)
  • deliver structured web data via an API or download
  • extract your web data behind login screens, two-factor authentication or CAPTCHA
  • automatically adjust pace and frequency to avoid IP blacklisting
  • schedule web data extraction (including schedule and return to retrieve)
  • ensure data is transmitted and stored securely
  • circumvent web scraping defenses
  • keep up with quickly evolving web browsers
  • validate web data extraction accuracy
  • provide audit trails of extracted web data
  • leverage experience extracting billions of web data records

Scraping requires hourly, salary and/or contract specialist in product management, software development and QA to build and maintain the same functionality that is available out of the box with Web Data Integration (WDI) tools as listed above.

Plus you to take on the risk of:
  • delays or never reaching a production-ready solution 
  • being out of compliance with data privacy and regulation policies such as GDPR and California Consumer Privacy Act
  • instability as you encounter frequent changes and challenges
  • cost overruns with building and maintenance
  • obsoleteness as severity of changes and challenges require a full rebuild tying up scarce and valuable software development resources
  • having critical IP addresses blacklisted
  • insecurely transmitting or storing data
  • overwhelming target web data source resources, resulting in blocking outsourcing to “gray hat” locales
​
To DIY, or Not DIY?

​
SMB and enterprise adoption of modern Web Data Integration (WDI) tools is on the rise. While there are many legacy, home-grown, outsourced and on-premise solutions still deployed to solve the “web data extraction needs” of competitive businesses, most do not deliver the modern ease of use, reliability and scalability companies need to make web data an integral part of their business.

Consider this: just as Stripe has done for online payments and Twilio for business communication, modern Web Data Integration (WDI) tools are now doing to empower organizations in all markets and of all sizes to capture and act on the endless ocean of web data.

REFERENCES
1: A. Griem, & O. Marenzi, Web Data Integration — Leveraging the Ultimate Dataset [White paper], retrieved March 2019, Opimas Research ​
Forward>>

    Weeldi

    Transform a website into an API in seconds w/ no coding required.

    Picture

    Archives

    March 2023
    January 2023
    November 2022
    October 2022
    September 2022
    August 2022
    July 2022
    May 2022
    April 2022
    March 2022
    February 2022
    January 2022
    December 2021
    November 2021
    October 2021
    September 2021
    August 2021
    July 2021
    June 2021
    May 2021
    April 2021
    February 2021
    January 2021
    November 2020
    September 2020
    July 2020
    May 2020
    March 2020
    February 2020
    January 2020

    Categories

    All
    Case Studies
    Videos
    Whitepapers

Why Weeldi?
Pricing
Get in Touch
About
Security
Privacy
Acceptable Use
Partners
Resources
Picture
Picture

​​© 2023 Weeldi LLC. All rights reserved.
  • Why Weeldi?
  • Pricing
  • Get in Touch
  • Sign In